Trained model vs generative AI
Welcome to Episode 4. Let us start with an honest temptation. You open a chat window, you type "give me a portfolio," and ten seconds later a large language model — a system like ChatGPT or Claude, trained to predict the next word in text — hands you a clean, confident answer. Names, weights, a short rationale. It reads like something a junior analyst would take a week to write. Now compare that to what we do to build a trained allocator. Weeks of gathering clean price and fundamental data. Weeks of splitting that data into training, testing, and validation windows so we never let the model peek at the future. Then the slow work of checking whether the thing actually holds up out of sample. One path takes ten seconds. The other takes a month. So the tempting question of this episode is simple, and we should say it out loud: is the shortcut real? Can we trust a spot allocation — a one-shot answer typed into a prompt — from a chatbot, instead of a trained, validated model? We are not here to mock the chatbot, and we are not here to worship it. Over the next half hour we will try to be fair. We will look at real evidence that these models can read financial language and extract genuine signal — that part is not hype. And then we will be precise about the gap between reading a document well and forecasting a market. Because those are two different jobs. A fluent answer is not a forecast. That is the line we will walk today. Let us begin where every comparison should begin: by putting both engines side by side and recalling exactly what each one is built to do.
The engines at a glance
Before we compare, let us recap the engines we have built across this series, because today's question only makes sense against them. We run three families. First, our research deep-learning models: LSTM networks with attention over ETFs. LSTM, long short-term memory, is a neural network that reads a sequence of prices through time; the attention part lets it weigh which ETFs matter for the decision. Second, candidate models: variants we test but have not promoted to research. Third, mechanical rules: simple, transparent recipes, momentum or regime filters, with no learning at all. Now, the frame for today. Notice what none of these is. None of them is a chatbot. Every one of them optimises portfolio weights on price data, and every one of them faces the same gate: a strict train, test, validation split before we trust it. That difference, machine that optimises and is gated versus model that converses, is the whole episode. Let us look at why.
Two different machines
Let's define both machines precisely, because they share the word "AI" and almost nothing else.
The first machine is a large language model, or LLM. At its core it is a next-token predictor. We train it on an enormous amount of text, and its one job is to continue language in a way that sounds plausible. "Next-token" simply means it guesses the next fragment of a word, then the next, then the next. Its objective — the thing it was optimised to do — is linguistic fluency. Nowhere in that training does it ever see a portfolio, a drawdown, or a risk budget. It was never asked to manage money; it was asked to sound right.
The second machine is a trained allocator — in our case, LSTM networks with attention over ETFs. This machine minimises a loss over price paths. "Loss" is a number that measures how bad an outcome is, and here that number trades off return against risk against drawdown, the peak-to-trough fall we care so much about. Its output is a set of weights that sum to one — an actual portfolio. And crucially, we judge it walk-forward: we test it on data it never saw during training, period after period, the way real money would have experienced it.
So notice the gap. Same word, "AI"; opposite objectives. One machine predicts words; the other optimises a portfolio under constraints. One is graded on whether the sentence flows; the other on whether the equity curve survives. When someone asks an LLM for an allocation, they are asking a fluency engine to imitate the output of an optimisation engine it was never trained to be.
Hold that distinction. In the next slide we look at the evidence — because, to be fair, LLMs genuinely can read signal from text. The question is what kind of signal, and under what conditions.
The evidence: LLMs CAN read signal
Let us be fair to these models, because they earn it. Here is real evidence that a large language model can read financial signal. In a study by Lopez-Lira and Tang, published in 2023 and updated in 2024, the authors fed news headlines to a large model — GPT-4 — and asked one simple question: is this good news or bad news for the stock? By "sentiment" we mean exactly that: a numeric score of how positive or negative a piece of text is. And here is the striking part. That sentiment score predicted the next day's stock return. Not perfectly, but with a real, measurable edge above chance.
What makes this genuinely impressive is that the ability is emergent. When they ran the same test on older, smaller models — GPT-1, GPT-2, and BERT, a language model from 2018 — those models could not do it. The signal appears only once the model is large enough. That is a real finding, and we should not wave it away.
But the authors themselves flag two honest caveats, and we will hold onto both. First, this is sentiment extraction — reading a headline and scoring its tone. It is not allocation. Nobody in this study asked the model, "here is my portfolio, give me the weights." Second, the edge compresses. As more market participants adopt the same trick, the profit from it shrinks toward zero, exactly as we would expect from any signal that becomes crowded.
So let us state it plainly. A large language model can extract signal from text. That is a genuine, useful capability, and it is not hype. The open question — the one that carries us into the rest of this talk — is whether reading a headline well ever becomes the same thing as producing an allocation we can trust. Those are two very different jobs, and next we will see why.
Look-ahead bias — the timeline
So the evidence says a language model can read signal from text. Now we have to ask the hard question: when we test that skill on the past, are we measuring skill, or are we measuring a leak? This brings us to look-ahead bias — using information that would not have been available at the moment of the decision. Glasserman and Lin, in 2023, showed this is the deep problem with backtesting a language model, and the diagram on this slide walks you through why.
Look at the two timelines. The top line is an honest model. On any backtest date — say we are standing in 2015 — an honest model may use only data up to that date. Nothing from the future crosses the line. The bottom line is the language model. That model was trained on a large body of text, and that text corpus extends past 2015 — often years past it. So when we ask the model to "predict" what happens in 2015, its training already contains the news articles, the filings, the commentary written after 2015. Its training cut-off — the date its knowledge stops — sits after the very event we asked it to forecast.
That is the trap. The model is not forecasting; it may be remembering. And remembering the answer looks exactly like skill on a backtest chart. It produces apparent skill that cannot exist live, because in real time the future text does not exist yet. Nobody in 2015 could read what was written in 2018.
This is not a bug we can patch by prompting more carefully. It is baked into how the model was built. So before we trust a single backtested number from a language model, we have to know where its knowledge ends. In the next slide we will see how this contamination gets worse when the model is also distracted by irrelevant text.
Look-ahead + the distraction effect
So the timeline problem we just drew is only half the story. Even when we get the dates right, a second contamination hides inside the text itself. Glasserman and Lin, in 2023, showed this with a clean, almost surgical experiment. They took news headlines and had a language model score them for sentiment — the emotional tone, positive or negative — to predict returns. It worked. Then they did one thing: they anonymised the company names. They stripped out "Apple," "Ford," "Lehman," and replaced them with a neutral placeholder. And the measured performance dropped. Sit with that. Part of what looked like forecasting was not forecasting at all. The model was recognising specific firms and quietly recalling what happened to them — because those outcomes are baked into the text it was trained on. The authors call this the distraction effect: the name itself distracts the model from the actual signal in the words, and pulls in memorised hindsight. Now, to be fair, some genuine signal survived the anonymisation — the model does read tone. But here is the uncomfortable consequence. In a naive historical backtest, you cannot cleanly separate the real forecasting from the memorised outcome. They are fused together in the same number. You get one performance figure, and no honest way to say how much of it is skill and how much is the model remembering the ending of a story it already read. This is, in our view, the single biggest reason a beautiful-looking LLM backtest may simply be an illusion. It is not that the model is lying to you. It is that the test itself is contaminated, and the contamination flatters the result. So before we celebrate any historical LLM performance number, we have to ask: did the model forecast this — or did it already know how it ends? Next, let us look at what a trained allocator actually produced.
What a trained allocator produced
So here is the other machine, drawn on the same axes. The blue line is our trained allocator's cumulative return; the grey line is the S&P 500 over the same period. Look at the gap that opens up between them. Now — the important thing is not the size of that gap. It is where the line came from. This track record did not come from fluent text. Nobody typed a clever prompt and read back a portfolio. This line is the output of a loss function — a number the model was forced to make smaller and smaller, measured on real price paths, day after day, over years of history. And the verdict you are looking at was read on held-out data: the test and validation window we defined in earlier episodes, data the model never saw while it was learning. That is the gate. Under the hood it is what we described before — LSTM networks with attention over the ETFs, meaning the model keeps a memory of the recent past and learns which instruments to pay attention to at each moment. I am staying deliberately high-level; the internals are not the point tonight. The point is three properties this curve has that a spot prompt does not. First, it is reproducible: run the same model on the same inputs and you get the same line, every time. Second, it is auditable: every allocation traces back to a rule fitted on past data, not to a sentence generated on the spot. Third — and this matters most for the coming slides — it is free of the look-ahead problem, because the model only ever saw past prices. It could not accidentally read tomorrow's news. Whatever its merits, and we will be honest about those, this is a forecast you can inspect. Hold that word — inspect — because next we ask whether an LLM's answer survives being run twice.
You cannot reproduce it twice
So the last slide showed us an allocation an LLM proposed. Now let us ask the question that decides whether it is a strategy at all: can we get it back? Reproducibility — the ability to run the same process on the same inputs and obtain the same result — is the bedrock of any testable strategy. If we cannot reproduce a decision, we cannot backtest it, we cannot audit it, and we cannot defend it to a client or a regulator. Here the large language model, the same kind of system we have been discussing, works against us by construction. Ouyang, Zhang, Harman and Wang, writing in the ACM Transactions on Software Engineering in 2024, ran a careful test. They gave the model the very same prompt, over and over, across eight hundred and twenty-nine coding tasks. And the outputs did not match. Depending on the task, somewhere between forty-seven and seventy-six percent of them came back non-identical — the same question, different answers. Now, you might say: set the temperature to zero, the setting that is supposed to make the model deterministic. They tried that. Temperature zero reduces the variation, but it does not eliminate it. The randomness is baked into how these systems generate text. Let us be precise about what this means for us. An allocation we ask for today, and cannot get back tomorrow, is not a process. It is a single draw from a slot machine. It may even be a good draw. But we cannot distinguish a good process from a lucky pull if we can never pull the same lever twice. A trained model, by contrast, is a fixed function: same weights, same inputs, same output, every time. That difference is not cosmetic — it is the line between something we can test and something we can only hope about. And it leads us to the next problem: even setting reproducibility aside, the LLM has no loss function, no risk model, and no notion of tax.
No loss, no risk model, no tax
So the prompt read the signal — but reading a signal is not building an allocation. Here is what a spot prompt, meaning a single question typed once into a chatbot, does not do. First, it does not minimise a loss. A trained model has an objective function: it searches for the weights that make some measured error as small as possible — that is what "argmin of a loss" means, the point where the error bottoms out. The prompt performs no such search. It writes the most plausible-sounding answer, not the one that provably reduces your risk. Second, it does not know your constraints. It does not know that here, in Italy, capital gains on most financial instruments are taxed at twenty-six percent, so a trade that looks attractive gross can be worse net than doing nothing. Third, it does not walk forward. It does not test the rule on data it has not seen, and it does not subtract transaction costs — the spreads and fees that quietly erode every rebalance. And fourth, it can hallucinate: invent a ticker that does not trade, hand you weights that do not sum to one hundred percent, or cite a statistic that never existed. That last one is dangerous precisely because the sentence around it reads perfectly. None of this is a flaw we can prompt our way out of. It is what the machine is: a fluent pattern-matcher over language, not an optimiser over risk and return. An allocation is disciplined optimisation under real constraints — a loss to minimise, a tax code to respect, costs to net out, and a walk-forward test to survive. The chatbot skips every one of those steps and still answers with total confidence. So next we will assemble what an honest pipeline actually looks like — where each of these missing pieces gets put back, on purpose.
The honest pipeline
So if a spot prompt is not the decision-maker, where does a language model actually belong? Let me draw the honest pipeline, left to right. It starts with raw text: news articles, earnings calls, regulatory filings — the messy prose no spreadsheet can hold. This is where a large language model earns its place. We use it to summarise that text and, crucially, to extract a sentiment feature — a single, disciplined number, say a score from minus one to plus one, capturing how negative or positive the language is. That is genuinely what the evidence supports: reading tone, not setting weights. Now watch the arrow. That feature does not go to a client. It flows downstream into a trained allocator — the LSTM networks with attention over ETFs we described earlier — where it sits beside price, volatility, and the other inputs. The allocator is the machine that minimises a loss function, walks forward through time so it is only ever tested on data it has not seen, nets out transaction costs and tax, and then faces the gate: the honest train, test, validation split we set as the standard. Only after all of that does it emit weights. Notice the division of labour. The language model is upstream — a research and feature tool, fluent, fast, tireless at reading. The decision stays downstream, with the validated allocator that carries a loss, a risk model, and a track record you can reproduce. Fluency is the input; discipline is the decision. This is not a compromise between the two engines — it is the correct wiring. It lets us take the one thing LLMs demonstrably do well and place it exactly where it cannot do harm. Next, let me show you that this is not a hunch: the domain models built this way are themselves trained, not merely prompted.
Domain LLMs are trained, not prompted
So if an LLM — a large language model, a system trained to predict the next word across enormous text — can genuinely read financial signal, where does that competence come from? Here is the honest answer: it comes from deliberate training on curated financial data, not from a clever prompt we type in the moment. Two examples make this concrete. BloombergGPT, from Wu and colleagues in 2023, was trained from the ground up on decades of Bloomberg's financial text alongside general data. FinGPT, from Yang and colleagues that same year, is the open-source counterpart — built to be fine-tuned on financial language rather than prompted cold. Both were engineered, over months, on domain data. That is the opposite of opening a chat window and asking, "How should I allocate my portfolio?" So we should be precise about what these systems are, even at their best. They are natural-language engines: they read and write financial language extremely well. They can classify a filing's tone, summarise an earnings call, extract a stance from a headline. That is real, and it is useful. But notice what they are still not. They are not portfolio optimisers. There is no risk model inside them — no covariance matrix, no volatility target, no drawdown constraint. They produce better words about markets; they do not produce an allocation decision that has been optimised against risk. That distinction is the whole point of this episode. Reading language well and deciding how to weight assets are two different jobs, done by two different machines. Even the strongest financial LLM sits firmly on the language side of that line. So when we compare, in a moment, a spot prompt against a trained allocator, remember: we are not comparing a weak model to a strong one. We are comparing a tool that writes about markets to a tool built to size positions in them.
LLM spot-prompt vs trained allocator
So let us put the two machines side by side, honestly, across the seven dimensions we have built up over the last slides. And to be precise about what we are comparing: a spot prompt to an LLM — a single question typed into a chat window, answered on the spot — against a trained allocator, a model fitted on past data to place weights. First dimension, risk-objective optimisation: the allocator minimises a defined objective — return penalised by risk; the spot prompt minimises nothing, it just produces fluent text. Second, look-ahead control — look-ahead being the model peeking at information it could not have had at the time: the allocator sees past data only, by construction; the spot prompt is contaminated, because, as Glasserman and Lin showed in twenty twenty-three, the LLM already knows what happened after the news it is reading. Third, reproducibility: the allocator gives the exact same weights from the same inputs; the LLM does not — Ouyang and colleagues, twenty twenty-four, found that even at temperature zero, the setting meant to remove randomness, the same prompt returns different answers on a large share of tasks. Fourth, the backtest and out-of-sample gate we spent Episode 3 defending: it is the core of the allocator; it is not native to a chat prompt. Fifth, awareness of risk, cost and tax: the allocator is judged net of our twenty-six percent Italian capital-gains tax; the prompt, by default, knows none of that. Sixth, transparency: the allocator is fixed weights plus logs you can audit; the prompt is an opaque draw you cannot fully inspect. And seventh, cost and latency: here, finally, the LLM wins — instant and cheap, while training the allocator is heavy. So the honest scorecard: on six of seven dimensions that decide whether you should trust an allocation, the trained model wins. The LLM wins on speed and price alone. Hold that thought, because next we will see what happens when a headline claim of LLM superiority meets replication.
A cautionary tale: the withdrawn paper
So far we've seen that language models can read a signal, but that reproducibility is where they stumble. Now let me tell you a cautionary tale that pulls both threads together. In 2024, a striking paper appeared from Kim, Muhn and Nikolaev — arXiv preprint number 2407.17866. It reported that a general-purpose language model, given only the numbers in a financial statement, could rival a trained machine-learning model at analysing that statement, and even beat professional analysts at predicting the direction of future earnings. That is a headline result. It circulated widely. It was exactly the kind of finding you would screenshot and send to your investment committee. And here is the turn: in February 2025, the paper was withdrawn. Not by a critic, not by a rival — by the authors themselves, after a coauthor tried to replicate it and found inconsistencies in the data. Let me be careful about the lesson, because it is easy to draw the wrong one. The lesson is not "language models are useless at reading statements." The earlier evidence we cited still stands on its own terms. The lesson is that this field is young and noisy. Headline claims arrive fast, they get amplified faster, and some of them retract. And notice what did the retracting: not a clever argument, but an attempt to run the thing again and get the same answer. Reproducibility — the same fault line we keep returning to. A demo persuades because it is fluent and it is singular; you see it work once. A gate persuades because it survives being repeated, on data the model never touched, by someone who wants it to fail. This is precisely why our discipline is to trust the gate and not the demo. In the next slides we'll turn that discipline directly onto language models themselves.
Where LLMs genuinely help — and do not
So let us be fair, and let us be precise. There is real work an LLM — a large language model, a system trained to predict text — does genuinely well, and we should use it there. It is excellent at summarising: hand it a two-hundred-page filing and it will give you a faithful digest in seconds. It drafts and checks code, catching the off-by-one error before you run the backtest. It can generate a sentiment feature — a numeric score of how positive or negative a document reads — that you then feed, as one input among many, into a trained model. And it makes a tireless research assistant and devil's advocate: ask it to argue against your own thesis and it will find the weak joints. These are the uses the evidence actually supports. Remember, the papers we cited — Lopez-Lira and Tang in 2023, and the work that followed — tested exactly this: sentiment signals extracted from news and filings, not one-shot allocation. Now the other column. There are things the LLM does not — yet — own. It does not own the allocation decision itself: how much weight goes to each ETF. It does not own the numeric risk optimisation, the step that balances expected return against volatility and correlation. It does not own the tax and cost model, the friction that quietly decides whether a strategy survives contact with reality. And it does not own the walk-forward verdict — the test on data the model never saw during training, which is the only honest judge we have. The rule is simple, and it is the spine of this episode: use the tool for what it is genuinely good at, and keep the decision inside the process. The LLM reads; the pipeline decides. In the next slide we make that split concrete — because the gate we built for our own models applies to the language model too.
The gate applies to LLMs too
So here is the point that ties this episode back to everything we built earlier in the series. We spent whole episodes defining one bar, one gate, that any strategy has to clear before we trust it with real money. Let me say it plainly one more time. A strategy passes only if its median alpha — that is, its return above a benchmark, taken at the middle of many runs, not the lucky best one — is positive across all three windows: the training window, the test window, and the validation window. And it has to stay positive after we subtract the 26 percent Italian tax on financial gains. And even then, it has to beat two things at once: the index itself, and the simplest free rule we can write down, like equal weight held and rebalanced. That is the gate. Now here is the honest part. The gate does not care how clever the source of the idea is. It does not care whether the weights came from a trained network, a coin flip, or a large language model — a system that predicts the next word from patterns in text. The gate only asks one question: does the edge survive out of sample? And this is exactly where a one-shot allocation prompt struggles today. Because of the look-ahead problem we walked through — the model having quietly seen the future — and because of non-determinism — the same prompt giving different answers on different days — demonstrating that a spot LLM allocation clears this gate, honestly, is very close to impossible right now. Not because the model is stupid. Because we cannot cleanly measure whether its edge is real or remembered. So the standard does not soften for a fluent answer. It holds. In the next slide, let us put all the evidence we have gathered onto one table.
The evidence on the table
Before we close, let's put every source on the table, because nothing we've claimed tonight rests on our opinion. It rests on published work you can go read yourself. When we said a large language model — an LLM, a text model trained to predict the next word — can genuinely read signal from news, that came from Lopez-Lira and Tang, 2023, who tested sentiment scores from headlines. When we warned about look-ahead bias, the model quietly using information it could not have known at the time, that was Glasserman and Lin, 2023, and the distraction effect from Ouyang and colleagues. Remember the fair framing: these papers score sentiment signals from news and filings — they do not test asking a model, in one shot, for portfolio weights. Please don't stretch them into "trust the allocation prompt." That leap is ours to avoid. When we said domain models are trained, not prompted, that's BloombergGPT, from Bloomberg's own team, and the open FinGPT project. And the cautionary tale — the paper that reported results, then was withdrawn — was Kim and colleagues. We name it not to mock anyone, but because a withdrawn result is exactly what honest evidence looks like when it corrects itself. Two honest caveats. First, the numbers we quoted are estimates, and they are paper-dependent — different data, different windows, different scoring. Treat them as direction, not decimals. Second, we cite these so you can check us, not so you take our word. Read the originals. If we got something wrong, the papers will tell you before we do. That is the whole spirit of this series: fluency is cheap, verification is not. So in our last slide, let's say plainly what all of this adds up to.
Fluency is not a forecast
So let us bring this episode home. We have tried all evening to be fair, so let us say the good part plainly first. A large language model, a system trained to predict the next word across enormous amounts of text, is a genuinely powerful reader and writer of language. It is a superb research assistant. It summarises a filing, drafts a hypothesis, explains a method, and, as the studies we walked through showed, it can extract real signal from news and company filings. As a feature-extractor, feeding a disciplined pipeline, it earns its place. None of that is in doubt, and none of it should be dismissed.
But here is the one line to carry out of this room. A fluent, confident allocation is not a forecast. When you type "give me a portfolio" and a model answers in a paragraph of calm, well-argued weights, remember what it is not doing. It is not optimising any risk objective, because it minimises no loss over returns. You cannot reproduce it twice, because the same prompt on another day gives you another answer. And its apparent track record is contaminated by look-ahead, because the model read the future while it was training. Eloquence is not evidence. The three failures we saw are not bugs to be patched away; they are what the tool is.
So the edge was never the model's eloquence. The edge is the disciplined, testable process around it, the same gate we defended from the first episode: hold out data, penalise the trial count, read the number once, on history the strategy never touched. Use the language model for what it is genuinely good at, and let the process, not the prose, decide the allocation.
Next episode, we ask a different question: performance across economic regimes. Who wins where, and why.
