Evidence Has a Trajectory

Chapter 9 · de Finetti 1937, Ville 1939, Wald 1947, Robbins 1967, Shafer 2021, Ramdas 2023

Chapters 4–8 built the abductive primitive. You can frame what to observe. You can generate hypotheses from a surprising observation. You can select which experiment to run next. But after the experiment runs and data arrives, how do you know the hypothesis is right?

The standard answer is the p-value: collect a fixed sample, compute a test statistic, compare to a threshold. If p < 0.05, declare significance. The p-value compresses the entire temporal structure of evidence into one number. That compression is mandatory. Peek at the data mid-experiment and the false positive guarantee breaks. The math demands you look once.

This chapter traces a 90-year lineage of work that removes that demand, and ends at its premise: evidence is a bet. Chapter 10 makes the bet formal, a measure of evidence you can compute at every observation and stop on whenever you want, with the error guarantee intact at every moment. The evidence becomes a trajectory. And the trajectory carries temporal patterns that a scalar discards.

Three snapshots that lied

Before the theory, three cases where compressing evidence to a snapshot produced catastrophic decisions.

Google: feedback loop

Google A/B-tested ad load: more ads per page, more revenue per session, ship it. The test assumed stationarity (the treatment effect stays constant) and independence (each user's behavior is unaffected by others'). Both assumptions reduce to one: no feedback. The treatment enters, the outcome exits, nothing loops back.

Long-term measurement revealed the loop. Users learned to ignore ads. Click-through rates degraded over months. Ad blindness spread as a learned behavior. Google cut mobile ad load by 50%. The snapshot said "more ads = more revenue." The trajectory said "more ads = more revenue now, less revenue later, and the crossover happens outside your test window."

Boeing: oscillation killed 346 people

The Maneuvering Characteristics Augmentation System (MCAS) read a single sensor snapshot: nose too high. Correct response: push it down. Correct logic, wrong sensor. The angle-of-attack sensor was faulty. MCAS pushed the nose down. Pilots trimmed up. MCAS pushed again. Two subsystems fighting. The oscillation diverged. Lion Air Flight 610. Ethiopian Airlines Flight 302. 346 dead.

Nobody was watching the trajectory. Each MCAS activation was locally correct. The snapshot said "nose too high, push down." The oscillation between MCAS and pilot inputs was the signal. A system that tracked the temporal pattern of its own corrections would have seen the fight. The snapshot saw only the current angle.

Vioxx: delayed cardiovascular risk

Rofecoxib (Vioxx) passed its 9-month trial: fewer gastrointestinal bleeds than naproxen, ship it. The cardiovascular risk emerged at 18 months, past the trial window. Cumulative thrombotic damage, invisible in the snapshot. 88,000–140,000 excess coronary events before the drug was pulled.

The trial tested for GI bleeds over 9 months. It found what it tested for. The cardiovascular trajectory was in the data (thrombotic risk accumulated monotonically), but the test window ended before the risk crossed the detection threshold.

Same structure, same failure

Each case has the same skeleton:

1. A system with feedback (users adapt, pilots fight the computer, cumulative organ damage).
2. A test that assumes no feedback (stationarity, independence, fixed time window).
3. A scalar summary (p-value, revenue delta, hazard ratio) computed at a single moment.
4. A decision shipped on the scalar.
5. A trajectory that would have revealed the danger, had anyone preserved it.

Meehl's structural flaw at industrial scale. In 1967 Meehl showed that directional predictions in soft psychology corroborate nothing, because the crud factor, the background reality that everything correlates a little with everything, guarantees significance. The same structure (directional prediction, snapshot window, no trajectory) now ships products and drugs. A/B testing culture inherited the pattern wholesale: directional predictions, fixed sample sizes, ship on significance.

Betting line

The alternative to snapshot testing has a 90-year lineage that ran parallel to Fisher's fixed-horizon view for decades before converging.

The lineage is older than ninety years, in two senses. Follow the word: probability descends from the Latin probare, to test or prove, the root behind probe, proof, approve, and probity. At the root, to be probable is to be probe-able, able to withstand a test. Follow the mathematics and it was born at a gaming table. Pascal and Fermat's 1654 letters solved a gambler's problem of how to split the stakes of an interrupted game; Huygens made the exchange the first printed treatise in 1657, On Reckoning in Games of Chance, and its founding quantity, expectation, is simply the fair price of a wager. Probability did not begin as a frequency or as a belief. It began as a bet.

Ian Hacking's The Emergence of Probability (1975) finds the term arriving already two-faced, one side the aleatory frequency of dice, the other the epistemic sense of an opinion approved and borne out under examination. Boole carried both faces into the algebra of logic when he extended it to probability in the Laws of Thought (1854), and Peirce, reading exactly that treatment, took the testing sense forward into his logic of inference. The twentieth century let the faces drift apart, until probability sounded like nothing but a long-run ratio and both the wager and the probe decayed into dead metaphor.

That drift is why the frequentist-versus-Bayesian quarrel feels forced: it asks which single face the word really wore, when the word predates both schools. The frequentist reads probability as a ratio and demands a reference class, a population for the ratio to count over; the Bayesian reads it as a degree of belief and demands a prior. The betting line below is neither, and older than either. Its first entry, de Finetti, still read the bet as a degree of belief; Ville kept the wager and dropped the reading, and it is Ville's line the rest of the table extends. In Shafer and Vovk's game-theoretic probability a hypothesis faces one operational question: can a bettor multiply capital against it? No reference class to posit, no prior to assume. An e-value is that bet's payoff, the probe made literal, the running record of how much testing the hypothesis has survived. It reunites the wager and the probe that the modern split drove apart, and returns probability to the gaming table where it was born.

The modern thread, in six steps:

De Finetti: evidence is a bet. Ville: test a hypothesis by trying to make money against it. Wald: you don't need a fixed sample size. Doob: the guarantee holds no matter when you stop. Robbins: confidence intervals hold at every peek. Ramdas, Shafer, and Grünwald unified it and named the central object.

The premise: a casino where the hypothesis is the house

Before the formal definition, the picture that motivates it. To test a hypothesis, open a casino in which that hypothesis is the house. The null hypothesis, the claim under test that nothing is going on, sets the odds: if it says the coin is fair, heads pays double; if it says the drug does nothing, recoveries pay out as if they arrive at the placebo rate. You walk in with one dollar and bet on what you actually expect to see. Each round of data settles the wager, and you may let winnings ride into the next round.

Your bankroll is the evidence. If the null is true, every bet in the house is fair or worse for you, so no strategy grows a dollar on average, and reaching twenty dollars is something a true null allows at most one time in twenty. If the null is false, the odds are mispriced, and a bettor who sees the mispricing compounds. Wealth won against a hypothesis is exactly as hard to come by as the hypothesis is close to true, which makes the bankroll a calibrated evidence meter.

This is what the bankroll buys over a p-value. A p-value is a verdict issued once, at a sample size fixed in advance; look twice and the guarantee is void. A bankroll is a running quantity. You can watch it every round, cash out whenever you choose, and carry winnings from one table to the next by simple multiplication. Chapter 10 makes the three casino facts formal: what counts as a fair bet (the definition), winnings carrying across tables (composition), and cashing out at will (anytime validity).

The bankroll also pays Chapter 8's debt. Peirce's economy of research says to spend where marginal information per dollar is highest and to abandon a line of inquiry when its return drops below the common frontier, and the limit was that the return could only be computed after the fact, from utility curves nobody has at the start. The bankroll makes the return a live reading. Run each open line as a wager against its own null and keep a tab of what its experiments cost; the running return is bankroll growth per dollar spent. A line whose bankroll compounds on a small tab is the one to fund. A line whose bankroll has stalled while its bill grows has priced itself out, and you can quit the moment the chart says so. That is the sequential reading of evidence Chapter 8 promised this chapter would deliver.

Exercises

💻 marks exercises meant for a keyboard. ★ marks open-ended problems with no single right answer.

9.1 In the casino picture, explain in two sentences why a true null cannot grow your bankroll on average, and why reaching twenty dollars from one licenses at most a one-in-twenty false alarm.

9.2 A fitness app A/B tests a streak badge and finds a 9% retention lift at week two, p < .01. Users soon learn to open the app for one second at 11:59pm to keep the streak. Write out the five-step skeleton of this failure, one line per step, and mark the step where the feedback enters.

9.3 A streaming service tests autoplay-next-episode against the null "no effect on weekly watch time." The bankroll against the null compounds for the first month, flattens during the second, and collapses during the third. Tell the story of what the users did, in the style of the ad-load case. What would a p-value computed at day 30 have said, what would one computed at day 90 have said, and why can't the two snapshots be reconciled without the trajectory?

9.4 💻 Build the casino for a die you suspect favors six. The null says the die is fair. Choose the bet the suspicion implies, then simulate 100 rolls under each truth (fair die, loaded die) in a spreadsheet or ten lines of Python, letting winnings ride. Plot both bankrolls. How many rolls does the loaded case need before the bankroll makes the case?

9.5 ★ Find a decision you or your team shipped on a snapshot: an A/B test, a diet that "worked" after two weeks, a hiring process change judged after one quarter. Reconstruct the evidence as a trajectory, as far back and forward as your records allow, and sketch its shape: steady decay, steady compounding, or rise-then-collapse like the ad-load case. Would a bankroll-watcher have shipped when you did, earlier, or never?

Sources