E-values

Chapter 10 · Doob 1953, Robbins 1967, Shafer 2021, Ramdas 2023, Grünwald 2024

E-values: definition from scratch

A p-value is the probability of seeing data at least as extreme as what you observed, assuming the null hypothesis is true. If p is small, the data is surprising under the null. Smaller p = more evidence against the null.

An e-value is the payoff of a bet against the null hypothesis. If the null is true, the expected payoff is at most 1:

E[e] ≤ 1    under the null

That's the entire definition. An e-value is a non-negative random variable whose expectation under the null is at most 1. Large values (e >> 1) mean the null is losing badly. An e-value of 20 means the data is 20 times more consistent with the alternative than the null.

The constraint E[e] ≤ 1 is the supermartingale property: the expected future value of the process is at most its current value. If you bet against the null and the null is true, your wealth cannot grow on average. If your wealth grows anyway, the null is losing.

Why "at most 1" and not "equals 1"

A martingale has E[e] = 1 exactly. A supermartingale relaxes this to E[e] ≤ 1. The relaxation matters because it makes e-values composable. If you run two sequential experiments producing e-values e₁ and e₂, their product e₁ × e₂ is also an e-value. The supermartingale property guarantees it: two honest bets composed are still an honest bet. This composition does not require any correction for multiple testing. It is exact.

P-values do not compose this way. Multiply two p-values and the result is not a valid p-value. Corrections (Bonferroni, Holm, Benjamini-Hochberg) restore validity but throw away statistical power. E-values compose for free.

Anytime validity

The deepest property: you can compute an e-value at every observation and the guarantee holds at each one. No peeking penalty. No alpha spending, the rationing of your error budget across looks. Look whenever you want, stop whenever you want, false positive rate controlled at every moment.

This is Doob's optional stopping theorem in action. A supermartingale evaluated at any stopping time is still a supermartingale. The math does not care when you look or why you stopped.

Contrast with a p-value. Compute p at n=100: 0.06, not significant. Peek again at n=200: 0.04, significant. Your false positive rate is no longer 5%. Peeking twice inflated it. The p-value guarantee requires exactly one look. The e-value guarantee holds at every look.

Trajectories expose feedback loops

A p-value compresses evidence into a scalar. An e-value stream preserves evidence over time. The system's dynamics live in the temporal structure. Compress to a scalar and they vanish.

Point an e-value stream at a system with feedback. The trajectory preserves temporal patterns that a terminal scalar discards.

Smoking and stress. The e-value climbs after each cigarette (relief), falls during withdrawal (stress spikes), climbs on relapse. The oscillation is the addiction cycle. A two-week snapshot says "smoking reduces stress, p < .05." The trajectory shows the loop.

Ad load. An e-value stream tracking revenue per session climbs for weeks (more ads = more revenue), flattens (ad blindness developing), then declines (degraded engagement). The inflection point marks the moment the feedback loop activates. A fixed-sample test that ends before the inflection sees only the climb.

MCAS. An e-value tracking "does the correction succeed?" oscillates. Push nose down (evidence for success), pilot trims up (evidence resets), MCAS fires again. The oscillation is two subsystems fighting. A snapshot at any single activation says "correction successful."

Evidence compounds across experiments

E-values compose multiplicatively across sequential experiments.

Run ten small experiments, each isolating one variable. Experiment A gives e₁ = 3 (suggestive). Experiment B gives e₂ = 4 (suggestive). Neither is strong alone. But e₁ × e₂ = 12, strong combined evidence, valid with no correction.

Change your test, your sampling strategy, your hypothesis mid-stream. The evidence still composes. The supermartingale guarantee holds regardless. Fixed-method testing cannot accumulate evidence across changing experiments. E-values can.

Code: e-value trajectory for a sequential test

A simple sequential test. Observations arrive one at a time. Under the null they are draws from N(0, 1), the standard normal. Under the alternative, N(μ, 1) for some μ > 0. At each step, compute the likelihood ratio (the simplest e-value) and track the trajectory.

The code below is written out so you can see every moving part. It is also packaged: pip install methodeutics gives you the same bankroll in three lines (Bankroll(normal(0, 0.5), alpha=0.05).extend(xs).plot()), with the threshold pre-committed and the crossing remembered. Source at github.com/kimjune01/methodeutics.

import math
import random

def e_value_trajectory(observations, mu_alt=0.5):
    """Compute cumulative e-value trajectory via likelihood ratio.

    E-value at step t = product of likelihood ratios up to t.
    Under H0: X_i ~ N(0,1). Under H1: X_i ~ N(mu_alt, 1).
    Likelihood ratio for one observation:
        L(x) = exp(mu_alt * x - mu_alt^2 / 2)
    Cumulative e-value: product of L(x_i) for i = 1..t.
    """
    trajectory = []
    log_e = 0.0

    for x in observations:
        # Log-likelihood ratio for one observation
        log_lr = mu_alt * x - (mu_alt ** 2) / 2
        log_e += log_lr
        trajectory.append(math.exp(log_e))

    return trajectory

def p_value_at_t(observations):
    """Two-sided z-test p-value computed at the end of the sample."""
    n = len(observations)
    if n == 0:
        return 1.0
    mean = sum(observations) / n
    z = mean * math.sqrt(n)
    # One-sided p-value via normal CDF approximation
    p = 0.5 * math.erfc(z / math.sqrt(2))
    return p


# --- Scenario 1: Null is true (no effect) ---
random.seed(42)
null_obs = [random.gauss(0, 1) for _ in range(100)]
null_traj = e_value_trajectory(null_obs)

print("=== Null is true (no real effect) ===")
print(f"E-value at t=10:  {null_traj[9]:.4f}")
print(f"E-value at t=50:  {null_traj[49]:.4f}")
print(f"E-value at t=100: {null_traj[99]:.4f}")
print(f"P-value at t=100: {p_value_at_t(null_obs):.4f}")
print()

# --- Scenario 2: Alternative is true (real effect, mu=0.3) ---
alt_obs = [random.gauss(0.3, 1) for _ in range(100)]
alt_traj = e_value_trajectory(alt_obs, mu_alt=0.5)

print("=== Alternative is true (mu=0.3) ===")
print(f"E-value at t=10:  {alt_traj[9]:.4f}")
print(f"E-value at t=50:  {alt_traj[49]:.4f}")
print(f"E-value at t=100: {alt_traj[99]:.4f}")
print(f"P-value at t=100: {p_value_at_t(alt_obs):.4f}")
print()

# --- Scenario 3: System with feedback (effect reverses at t=50) ---
feedback_obs = (
    [random.gauss(0.5, 1) for _ in range(50)] +  # positive effect
    [random.gauss(-0.3, 1) for _ in range(50)]   # effect reverses
)
feedback_traj = e_value_trajectory(feedback_obs, mu_alt=0.5)

print("=== Feedback system (effect reverses at t=50) ===")
print(f"E-value at t=25:  {feedback_traj[24]:.4f}  (rising)")
print(f"E-value at t=50:  {feedback_traj[49]:.4f}  (peak)")
print(f"E-value at t=75:  {feedback_traj[74]:.4f}  (falling)")
print(f"E-value at t=100: {feedback_traj[99]:.4f}  (collapsed)")
print(f"P-value at t=50:  {p_value_at_t(feedback_obs[:50]):.4f}  (snapshot at peak)")
print(f"P-value at t=100: {p_value_at_t(feedback_obs):.4f}  (snapshot at end)")

Output:

=== Null is true (no real effect) ===
E-value at t=10:  0.1432
E-value at t=50:  0.0005
E-value at t=100: 0.0001
P-value at t=100: 0.2802

=== Alternative is true (mu=0.3) ===
E-value at t=10:  2.3119
E-value at t=50:  68.6328
E-value at t=100: 73236.5661
P-value at t=100: 0.0000

=== Feedback system (effect reverses at t=50) ===
E-value at t=25:  122.3931  (rising)
E-value at t=50:  1161.3707  (peak)
E-value at t=75:  1.5770  (falling)
E-value at t=100: 0.0027  (collapsed)
P-value at t=50:  0.0001  (snapshot at peak)
P-value at t=100: 0.0942  (snapshot at end)

Three scenarios. Three stories.

Null is true: the e-value drops below 1 immediately and keeps falling. By t=100, e = 0.0001. The bet against the null is thoroughly lost. Under the null, honest betting cannot grow in expectation. Individual paths fluctuate, but the trend is downward.

Alternative is true: the e-value climbs steadily. By t=100, e = 73,237. You could have stopped at t=50 (e = 69, already strong) or continued. The guarantee held at every peek. Monotone climb: accumulating.

Feedback system: the Google ad-load scenario. The effect is real for the first 50 observations, then reverses. The e-value climbs to 1,161 at t=50, then collapses as the reversal erodes the accumulated evidence. By t=100, e = 0.003. Meanwhile: p at t=50 = 0.0001 (highly significant), p at t=100 = 0.09 (borderline). Two snapshots, two conclusions, no way to reconcile them. The e-value trajectory tells the whole story: rise, peak, collapse. The shape of the trajectory is the shape of the system.

The p-value's guarantee demands one look

A p-value controls the type I error rate, the false positive rate, at a single, predetermined stopping time. Fix n in advance, compute once: P(p < α | H₀) = α. Peek at t=50 and again at t=100, and you get two chances to cross the threshold. The error rate inflates.

An e-value controls the error rate uniformly over all stopping times. By Markov's inequality and the supermartingale property, for any stopping time τ:

P(eτ ≥ 1/α) ≤ α    under the null

This holds for any stopping time: fixed, random, data-dependent, adversarial. The proof is three lines from Doob's optional stopping theorem. The p-value guarantee requires the stopping time to be independent of the data. The e-value guarantee does not.

The tradeoff: at any single sample size, a p-value test is more powerful. If you commit to looking exactly once at exactly n observations, you should use it. The e-value pays a constant-factor efficiency loss at each time point. What it buys is the entire trajectory.

Connection to Part II

The economy of research (Chapter 8) selects which experiment to run. The e-value tracks what the experiment says over time. The two connect directly:

1. Generate hypotheses via abduction (Chapters 4–6).
2. Select the next experiment via information gain per unit cost (Chapter 8).
3. Run the experiment and track the e-value trajectory (Chapters 9–10).
4. Read the trajectory's shape to classify the system's dynamics (Chapter 11).

Step 3 is new. Before e-values, step 2 fed directly into a terminal decision: run, compute p, decide. Now a trajectory sits between the experiment and the decision, carrying information the terminal scalar discards.

Trajectory shape is ambiguous without classification

You have the trajectory. It oscillates. Is the system fighting itself (MCAS vs. pilots), or is your measurement just noisy?

Structural oscillation (cyclic dynamics in the system) and statistical oscillation (weak effect plus finite-sample noise) look the same in the raw trajectory. The trajectory alone cannot distinguish them.

Classifying whether evidence is converging, diverging, oscillating, or chaotic requires tools from dynamical systems theory: convergence rates, Lyapunov exponents, stability analysis. These tools apply directly to evidence trajectories. Chapter 11 builds the classification scheme.

Exercises

💻 marks exercises meant for a keyboard. ★ marks open-ended problems with no single right answer.

10.1 A colleague reports e = 15 against the null after 40 observations and asks whether looking again at observation 60 would spoil the guarantee, the way peeking spoils a p-value. Answer in two sentences, naming the property that settles the question.

10.2 You suspect a coin comes up heads 60% of the time; the null says it is fair. Bet the likelihood ratio each toss: heads multiplies your e-value by 0.6/0.5 = 1.2, tails by 0.4/0.5 = 0.8. The coin comes up H, H, T, H, H. Compute the e-value after each toss. After which toss is the evidence strongest so far? How many consecutive heads from the start would you need to cross e = 20, and what does that number say about how much evidence a 60%-vs-50% distinction carries per toss?

10.3 Lab A reports e = 3 against the null that a drug leaves a biomarker unchanged. Lab B, working independently with a different assay, reports e = 4 against the same null. (a) State the combined evidence and the property that licenses the combination. (b) A third lab reports p = 0.04 from a fixed-sample test; explain why it cannot be folded in the same way. (c) Lab B stopped collecting the moment its e-value looked good. Does the product survive? Name the theorem that answers this.

10.4 💻 In the chapter's simulation, move the reversal from t = 50 to t = 90 and rerun. How high does the peak get, and does the collapse finish before t = 100? Then restore the reversal to t = 50 and vary the bet: rerun with mu_alt = 0.1 and mu_alt = 1.0. Which bet loses less of its peak after the reversal? (Hint: a smaller mu_alt is a smaller stake per observation, in both directions.)

10.5 ★ Pick a metric you already watch: weekly revenue, resting heart rate, test-suite flake rate. Define a null and a simple alternative, install the companion package (pip install methodeutics), and run a Bankroll on it for a month. Before you start, write down the e-value at which you will act. At the end, report the trajectory's shape and whether the pre-committed threshold changed how it felt to look at the data every day.

Sources