Economy of Research

Chapter 8 · Peirce 1879, 1903, Chamberlin 1890, Platt 1964, de Kleer 1987, Hintikka 1999

Hypothesis generation is cheap. Testing is expensive. Every experiment costs time, compute, reagents, or attention. A single surprising observation can spawn dozens of candidate explanations. Running all of them is never feasible. Which subset do you test, in what order, and when do you stop?

Peirce formalized this in 1879. Chamberlin addressed the cognitive prerequisite in 1890. Platt operationalized it in 1964. De Kleer and Williams automated it in 1987. Hintikka gave it game-theoretic foundations in the 1990s. The machine-learning literature keeps reinventing the same criteria without connecting back to abduction.

Peirce: marginal return on research investment (1879)

In "Note on the Theory of the Economy of Research," Peirce made the allocation problem explicit: given a fixed budget, how do you distribute it across experiments to maximize what you learn?

His answer used marginal analysis. Each dollar spent on an experiment yields some increment of knowledge. That increment diminishes as you pour more into the same line of inquiry. The optimal allocation equalizes marginal return per dollar across all active experiments. If A yields more information per dollar than B, shift budget from B to A until the marginals equalize.

Peirce derived a specific result: optimal sample size depends on the ratio of information utility to observation cost. High-utility, low-cost experiments get large samples. Low-utility, high-cost experiments get small samples or none. Obvious in retrospect, but Peirce formalized it 68 years before Wald.

"The doctrine of economy, in general, treats of the relations between utility and cost. The economy of research is that particular application of it which gives the rules for the expenditure of money, energy, and time upon the different elements of an investigation."

— Peirce, "Note on the Theory of the Economy of Research" (1879)

Wald (1947) asked: given a stream of data, when should you stop collecting? Peirce asked the prior question: which stream should you open? Wald's work became a field. Peirce's note was ignored until historians rediscovered it in the 1960s.

Two limits kept the note unusable in its own century. Its utility term has no measure: the mathematics assumes you can put a number on the value of knowing something more exactly, and nobody in 1879 could produce one. And it optimizes against utility and cost curves it treats as known before the work begins, when in a real investigation you learn the cost of a result by paying for it, tracing the curve rather than receiving it. Neither limit is fatal, because the note was never delivering a number. It was isolating a form: spend so that marginal value per unit cost is equal across the active lines, and abandon a line when its marginal drops below the common frontier. Shannon's entropy, reaching this problem through Lindley and de Kleer's diagnostic engine below, later supplied the missing information term. The second limit dissolves in Chapter 9, where the sequential reading of evidence lets you quit the instant the running return crosses the frontier, without ever having had to know the frontier in advance.

Peirce again: the leading consideration (1903)

The 1879 note reads like a budgeting aside. By the 1903 Lowell lecture on abduction it had become the whole doctrine of hypothesis selection: "the leading consideration in Abduction, in all cases, ... is the question of Economy, — Economy of money, time, thought, and energy" (MS [R] 475; CP 5.600). Economy is no longer one consideration among several. Peirce argues there are no other serious ones, because "abduction commits us to nothing. It merely causes a hypothesis to be set down upon our docket of cases to be tried." Adding a candidate is free; only testing spends. Whatever ranks the candidates must therefore be an accounting of the testing budget.

The lecture also states the selection rule this chapter automates:

"If two hypotheses present themselves, one of which can be satisfactorily tested in two or three days, while the testing of the other might occupy a month, the former should be tried first, even if its apparent likelihood is a good deal less."

— Peirce, Eighth Lowell Lecture (1903), MS [R] 475

And it takes a side that still divides the field: "It is a very grave mistake to attach much importance to the antecedent likelihood of hypotheses, except in extreme cases; because likelihoods are mostly merely subjective, and have little real value." His extreme case is a man offering to turn lead into gold: "My dear sir, I haven't time to make gold." He prices even that refusal in time. Bayesian experimental design (below) leans on exactly what Peirce refused to lean on.

The refusal is principled, and he argued it for decades. For Peirce a probability is a ratio of frequencies, and a ratio needs a reference class. The same lecture supplies the image: "If you could put all possible universes into a bag and after shaking them well up could draw out universe after universe, you might form statistics as to the proportion of universes in which the proposition is true. But since you cannot do that, to speak of the probability of the truth of a general proposition ... leaves us at a loss to conjecture what the other term of that ratio can be." A hypothesis has no reference class, so its probability is undefined rather than merely imprecise. "The Probability of Induction" (1878) makes the case against probability as degree of belief; "A Theory of Probable Inference" (1883) draws the consequence: the conclusion of an ampliative inference, one whose conclusion outruns its premises, carries no probability. What can be graded is the method that produced it, by how reliably it leads to truth. Which conjecture to spend on next is a question the probability calculus cannot receive. Economy restates it in units the lab has.

The objection generalizes past Peirce: some questions outrun the data type they are asked of. Hume observed that causation is never in the data, only conjunction. Statistics compressed that into correlation is not causation. Holland's "no causation without manipulation" (1986) and Cartwright's "no causes in, no causes out" (1989) sharpened it into slogans. Pearl made it formal: interventional questions are unanswerable from observational distributions without causal assumptions that observation cannot itself test. The remedy is the same in both cases, and Peirce had already prescribed it: "Every hypothesis should be put to the test by trying, if possible, to make verifiable predictions" (MS [R] 475). Perturbation is required; the only live question is its price. Buy the answer with an experiment, and buy the cheapest decisive one first.

The lecture credits Ernst Mach as having "done the most to show the importance in logic of the consideration of Economy," while noting he "had written a paper on the subject as early as 1878": the note above, published in 1879.

Chamberlin: breadth before depth (1890)

Chamberlin's "Method of Multiple Working Hypotheses" addresses the cognitive prerequisite. You cannot select optimally among hypotheses you haven't generated. And you won't generate competitors to your favorite if you have one.

His argument is structural. A scientist holding one ruling theory unconsciously steers experiments toward confirmation. Fitting data is amplified; contradicting data is explained away. The remedy: hold multiple hypotheses simultaneously, without ranking, and let data sort the field.

Chamberlin supplies the input to Peirce's economy. Before you can ask "which experiment next?" you need a hypothesis set broad enough that the true explanation is likely among them.

Platt: strong inference as a selection protocol (1964)

Platt operationalized Chamberlin into a loop: enumerate alternative hypotheses, design a crucial experiment that excludes at least one, run it, repeat. His contribution is the crucial experiment: one whose outcome eliminates hypotheses regardless of which way it goes.

Crucial experiments are maximally economical. A non-crucial experiment can confirm a hypothesis without eliminating alternatives; it spends budget without shrinking the hypothesis space. A crucial experiment guarantees the space shrinks every round. Platt observed that fields practicing strong inference (molecular biology, particle physics) moved faster than fields that didn't (psychology, ecology). The rate difference tracks the fraction of experiments that are crucial.

But Platt's framework is qualitative. He tells you what a good experiment looks like. He does not tell you how to rank several crucial alternatives. For ranking, you need de Kleer.

De Kleer & Williams: GDE and computed experiment selection (1987)

The General Diagnostic Engine (GDE) computed which measurement to take next from the structure of remaining hypotheses. The context was circuit diagnosis. A chip has a fault, multiple components could be responsible, and each probe has a cost.

GDE maintains candidate diagnoses: minimal sets of faulty components consistent with all observations so far. For each possible measurement, it computes how much probing that node reduces the entropy of the diagnosis distribution, the remaining uncertainty measured in bits, then selects the measurement with the highest information gain per unit cost.

The algorithm is straightforward:

1. Enumerate remaining candidate diagnoses, each with a probability.
2. For each possible measurement, compute the expected posterior entropy over diagnoses.
3. Subtract from the current entropy to get expected information gain.
4. Divide by measurement cost. Select the measurement with the highest ratio.

GDE realizes the same structure as Peirce's economy (marginal information per dollar), Platt's crucial experiments (maximize elimination), and Shannon's information theory (entropy reduction) in a single algorithm. The domain was circuit diagnosis, but the architecture generalizes. Replace "component" with "hypothesis" and "probe" with "experiment" and you have economy of research, automated.

Hintikka: inquiry as a game (1988/1999)

Hintikka recast investigation as a two-player game between the Inquirer and Nature. The Inquirer asks questions (runs experiments). Nature answers (provides data). The goal: reach a definite conclusion at minimum total cost.

Each question is an abductive act: "If H were true, what would I observe at point X?" The answer supports H or eliminates it. The optimal strategy is economy of research expressed as a game tree.

The game-theoretic framing captures what information theory misses: the next question depends on previous answers. GDE recomputes from scratch after each measurement. Hintikka encodes the dependency structure. The Inquirer's strategy is a policy, a function from observation history to the next question. This is the difference between a sorted list and a decision tree.

Modern connections

Machine learning has reinvented economy of research under several names, each formalizing "which experiment next?" without citing Peirce or Hintikka.

Active learning. Which unlabeled point should an oracle label next? The standard criterion (uncertainty sampling, query-by-committee) selects the point the model is most uncertain about. This is GDE's entropy-reduction criterion restricted to a single model class.

Bayesian experimental design. Choose the experiment that maximizes expected information gain about model parameters. Lindley (1956) formalized this; Chaloner and Verdinelli (1995) surveyed the field. The criterion is GDE's, applied to continuous parameter spaces instead of discrete diagnostic candidates.

Multi-armed bandits. Allocate trials across K arms, the candidate options, to maximize cumulative reward (exploration-exploitation) or identify the best arm (pure exploration). The pure-exploration variant is Peirce's economy of research: fixed budget, maximize what you learn.

None of these literatures connects experiment selection back to abduction. The hypothesis set is taken as given. This matters because the quality of the hypothesis set determines the ceiling of any selection strategy. Abduction is where that set comes from.

Code: information gain per unit cost

Given N hypotheses with prior probabilities and a set of possible experiments, each with a cost and a likelihood of each outcome under each hypothesis, compute which experiment maximally reduces uncertainty per dollar spent.

import math

def entropy(probs):
    """Shannon entropy of a probability distribution."""
    return -sum(p * math.log2(p) for p in probs if p > 0)

def expected_posterior_entropy(priors, likelihoods):
    """Expected entropy after running an experiment.

    likelihoods[i][k] = P(outcome_k | hypothesis_i)
    """
    n_hypotheses = len(priors)
    n_outcomes = len(likelihoods[0])

    # P(outcome_k) = sum_i P(outcome_k | H_i) * P(H_i)
    p_outcomes = []
    for k in range(n_outcomes):
        p_k = sum(likelihoods[i][k] * priors[i] for i in range(n_hypotheses))
        p_outcomes.append(p_k)

    # For each outcome, compute posterior entropy
    expected_H = 0.0
    for k in range(n_outcomes):
        if p_outcomes[k] == 0:
            continue
        # Posterior: P(H_i | outcome_k) = P(outcome_k | H_i) * P(H_i) / P(outcome_k)
        posterior = [
            likelihoods[i][k] * priors[i] / p_outcomes[k]
            for i in range(n_hypotheses)
        ]
        expected_H += p_outcomes[k] * entropy(posterior)

    return expected_H

def select_experiment(priors, experiments):
    """Select the experiment with highest information gain per unit cost.

    experiments: list of (name, cost, likelihoods) tuples
    Returns sorted ranking.
    """
    current_H = entropy(priors)
    results = []

    for name, cost, likelihoods in experiments:
        post_H = expected_posterior_entropy(priors, likelihoods)
        info_gain = current_H - post_H
        ratio = info_gain / cost
        results.append((name, info_gain, cost, ratio))

    results.sort(key=lambda x: x[3], reverse=True)
    return results


# --- Example: 4 hypotheses, 3 possible experiments ---

priors = [0.50, 0.25, 0.15, 0.10]

experiments = [
    # (name, cost, likelihoods[hypothesis][outcome])
    # Experiment A: cheap but only discriminates H1 vs rest
    ("A: probe signal line", 1.0, [
        [0.9, 0.1],   # H1: 90% positive
        [0.2, 0.8],   # H2: 20% positive
        [0.3, 0.7],   # H3: 30% positive
        [0.25, 0.75],  # H4: 25% positive
    ]),
    # Experiment B: expensive but discriminates all four
    ("B: full spectrum analysis", 5.0, [
        [0.8, 0.1, 0.1],  # H1
        [0.1, 0.8, 0.1],  # H2
        [0.1, 0.1, 0.8],  # H3
        [0.05, 0.05, 0.9], # H4
    ]),
    # Experiment C: moderate cost, sharp on H2 vs H3
    ("C: thermal imaging", 2.0, [
        [0.5, 0.5],   # H1: uninformative
        [0.95, 0.05], # H2: hot
        [0.05, 0.95], # H3: cold
        [0.5, 0.5],   # H4: uninformative
    ]),
]

print(f"Prior entropy: {entropy(priors):.3f} bits\n")
print(f"{'Experiment':<30} {'IG (bits)':>10} {'Cost':>6} {'IG/Cost':>10}")
print("─" * 60)

for name, ig, cost, ratio in select_experiment(priors, experiments):
    print(f"{name:<30} {ig:>10.3f} {cost:>6.1f} {ratio:>10.3f}")

Output:

Prior entropy: 1.749 bits

Experiment                         IG (bits)   Cost    IG/Cost
────────────────────────────────────────────────────────────────
A: probe signal line                   0.386    1.0      0.386
C: thermal imaging                     0.319    2.0      0.160
B: full spectrum analysis              0.706    5.0      0.141

Experiment B has the highest absolute information gain (0.706 bits). A naive strategy picks B. But B costs five times as much as A, and per unit cost A delivers 2.7x more information. Run A first.

After running A and updating posteriors, the ranking changes. If A's outcome concentrates probability on H2 and H3, experiment C (which sharply discriminates exactly those two) jumps to the top. The optimal sequence is adaptive. This is Hintikka's point: the next question depends on the previous answer.

Three orderings, three failure modes

Three natural strategies for experiment selection, each producing a different sequence.

The third strategy is the best single-step heuristic. GDE uses it. It is still myopic. It optimizes one step ahead without considering the trajectory. Hintikka's game tree optimizes the full sequence, but computing the optimal policy is exponential in the number of experiments. Under independence and uniform costs, information gain is submodular (each added experiment gains less than the one before), and greedy selection stays within a constant factor of optimal. Correlated outcomes and variable costs weaken this guarantee, but greedy remains hard to beat in practice.

Gap: evidence trajectory

Economy of research selects which experiment to run. It says nothing about what to do with the evidence once it arrives. You run experiment A, observe the outcome, update your beliefs. But how do you know the update is accumulating toward a conclusion rather than oscillating between hypotheses?

A sequence of experiments might produce: evidence for H1, then against H1, then for H1 again. The posterior oscillates. Are you converging or chasing noise? Economy of research cannot answer this. It selects the next experiment but cannot read the trajectory of evidence across experiments.

That gap motivates Chapter 9. Evidence has a trajectory, and you need tools to read its shape.

Exercises

💻 marks exercises meant for a keyboard. ★ marks open-ended problems with no single right answer.

8.1 Peirce says "abduction commits us to nothing. It merely causes a hypothesis to be set down upon our docket of cases to be tried." In two sentences: why does this make economy the leading consideration in hypothesis selection, rather than one consideration among several?

8.2 Two lines of inquiry with diminishing returns. Successive dollars spent on X yield 10, 8, 6, 4, 2 units of information; successive dollars on Y yield 9, 7, 5, 3, 1. (a) Allocate a $6 budget dollar by dollar to maximize total information, and verify the final allocation equalizes marginal return across the two lines. (b) A third line Z appears, yielding a constant 5 units per dollar with no diminishing returns. Explain why Y never receives a fourth dollar at any budget, in the chapter's terms: Y's marginal has dropped below the common frontier.

8.3 Your sourdough didn't rise. Three hypotheses: dead starter (you rate it likely; the float test takes a day), oven too cool during proofing (unlikely; checking with a thermometer takes 5 minutes), bad flour (moderate; a side-by-side bake takes an hour). Order the tests by Peirce's selection rule and defend the order against the objection that the starter is the most probable culprit. Which of the three failure modes in "Three orderings, three failure modes" does likeliest-first commit here?

8.4 💻 Two extensions to the chapter's code. (a) Design an experiment D with cost and likelihoods that beats A on information gain per unit cost, and explain what makes it cheapest-decisive. (b) Implement the adaptive step: suppose A returns its positive outcome, update the priors by Bayes' rule, and re-rank B, C, D. Does C jump the queue as the chapter predicts?

8.5 ★ Write out the open questions in your current project as a docket. For each, estimate its testing cost and a rough information value, then compute the ratio. Find the expensive favorite you were about to run first, and the cheap decisive test Peirce would run instead. Run the cheap one this week and record whether the ranking of everything else changed.