Research method, drawn as a working cycle

Strong inference, one step at a time

Strong inference is a way to do research: write several rival explanations, design an experiment that rules some of them out, then repeat with what is left. Press Play to watch an invented ML project move through the workflow, and see what each step needs, does, produces and stores.

An untested concept. Nobody has run this workflow on a real problem. Every run and number in the walkthrough is invented, except the two facts marked (real). The “needs” and “outputs” fields are this page’s summary of the diagram and its explanation.

Written before the run The run After the result Run not clean Arrow keys step, space plays
Next step Back to an earlier step Recycle New objective Read or write the tree A Popper rule applies here

The notebook, as of this moment

This is what the project has kept so far. Nothing is erased: an excluded hypothesis stays with the run that excluded it, so nobody walks that road again. Scrub the timeline and watch it grow.

Logical tree

A node is a question and its hypotheses. An experiment is an edge, and its outcome picks the branch.

Project knowledge

What holds, with its runs and conditions. The next project reads this at step 2.

Findings

What nobody predicted. A finding suggests a question and supports nothing yet. Select a stage in its trail to jump there.

Rescue log and hygiene

Every not-clean run is logged with its cause. A hypothesis saved repeatedly is excluded.

How knowledge is kept, and what limits it

Each kind of record is written at one step and read at another. Each can also be undone: nothing in the tree is final.

RecordWhat it saysWrittenReadWhat can undo it
ExclusionThis hypothesis failed, in this run, on a test of this severity.Step 9Tree, at steps 2, 3 and 4A change of conditions or a contradicting result reopens it as a finding.
SurvivorBest of this family, not proven. “None of the above” stays live.Step 9Step 10, where it must pass its own risky testA failed use at step 10, or what the Stop report names as still able to refute it.
FindingSomething no hypothesis predicted, with its size and run.Step 8, kept at step 9Step 3 every cycle, step 4 after a recycleA decision with a reason: promote, defer or drop.
Rescue logA run was not clean. Its cause, and the declared auxiliary that was blamed.The not-clean branchWhenever a hypothesis keeps being savedIt does not go away. A hypothesis saved repeatedly is excluded.
Project knowledgeWhat holds, the runs behind it, and the conditions they ran under.Step 10Step 2 of the next projectThe same rule as exclusions: it holds under its conditions.

The life of a finding

  1. SeenStep 8 · openThe standing checks show it. It is written with its size and run.
  2. KeptStep 9 · openIt is recorded, whatever the gates decide about the hypotheses.
  3. ReopenedAny time · openChanged conditions turn an old exclusion into a new finding.
  4. DecidedStep 3 · 3 waysRepeats and needed: promote. Repeats, not needed: defer. Does not repeat: drop.
  5. TestedSteps 4 to 7 · promotedIt becomes a how or why question with rivals and a new experiment.
  6. UsedStep 10 · closedThe surviving explanation changes something, and the change is measured.
  7. ReportedStop · unchangedEvery finding is listed with its status, so stopping cannot lose one.

Three obligations hold this together: step 9 needs a findings entry, even if empty; step 3 needs a decision on each open finding; Stop needs every finding’s status. These are rules of the method. For now they rest on the researcher’s discipline, and nothing checks them.

Where each piece comes from

SourceWhat it adds
Platt, 1964Alternative hypotheses, the crucial experiment, the clean result, recycling, and the logical tree in a daily notebook.
Chamberlin, 1890Hold every rational explanation at once, and allow several causes in varying proportions.
Fudge, 2014The front end: observe, question, predict, and the gates on each.
Popper, 1963Start from a problem, forbid outcomes, risk the prediction, never rescue ad hoc. Marked with an orange dot on the map.
Railsback“None of the above” as a placeholder, and untested is not excluded.
Lakatos, Mayo, MeehlRepeated rescue degenerates; severity; a direction alone risks little. Their notes rest on summaries, one posted section and memory, not on a primary text.
This workflowSteps 8 and 10, the path of a finding, and the rule for reopening exclusions. No source has these; the explanation marks them as inference.

Left out on purpose

These wait until a first working skeleton shows a need.

  • Independent checking: who runs, who judges, who sees what
  • Reusing data across cycles, which can overfit even a holdout set
  • Making sure rival hypotheses really differ
  • Choosing what is worth doing next, by value and cost
  • Whether the measure captures the objective
  • Whether all these rules pay for their cost
  • Thresholds: how many hypotheses, seeds or checks are enough
  • Recording pruned hypotheses
  • Outside criticism and independent replication