The evidence
What runs today
The longitudinal machinery is not a design document. It runs.
Running
- state that persists per person across sessions
- evidence → versioned hypothesis → prediction → revision
- a full revision history, with provenance and audit
- current state that changes later system behaviour
- evidence, state transitions and hypothesis history open to inspection
- a deployed TypeScript and Node service on PostgreSQL with pgvector
It was built for a different product — a relational AI application, not professional decision-making — and it has had no external users. Both facts are worth stating early rather than discovering later. Neither is a reason to discount it: what a prototype can demonstrate is that the inference architecture exists and runs, which is a narrower claim than the one the rest of this site is making, and a real one.
One hypothesis, with its revisions
This is what a belief about a pairing looks like from the inside: bounded, versioned, carrying its own evidence, and keeping the record of having been wrong instead of overwriting it.
- Scope
- person model task class order: verdict first
- Claim
- On this task class, with this model, when the verdict arrives first, this person stops catching a particular kind of error.
- Evidence
- ⟨n⟩ adjudicated cases · ⟨n⟩ of them with the verdict withheld
- Confidence
0.78→0.41- Prediction
- Withholding the verdict on this task class raises detection of that error class.
Revisions — kept, not overwritten
- v1 · ⟨date⟩Formed from ⟨n⟩ cases. Confidence 0.62.
- v2 · ⟨date⟩Prediction held on ⟨n⟩ further cases. Confidence 0.62 → 0.78.
- v3 · ⟨date⟩Prediction failed on ⟨n⟩ cases after the model was upgraded. Confidence 0.78 → 0.41. Scope flagged: the finding may not survive the new model.
What this proves, and what it does not
Proved
That the inference architecture exists: that state can persist per person across sessions, that beliefs can be versioned and revised against outcomes, and that the revision history survives inspection.
Not proved
Enterprise outcomes. It was not built for professional decision-making, it has had no external users, and nothing here shows that a decision made this way is a better decision.
That gap is the whole of what remains to be tested, and the next two sections are the plan for testing it.
Building now
What is being built now
- task-bounded professional signals in place of the prototype’s domain-specific inference
- capture of judgment and confidence before the model speaks, the model’s influence on the decision, the task and model context, the response to a change in how the case was run, and the adjudicated outcome
- longitudinal state connected to the policy that decides how the next case runs
- repeated cases instrumented so the outcome can test the prediction
- a first workflow to run all of it against
Planned test
How I intend to falsify it
Four arms, in increasing order of what they assume. Measured on decision quality, on whether reliance on the model matches when the model is right, on how many of the model’s errors are caught, and on review time.
- Ageneric AI assistance
- Bstrong universal interaction design, the same for everyone
- Cstrong shallow personalization
- DJanus, with longitudinal state
The comparison that decides the product is D against C. If rich longitudinal state does not materially beat strong shallow personalization, the honest response is to simplify the product, not to defend the architecture.
Worth saying plainly, because it is the part that usually goes unsaid: only the third comparison is risky. A→B asks whether interaction design changes the outcome at all, and B→C whether personalization does, and the prior evidence for both is strong. So the test has a floor: even where D and C come out level, the run has still measured whether the order and form of handover between a person and an agent change the result, which is worth knowing on its own.
Not claimed
Not claimed
- No enterprise outcome has been validated with external users.
- It is not known whether rich longitudinal state beats strong, simple personalization. That is the test above, not a result being held back.
- Nothing here shows that adapting how decisions are run preserves anyone’s capability over time. That is the claim this project would most like to make and the one furthest from evidence.
- Findings expire when the model underneath changes. Some of what is learned travels — how well someone’s confidence tracks their accuracy outlives any one model — but the sharp findings are the perishable ones. So this works only if findings form faster than models are replaced. That inequality is the bet, and it is not settled.
- The hypotheses are written by a model, not by a person reading cases. A model will find a pattern in three examples as readily as in three hundred, which is the failure this product exists to catch, turned on its author. Two things constrain it: a hypothesis must commit to a prediction before it can change how a case runs, and it must survive outcomes it did not see. One thing does not: no person reviews a hypothesis before it takes effect.
What would settle it
None of this gets settled by more building. It gets settled by running the comparison on real work, and that takes a partner.
- What it would need
- A decision your people already make repeatedly, where an AI already has a part in it, and where somebody eventually establishes what was actually true. Access to that workflow for long enough that the outcomes come back — in most of the work that fits, that is the constraint setting the clock, not the engineering.
- What you would get, either way
- A measurement of whether the order and form of handover changes your outcomes. Nobody currently runs that measurement, and it is worth having whether or not Janus turns out to be the right way to act on it. And an answer on whether the elaborate version beats the simple one — including if the answer is no, in which case you have been saved the elaborate version.
- What would end it
- A null result on the comparison that decides the product. What I would do then is written above, and I would rather be held to it than believed about it now.