Lock interpretation before results
The seven stages prevent choosing the most favourable meaning of an ambiguous rule after seeing its performance. Dataset provenance and experiment registries preserve the path to each verdict.
Flagship / Engineering case study
Assay
A falsification platform for trading strategy claims.
Research · Local tool
Python / Pydantic / pytest / OANDA API
One relationship check changed the verdict.95% bootstrap CI · mean R
Before correction · invalidated
- Trades
- 238
- Win rate
- 48.7%
- 95% CI on mean R
- [+0.145, +0.461]
The interval excludes zero, but the underlying trades include invalid stops. These results were superseded.
Source: Assay DEF0006 research record. R expresses return relative to risk. These are historical before/after findings, not a forecast.
S001 · EUR/USDSession sweep + FVGFAIL
S002 · NAS100Opening-range breakoutFAIL
S003 · NAS100EMA + VWAP bounceFAIL
S004 · NAS100 / EUR/USDFirst Candle RuleFAIL
01 / The problem
The problem
An attractive backtest can reward a convenient interpretation, a hidden engine defect or tuning on the same data used to judge success. Statistical confidence is meaningless if the measurement is wrong.
02 / What I built
What I built
Assay takes an informal strategy from a video, forum or written claim through seven ordered stages. Like an assay of ore, it tests whether the claimed value is really there. Ambiguities are mechanically resolved before results are visible. Typed registries connect hypotheses, experiments, datasets, defects and permanent verdicts.
03 / Architecture
Architecture
- 1 · Capture
- 2 · Interpret
- 3 · Specify → resolve ambiguities before seeing results
- 4 · Implement
- 5 · Run
- 6 · Verify → reconstruct sampled trades
- 7 · Verdict
04 / Key decisions
Key decisions
Refuse a convenient PASS
The only net-positive full-pipeline variation was driven heavily by the same in-sample data used to select it. It was recorded as needing more data, with an explicit warning against further tuning on that sample.
Wait for a real second consumer
Later strategies exposed assumptions baked into the original StrategyPlugin abstraction. They were not forced into it. Only execution primitives with demonstrated shared use were generalized.
05 / What went wrong
What went wrong
DEF0006 · A statistically significant edge that disappeared
- Problem
- A baseline showed 238 trades, a 48.7% win rate, 97 passing tests and a bootstrap confidence interval excluding zero.
- Why it happened
- The stop was calculated before a later forward search found the entry. Without a relationship check, the stop could land on the profitable side of entry.
- How it was detected
- Manual candle-by-candle reconstruction found a winning trade recorded as exiting at its initial protective stop. That relationship was impossible for this setup.
- Response
- Reject a setup when its resolved stop is on the wrong side of entry. Trades fell to 176, win rate to 37.5%, and the confidence interval crossed zero. Prior results were marked affected.
- Lesson
- Passing tests and statistically significant results are necessary, not sufficient. Independently reconstruct the measurement.
06 / Evidence
Evidence
- Around 1.9 million real one-minute candles across multiple markets.
- Four strategy claims, four FAIL verdicts, preserved in research registries.
- DEF0006 before/after values are recorded in the project research evidence, not simulated for this portfolio.
07 / Current status & limitations
Current status & limitations
Research · Local toolLocal research infrastructure, intentionally without a hosted app. These are findings about the tested claims and datasets, not a profitable trading product. Further evidence must come from fresh data rather than repeated in-sample tuning.
Evidence basis: project implementation review and research notes supplied for this portfolio, September 2026. Project tests described here are distinct from the portfolio’s own checks.