We ran 367 quant experiments looking for a real edge. Each leakage-test fix revealed the next escape route the model took. Here's what that taught us about ML harness design.

This is a write-up of a long research arc — 12 phases of work, 367 deterministic experiments, 54 documented learnings. The headline finding is a negative one: no measurable edge on Indian large-cap equities with publicly-available features at our universe size and holdout horizon. The interesting finding is structural.

Every layer of leakage testing we added revealed the next escape route the model was taking. Fix one class of overfit, the optimizer finds another. The pattern itself is the result.

If you're building any ML system where the cost of being wrong is high — fraud detection, medical signals, credit scoring, anything where a "promising" result can mislead — this is the harness shape we'd recommend.

The setup, briefly

We were testing whether daily news, fundamentals, macro signals, regulatory events, and price action could predict next-day cross-sectional returns on Indian large-caps (Nifty 50 / 100). Standard ML harness: walk-forward training, deterministic experiments, seed-stable verdicts, leakage tests.

The literature on Indian-equity edges suggests honest post-cost Sharpe is in the 0.4-0.7 range over 18+ years of data. Our cleaner-than-realistic noise floor is ±1.55. So any honest signal would be within or below our noise band.

That's the baseline. Here's what happened.

Phase 6.A — naive leakage tests pass on garbage

Our first version of look_ahead_cheat_test planted next-day returns as a feature column, retrained, and checked whether Sharpe lifted past 5×. Sound by construction.

It didn't work. Models trained on raw multi-magnitude features (fundamentals at scale 5-30, returns at scale 0.01) silently missed the cheat. The first-layer linear weights were dominated by the largest-magnitude feature. The planted return signal was swamped.

The fix: pass cheat columns through the same _causal_standardise transform as honest features. Cheats now compete on equal footing post-standardisation.

# leakage_tests.py — the seam that mattered
def look_ahead_cheat_test(features, returns, transform_fn=None):
    cheat_col = returns[:, 1:].flatten()  # plant next-day returns
    augmented = np.concatenate([features, cheat_col], axis=-1)
    if transform_fn:  # ← apply the same scaling as honest features
        augmented = transform_fn(augmented)
    sharpe_with_cheat = train_and_score(augmented, returns)
    return sharpe_with_cheat / max(abs(sharpe_honest), 1e-8) >= 5.0

A leakage test that doesn't share the feature-transform path with honest training is testing a different model than the one you're shipping.

Phase 6.E — the position-floor pathology

After 6.A, models started reporting Sharpe near zero. Honest result, presumably.

It wasn't. The negative_sharpe loss function we'd been using is scale-invariant in positions. The optimizer's cheapest path to a clean leakage verdict is to collapse positions toward zero — Sharpe is undefined when there's no PnL, so the harness reports ≈ 0, and the verdict is "no edge."

A model that learns to do nothing passes every leakage test. That isn't a feature.

The fix lives in the harness, not the loss. walk_forward.py now uses Python signature inspection to detect which losses declare a pos parameter, and passes positions to those losses. The sharpe_with_position_floor(α=0.5) regularizer penalizes degenerate positions without overwhelming the Sharpe objective.

Lesson: position-magnitude regularization belongs in the harness contract, not in any single loss function.

Phase 8 — the static-features test

By Phase 8 we were at Nifty 50 scale. Models passed shuffled_target, passed cheat tests, produced honest small-Sharpe results. Looked clean.

We added a 5th leakage test, static_features_test: replace every feature column with its per-(stock) time-mean, holding it constant across the holdout. A real time-edge model should produce ≈ zero positions when its inputs are constant. A ticker-memoriser keeps its baseline allocations.

# stability_tests.py — static_features_test
def static_features_test(features, returns):
    # collapse each (stock, feature) to its time-mean
    static = features.mean(axis=0, keepdims=True).repeat(T, axis=0)
    sharpe_static = train_and_score(static, returns)
    return abs(sharpe_static) < 2 * standard_error  # pass if no edge on static

Every model that had passed all four prior tests, failed this one. Best |sharpe| on static features = 1.66, well outside the noise band. Models were learning ticker identity — "always go long high-news-density tickers" — and presenting it as a time-edge signal.

The escape route: when features carry persistent per-stock baselines (vol_60d, fundamentals, even news article count), the model extracts ticker-identity signal from feature covariance structure even after we per-stock-z-score the levels.

Phase 12 — the window-position artifact

By Phase 12 we had models passing five leakage tests cleanly. At n_holdout=21, our best candidate produced Sharpe +5 with all flags green.

Window-robustness check (the 6th leakage test): retrain the same configuration with n_holdout=180. The Sharpe collapsed to -0.14.

# stability_tests.py — window_robustness_test
def window_robustness_test(config, holdout_lens=[21, 60, 180]):
    sharpes = [train_and_score(config, n_holdout=h) for h in holdout_lens]
    # A real edge produces consistent Sharpe across windows
    return statistically_consistent(sharpes)

The model was fitting a feature-distribution drift that existed only in the short window. Walk-forward retrained every 5 days; 21-day holdout = ~5 retrains; 180-day = 36 retrains. The 21-day "edge" was a window-position artifact, structurally identical to the ones we'd caught at 5-stock scale 9 phases earlier.

The pattern

Layer of leakage testing	Optimizer's next escape route
Standardize features	Position collapse to zero (scale invariance)
Ratio co-condition	Position-aware losses required
Cross-sectional de-mean	Feature-magnitude latching
Per-stock z-score	Feature-covariance memorisation
Static-features test	Window-position artifacts
Window robustness	(Bootstrap-calibrated bounds next; not done)

Each fix revealed the next escape route. The optimizer is more creative than the harness designer.

Practical principles

1. Every leakage test must share its full pipeline with honest training. Standardisation, regularisation, every preprocessing transform. A leakage test that diverges from the training path is testing a different model than the one shipping.
2. Position-magnitude regularisation belongs in the harness. Not in losses. Losses don't see positions structurally; the harness does.
3. Static-features test is criminally underused. If your model uses inputs that carry persistent per-entity baselines (per-customer features, per-store features, per-stock features), test that the model goes near-zero when those inputs are held constant.
4. Window-robustness should be a leakage test, not a robustness check. A "promising" result that exists only at one holdout length is an artifact, not an edge.
5. Bootstrap-calibrate your bounds. Closed-form 2·SE bounds understate empirical variance by ~6× at small sample sizes. We learned this the hard way at Phase 12.A.

The honest finding

After 12 phases, 367 experiments, 54 documented learnings: no measurable edge on Indian large-cap equities with the features we tried, at our universe size and holdout horizons.

The methodology itself is the deliverable. Anyone running ML on financial signals — or any other domain where the cost of false-positive findings is high — could apply this harness shape to a different asset class, a different feature set, or a different prediction horizon.

A negative result with a sharp characterization of why is research output. It constrains the hypothesis space for the next attempt.

Where the rest lives

The full methodology, the 54 learnings, and what we'd do next at: 🔗 deemwar.com

— deemwar

References

• Marcos López de Prado — Advances in Financial Machine Learning (Wiley, 2018). Publisher page
• Bailey, Borwein, López de Prado, Zhu — Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (Notices of the AMS, 2014). AMS Notices PDF · Preprint PDF — SSRN copy returns 403 to scripted clients, so linking the author's preprint and the AMS publication.
• Bailey & López de Prado — The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality (Journal of Portfolio Management, 2014). Preprint PDF
• Bailey, Ger, López de Prado, Sim, Wu — Statistical Overfitting and Backtest Performance (Risk-Based and Factor Investing, Elsevier, 2015). Preprint PDF
• Hudson & Thames Research — Does Meta Labeling Add to Signal Efficacy? (Triple-Barrier Method) — practitioner walkthrough of López de Prado's labeling and purged-CV pipeline. Article
• deemwar — Stock Research: methodology, leakage tests, and the 54 learnings. stock-research.deemwar.com