Tool 1: Feature Engineering Explorer

ml-feature-explorer.mathandmarkets.com

This is the companion to Post 95. It lets you do what I did in that post — test 20 candidate features and figure out which ones are signal and which ones are noise — but interactively.

The Feature Explorer tab shows all 20 features ranked by Information Coefficient. Click any feature to select or deselect it. The scoreboard at the top updates in real-time: how many features you’ve selected, how many are above the noise floor (IC > 0.03), your portfolio’s combined IC, and how many redundancy warnings you’ve triggered. Hover over any feature to see what it measures and why it matters.

The thing to watch: redundancy warnings. If you select VIX z-score, Realized vol z, AND VIX term structure, you’ll get a warning that you have three volatility features with high inter-correlation. The model can’t distinguish their individual contributions. Drop one.

The Stationarity Lab tab shows one feature (VIX) rendered four different ways: raw level, z-score, percentile rank, and rate of change. Toggle between them and notice that the raw level drifts — its mean shifts over time. The other three are stationary. The Python code for each transformation is shown alongside so you can apply the same logic to your own features.

If you take one thing from this tab: never feed a raw level into a model.

The IC Decay tab shows your selected features losing predictive power over time. Each feature decays at a different rate — VIX z-score holds its IC for roughly 8 years, while OFI proxy drops to the noise floor in about 3. This tab tells you how often your model needs retraining based on the features you’ve chosen.

The Model Playground tab is where you feel overfitting with your hands. Three sliders: number of trees, max depth, and learning rate. As you increase any of them, the in-sample Sharpe climbs — it can hit 1.0, 1.5, even 2.0+. The out-of-sample Sharpe peaks around 200 trees / depth 4 / learning rate 0.05, then collapses. The bar chart shows the gap at every tree count. The verdict text changes from green (”healthy”) to yellow (”moderate overfit”) to red (”your model has memorized the training data”).

Crank the depth to 15 and watch the IS Sharpe hit 2.0+ while OOS drops below 0.3. That’s the chart that should be tattooed on every ML practitioner’s forearm.

Tool 2: SHAP Explorer

ml-shap-explorer.mathandmarkets.com

This is the companion to Post 96. It shows what the XGBoost model actually learned — not just which features are important (the bar chart) but how each feature affects the model’s output (the dependence plots).

The sidebar ranks all 11 features by mean |SHAP value|. Click any feature to load its dependence plot in the main panel.

The dependence scatter shows the feature’s value on the x-axis and its SHAP contribution on the y-axis. Each dot is one day. The shape of the cloud tells you the relationship — and the key discovery is that many relationships are non-linear.

Start with VIX z-score. You’ll see a shape that isn’t a straight line. Below zero, SHAP is mildly positive (buy the calm). Between 0 and 2.5, it’s strongly negative (reduce allocation). But above 2.5 — the extreme tail — it bends back up. The model discovered that extreme VIX readings predict mean-reversion, not continuation. That’s the signal my if-else rules missed for 94 posts.

Toggle the GEX split button in the top right. This colors each dot by whether GEX was positive (green) or negative (red) on that day. The two populations separate — negative GEX makes every VIX level worse. The model treats VIX = 2.0 with negative GEX the same as VIX = 2.5 with positive GEX. That’s an interaction effect that linear IC analysis from Post 95 couldn’t detect.

The Spread z-score dependence plot is another one worth examining with GEX split on. Wide spreads with positive GEX have almost no effect — dealer hedging absorbs the microstructure stress. Wide spreads with negative GEX are strongly negative. The interaction is the signal, not the individual feature.

Tool 3: Decision Tree Builder

ml-tree-builder.mathandmarkets.com

This one is a game. Can you build a hand-tuned decision tree that beats XGBoost?

The left panel is your rule builder. Click “+ Add Rule” to create a split. Each rule has four components: a feature (VIX z-score, GEX regime, momentum, etc.), a direction (greater than or less than/equal), a threshold, and an allocation (0-100%). Rules are evaluated top to bottom — the first matching rule wins. Days that don’t match any rule get full allocation.

The right panel updates live. You’ll see a heatmap strip showing your allocation over time (green = full, red = minimum), an equity curve comparing your tree to the V6 base, and a results table benchmarking you against XGBoost (Sharpe 0.56) and the if-else rules from Post 86 (Sharpe 0.50).

To get started: try adding one rule — VIX z-score > 1.5, allocation 20%. This single rule captures the three stress periods and should push your Sharpe from the base level up toward 0.4-0.5. Then add a second rule for GEX regime <= 0, allocation 50%. This catches days where dealer positioning amplifies moves even when VIX isn’t extreme.

The verdict bar at the bottom tells you where you stand. If you beat XGBoost with fewer rules than it has trees, the verdict will say so — and that’s the point. Sometimes domain expertise in 3 rules beats 200 trees and 240,000 splits.

The hint if you get stuck: the stress periods are around days 130-170, 370-405, and 570-615. The model catches them by combining VIX z-score, momentum direction, and spread width. Try adding a rule that captures negative momentum during high VIX — that’s the regime the model weights most heavily.

What’s Coming Next

These three tools cover Posts 95-96. The next two posts — Post 97 (The Overfitting Minefield) and Post 98 (The V6 ML Layer) — will bring additional interactive components. The walk-forward validation visualizer is in development, and the final post will include a head-to-head simulator where you can compare the ML layer, the if-else rules, and the V6 base across different market regimes.

All tools are at ml-tools-hub.mathandmarkets.com.

Not investment advice. These are simply tools to experiment and play with. Do your own due diligence.

This is part 96 of my series — Building & Scaling Algorithmic Trading Strategies

Part 2 of the ML for Trading series. Part 1: Feature engineering — 20 features, only 7 survive.

The Problem a Linear Model Can’t Solve

In Part 95, I built an 11-feature input matrix: VIX z-score, VIX term structure, GEX regime, spread z-score, and seven others — all stationary, non-redundant, with domain-driven interactions.

Now: what model should consume these features?

The instinct is to reach for the deepest tool in the box. Transformers. LSTMs. Attention mechanisms. But this problem doesn’t need depth. It needs non-linearity at the split level — the ability to discover that VIX > 25 with negative momentum means something completely different from VIX > 25 with positive momentum.

Left: the actual regime structure in simulated data. Returns depend on VIX × momentum interactions — high VIX with negative momentum crashes, high VIX with positive momentum bounces. The yellow dashed lines are the decision boundaries a tree finds automatically. Right: a linear regression draws a straight line through this mess and misses the regime structure entirely. Trees capture the step-function reality.

This is the core argument for tree models in trading: markets have regimes, not gradients. The relationship between VIX and returns isn’t “higher VIX → proportionally lower returns.” It’s “VIX below 18 = one regime, VIX 18-25 = another, VIX above 25 = a third.” Trees find these thresholds automatically.

What the Tree Learns

Here’s a simplified decision tree trained on V6’s feature matrix:

A single tree from the ensemble, pruned to depth 3 for readability. The root split is on VIX z-score (the strongest feature from Part 95). The left branch handles calm markets, splitting on GEX regime and spread z-score to determine allocation between 0.6× and 1.0×. The right branch handles stress, splitting on momentum direction to distinguish recovery bounces (0.5×) from active crashes (0.2×).

Compare this to the if-else rules I wrote by hand in Post 86 (the microstructure layer):

# My hand-tuned rules (50 lines)
if gex_positive and spread_z < 1.0:
    allocation = 1.0
elif vix > 25 and momentum < 0:
    allocation = 0.3
elif vix > 25 and momentum > 0:
    allocation = 0.5
else:
    allocation = 0.7

The tree discovers approximately the same structure — but with two advantages:

1. The thresholds are optimized. I picked VIX > 25 and spread_z < 1.0 based on intuition. The tree picks VIX z-score > 1.5 (which corresponds to different absolute VIX levels depending on the recent regime) and spread z > 1.0. The tree’s thresholds are fit to the data; mine were fit to my gut.

2. The interactions are automatic. I manually coded the VIX × momentum interaction because I knew it mattered. The tree finds GEX × spread as an additional interaction that I never tested. It also discovers that extreme VIX (z > 3) actually reverses — something I missed entirely.

SHAP: What the Model Actually Uses

The tree has 1,200 estimators, each with up to 200 splits. That’s 240,000 decision points. What is all that complexity doing?

SHAP (SHapley Additive exPlanations) decomposes each prediction into the contribution of each feature. Instead of a black box that says “allocate 0.3×”, SHAP says “VIX z-score pushes the allocation down by 0.15, GEX pushes it down by 0.08, but momentum pushes it up by 0.05, net = 0.3×.”

Mean absolute SHAP values across the test set. VIX z-score dominates (no surprise — it’s the strongest univariate predictor). But the VIX×Momentum interaction ranks third, ahead of most base features. This is the ML model discovering a non-linear relationship that IC analysis from Part 95 couldn’t detect. Top 5 features account for ~80% of the model’s output.

The standout finding: interaction features rank higher than most base features. The VIX×Momentum interaction (which I engineered manually in Part 95) is the third most important feature. The GEX×Spread interaction is sixth. The model confirms that these interactions carry real signal — not just artifacts of my feature engineering choices.

The Non-Linear Discoveries

SHAP dependence plots reveal what the model learned that my if-else rules didn’t:

Left: VIX z-score dependence, colored by GEX regime. The relationship is NOT monotonic. Below z = 0 (calm), the effect is mildly positive (buy the dip). Between 0 and 2.5 (stress), it’s strongly negative (reduce allocation). Above 2.5 (extreme stress)... the model discovers that extreme VIX readings actually reverse. The curve bends back up. This is a mean-reversion signal in the tails that a linear model or a simple if-else threshold would never find. Right: the Spread × GEX interaction. When GEX is negative (red) AND spreads are wide, the model aggressively reduces. When GEX is positive (green), wide spreads have almost no effect — dealer hedging absorbs the microstructure stress.

The extreme-VIX reversal is the most interesting discovery. My hand-tuned rules treated VIX > 25 as uniformly bad. The tree says: “VIX at 25-35 is bad. VIX at 40+ means the crash is almost over — start buying back in.” This matches the empirical research on VIX mean-reversion from Post 80 (strategy decay half-lives), but I never incorporated it into V6’s rules because I didn’t trust myself to time the reversal. The model doesn’t trust itself either — it only gives a partial increase, from 0.2× to maybe 0.35× — but it’s an edge I was leaving on the table.

Model Selection: Why XGBoost Wins

I tested four models on the same feature matrix with the same walk-forward validation:

Left: four models compared on OOS Sharpe, direction accuracy, overfit gap, and training time. XGBoost wins on Sharpe (0.71) with a moderate overfit gap (0.08). Random Forest is close behind (0.62) with a larger overfit gap (0.12). Logistic regression is simplest but misses the non-linearity. The neural net overfits badly (0.25 overfit gap) and is 10× slower. Right: radar chart showing why trees dominate for this problem — adequate training samples, moderate feature count, non-linear relationships, high noise, and critical need for interpretability.

Why neural nets fail here:

Too few samples. V6 has roughly 2,500 days of data. After walk-forward splits, each training window is 500-800 days. Neural nets need 10,000+ samples to generalize. Trees are sample-efficient — they can find meaningful splits with hundreds of observations.

Too much noise. Financial returns have a signal-to-noise ratio of roughly 0.05 (the IC from Part 95). Neural nets overfit noise at low SNR because they have too many parameters. Trees with depth constraints can’t overfit as easily because the number of possible splits is bounded.

Interpretability. When the model says “reduce to 0.2×”, I need to know why. SHAP on a tree ensemble is computationally exact. SHAP on a neural net is approximate and expensive. If I can’t explain the model’s decision, I won’t trade on it.

The Regime Map

Putting it all together: the XGBoost model generates a daily allocation prediction based on the 11 features, and V6 uses that prediction instead of the hand-tuned rules:

Three panels. Top: price action with stress periods shaded. Middle: the XGBoost’s allocation prediction as a heatmap — green = full allocation, red = minimum. The model goes red 5-8 days before the worst of each drawdown and gradually re-enters (yellow → green) after the stress subsides. Bottom: cumulative P&L. XGBoost improves the Sharpe from 0.33 to 0.56 and cuts max drawdown from -34% to -17%.

The heatmap in the middle panel is the key visualization. It shows the model’s regime classification in real-time — not as a binary on/off but as a continuous allocation between 0.1× and 1.0×. The gradual re-entry after stress (the yellow-to-green transition) is something my hand-tuned rules didn’t do well — I used binary switches (in or out) rather than continuous scaling.

The Hyperparameter Trap

Before you get excited: most of the Sharpe improvement is fragile. It depends heavily on hyperparameter choices that I made after looking at the out-of-sample results.

Left: number of trees vs. Sharpe. In-sample Sharpe climbs to 1.95 with 5,000 trees. Out-of-sample peaks at 200 trees (0.71) then declines to 0.42 with 5,000 trees. The widening gap is pure overfitting. Right: max tree depth shows the same pattern. Depth 4 is optimal OOS (0.71). Depth 15 gives an IS Sharpe of 2.20 and an OOS Sharpe of 0.15.

This is the central problem of ML for trading: the model that looks best in-sample is the worst out-of-sample.

A researcher who only reported the depth-15 IS Sharpe of 2.20 would have a “stunning” result. The actual OOS Sharpe of 0.15 tells you it’s garbage. The only honest approach is walk-forward validation with strict separation between parameter tuning and performance evaluation — which is what Part 97 (the overfitting minefield) will cover in depth.

For now, the practical takeaway: shallow trees, few estimators, heavy regularization. My final XGBoost configuration:

model = XGBRegressor(
    n_estimators=200,     # not 1,200 — OOS peaks here
    max_depth=4,          # shallow trees generalize better
    learning_rate=0.05,   # slow learning prevents overfitting
    subsample=0.7,        # row sampling (bagging)
    colsample_bytree=0.7, # column sampling (feature bagging)
    reg_alpha=1.0,        # L1 regularization
    reg_lambda=5.0,       # L2 regularization (heavy)
    min_child_weight=50,  # minimum samples per leaf
)

Every parameter choice limits the model’s ability to memorize the training data. The resulting model is weaker in-sample but more honest out-of-sample.

The Uncomfortable Question

The XGBoost improves V6’s Sharpe from 0.33 to 0.56. That’s meaningful. But the 50-line microstructure layer from Post 86 improved it from roughly 0.33 to roughly 0.50 (varying by simulation).

The marginal improvement of ML over hand-tuned rules is about +0.06 Sharpe. For that, you get:

Additional complexity:
  - Model training pipeline (data prep, validation, tuning)
  - Feature maintenance (11 features, daily updates)
  - Periodic retraining (quarterly, per Part 95's IC decay)
  - Monitoring for model drift
  - SHAP analysis for each trade decision

versus:

50 lines of if-else that anyone can read, modify, and debug

Is +0.06 Sharpe worth that operational burden? Part 98 will give the definitive answer. But I’ll preview my current thinking: it depends on your scale. At $100K, the if-else rules win on simplicity. At $10M+, the ML layer starts to justify its overhead.

Up Next

Part 97: The Overfitting Minefield — Walk-forward validation, combinatorial purged cross-validation, the deflated Sharpe ratio, and a rigorous framework for answering “is this result real or noise?”

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 94 of my series — Building & Scaling Algorithmic Trading Strategies

This begins a 4-part series on machine learning for trading. Part 1: feature engineering. Part 2: tree models for regime classification. Part 3: the overfitting minefield. Part 4: does the ML layer actually help V6?

🔧 Interactive tool: ML Feature Engineering Explorer — select from all 20 candidate features, check which ones beat the noise floor, watch their predictive power decay over time, and see the Python transforms for making raw data ML-safe.

The Dirty Secret of ML for Trading

Here’s what nobody tells you in the scikit-learn tutorials: the model doesn’t matter.

Not at first. Not until you’ve spent 80% of your time on the input. A random forest trained on good features will crush a transformer trained on garbage. A logistic regression with the right 8 features will beat a neural network with 200 noisy ones. The model learns patterns in whatever you feed it. If you feed it noise, it learns noise.

This post is about the 80% — feature engineering. Specifically: how to convert 93 posts of trading signals, macroeconomic indicators, and microstructure data into a clean, stationary, non-redundant feature set that a model can actually learn from.

The Feature Taxonomy

Over the last 15 posts (Parts 80-94), we built signals across five categories. Here they all are:

Twenty candidate features organized by category. Momentum/trend features capture directional persistence. Volatility/regime features capture market stress. Microstructure features capture order flow and dealer positioning. Carry/macro features capture the yield environment. Cross-asset features capture inter-market relationships.

Each of these features was developed and discussed in a specific post. They’re not random — they have theoretical grounding and empirical motivation. But that doesn’t mean they all belong in an ML model. Many are redundant, some are non-stationary, and a few might be pure noise in a multivariate context.

The job of feature engineering is to separate the signal from the noise, the unique from the redundant, and the stationary from the drifting.

Problem 1: Redundancy

The first thing you notice when you stack these features into a correlation matrix:

Pairwise correlations across 12 features. VIX and 21-day realized vol are 70% correlated — they’re measuring the same thing. 50-day and 200-day momentum have 50% overlap. Credit spreads and VIX are 40% correlated. Feeding all of these into a model gives you multicollinearity — the model can’t separate the contribution of each feature.

The fix is straightforward: for each pair with |r| > 0.7, drop one. The decision of which to drop is domain-driven:

Between VIX and realized vol, keep VIX — it’s forward-looking (implied) while RV is backward-looking (realized). The forward-looking version is more useful for predicting future returns.

Between Mom(50) and Mom(200), keep the ratio (50/200) rather than either level — the crossover signal carries the information of both in a single feature.

Between credit spreads and VIX, keep both — the correlation is moderate (0.4) and they capture different aspects of risk (credit vs. equity volatility). But if forced to choose, credit spreads add incremental information that VIX doesn’t capture.

After the correlation filter: 20 candidates → ~14 features.

Problem 2: Stationarity

This is where most retail ML projects fail silently.

Top: raw VIX level over 5 years. The mean drifts upward — a model trained on the first 2 years would learn “VIX above 18 is high” but by year 5, VIX is regularly above 20 even in calm markets. Bottom: VIX z-score (126-day) is stationary — mean zero, stable variance. A z-score of +2 means the same thing in year 1 and year 5.

Non-stationary features poison ML models because the relationship between feature and target shifts over time. A model trained in a low-VIX era will have learned that VIX = 20 is “stressed.” Deploy it in a high-VIX era where VIX = 20 is “calm,” and it will persistently overestimate risk.

The rule: never feed a raw level into a model. Always transform to a stationary representation.

Four transformations, from simplest to most robust:

# 1. Z-score (most common)
z = (x - x.rolling(63).mean()) / x.rolling(63).std()

# 2. Percentile rank (non-parametric, handles skew)
pct = x.rolling(252).apply(lambda w: (w.iloc[-1] > w).mean())

# 3. Rate of change (captures dynamics)
roc = x.pct_change(21)

# 4. Binary regime (simplest, most robust)
regime = (x > x.rolling(126).median()).astype(int)

Which to use? Z-scores work for normally distributed features (vol, spreads). Percentile ranks work for everything but lose granularity. Rates of change work for trending features (momentum, yield curve). Binary regimes sacrifice precision for robustness — which matters when the model will see out-of-sample data that looks nothing like the training set.

Problem 3: Individual Features Are Weak

Here’s the uncomfortable truth about financial prediction: most features have an information coefficient (IC) — the rank correlation between the feature today and the return over the next week — of approximately 0.02 to 0.08.

This is a lesson I’ve had to learn over and over again, but I still keep repeating the same mistake.

Univariate IC for each candidate feature. VIX level is the strongest predictor at IC ≈ 0.08 — which means it explains roughly 0.6% of the variance in next-week returns. Most features are below 0.05. Some are below 0.02, which is indistinguishable from noise.

For context: an IC of 0.05 means that if you ranked all days by the feature value and sorted them into quintiles, the top quintile would outperform the bottom quintile by roughly 2-3% annualized. That’s meaningful for a systematic strategy but invisible on any individual trade.

This is why ML helps: individual features are weak, but their combination can be strong. If VIX, term structure, GEX, and spread z-score each independently predict returns at IC ≈ 0.05 and they capture different dimensions of risk, a model that combines them might achieve IC ≈ 0.10-0.12 through the aggregation of weakly correlated weak predictors.

This is the “wisdom of crowds” principle applied to features. No single expert (feature) is very good. But the committee (model) can be substantially better than any individual member, provided the members aren’t all saying the same thing (low inter-feature correlation).

The Transformation Toolkit

One raw feature can produce multiple transformed features. The question is which transformations capture useful information:

Six representations of a single input (VIX). Raw level (non-stationary, avoid). Z-score (stationary, captures deviation from recent norm). Percentile rank (robust, captures relative positioning). Rate of change (captures acceleration/deceleration). Binary regime (simplest, most robust). Interaction term (VIX_z × momentum, captures non-linear dependencies).

Interaction Features

Interaction features are where domain expertise meets statistical power. Two features that are individually weak can be jointly powerful if their interaction captures a meaningful dynamic:

VIX_z × Momentum: High VIX with negative momentum = crash regime. High VIX with positive momentum = recovery bounce. The interaction separates these two regimes that VIX alone cannot distinguish.

GEX × Spread_z: Negative GEX with wide spreads = maximum microstructure stress. Negative GEX with normal spreads = moderate stress. The interaction captures severity better than either feature alone.

Yield_slope × Credit_spread: Steep curve with tight credit = risk-on carry environment. Steep curve with wide credit = economic distress. The interaction disambiguates.

The danger of interactions: every pair you add is a new feature that the model can overfit to. With 14 base features, you have 91 possible interactions. Adding all of them is a recipe for overfitting. Add only the interactions that have clear economic interpretation.

The Selection Pipeline

Putting it all together:

Starting with 20 candidates, the pipeline produces 8-10 final features through five stages. Each stage has a clear purpose and a clear rule for what survives.

The output of this pipeline is the feature matrix X that gets fed into the model in Part 2. I’d say that the pipeline is not optional — skipping any stage (especially stationarity testing and redundancy removal) will produce a model that looks great in-sample and falls apart out-of-sample.

The Half-Life of Features

One more complication: features decay, just like strategies (Post 80).

Information coefficient over time for five features. VIX level maintains predictive power longest (structural, rooted in risk aversion). OFI proxy decays fastest (microstructure signals get crowded). GEX regime sits in between. All features trend toward the noise floor of IC ≈ 0.02 eventually.

This connects directly to the Strategy Decay series. The alpha in a feature comes from an informational edge. As more people learn about and trade on GEX signals (thanks partly to the popularization by SpotGamma and others), the feature’s predictive power decays. VIX decays more slowly because it’s structural — risk aversion doesn’t get arbitraged away.

The implication for ML: your model needs periodic retraining not just because the model degrades, but because the features themselves degrade. A model trained in 2020 using GEX as a strong feature might find that GEX has an IC of 0.01 by 2025 — below the noise floor. The model needs to downweight or drop it.

This is the feature engineering → model training → retraining feedback loop. We’ll build the retraining framework in Part 3.

What the Feature Matrix Looks Like

After the full pipeline, here’s the concrete feature matrix for V6:

features = {
    # Volatility / Regime (3 features)
    'vix_z':        vix_zscore_63d,
    'vix_ts':       vix_1m_div_3m,
    'rv_z':         realized_vol_z_63d,  # keep if corr with vix_z < 0.7
    
    # Microstructure (2 features)
    'gex_regime':   gex_binary,           # +1 / -1
    'spread_z':     spread_zscore_21d,
    
    # Momentum (2 features)
    'mom_ratio':    ma50_div_ma200,
    'mom_roc':      price_roc_21d,
    
    # Carry / Macro (2 features)
    'yield_slope_z': yield_10y2y_zscore,
    'credit_z':     hy_ig_spread_zscore,
    
    # Interactions (2 features)
    'vix_x_mom':    vix_z * mom_roc,
    'gex_x_spread': gex_regime * spread_z,
}

# Target: next 5-day V6 excess return (over SPY)
target = v6_return_5d - spy_return_5d

Eleven features that are clean, stationary, non-redundant, with domain-driven interactions. This is what gets fed into Part 2’s tree models.

Up Next

Part 2: Tree Models for Regime Classification — Random forests, XGBoost, and why tree-based models are the right tool for this problem. We’ll train, validate, and stress-test the models using the feature matrix we just built.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 94 of my series — Building & Scaling Algorithmic Trading Strategies

Final part of the Synthetic Replication series. Part 1: Building blocks. Part 2: Replicating strategies. Part 3: Vol surface.

The Question That Started This Series

Four posts ago, I asked: “What payoff shape would improve V6’s risk/return profile?” Now it’s time to answer.

V6’s current profile is approximately linear-with-a-kink: leveraged upside through TQQQ, partial downside protection from the VIX regime filter and TLT hedge. The regime filter adds some convexity — it exits before the worst of most drawdowns — but it’s not a hard floor. A fast crash that outpaces the momentum signal hits V6 at full leverage.

The goal of an options overlay: add a hard floor without killing the upside.

Left: V6’s current return profile (orange) vs. the desired profile (green), plotted against market returns. The gap is in the left tail — V6 is exposed to drawdowns beyond -15% where the regime filter hasn’t yet fired. Right: the return distribution shift. The overlay truncates the 1st percentile from approximately -40% to -20%.

The Three-Leg Overlay

After working through the building blocks (Part 1), strategy replication (Part 2), and vol surface analysis (Part 3), here’s the specific overlay design:

Three panels showing each leg individually and then combined. Leg 1 (green): long 10% OTM put — the tail hedge, costs ~1.2%/quarter. Leg 2 (orange): short 8% OTM call — income to offset the hedge cost, earns ~0.8%/quarter but caps gains beyond +8%. Combined (yellow): a put spread collar with ~20bps net quarterly cost.

Leg 1: The Tail Hedge (Long 10% OTM Put)

This is the core of the overlay. A quarterly 10% OTM put on SPY (or QQQ, depending on V6’s current allocation) provides a hard floor on losses.

Why 10% OTM? Three reasons. First, V6’s regime filter typically catches drawdowns of 5-8% before they get worse. The put doesn’t need to protect against normal volatility — it needs to protect against the gap down that outpaces the momentum signal. Second, 10% OTM puts are on the steep part of the skew curve — expensive, but not as expensive as 5% OTM (which trades at much higher IV). Third, the historical frequency of 10%+ quarterly drawdowns in QQQ is roughly once every 2-3 years, which keeps the premium manageable.

Cost would be approximately 1.0-1.5% of notional per quarter, depending on IV and skew.

Leg 2: The Income (Short 8% OTM Call)

Selling OTM calls generates premium to offset the tail hedge cost. V6 doesn’t need the last 8% of quarterly upside — most of V6’s alpha comes from compounding 1-2% monthly, not from catching +10% rips.

The call strike is set at 8% OTM because historically, TQQQ gains more than 8% in a single quarter about 15% of the time. The short call caps participation in those rare events but generates consistent income the other 85%.

Income: approximately 0.6-1.0% per quarter.

Leg 3: The Short Far-OTM Put (Funding)

Selling a 20% OTM put further reduces the net cost of the overlay. At 20% OTM, the put is deeply out of the money and expires worthless in roughly 98% of quarters. The premium is small ($0.3-0.5% per quarter) but it partially funds the tail hedge.

The trade-off: protection stops below the 20% OTM level. If QQQ drops more than 20% in a quarter, you’re exposed again below that floor. This is an acceptable trade-off because a drawdown that large typically triggers V6’s regime filter well before it reaches 20%.

Net Cost

Tail hedge cost:       -1.2% per quarter
Call premium income:   +0.8% per quarter
Far put premium:       +0.3% per quarter
───────────────────────────────────────
Net cost:              -0.1% per quarter (~0.4% annualized)

At 40 bps annualized, the overlay is nearly self-funding. The short call and short far put together pay for most of the tail hedge. This is the power of the put spread collar structure — you’re not just buying insurance, you’re reshaping your return distribution by giving up extreme upside to fund extreme downside protection.

Conditional Engagement: Not Always On

The overlay shouldn’t run on autopilot. Its value depends on market conditions:

The key rule: buy protection when it’s cheap, not when it’s needed.

When VIX is below 18 and skew is below the historical median, 10% OTM puts cost roughly 1.0% per quarter. When VIX is above 30 and skew is elevated, the same puts cost 3-4%. By the time you “need” the protection, it’s too expensive to be worth it — and V6’s regime filter is already scaling down anyway.

This is the paradox of tail hedging: the optimal time to hedge is when nothing is happening, because that’s when hedges are cheap. The worst time to hedge is during the crisis, because that’s when hedges are expensive. Discipline means buying the put in January when VIX is 13, not in March when VIX is 35.

Backtest Results

Top: 5-year simulated equity curves. V6 base (gray) and V6 with overlay (green). The overlay gives up some raw return but compresses drawdowns significantly. Bottom: drawdown comparison — every stress period (red shading) shows reduced depth with the overlay.

The improvement is consistent with what we’d expect from a well-designed tail hedge: modest return reduction, meaningful drawdown compression, and a Sharpe ratio that improves because the volatility reduction exceeds the return drag.

Is It Worth It?

This is the question that determines whether the overlay goes from theory to implementation. The answer depends on what you value.

Left: break-even analysis. At 1% annual hedge cost, you need roughly 2 crashes per decade of 20%+ to justify the protection — consistent with historical frequency. Right: optimal hedge budget peaks at 1.0-1.5% annualized, where the Sharpe improvement from volatility reduction exceeds the return drag.

The Case For

The overlay converts a -45% max drawdown into a -30% max drawdown. The math on this is unambiguous: a -45% drawdown requires +82% to recover (3+ years at V6’s CAGR). A -30% drawdown requires +43% to recover (under 2 years). The overlay saves roughly 1-1.5 years of recovery time per major crash.

Over a 20-year compounding horizon, that recovery time difference compounds into a meaningful terminal wealth advantage — even though the overlay costs 40 bps per year.

The Case Against

V6 already has three layers of drawdown protection: the VIX regime filter, the momentum threshold, and the CUSUM monitoring system. Adding a fourth layer (options) introduces:

Operational complexity. Options require rolling, strike selection, and expiry management. This is a weekly task, not daily.

Execution risk. Options bid-ask spreads on TQQQ are wide. The theoretical 40 bps annual cost could easily become 80-100 bps after execution costs.

Marginal improvement. The first three protection layers (regime, momentum, CUSUM) capture most of the available Sharpe improvement. The options overlay adds the fourth layer at 4× the implementation complexity for the smallest marginal gain.

The Honest Assessment

Each V6 enhancement layer plotted by Sharpe improvement and implementation complexity. Execution timing (Part 84) gives the highest ROI: +0.08 Sharpe for minimal effort. The options overlay gives +0.07 Sharpe for 4× the complexity of any other layer. The curve of diminishing returns is steep.

Here’s what I’m actually going to do:

Implement the tail hedge only. No short call, no short far put. Just a quarterly 10% OTM put when VIX is below 18 and skew is below the 63-day median. Cost: ~1.0% per quarter, ~4% annualized (not self-funded without the other legs). Run this in paper mode for 6 months.

Defer the full collar. The short call and short put legs add complexity and execution risk that I don’t think is justified yet. They save 60-80 bps of annual hedge cost, but they also introduce path-dependent risks (getting called away on a TQQQ rally, getting assigned on a far put during a crash). I need more experience with the mechanics before adding these legs.

Monitor skew as a signal. Whether or not the overlay is running, skew z-score (from Part 92) is a useful re-entry signal after selloffs. I’m adding it to the dashboard alongside the VIX regime, CUSUM, and microstructure composite.

The 14-Post Series Arc: What We Built

Stepping back from this post, here’s the full arc of Posts 80-93:

Strategy Decay (80-82): Half-lives of alpha, CUSUM detection, adaptive strategies. Core insight: all strategies decay, the question is how fast and whether you can detect it.

Microstructure Edge (83-86): Order flow toxicity, auction mechanics, dealer gamma, signal layer. Core insight: the plumbing of markets affects strategy performance, and GEX is the highest-impact single signal.

Carry Trade (87-89): Unified carry framework, crash timing, multi-asset carry. Core insight: V6 is already a carry trader — understanding this changes risk attribution.

Synthetic Replication (90-93): Put-call parity, strategy replication, vol surface, options overlay. Core insight: options add convexity but with diminishing returns relative to complexity.

Across 14 posts, V6’s architecture has gone from a simple TQQQ/TLT allocator with a VIX regime filter to a layered system with momentum, regime detection (CUSUM), microstructure conditioning (spread + GEX), carry optimization, and now a potential options overlay.

Each layer improved the Sharpe ratio. But each layer also added complexity. The execution timing fix (Part 84) gave +0.08 Sharpe for essentially zero effort. The options overlay gives +0.07 Sharpe for significant ongoing operational work. The law of diminishing returns is real, and knowing when to stop adding layers is as important as knowing which layers to add.

Sometimes the smartest thing you can build is nothing (which is really, really hard for someone like me who keeps poking at things to see if it does anything at all..)

This concludes the Synthetic Replication series (Posts 90-93) and the full 14-post arc spanning strategy decay, microstructure, carry, and synthetic replication.

My next series will be on Machine Learning for Trading.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 93 of my series — Building & Scaling Algorithmic Trading Strategies

Part 3 of the Synthetic Replication series. Part 1: Building blocks. Part 2: Replicating hedge fund strategies.

The Lie at the Heart of Options Pricing

Black-Scholes assumes one number — σ — describes the volatility of the underlying. Every option on the same stock, at every strike and every expiry, should use the same σ.

This has been empirically false since approximately October 19, 1987.

Before the ‘87 crash, the vol “smile” was roughly flat. After it, puts became permanently expensive relative to calls. The market learned that crashes happen, and it never forgot. That fear — that specific, priced-in fear of a left-tail event — is visible in every options chain you pull up today.

Left: Black-Scholes assumes flat vol (dashed gray). Reality is the smirk (orange) — OTM puts trade at significantly higher implied vol than ATM or OTM calls. That 8-10 vol point premium on the 20% OTM put is the market’s price for crash insurance. Right: different asset classes have different smile shapes. Equities smirk left (crash fear dominates). FX smiles symmetrically (both tails feared). Commodities skew right (spike fear dominates — no one worries about oil going to zero, they worry about it going to $200).

The shape of the smile tells you what the market is afraid of. And when the shape changes, it tells you the fear is shifting.

The Surface: Three Dimensions of Fear

The volatility smile is a 2D slice. The full picture is a surface — implied vol varies across both strike (moneyness) and time to expiry.

The vol surface for a typical equity index. The left-back corner (OTM puts, short-dated) is the most expensive — near-term crash protection commands the highest premium. The right-front corner (OTM calls, short-dated) is the cheapest. The surface flattens as you move to longer expirations — skew “decays” with time.

Two key features of the surface:

1. Skew is steeper for shorter expirations. A 7-day 90% moneyness put might trade at 35% IV while the same strike at 365 days trades at 24%. The short-dated put is more expensive per unit of time because short-dated crash fear is more acute — if a crash happens this week, you need protection now.

2. The surface is not static. After a selloff, the entire surface shifts up and steepens. After a long calm period, it shifts down and flattens. The surface is a living map of the market’s collective anxiety.

Term Structure: Near vs. Far

The second dimension of the surface — how IV varies with expiration at a fixed strike — is the term structure.

Left: three term structure shapes. Normal (contango, green): longer-dated options have higher IV because more uncertainty over longer horizons. Inverted (backwardation, red): near-term IV is elevated — a stress event is pricing into the front month. Flat (gray): no informational content. Right: the VIX term structure ratio (1-month VIX / 3-month VIX) over time. Above 1.0 = inverted = stress. Below 1.0 = contango = calm.

The term structure is a regime indicator, just like the VIX level itself. But it contains information that VIX alone misses:

VIX can be at 20 in both contango and backwardation. In contango (1M VIX = 20, 3M VIX = 22), the market expects the current level to persist — orderly uncertainty. In backwardation (1M VIX = 20, 3M VIX = 18), the market expects near-term stress to resolve — someone is paying up for immediate protection.

This links directly to the carry trade framework from Part 88: contango is “harvest” mode (sell near-term vol, it decays toward the lower long-term level). Backwardation is “retreat” mode (near-term vol is elevated for a reason).

Trading the Surface: Four Dimensions

Every vol surface trade targets one of four dimensions:

Let me unpack the two most tradeable dimensions.

Skew Trades: Selling Expensive Fear

Equity skew — the premium of OTM puts over OTM calls — is the most persistent and most tradeable feature of the vol surface.

The core trade: sell expensive put vol, buy cheap call vol. This is a risk reversal (Part 2), but now motivated by vol surface analysis rather than directional conviction.

Left: the put ratio spread — sell one expensive near-ATM put, buy two cheaper far-OTM puts. You collect a net premium while maintaining downside protection below 85. Right: a risk reversal in two different skew regimes. When skew is steep (post-selloff), the short put is so expensive that you get PAID net premium for a bullish bet. When skew is flat, the same structure costs zero.

Why Steep Skew Is an Opportunity

After a selloff, skew overshoots. Put IV spikes to 30-35% while ATM vol is at 22% and call IV is at 18%. The 12-17 vol point skew is the market pricing in a continuation of the decline that is, on average, too pessimistic.

Research by Bollen and Whaley (2004) and Xing, Zhang, and Zhao (2010) documents that extreme skew predicts mean-reversion — steep skew is followed by declining skew as the panic premium dissipates. The half-life of a skew overshoot is approximately 2-4 weeks.

The Skew Mean-Reversion Signal

Top: 25-delta skew (put IV minus call IV) over time. It spikes during selloffs and mean-reverts within 2-4 weeks. Bottom: z-score of skew with trading thresholds. When skew z-score exceeds 2.0 (red markers), selling skew via risk reversals or put ratio spreads has historically been profitable as skew normalizes.

The trade: when skew z-score exceeds 2.0, sell the 25-delta put and buy the 25-delta call (risk reversal). Hold for 2-4 weeks or until the z-score returns below 0.5. The risk: skew is elevated for a reason — if the crash continues, you’re short puts into a falling market.

Calendar Trades: Trading Time

Calendar spreads trade the term structure — they’re bets on whether near-term vol will converge toward long-term vol, or vice versa.

Left: the calendar spread payoff at front-month expiry. Maximum profit occurs near the strike price, where the back-month call retains maximum time value while the front-month call expires worthless. Right: regime-dependent calendar direction — sell expensive front-month vol when term structure is inverted, sell expensive back-month vol when term structure is steep.

The key insight: calendar spreads are not directional bets. They’re bets on the relative speed of time decay between two expirations. In a normal term structure, the front month decays faster (higher theta per day) — a standard calendar profits from this differential.

Calendar Trades for V6

V6 doesn’t currently trade calendars, but the term structure informs two decisions:

1. Expiry selection for the overlay (Part 4). When term structure is steep (contango), longer-dated options are expensive relative to near-term — buy shorter-dated protection. When inverted, shorter-dated options are overpriced — buy longer-dated protection.

2. Carry regime timing. VIX term structure inversion is one of the best carry crash indicators (Part 88). The calendar spread framework explains why — inversion means the market is paying more for near-term protection than long-term, which signals acute stress.

What the Surface Tells My V6

Here’s the practical extraction for a daily equity allocator:

Level (ATM IV / VIX): V6 already uses this via the VIX regime filter. High ATM vol = reduce allocation. This is the most important dimension and my V6 already captures it!

Skew (25-delta put-call IV spread): Extreme skew overshoots predict mean-reversion. For V6, this informs when to add back exposure after a selloff — if skew is extremely elevated but the selloff has paused, the market is pricing in too much further downside. This is a potential re-entry signal.

Term structure (VIX 1M/3M ratio): Inversion signals acute stress. V6 should treat term structure inversion as an independent warning alongside VIX level — you can have VIX at 22 with a normal term structure (fine) or VIX at 22 with an inverted structure (danger).

Convexity (wing pricing): Less relevant for V6’s daily allocation as it stands today (but that might change…) More relevant for the options overlay in Part 4, where we’ll use wing pricing to determine whether tail hedges are cheap or expensive.

The Honest Limitations

Vol surface trading is deep water. A few caveats:

1. Skew is expensive to trade. The bid-ask on OTM options is wide — 5-15% of the option’s value. A lot of the theoretical edge in skew trading gets eaten by spreads — unless you are doing this at scale and strategically, this is a significant consideration, especially for retail traders.

2. The surface moves. By the time you compute your z-scores and place the trade, the surface has shifted. Institutional vol desks have real-time feeds and sub-second execution. Retail has end-of-day data and limit orders.

3. Gamma risk. Short-dated skew trades have enormous gamma. A 1% move in the underlying can change the P&L of a weekly put spread by 50% of its maximum value. Position sizing must account for this.

4. Model risk. Everything above assumes Black-Scholes implied vols are the right metric. They’re not — they’re a convention. More sophisticated models (stochastic vol, local vol, rough vol) fit the surface better but require infrastructure that’s beyond most retail setups. It’s doable but you’ll need to invest significant time, energy, infrastructure and need a better command of the model.

For V6, the surface is primarily an input — a richer version of the VIX signal — rather than a tradeable opportunity in its own right. In Part 4, we’ll use surface information to design the options overlay, but we won’t try to trade the surface directly.

Up Next

Part 4: The V6 Options Overlay — Everything from Parts 90-92 converges into a specific overlay design. Tail hedge sizing using wing pricing, covered call income conditioned on skew, and the final assessment: is the added complexity worth the improved Sharpe?

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 92 of my series — Building & Scaling Algorithmic Trading Strategies

Part 2 of the Synthetic Replication series. Part 1: Put-call parity and the building blocks.

Hedge Funds Are Just Payoff Shapes

Strip away the marketing and the 2-and-20 fee structure, and every hedge fund strategy is a payoff shape. Trend following produces convex returns — small losses, occasional large gains. Covered call funds produce concave returns — steady income, occasional large losses. Tail risk funds produce extreme convexity — constant small premium drain, massive gain in crashes.

You can replicate any of these shapes with the four building blocks from Part 1. This doesn’t mean you can replicate the skill of a good hedge fund manager — timing, security selection, and risk management still matter. But you can replicate the structure, and the structure explains most of the return characteristics.

Strategy 1: Risk Reversal as Synthetic Momentum

A risk reversal — long an OTM call, short an OTM put at the same expiry — creates a zero-cost (or near-zero-cost) directional bet with unlimited upside and significant downside.

Left: the risk reversal payoff. Long the 105 call, short the 95 put, for approximately zero net premium. If the stock rallies past 105, you participate fully. If it drops below 95, you’re obligated to buy at 95 — taking losses as if you owned the stock. The “dead zone” between 95 and 105 is where nothing happens. Right: when a momentum signal fires (50-day MA crosses above 200-day), you enter the risk reversal instead of buying stock.

Why This Replicates Momentum

A momentum strategy says: “I think the stock will continue its recent trend.” A risk reversal says exactly the same thing with different capital characteristics:

Stock purchase: $10,000 capital for 100 deltas. Linear exposure. Full downside.

Risk reversal: ~$0 net premium for ~50 deltas initially (grows toward 100 if the stock rallies). Defined dead zone. Full downside below the put strike.

The risk reversal gives you leverage without margin. You get directional exposure at a fraction of the capital, and the OTM call gives you positive convexity — your gains accelerate as the stock moves in your favor because the call’s delta increases.

The trade-off: the dead zone. If the stock moves modestly (stays between 95 and 105), the risk reversal returns approximately nothing. The stock purchase returns whatever the stock does. For momentum signals that predict large moves, the dead zone is a feature — you avoid paying up for small, uncertain moves. For signals that predict small, consistent moves, the dead zone kills you.

When to Use It

Risk reversals work best when:

• Your momentum signal has high conviction but low frequency (a few signals per year)

• You want exposure without committing full capital

• You’re willing to accept the short put obligation as “buying the dip” if wrong

Strategy 2: Covered Call as Systematic Vol Selling

The covered call — own the stock, sell an OTM call — is the most popular retail options strategy. It’s also a misunderstood one. Most people think they’re “generating income from their stock.” What they’re actually doing is selling volatility.

Left: the covered call payoff. You participate in gains up to the 105 strike, then give up additional upside. The premium ($3) provides a small buffer against losses. Right: systematic covered call writing vs. buy-and-hold over 5 years. The covered call has higher Sharpe and lower vol, but lower total return — you’ve traded upside for income.

The Vol Selling Connection

When you sell a call against your stock, you’re implicitly saying: “I believe realized vol will be less than implied vol over this period.” If the stock doesn’t move much (low realized vol), you keep the premium. If it moves a lot (high realized vol), the sold call costs you in either direction — you miss the upside, or the premium doesn’t cover the downside.

This is the same trade as the volatility risk premium we discussed in Part 87 (Carry series). The covered call is carry. The premium you collect each month is the “income” component, and the capped upside is the price of that income.

When to Use It

Covered calls work best when:

• You’re holding the stock anyway and want income

• Implied vol is elevated relative to your expected realized vol

• You have a target sell price (the strike is your “limit order with a bonus”)

Covered calls work worst when:

• The stock is in a strong trend (you cap your participation)

• IV is low (the premium isn’t worth the opportunity cost)

Strategy 3: Collar as Tail Hedge

The collar — own the stock, buy an OTM put, sell an OTM call — truncates both tails of your return distribution. It’s the simplest implementation of tail hedging.

Left: the 90/110 collar payoff. You participate between 90 and 110, with losses floored at -$14 and gains capped at +$14. If the put and call premiums offset, it’s zero cost. Right: the return distribution shifts — both the left tail (crashes) and right tail (rallies) are truncated.

The Put Spread Collar: A More Practical Version

The plain collar is expensive because the OTM put costs more than the OTM call provides (skew — puts are more expensive than calls at equal distance from ATM). A more capital-efficient version uses a put spread instead of a standalone put:

The put spread collar: long the 95 put, short the 80 put, short the 110 call. Cheaper than the plain collar because selling the 80 put funds part of the 95 put. The trade-off: protection stops at $80. Below that, you’re exposed again. This is the “defined-range protection” approach — cheaper, but with a gap risk if the crash exceeds your protective range.

When to Use It

Collars work best when:

• You have a concentrated stock position you can’t or don’t want to sell (tax, vesting)

• You want to define your risk range for a specific period (earnings, macro event)

• You’re willing to trade upside for peace of mind

Strategy 4: Iron Condor as Defined-Risk Vol Selling

The iron condor — sell an OTM put, buy a further OTM put, sell an OTM call, buy a further OTM call — is the defined-risk version of vol selling. It profits when the stock stays within a range.

Left: the iron condor payoff vs. the short straddle. Same bet (stock stays flat) but the iron condor has a defined maximum loss of $1 vs. the straddle’s unlimited risk. Right: monthly iron condor return distribution. High win rate (~75%), small average win, larger average loss. Classic carry profile.

The iron condor is the retail-accessible version of what systematic vol selling funds do. It captures the volatility risk premium with defined risk — you know your maximum loss before entering the trade.

The return distribution tells the full story: many months of small wins, punctuated by fewer but larger losses. This is the carry profile from Part 87, expressed through options instead of interest rates or credit spreads.

The Rosetta Stone: Strategy → Options

Every hedge fund archetype maps to an options construction. The mapping isn’t perfect — a Renaissance Technologies quant is doing more than selling iron condors — but the payoff structure is replicable.

The deeper insight: by choosing your options construction, you’re choosing where you sit on the convexity spectrum.

Convexity: The Only Free Lunch Left

The most important concept in this post — arguably in the entire series — is convexity: the curvature of your payoff relative to the market.

Three strategies plotted against market returns. Gray: linear (buy & hold) — you get whatever the market gives you. Orange: concave (covered call) — you do better than the market in small moves but worse in large moves, especially on the upside. Green: convex (risk reversal / trend following) — you do worse in small moves but better in large moves, especially preserving capital in crashes.

The key:

Concave strategies (covered calls, short straddles, credit spreads) earn steady income in normal markets but get destroyed in tail events. They’re selling insurance.

Convex strategies (long options, trend following, risk reversals) bleed slowly in normal markets but profit from large moves. They’re buying insurance.

Neither is inherently better. The question is which one fits your strategy’s risk profile.

For V6 specifically: V6 has embedded convexity from its momentum signals — it exits before large drawdowns complete. But it also has embedded concavity from the VRP carry component. Understanding this mix is what Part 4 will address when we design the options overlay.

The Concavity Trap

A warning: most retail options strategies are concave. Covered calls, credit spreads, iron condors, cash-secured puts — they all have the same profile: steady income, occasional large loss. They feel great 80% of the time and terrible the other 20%.

The danger isn’t the strategy itself — it’s the psychology. When you’ve collected 10 months of premium, the 11th month’s -15% drawdown feels like a “freak event” rather than the expected cost of doing business. The premium you collected over 10 months was compensation for this event. If you didn’t earn enough premium to cover it, you were underpaid for the risk.

Before implementing any concave strategy, calculate your expected loss-to-premium ratio:

Expected loss frequency × Average loss size ÷ 
Annual premium collected

If this ratio > 0.8, the strategy isn't earning enough
premium to justify the risk.

Up Next

Part 3: Vol Surface Arbitrage — When the implied vol at different strikes and expirations is “wrong,” how do you express a view on skew, term structure, or convexity? The math of the volatility surface and where the opportunities live.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 91 of my series — Building & Scaling Algorithmic Trading Strategies

This begins a 4-part series on synthetic replication. Part 1: the building blocks. Part 2: replicating hedge fund strategies. Part 3: vol surface arbitrage. Part 4: an options overlay for V6.

Options Are Just Payoff Shapes

Most people learn options as a catalog of strategies: covered calls, protective puts, iron condors, butterflies. Each has a name, a diagram, and a use case. This is like learning architecture by memorizing floor plans.

The better way: understand the four building blocks, learn the one equation that connects them, and then design whatever payoff shape you need.

The Four Bricks

Every option position in existence is a combination of four elemental payoffs:

Top left: Long Call — unlimited upside, capped downside. Top right: Long Put — profits from decline, capped loss. Bottom left: Short Call — income if flat/down, unlimited risk if up. Bottom right: Short Put — income if flat/up, large risk if down. Every strategy you’ve ever seen is a linear combination of these four payoffs plus long or short stock.

The key insight: these four shapes are linearly independent. Any payoff diagram you can draw — literally any combination of slopes, kinks, and flat sections — can be constructed by adding these together with appropriate strikes and sizes.

This is why options are sometimes called “financial LEGO.” Four brick types, infinite possible structures.

The One Equation: Put-Call Parity

In 1969, Hans Stoll published the relationship that connects everything:

C - P = S - K·e^(-rT)

Call price minus put price equals stock price minus the present value of the strike. Same strike, same expiry. Always. In every market, for every stock, at every moment.

Left: visual proof. The green dashed line (long call payoff) plus the red dashed line (short put payoff) equals the yellow solid line — which is exactly the payoff of owning the stock. Right: rearrange the equation to build five synthetic positions from the same identity.

Why This Matters

Put-call parity isn’t just an academic identity. It’s a construction manual. Rearranging one equation gives you five synthetic positions:

Synthetic Long Stock = Long Call + Short Put + Cash

You don’t need to buy stock. Buy the call, sell the put at the same strike, set aside the present value of the strike in cash. At expiry, you’ll have exactly the same P&L as owning the stock. Why would you do this? Capital efficiency — the options might require less capital than buying the stock outright.

Synthetic Long Call = Long Stock + Long Put

If you own the stock and buy a put, you’ve created a call option. This is the “protective put” — but reframed as building a call from its components. Conceptually, a protective put isn’t “buying insurance on your stock” — it’s “building a synthetic call option.”

Synthetic Bond = Long Stock + Long Put + Short Call

The “conversion” — you hold the stock, buy a put, and sell a call at the same strike. Your payoff is completely flat: the stock gains are capped by the short call, and losses are floored by the long put. The result is a risk-free position that earns the risk-free rate. You’ve synthesized a Treasury bond from options and stock.

The Six Essential Synthetics

Six positions built from the four building blocks. Each panel shows the component payoffs (dashed) and the resulting combined payoff (solid). Top row: synthetic long stock, synthetic short stock, synthetic call. Bottom row: synthetic put, long straddle, collar. The dashed lines always add up to the solid line — that’s the construction at work.

The straddle and collar are particularly important for what comes in Parts 2-4:

The straddle (long call + long put, same strike) creates a V-shaped payoff that profits from movement in either direction. You’re buying volatility. You don’t care which way the stock moves — you just need it to move more than the combined premium you paid.

The collar (stock + long OTM put + short OTM call) clips both tails of the stock’s return distribution. You give up upside beyond the call strike and gain protection below the put strike. If the premiums offset, it’s “free” — you’ve reshaped your risk without paying cash.

Why Capital Efficiency Matters

The practical motivation for synthetic positions: you can get the same directional exposure with dramatically different capital requirements.

Five ways to get bullish exposure to a $100 stock. Buying 100 shares costs $10,000 for 100 deltas of exposure (1.0 delta per $100). A synthetic long costs $1,500. A long call costs $500 for 55 deltas. A bull call spread costs $300 for 45 deltas. The capital efficiency ranges from 1.0 to 15.0 deltas per $1,000 deployed.

This efficiency has a cost: defined expiry. Stock doesn’t expire; options do. But for a strategy like V6 that rebalances regularly, the expiry constraint is manageable — you’d roll the position quarterly.

When Parity Breaks: Arbitrage Signals

Put-call parity should hold exactly. When it doesn’t, someone is making a mistake — and someone else can profit.

Left: the conversion — stock + put - call produces a flat payoff (risk-free). If the flat payoff exceeds the risk-free rate, you’ve found free money. Right: what parity violations tell you about mispricing.

In practice, transaction costs eat most of the arbitrage. But the direction of the violation tells you something useful:

If calls are expensive relative to parity → implied vol on calls is elevated → the market is pricing more upside risk than the put side.

If puts are expensive relative to parity → implied vol on puts is elevated → the market is pricing more downside risk (skew is steep).

This links directly to the vol surface discussion in Part 3.

The Replication Menu

Here’s where this series is going. Every strategy in the next three posts uses the same four building blocks:

The replication principle: any payoff you want can be constructed from options. The question isn’t “which named strategy should I use?” — it’s “what payoff shape do I want, and how do I build it?”

Payoff Engineering: Design, Then Build

This is the mindset shift. Instead of asking “should I buy a covered call?” ask “what do I want my P&L to look like?”

Three investor goals, three custom payoffs. Left: “I want upside with capped downside” → long call (max loss = premium). Center: “I want income if the stock stays flat” → short straddle (profit in the center, loss in the tails). Right: “I want crash protection at zero cost” → risk reversal (sell upside call to fund downside put).

Each of these payoffs is a direct expression of a view:

The long call says: “I’m bullish and want defined risk.” The short straddle says: “I think realized vol will be lower than implied vol.” The risk reversal says: “I’d rather have crash protection than participate in a rally.”

The same framework applies to V6. In Part 4, we’ll ask: “What payoff shape would improve V6’s risk/return profile?” — and then build it from the bricks.

What’s Next

If you’re comfortable with payoff diagrams and put-call parity, you have everything you need for the next three posts:

Part 2 will use these bricks to replicate hedge fund strategies — risk reversals as synthetic momentum, covered calls as systematic vol selling, collars as tail hedging.

Part 3 goes deeper into the vol surface — when the implied vol at different strikes and expirations is “wrong,” how to express a view on skew, term structure, or convexity.

Part 4 brings it home to V6 — designing a specific options overlay that improves V6’s Sharpe by adding convexity in the tails.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 90 of my series — Building & Scaling Algorithmic Trading Strategies

Final part of the Carry Trade series. Part 1: What carry is. Part 2: When carry blows up.

The Diversification Illusion

The institutional pitch for multi-asset carry is elegant: combine carry streams from FX, bonds, vol, equities, and commodities. Since each has different risk drivers, the portfolio should be well-diversified, smoothing out drawdowns.

In normal markets, this works. But in crises — like right now! — it doesn’t.

Lower triangle: normal-period correlations across carry strategies. They’re low (0.05-0.35) — genuine diversification. Upper triangle: crisis-period correlations. They spike to 0.40-0.75 — the diversification vanishes precisely when you need it most.

This is Longin and Solnik’s (2001) “correlation breakdown” problem applied to carry. During the 2008 GFC, every carry trade blew up simultaneously: FX carry (yen unwind), credit carry (spread explosion), vol carry (VIX spike), commodity carry (oil collapse). The supposedly diversified multi-asset carry portfolio lost 25-40% in three months.

Why? Because all carry trades share the same underlying risk factor: liquidity preference. When investors flee to cash, every asset that was paying you to hold risk reprices at once. The individual carry streams have different betas to this common factor in normal times but converge to beta ≈ 1 in stress.

Multi-Asset Carry: Better Than Single-Asset, Worse Than You’d Hope

Despite the correlation breakdown, multi-asset carry still beats concentrated carry:

Individual carry streams (thin lines) and the equal-weight portfolio (yellow). The portfolio smooths out idiosyncratic volatility in normal periods, but all streams crash together during stress events. Still, the portfolio never falls as far as the worst individual stream — FX carry or vol premium would have much worse standalone drawdowns.

The math works out to roughly a 30-40% drawdown reduction from diversification — meaningful but not transformative. You’re turning a -40% max drawdown into a -25% max drawdown. Better, but not the -10% you’d get from genuinely uncorrelated assets.

The Carry-Momentum Hedge

Here’s the most important insight in this series: carry and momentum are natural hedging partners.

Carry earns money when nothing changes (steady collection of premium). Momentum earns money when things change (riding trends). Carry crashes happen when regimes shift violently — which is exactly when momentum signals fire and momentum strategies profit.

Asness, Moskowitz, and Pedersen (2013) documented this formally: carry and momentum have near-zero correlation in normal times and negative correlation during crises. When carry crashes, momentum profits from the same trend that’s destroying the carry trade.

Orange: carry-only strategy. Blue: momentum-only strategy. Green: 60/40 blend. The correlation between carry and momentum returns is approximately zero. More importantly, during the crash periods, momentum tends to offset carry losses — the combined strategy has smaller drawdowns than either standalone.

This is V6’s secret weapon, whether or not it was designed this way. V6’s momentum signals (the 0.1% threshold, the regime switching) provide exactly the crisis-period hedge that a carry position needs. When the market regime shifts and carry starts bleeding, the momentum signal says “get out” — and the exit itself generates momentum profit.

Normal market:
  V6 carry:     accruing daily (+)
  V6 momentum:  mostly flat, small contributions
  Combined:     steady accumulation

Crisis:
  V6 carry:     losing money as spreads widen (-)
  V6 momentum:  fires regime switch signal (+)
  Combined:     momentum offsets carry loss, net damage limited

This carry-momentum interaction explains why V6 has performed better than a naive leveraged equity strategy. It’s not just the momentum alpha — it’s the momentum’s role as a carry crash hedge.

What V6 Should Actually Implement

So if we are concrete about what carry enhancements that are worth pursuing:

Tier 1: Implement Now (Low Complexity, Low Risk)

Optimize TLT entry timing. When the yield curve is steep (10Y - 2Y > 100bps), TLT carry is elevated. Weight toward TLT more aggressively in these periods. When the curve is flat or inverted, the roll-down carry is zero or negative — reduce TLT and hold more cash. This requires only FRED yield curve data (free) and a simple conditional rule.

Expected improvement: 30-50 bps annualized.

Consider TIPS allocation. TIPS (Treasury Inflation-Protected Securities) offer real yield carry plus an inflation hedge. When real yields are positive (as they have been since 2022), TIPS carry is attractive. This adds a genuinely different carry stream — real rates rather than nominal rates — with low correlation to TLT.

Expected improvement: 50-100 bps annualized with better inflation hedging.

Tier 2: Evaluate Next Quarter (Medium Complexity)

Explicit VRP harvesting. V6 currently captures VRP implicitly through the VIX regime filter. An explicit overlay — selling 30-delta puts on SPX, sized at 5% of portfolio — would formalize the VRP harvest. But this requires options-level execution, margin management, and the tail risk discussed in Part 2.

Not ready for V6 today. Would require a separate paper-trading phase of at least 6 months.

Tier 3: Research Only (High Complexity)

FX carry via currency ETFs and covered call overlays are theoretically appealing but add significant complexity. FX carry requires managing currency risk exposure that V6 currently avoids entirely. Covered calls cap the upside on TQQQ — problematic for a strategy designed to capture leveraged equity returns.

These belong in a different strategy, not bolted onto V6.

The Complete Carry Decomposition for V6

Bringing together all three posts in this series, here’s V6’s full carry picture:

Carry Source          Current    After Tier 1   Mechanism
──────────────────────────────────────────────────────────
TQQQ dividend         0.3%       0.3%          Equity carry
TLT coupon + roll     2.8%       3.1%          Bond carry (optimized)
TIPS real yield        —         0.8%          Real rate carry (new)
VRP (implicit)        1.5%       1.5%          Vol risk premium
Funding cost         -1.2%      -1.4%          Borrowing cost
──────────────────────────────────────────────────────────
Total carry           3.4%       4.3%          +0.9% improvement

An additional 90 bps of carry accruing daily, before any changes to the momentum signals or microstructure layer. Across a 10-year horizon at compound rates, 90 bps per year compounds to roughly 9% more total return — not nothing.

The Series Summary: Three Truths About Carry

1. You’re already a carry trader. Every buy-and-hold investor earns carry (dividends, coupons). V6 earns ~3.4% annualized from carry before any alpha. Understanding this decomposition changes how you diagnose performance.

2. Carry timing is more important than carry level. A 5% carry that you hold through a -40% crash costs you 35%. A 3% carry that you scale down before the crash nets you 3%. The regime framework from Part 2 captures this — and V6’s existing VIX filter already does it.

3. Carry and momentum are complements, not substitutes. The carry-momentum interaction is V6’s structural edge. Carry provides the floor. Momentum provides crash protection. Together, they compound better than either alone.

This concludes the Carry Trade series (Parts 87-89) and the second major content arc after Microstructure Edge. Across these 10 posts (Parts 80-89), I’ve covered strategy decay, detection and adaptation, order flow and market microstructure, dealer gamma exposure, and carry across asset classes — all connected back to V6’s architecture.

Next up: the Synthetic Replication series (4 parts) on building anything from options.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 89 of my series — Building & Scaling Algorithmic Trading Strategies

Part 1: What carry actually is. This is part 2 of the Carry Trade series. Part 3: building a multi-asset carry portfolio.

Three Ways to Die

Every carry trade dies the same way — price changes overwhelm the income — but the death comes in three distinct flavors:

Left: the FX carry unwind (2008-style). A year of steady carry income erased in 30 days as the yen strengthened 25% against carry currencies. Center: Volmageddon (Feb 5, 2018). XIV lost 96% of its value in a single session when the VIX doubled. Right: the 2022 bond carry reversal. Not a crash but 18 months of grinding losses as the Fed hiked 500bps — the carry wasn’t wrong, the regime changed.

These three patterns — fast crash, flash crash, and slow bleed — require different detection methods and different responses.

The fast crash (FX carry, equity carry): A risk-off event triggers unwinding of crowded carry positions. Leveraged carry traders hit margin calls and are forced to sell, creating a feedback loop. The signature: cross-asset correlations spike to near 1.0 as everything carry-related sells off simultaneously.

The flash crash (vol selling): A structural break in the instrument itself. XIV, the short-VIX product, wasn’t just hurt by the VIX spike — its own termination triggered forced covering that amplified the spike. The signature: non-linear, gap-driven, over in hours.

The slow bleed (bond carry, commodity carry): The macro regime shifts. Rate hikes make bond carry negative. Contango in oil makes commodity carry negative. The carry is still accruing, but it’s overwhelmed by adverse price changes month after month. The signature: no single event, just persistent negative returns. Hardest to time because there’s no catalyst to react to.

Early Warning Signals

Can you see carry crashes coming? Well, partially. Research by Brunnermeier, Nagel, and Pedersen (2009) identifies several precursors:

Three panels. Top: carry strategy equity with crash at day 300. Middle: cross-asset correlation rises ~20 days before the crash. Bottom: VIX term structure inverts ~10 days before. Both signals provide warning, but neither is precise — the lead time varies and false alarms are common.

The Warning Dashboard

Four signals to monitor, ranked by reliability:

1. VIX term structure inversion (most reliable). When the front-month VIX exceeds the second-month, it signals that the market is pricing immediate stress. This inverts 70-80% of the time before significant carry drawdowns. Lead time here is 5-15 days.

2. Cross-asset correlation spike. When the 21-day rolling correlation across carry strategies (FX, bonds, equities) exceeds 0.5, it signals that the “carry factor” is under stress. Lead time is 10-30 days.

3. Credit spreads widening. IG and HY credit spreads are a leading indicator of funding stress. When spreads widen by more than 1 standard deviation from their 63-day mean, carry trades become more fragile. Lead time would be 5-20 days (big challenge here is the width of time).

4. Carry crowding indicators. CFTC positioning data shows speculative positioning in carry currencies and vol instruments. Basically, extreme crowding increases crash risk. Lead time is weeks to months, but pretty imprecise.

As you can see, none of these are precise enough to time the bottom tick. But collectively, they tell you when to scale down from 100% to 30-50% — and that’s enough to avoid the worst of the damage.

The Carry Regime Framework

The framework maps to V6’s existing VIX regime filter. In the green regime (VIX < 20, steep curve, low correlation), V6 is already running at full allocation — and carry is accruing. In the red regime (VIX > 25, inverted term structure), V6 should already be scaling down — which protects carry too.

The insight: V6’s regime filter is already a carry timing mechanism, even though it wasn’t designed as one. The VIX regime tracks the same underlying conditions that determine carry profitability.

Carry Returns Are Regime-Dependent

This point deserves its own chart:

Carry returns conditioned on VIX level. When VIX is below 15, carry earns ~5.2% annualized with a 0.87 Sharpe. When VIX exceeds 30, carry loses 8.5% with a -0.28 Sharpe. The strategy only “works” in calm markets — which is fine, because that’s most of the time.

The implication is clear: carry is not a strategy you run all the time. It’s a strategy you run when conditions are favorable and shut down when they’re not. The timing decision — harvest vs. retreat — accounts for more of the total return than the carry level itself.

Timed vs. Untimed Carry

What happens when you apply regime scaling?

Gray: untimed carry (always on). Green: timed carry (scale to 30% when signals flash). Timed carry sacrifices some return but dramatically improves the Sharpe and cuts the worst drawdowns. The regime signals don’t need to be perfect — even a crude VIX > 25 filter captures most of the benefit.

The improvement is similar to what we saw with GEX conditioning in Part 85 (Microstructure series): you’re not predicting crashes, you’re reducing exposure when the environment is hostile. The carry still works — you’re just not sitting at full size when it’s most likely to reverse.

The Connection to V6’s Architecture

V6 already does carry timing — it just doesn’t call it that. Here’s the mapping:

V6 Component          → Carry Timing Function
─────────────────────────────────────────────────
VIX < 20: full TQQQ   → Harvest equity carry + VRP
VIX > 30: shift to TLT → Harvest bond carry instead
CUSUM alarm            → Scale down all carry exposure
Momentum threshold     → Avoid negative price-change periods
Microstructure layer   → Scale down when execution costs
                         erode carry income

Every layer of V6 that we’ve built across 89 posts — the regime filter, the momentum signals, the CUSUM monitoring, the microstructure layer — is also a carry timing mechanism. This isn’t a coincidence. Good strategies manage their carry exposure whether or not they explicitly think about it that way.

Up Next

Part 3: Building the Multi-Asset Carry Portfolio — Can we add explicit carry streams to V6? Bond curve optimization, the carry-momentum interaction, and what it would take to build a proper multi-asset carry strategy.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

Thanks for reading Math & Markets! Subscribe for free to receive new posts and support my work.

This is part 88 of my series — Building & Scaling Algorithmic Trading Strategies

This begins a 3-part series on carry. Part 1: what carry actually is. Part 2: when carry blows up. Part 3: building a multi-asset carry portfolio.

The Unifying Equation

Every asset you can hold generates two types of return: the carry you earn from simply holding the position, and the price change you get from the market moving. Most people focus on the price change. The professionals focus on the carry.

Total Return = Carry + Price Change

This equation is trivially simple but its implications are pretty crazy.

Carry is the return you’d earn if nothing changes — if rates stay the same, if exchange rates don’t move, if the yield curve maintains its shape. It’s the “income” component of return. Price change is everything else — the unpredictable part.

What makes carry powerful: it’s knowable in advance. You can calculate today exactly how much carry you’ll earn tomorrow. You can’t do that with price changes. This asymmetry — one component is predictable, the other isn’t — is the foundation of an entire class of strategies.

Four carry strategies over 20 simulated years. The volatility risk premium (purple) has the highest carry but the worst drawdowns. FX carry (orange) shows the classic “escalator up, elevator down” pattern. Bond roll-down (blue) is the steadiest. Equity dividend carry (green) has the lowest carry but the least drama.

The Five Faces of Carry

Carry shows up in every asset class. The name changes, the math is the same:

In every case, you earn a predictable income for holding a position. And in every case, price changes can overwhelm that income — sometimes catastrophically.

So let’s walk through each one…

FX Carry: The Fama Puzzle

FX carry is the most studied and the most intuitive. Borrow in a low-rate currency (JPY, CHF), invest in a high-rate currency (AUD, BRL, TRY). The interest rate differential is your carry.

Standard economics says this shouldn’t work. Uncovered Interest Parity (UIP) predicts that high-rate currencies should depreciate at exactly the rate of the interest differential, leaving carry traders with zero expected profit. A 10% rate in Turkey should be offset by a 10% depreciation of the lira.

The reality is different. Fama (1984) showed that high-rate currencies actually appreciate slightly on average, making the carry trade doubly profitable — you earn the rate differential AND a small FX gain. This anomaly, known as the “Fama puzzle” or the “forward premium puzzle,” has persisted for 40+ years across currencies and time periods.

Left: FX carry returns by pair. Note TRY/USD: 25% rate differential but negative total return — the lira depreciation overwhelmed the carry. High carry ≠ guaranteed profit. Right: the Fama puzzle visualized. UIP predicts the gray line (high rates → proportional depreciation). Reality (orange line) has a slope of about -0.3, not -1.0. The gap between the lines is the carry profit.

Why Does This Persist?

Three theories, none fully satisfying:

Peso problem. Carry trades work most of the time but occasionally blow up catastrophically (peso devaluation, yen carry unwind, etc.). The average return compensates for crash risk that hasn’t been fully sampled.

Risk premium. Investors demand compensation for holding higher-risk currencies. Just as equities earn a premium over bonds, high-rate currencies earn a premium over low-rate currencies.

Slow-moving capital. Central bank interventions and institutional frictions prevent immediate arbitrage. By the time capital moves to exploit the anomaly, the opportunity has partially closed but never fully disappears.

For our purposes, the mechanism matters less than the empirical fact: carry has been a profitable strategy across currencies for decades, with a Sharpe of approximately 0.4-0.6 in G10 currencies — similar to equities but with very different risk characteristics.

Bond Roll-Down: Riding the Curve

Bond carry is subtler than FX carry. It has two components:

1. Coupon (yield) carry. You earn the bond’s yield minus your funding cost. A 10-year Treasury yielding 4.2% funded at a 4.5% repo rate gives you -0.3% of yield carry. Not great.

2. Roll-down carry. As a bond ages, it moves to a shorter maturity on the yield curve. If the curve is upward-sloping, shorter maturities have lower yields. Lower yields mean higher prices. You profit from this mechanical price appreciation.

Left: the roll-down mechanism. Buy a 5-year bond at 3.70%. After one year, it’s a 4-year bond. If the yield curve hasn’t changed, a 4-year bond yields 3.75% — but you bought at 3.70%, so you’ve gained. The price appreciation from this yield decline is the roll-down carry. Right: decomposition of bond return. Coupon provides 3.7%, roll-down adds 0.5%, price change subtracts 0.3%, for a total of 3.9%.

Roll-down carry is positive whenever the yield curve is upward-sloping (normal) and negative when it’s inverted. Since the curve is normal roughly 80% of the time, bond carry is usually positive.

The Duration Bet

Bond carry is partly compensation for duration risk — the risk that rates rise and your bond loses value. A strategy that systematically harvests bond carry is implicitly betting that rates won’t rise faster than the carry compensates. When rates spike (2022), bond carry gets destroyed. When rates are stable or declining, it compounds steadily.

This is directly relevant to V6: TLT (the 20+ year Treasury ETF) earns carry from both coupon and roll-down. V6’s TLT allocation isn’t just a hedge — it’s a carry position.

The Volatility Risk Premium: Selling Insurance

The volatility risk premium (VRP) is the most lucrative and most dangerous carry trade.

The VRP exists because implied volatility (what options are priced at) systematically exceeds realized volatility (what actually happens). The gap — typically 3-5 percentage points of volatility — is the “insurance premium” that option buyers pay and option sellers collect.

Top: implied vol (orange) vs. realized vol (blue). The green shading shows the VRP when it’s positive (profit for short vol). Red shading shows when realized exceeds implied (loss). Bottom: cumulative VRP P&L. Steady upward grind interrupted by sharp drawdowns — the quintessential carry profile.

Why VRP Exists

Unlike the Fama puzzle, the VRP has a clean economic explanation: risk aversion.

Option buyers are purchasing insurance against tail events. Insurance is always priced above its actuarial value — that’s how insurance companies make money. Similarly, options are priced above their expected payoff. The seller (the “insurance company”) collects this premium.

The VRP is robust because it’s structural, not informational. As long as investors are risk-averse — which is basically always — they’ll overpay for downside protection, and vol sellers will collect the premium.

V6 captures VRP implicitly through its VIX regime filter. When VIX is elevated (options are expensive), V6 shifts allocation — effectively acting as a partial buyer of the vol premium. When VIX is low, V6 is fully invested — effectively neutral on vol. Understanding this as a carry trade changes how you think about V6’s risk.

The Skewness Problem

Every carry trade shares a statistical fingerprint: positive mean, negative skew.

Return distributions for four carry strategies. FX carry and vol premium have extreme negative skew (-0.5 to -0.8) and excess kurtosis (3-8). Bond roll-down and equity dividends are more symmetric. The fat left tails are the “elevator down” — rare but devastating losses.

This is not a coincidence. Carry trades earn a premium precisely because they expose you to left-tail risk. The Sharpe ratio of a carry trade overstates its risk-adjusted return because the Sharpe assumes normal returns. With negative skew, you’re earning the average return most of the time but occasionally losing 5-10× the average in a single event.

The practical implication: never size a carry trade using the Sharpe ratio. Use a tail-risk-adjusted metric like the Sortino ratio, or simulate the worst-case drawdown and size to survive it.

V6’s Hidden Carry

Here’s the punchline of this post: V6 is already a carry trader, and understanding this changes how you analyze its returns.

V6 earns approximately 3.4% annualized carry from four sources: TQQQ’s dividend yield (0.3%), TLT’s coupon and roll-down (2.8%), implicit VRP from the VIX regime filter (1.5%), minus funding cost (-1.2%). This carry accrues before any alpha from momentum signals or regime switching.

When V6 has a good year, how much is carry and how much is alpha? When V6 has a bad year, is the carry still working and the alpha failing, or has the carry trade itself reversed (as bond carry did in 2022)?

Decomposing returns into carry and alpha is essential for understanding whether V6 is working for the right reasons. If the carry component accounts for most of the return, V6 is primarily a carry trade with alpha on top. If the alpha dominates, V6 is primarily a directional strategy with carry as a bonus.

Based on the math above, V6’s ~24% CAGR decomposes roughly as:

Carry component:     ~3.4% (predictable, accrues daily)
Momentum alpha:      ~8-10% (from regime switching, variable)
Leveraged beta:      ~10-12% (from TQQQ's 3× equity exposure)
Interaction effects:  ~1-2% (timing of carry vs. alpha)

The carry is the floor. Even when the momentum signals are wrong, V6 earns ~3.4% from holding its positions. That’s not nothing — it’s the difference between a bad year being -5% versus -8%.

Up Next

Part 2: When Carry Blows Up — The “escalator up, elevator down” problem formalized. Can regime signals warn you before a carry crash? We’ll look at FX carry unwinds, Volmageddon, and the 2022 bond rout through the lens of carry timing.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This post is based on an ask by one of my readers. This is also something I have previously experimented with: Retraining the Hybrid Ensemble on Synthetic Regimes // Stress-Testing V6 with Synthetic Markets (Part 1: Building the Stochastic Model) // Stress-Testing V6 with Synthetic Markets (Part 2: Results and Failure Modes)

I have previously written about building a synthetic SPY generator to stress-test a trading strategy against market scenarios that historical data simply doesn’t cover. The questions and responses I got were mostly variations of: “how did you actually build that thing?”

This post answers that question, properly, without the strategy context getting in the way. It is a standalone guide to synthetic financial data: the math behind the models, the code to implement them, the statistical tests to validate them, and the honest accounting of what synthetic data can and cannot do.

If you have ever asked “what if we got a crash that’s different from 2020?” or “how would this hold up in a 2000-style slow grind?”—this is the machinery that lets you answer those questions empirically rather than with hand-waving.

BLUF: If you are impatient and just want to get to the code, here you go:

https://github.com/kniyer/stochastic-data-generation

1. Why Bother With Synthetic Data?

Before building anything, it is worth being precise about the problem synthetic data solves.

The standard backtesting approach runs a strategy against historical price series and reports performance metrics. This works reasonably well if the historical window contains enough variety, but in practice, it rarely does. A 2015–2025 window gives you exactly one flash crash, one sustained bear market, one rate-hiking cycle, and zero prolonged stagflation periods. If your strategy has never seen something, you have no evidence it handles it well or badly—you just have silence.

The academic literature sometimes calls this the Backtest Overfitting Problem. The more you tune a strategy to a specific historical dataset, the better it looks in-sample and the less informative that performance becomes about the future. Synthetic data provides a partial remedy: if the generator captures the statistical structure of real markets rather than their specific history, you can run your strategy against thousands of plausible futures and identify failure modes that no finite historical window can reveal.

There is an important caveat I will state now and return to at the end: synthetic data is excellent for stress-testing and scenario analysis, and much more dangerous as training data. Real price series carry information about actual economic relationships—information that stochastic simulators systematically omit. Use synthetic data for testing. Be very careful about training on it.

2. The Statistical Fingerprint of Real Markets

The generator needs to reproduce what real markets actually look like statistically. Before writing a single line of simulation code, it is worth cataloguing exactly which properties matter.

Using 30+ years of SPY daily returns, the key statistics shake out like this:

Property                    SPY (1993-2025)    Gaussian Assumption    Gap
──────────────────────────────────────────────────────────────────────────
Annualized Return             ~10.1%             N/A                  —
Annualized Volatility         ~18.7%             N/A                  —
Sharpe Ratio                  ~0.54              N/A                  —
Skewness                      -0.55              0.00                 Large
Excess Kurtosis               11.72              0.00                 Enormous
Lag-1 Autocorrelation         -0.08              0.00                 Moderate
Lag-1 Vol Autocorrelation      0.25              0.00                 Large
Tail Events (>3σ)              ~1.2%             0.27%                4× more

Each row in that table is a failure mode for naive Gaussian simulation. Let me explain why each one matters.

Skewness (-0.55): Returns are not symmetric. Large down days are more common than large up days of equivalent magnitude. A coin-flip model misses this entirely.

Excess kurtosis (11.72): This is the fat-tail signature. Under normality, a 3-sigma daily move should happen 0.27% of the time—about once every 380 trading days. In actual SPY data it happens closer to 1.2% of the time, or once every 83 days. Market crashes are not 10-sigma events that happen once a century; they are fat-tail events that happen every few years.

Lag-1 autocorrelation (-0.08): A weak but real negative serial correlation. Large up days slightly predict modest reversal, and vice versa. Markets exhibit short-term mean reversion alongside longer-term momentum.

Lag-1 volatility autocorrelation (0.25): This is volatility clustering, and it is one of the most robust empirical facts in all of finance. Volatile periods tend to stay volatile. Calm periods tend to stay calm. A market that was turbulent yesterday is more likely to be turbulent tomorrow—not because of any predictive model, but just because that is how markets structurally behave.

Any synthetic generator that ignores these properties will produce price paths that look plausible in charts but fail immediately when you run statistical tests on them.

3. The Stochastic Model Stack

No single model captures all of these properties. The right approach is to build several models, each targeting specific statistical features, then combine them into a single generator.

3.1 The Baseline: Geometric Brownian Motion

GBM is where every quantitative finance course starts. The stochastic differential equation is:

dS = μS dt + σS dW

Here S is price, μ is drift, σ is volatility, and dW is a Wiener process (standard Brownian motion increment). In discrete time over daily steps:

r_t = μ·Δt + σ·√Δt·Z_t

where Z_t ~ N(0,1)

GBM gets two things right: positive drift over time, and the intuition that percentage returns rather than absolute price changes should be normally distributed. It gets everything else wrong. Constant volatility, Gaussian tails, no serial correlation. But it is the skeleton on which everything else hangs.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats

def simulate_gbm(mu: float, sigma: float, S0: float, T: int, seed: int = 42) -> np.ndarray:
    """
    Simulate a single price path using Geometric Brownian Motion.
    
    mu:    Annual drift (e.g. 0.10 for 10%)
    sigma: Annual volatility (e.g. 0.187 for 18.7%)
    S0:    Starting price
    T:     Number of trading days
    
    Returns array of length T+1 (including starting price).
    """
    rng = np.random.default_rng(seed)
    dt = 1 / 252  # daily step
    
    daily_mu = mu * dt
    daily_sigma = sigma * np.sqrt(dt)
    
    log_returns = daily_mu - 0.5 * sigma**2 * dt + daily_sigma * rng.standard_normal(T)
    prices = S0 * np.exp(np.cumsum(np.insert(log_returns, 0, 0)))
    
    return prices

3.2 GARCH(1,1) for Volatility Clustering

GARCH (Generalized Autoregressive Conditional Heteroskedasticity) is the standard model for volatility clustering. The key insight is that tomorrow’s variance is a weighted combination of three things: a long-run baseline, today’s squared shock, and today’s estimated variance.

σ²_t = ω + α·ε²_(t-1) + β·σ²_(t-1)

The parameters:

• ω is the long-run variance floor (makes sure σ² never collapses to zero)

• α (ARCH term) captures how strongly recent shocks feed into tomorrow’s variance

• β (GARCH term) captures how strongly today’s estimated variance persists into tomorrow

For SPY, reasonable calibrated values are α = 0.09, β = 0.89. Their sum, α + β = 0.98, is close to 1—this is characteristic of high persistence, meaning volatility shocks decay slowly. After a 30% crash, it typically takes weeks for realized volatility to return to its baseline level. The GARCH model replicates this.

def simulate_garch(mu: float, omega: float, alpha: float, beta: float,
                   S0: float, T: int, seed: int = 42) -> tuple[np.ndarray, np.ndarray]:
    """
    Simulate price path with GARCH(1,1) volatility dynamics.
    
    Returns (prices, conditional_volatilities).
    """
    rng = np.random.default_rng(seed)
    dt = 1 / 252
    
    # Long-run variance implied by parameters
    long_run_var = omega / (1 - alpha - beta)
    
    variances = np.zeros(T)
    returns = np.zeros(T)
    variances[0] = long_run_var
    
    for t in range(T):
        # Draw return shock
        z = rng.standard_normal()
        returns[t] = mu * dt + np.sqrt(variances[t] * dt) * z
        
        # Update variance for next period
        epsilon_sq = (returns[t] - mu * dt) ** 2
        if t + 1 < T:
            variances[t+1] = omega + alpha * epsilon_sq / dt + beta * variances[t]
    
    prices = S0 * np.exp(np.cumsum(np.insert(returns, 0, 0)))
    vols = np.sqrt(variances * 252)  # annualize
    
    return prices[1:], vols


# SPY-calibrated parameters
omega_spy = 0.000002   # long-run daily variance floor
alpha_spy = 0.09
beta_spy  = 0.89

3.3 Fat Tails with the Student’s t Distribution

The simplest and most effective fix for fat tails is replacing the Gaussian noise term with draws from a Student’s t distribution. As degrees of freedom ν decreases, the tails get fatter. At ν → ∞, you recover the Gaussian.

The mapping between excess kurtosis and degrees of freedom is:

Excess Kurtosis = 6 / (ν - 4),    for ν > 4

Solving for ν:
ν = 6 / Excess_Kurtosis + 4

For SPY’s excess kurtosis of 11.72:

ν = 6 / 11.72 + 4 ≈ 4.51

At ν = 4.51, extreme events occur roughly 4× more often than a Gaussian would predict—matching the empirical tail frequency in the historical data. The t-distribution also produces negative skew when combined with GBM drift dynamics, which partially addresses that -0.55 skewness signature.

def draw_fat_tail_shocks(T: int, nu: float, seed: int = 42) -> np.ndarray:
    """
    Draw T noise samples from a Student's t distribution scaled to unit variance.
    
    nu: degrees of freedom (lower = fatter tails)
        nu=4.51 matches SPY's excess kurtosis of ~11.7
        nu=6.0  matches moderate fat tails
        nu=30   is close to Gaussian
    """
    rng = np.random.default_rng(seed)
    
    # Student's t with variance = nu/(nu-2), normalize to unit variance
    raw = rng.standard_t(nu, size=T)
    scale = np.sqrt((nu - 2) / nu)
    
    return raw * scale


# Quick sanity check on tail properties
def report_tail_stats(shocks: np.ndarray, nu: float) -> None:
    kurtosis = stats.kurtosis(shocks)
    theoretical_excess_kurtosis = 6 / (nu - 4) if nu > 4 else float('inf')
    tail_prob = np.mean(np.abs(shocks) > 3)
    
    print(f"Degrees of freedom:          {nu:.2f}")
    print(f"Empirical excess kurtosis:   {kurtosis:.2f}")
    print(f"Theoretical excess kurtosis: {theoretical_excess_kurtosis:.2f}")
    print(f"Empirical |Z| > 3 prob:      {tail_prob:.4f}  (Gaussian: 0.0027)")

3.4 Mean Reversion with the Ornstein-Uhlenbeck Process

The Ornstein-Uhlenbeck (OU) process is the standard continuous-time mean-reverting model:

dX = θ(μ - X) dt + σ dW

Here θ controls the speed of mean reversion—how strongly deviations from μ get pulled back. In discrete time:

X_t = X_(t-1) + θ(μ - X_(t-1))·Δt + σ·√Δt·Z_t

For equity returns, the mean-reversion component is subtle (the lag-1 autocorrelation of -0.08 implies θ is small). For VIX, mean reversion is much more pronounced—VIX always drifts back toward its long-run average of ~18, even after spikes to 80.

def simulate_ou_process(theta: float, mu: float, sigma: float,
                         X0: float, T: int, seed: int = 42) -> np.ndarray:
    """
    Simulate Ornstein-Uhlenbeck mean-reverting process.
    
    theta: mean-reversion speed (higher = faster reversion)
    mu:    long-run mean
    sigma: diffusion coefficient
    X0:    starting value
    T:     number of steps
    
    Useful for VIX simulation, interest rates, or adding mean-reversion
    adjustment to equity return models.
    """
    rng = np.random.default_rng(seed)
    dt = 1 / 252
    
    path = np.zeros(T + 1)
    path[0] = X0
    
    for t in range(T):
        drift = theta * (mu - path[t]) * dt
        diffusion = sigma * np.sqrt(dt) * rng.standard_normal()
        path[t + 1] = path[t] + drift + diffusion
    
    return path


# VIX-calibrated OU parameters
vix_theta = 4.5    # moderately fast mean reversion
vix_mu    = 18.0   # long-run VIX average
vix_sigma = 25.0   # high diffusion (VIX is volatile itself)

3.5 Regime Switching with a Markov Chain

The most powerful addition to the model is regime switching. Real markets cycle through meaningfully different states—calm bull markets, panicked crashes, grinding bear markets, choppy sideways periods—and each regime has its own return and volatility characteristics.

The Markov chain assumption is that tomorrow’s regime depends only on today’s regime (not on the full history), with fixed transition probabilities. Estimated from SPY data, the regime distribution and transition matrix look like this:

Regime Distribution:
────────────────────────────────────────────────────────────
Regime       Base Freq    Annual Drift    Daily Volatility
────────────────────────────────────────────────────────────
Bull          48.3%         +15%              0.7%
Bear          19.5%         -12%              1.3%
Crisis         6.2%         -35%              2.5%
Choppy        26.0%          +2%              1.5%

Transition Matrix (rows = current, cols = next):
──────────────────────────────────────────────────────
From/To       Bull    Bear    Crisis    Choppy
──────────────────────────────────────────────
Bull          0.95    0.03    0.01      0.01
Bear          0.10    0.85    0.03      0.02
Crisis        0.15    0.25    0.55      0.05
Choppy        0.20    0.15    0.05      0.60

The diagonal dominance reflects persistence—once in a regime, you tend to stay there. But crisis regimes are inherently unstable (diagonal of only 0.55), reflecting the empirical observation that crashes don’t last forever.

import numpy as np

# Regime parameters
REGIME_PARAMS = {
    # regime_id: (annual_drift, daily_vol, name)
    0: (0.15,  0.007, 'Bull'),
    1: (-0.12, 0.013, 'Bear'),
    2: (-0.35, 0.025, 'Crisis'),
    3: (0.02,  0.015, 'Choppy'),
}

# Calibrated transition matrix
TRANSITION_MATRIX = np.array([
    [0.95, 0.03, 0.01, 0.01],  # from Bull
    [0.10, 0.85, 0.03, 0.02],  # from Bear
    [0.15, 0.25, 0.55, 0.05],  # from Crisis
    [0.20, 0.15, 0.05, 0.60],  # from Choppy
])

def simulate_regime_sequence(T: int, initial_regime: int = 0,
                              seed: int = 42) -> np.ndarray:
    """
    Simulate a sequence of regime labels using a Markov chain.
    Returns array of regime indices, length T.
    """
    rng = np.random.default_rng(seed)
    regimes = np.zeros(T, dtype=int)
    regimes[0] = initial_regime
    
    for t in range(1, T):
        current = regimes[t - 1]
        regimes[t] = rng.choice(4, p=TRANSITION_MATRIX[current])
    
    return regimes

4. Assembling the Full Generator

Now we combine all five components into a single function. The execution order matters:

1. Draw the regime sequence from the Markov chain

2. For each day, look up that regime’s drift and base volatility

3. Apply the GARCH update to get the conditional volatility

4. Draw the return shock from Student’s t (not Gaussian)

5. Add a small mean-reversion adjustment

6. Compute the price change via GBM dynamics

import numpy as np
import pandas as pd
from scipy import stats


class SyntheticMarketGenerator:
    """
    Full stochastic market generator combining:
    - Regime-switching Markov chain
    - GARCH(1,1) volatility dynamics
    - Student's t fat tails
    - Ornstein-Uhlenbeck mean reversion
    - GBM baseline
    """

    def __init__(
        self,
        # GBM baseline
        mu: float = 0.10,
        sigma: float = 0.187,
        # GARCH
        garch_omega: float = 2e-6,
        garch_alpha: float = 0.09,
        garch_beta: float = 0.89,
        # Fat tails
        t_nu: float = 4.51,
        # Mean reversion
        mr_theta: float = 0.08,
        # Regime switching
        use_regimes: bool = True,
        transition_matrix: np.ndarray = None,
        regime_params: dict = None,
    ):
        self.mu = mu
        self.sigma = sigma
        self.garch_omega = garch_omega
        self.garch_alpha = garch_alpha
        self.garch_beta = garch_beta
        self.t_nu = t_nu
        self.mr_theta = mr_theta
        self.use_regimes = use_regimes

        self.transition_matrix = transition_matrix if transition_matrix is not None else \
            np.array([
                [0.95, 0.03, 0.01, 0.01],
                [0.10, 0.85, 0.03, 0.02],
                [0.15, 0.25, 0.55, 0.05],
                [0.20, 0.15, 0.05, 0.60],
            ])

        self.regime_params = regime_params if regime_params is not None else {
            0: {'drift': 0.15,  'vol_mult': 0.7,  'name': 'Bull'},
            1: {'drift': -0.12, 'vol_mult': 1.3,  'name': 'Bear'},
            2: {'drift': -0.35, 'vol_mult': 2.5,  'name': 'Crisis'},
            3: {'drift': 0.02,  'vol_mult': 1.5,  'name': 'Choppy'},
        }

    def generate(
        self,
        T: int = 252,
        S0: float = 100.0,
        initial_regime: int = 0,
        seed: int = 42,
    ) -> pd.DataFrame:
        """
        Generate T days of synthetic price data.

        Returns DataFrame with columns:
            price, daily_return, log_return, regime, regime_name,
            conditional_vol (annualized), garch_var
        """
        rng = np.random.default_rng(seed)
        dt = 1 / 252

        # --- Step 1: regime sequence ---
        regimes = np.zeros(T, dtype=int)
        regimes[0] = initial_regime
        if self.use_regimes:
            for t in range(1, T):
                regimes[t] = rng.choice(4, p=self.transition_matrix[regimes[t - 1]])

        # --- Step 2: GARCH variance initialization ---
        long_run_var = self.garch_omega / (1 - self.garch_alpha - self.garch_beta)
        garch_var = np.zeros(T)
        garch_var[0] = long_run_var

        # --- Step 3: Draw fat-tail shocks ---
        raw_shocks = rng.standard_t(self.t_nu, size=T)
        shocks = raw_shocks * np.sqrt((self.t_nu - 2) / self.t_nu)  # unit variance

        # --- Step 4: Build returns ---
        log_returns = np.zeros(T)
        prev_log_return = 0.0

        for t in range(T):
            regime = regimes[t]
            regime_drift = self.regime_params[regime]['drift'] if self.use_regimes else self.mu
            regime_vol_mult = self.regime_params[regime]['vol_mult'] if self.use_regimes else 1.0

            # Effective daily volatility from GARCH, scaled by regime
            daily_vol = np.sqrt(garch_var[t] * dt) * regime_vol_mult
            daily_vol = max(daily_vol, 0.001)  # floor at 0.1% to prevent collapse

            # Mean reversion component (pulls return toward zero after extremes)
            mr_adj = -self.mr_theta * prev_log_return * dt

            # GBM drift term for this regime
            drift_term = regime_drift * dt - 0.5 * (daily_vol ** 2)

            # Combined return
            log_returns[t] = drift_term + daily_vol * shocks[t] + mr_adj
            prev_log_return = log_returns[t]

            # GARCH update for next period
            if t + 1 < T:
                epsilon_sq = (log_returns[t] - regime_drift * dt) ** 2
                garch_var[t + 1] = (
                    self.garch_omega
                    + self.garch_alpha * epsilon_sq / dt
                    + self.garch_beta * garch_var[t]
                )
                garch_var[t + 1] = max(garch_var[t + 1], 1e-8)

        # --- Step 5: Construct price series ---
        prices = S0 * np.exp(np.cumsum(log_returns))
        daily_returns = np.exp(log_returns) - 1

        df = pd.DataFrame({
            'price':           prices,
            'daily_return':    daily_returns,
            'log_return':      log_returns,
            'regime':          regimes,
            'regime_name':     [self.regime_params[r]['name'] for r in regimes],
            'conditional_vol': np.sqrt(garch_var * 252),
            'garch_var':       garch_var,
        })
        df.index.name = 'day'

        return df


# Basic usage
gen = SyntheticMarketGenerator()
df = gen.generate(T=252, S0=450.0, seed=99)
print(df.head(10))
print(f"\nTotal return: {(df['price'].iloc[-1] / df['price'].iloc[0] - 1) * 100:.1f}%")
print(f"Max drawdown: {((df['price'] / df['price'].cummax()) - 1).min() * 100:.1f}%")

5. Generating Correlated Multi-Asset Paths

A single synthetic SPY path is not very useful on its own. Real trading strategies involve multiple assets with specific correlation structures. The standard approach is Cholesky decomposition of the correlation matrix.

If you have assets with correlations described by matrix Σ, and you want correlated shocks Z₁, Z₂, ..., Zₙ, you compute the Cholesky factor L such that LLᵀ = Σ, then generate independent standard normals W and multiply: Z = LW.

For a three-asset portfolio (equity index, long-term bonds, VIX), calibrated correlations look like:

            Equity    Bonds    VIX
Equity       1.00     -0.40   -0.75
Bonds       -0.40      1.00    0.30
VIX         -0.75      0.30    1.00

The equity-VIX correlation of -0.75 is the most important: when stocks fall, fear spikes. The equity-bond correlation of -0.40 captures the flight-to-quality dynamic. These correlations are not fixed in practice—they tend to strengthen during crises—but as a baseline they are reasonable.

def generate_correlated_paths(
    T: int = 252,
    corr_matrix: np.ndarray = None,
    asset_params: list[dict] = None,
    seed: int = 42,
) -> pd.DataFrame:
    """
    Generate correlated synthetic paths for multiple assets.

    corr_matrix: (n x n) correlation matrix
    asset_params: list of dicts with keys 'name', 'mu', 'sigma', 'S0'

    Returns DataFrame with one column per asset (prices).
    """
    if corr_matrix is None:
        corr_matrix = np.array([
            [ 1.00, -0.40, -0.75],
            [-0.40,  1.00,  0.30],
            [-0.75,  0.30,  1.00],
        ])

    if asset_params is None:
        asset_params = [
            {'name': 'SPY',  'mu': 0.10,  'sigma': 0.187, 'S0': 450.0},
            {'name': 'TLT',  'mu': 0.04,  'sigma': 0.150, 'S0': 95.0},
            {'name': 'VIX',  'mu': 0.00,  'sigma': 0.800, 'S0': 18.0},
        ]

    rng = np.random.default_rng(seed)
    n_assets = len(asset_params)
    dt = 1 / 252

    # Cholesky decomposition for correlated shocks
    L = np.linalg.cholesky(corr_matrix)

    # Independent normals -> correlated
    independent = rng.standard_normal((T, n_assets))
    correlated = (L @ independent.T).T  # shape (T, n_assets)

    prices = np.zeros((T + 1, n_assets))
    for i, params in enumerate(asset_params):
        prices[0, i] = params['S0']

    for t in range(T):
        for i, params in enumerate(asset_params):
            mu, sigma = params['mu'], params['sigma']
            log_ret = (mu - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * correlated[t, i]
            prices[t + 1, i] = prices[t, i] * np.exp(log_ret)

    df = pd.DataFrame(
        prices[1:],
        columns=[p['name'] for p in asset_params],
    )
    df.index.name = 'day'

    return df


# Generate and inspect correlations
paths = generate_correlated_paths(T=252, seed=42)
log_rets = np.log(paths / paths.shift(1)).dropna()
print("Realized correlations:")
print(log_rets.corr().round(2))

6. Adding Synthetic VIX

VIX deserves special treatment because it is not a price series—it is a volatility index with specific structural properties. It is mean-reverting (always drifts back toward ~18), spikes sharply during equity drawdowns (negative correlation with returns), and has a strong asymmetry: it rises faster than it falls.

The jump-diffusion extension of the OU process captures this:

dVIX = κ(V̄ - VIX)dt + σ_VIX·dW + J·dN

where dN is a Poisson jump process that fires when equity returns exceed a negative threshold, and J is the jump size (calibrated to the typical VIX spike magnitude of 8–15 points).

def simulate_vix(
    equity_returns: np.ndarray,
    kappa: float = 4.5,       # mean reversion speed
    vbar: float = 18.0,       # long-run mean
    sigma_vix: float = 4.0,   # diffusion coefficient
    vix0: float = 18.0,       # starting VIX
    jump_threshold: float = -0.02,    # equity return that triggers jump
    jump_mean: float = 10.0,          # average VIX jump size
    jump_std: float = 3.0,            # jump size dispersion
    seed: int = 42,
) -> np.ndarray:
    """
    Simulate VIX path given realized equity returns.

    VIX spikes when equity returns fall below jump_threshold,
    and mean-reverts toward vbar otherwise.
    """
    rng = np.random.default_rng(seed)
    T = len(equity_returns)
    dt = 1 / 252

    vix = np.zeros(T + 1)
    vix[0] = vix0

    for t in range(T):
        # Mean reversion drift
        drift = kappa * (vbar - vix[t]) * dt

        # Diffusion
        diffusion = sigma_vix * np.sqrt(dt) * rng.standard_normal()

        # Jump if equity crash
        jump = 0.0
        if equity_returns[t] < jump_threshold:
            severity = abs(equity_returns[t]) / abs(jump_threshold)
            jump_size = rng.normal(jump_mean * severity, jump_std)
            jump = max(jump_size, 0)  # VIX can only jump up, not down

        vix[t + 1] = max(vix[t] + drift + diffusion + jump, 5.0)  # floor at 5

    return vix[1:]


# Pair with equity returns
gen = SyntheticMarketGenerator()
equity_df = gen.generate(T=252, seed=42)
vix_series = simulate_vix(equity_df['daily_return'].values, seed=42)

equity_df['vix'] = vix_series

print(f"VIX range: {vix_series.min():.1f} - {vix_series.max():.1f}")
print(f"VIX mean:  {vix_series.mean():.1f}")
print(f"Corr(equity returns, VIX changes): "
      f"{np.corrcoef(equity_df['daily_return'], np.diff(vix_series, prepend=vix_series[0]))[0,1]:.2f}")

7. Scenario Engineering

The power of synthetic data is not just generating “random” markets—it is generating specific market conditions to probe strategy behavior. Scenario engineering means overriding the base generator’s parameters to produce targeted environments.

Here are eight canonical scenarios with the specific parameter modifications that create them:

SCENARIOS = {
    'crash': {
        'description': 'Sudden equity crash with gradual recovery',
        'override': {
            'initial_regime': 2,  # start in Crisis
            'transition_override': {
                # Keep crisis regime sticky for first 40 days,
                # then force transition toward Bear/Bull recovery
            },
        },
        'drift_override': None,  # use regime drift
        'vol_override': None,
    },

    'choppy': {
        'description': 'High volatility, no directional trend',
        'mu_override': 0.02,
        'sigma_override': 0.35,
    },

    'stagflation': {
        'description': 'Slow grind down with moderate vol',
        'mu_override': -0.08,
        'sigma_override': 0.15,
    },

    'bull': {
        'description': 'Strong sustained uptrend',
        'mu_override': 0.20,
        'sigma_override': 0.12,
    },

    'slow_bleed': {
        'description': 'Gradual decline, low volatility (momentum killers)',
        'mu_override': -0.105,
        'sigma_override': 0.12,
    },

    'flash_crash': {
        'description': 'Sharp V-shaped crash and rapid recovery',
        'crash_days': 10,
        'crash_magnitude': -0.37,
        'recovery_days': 20,
    },

    'sideways': {
        'description': 'Range-bound market, high mean reversion',
        'mu_override': 0.00,
        'sigma_override': 0.10,
        'mr_theta_override': 0.50,
    },

    'bear': {
        'description': 'Sustained bear market with elevated vol',
        'mu_override': -0.15,
        'sigma_override': 0.25,
    },
}


def generate_scenario(
    scenario_name: str,
    T: int = 252,
    S0: float = 100.0,
    seed: int = 42,
) -> pd.DataFrame:
    """
    Generate a scenario-specific price path.
    """
    if scenario_name not in SCENARIOS:
        raise ValueError(f"Unknown scenario: {scenario_name}. "
                         f"Choose from {list(SCENARIOS.keys())}")

    scenario = SCENARIOS[scenario_name]

    if scenario_name == 'flash_crash':
        return _generate_flash_crash(
            T=T, S0=S0,
            crash_days=scenario['crash_days'],
            crash_magnitude=scenario['crash_magnitude'],
            recovery_days=scenario['recovery_days'],
            seed=seed,
        )

    # For standard scenarios, override generator parameters
    gen = SyntheticMarketGenerator(
        mu=scenario.get('mu_override', 0.10),
        sigma=scenario.get('sigma_override', 0.187),
        mr_theta=scenario.get('mr_theta_override', 0.08),
        use_regimes=False,  # fixed parameters override regime switching
    )
    return gen.generate(T=T, S0=S0, seed=seed)


def _generate_flash_crash(
    T: int, S0: float,
    crash_days: int, crash_magnitude: float,
    recovery_days: int, seed: int,
) -> pd.DataFrame:
    """
    Inject a programmatic flash crash into an otherwise normal path.
    """
    gen = SyntheticMarketGenerator(mu=0.10, sigma=0.12, use_regimes=False)
    df = gen.generate(T=T, S0=S0, seed=seed)

    crash_start = T // 4
    crash_end = crash_start + crash_days
    recovery_end = crash_end + recovery_days

    pre_crash_price = df['price'].iloc[crash_start - 1]
    post_crash_price = pre_crash_price * (1 + crash_magnitude)

    # Inject crash: linear decline over crash_days
    crash_prices = np.linspace(pre_crash_price, post_crash_price, crash_days)
    df.loc[crash_start:crash_end - 1, 'price'] = crash_prices

    # Inject recovery: exponential recovery over recovery_days
    if recovery_end <= T:
        recovery_prices = np.linspace(post_crash_price, pre_crash_price, recovery_days)
        df.loc[crash_end:recovery_end - 1, 'price'] = recovery_prices

    return df

8. Generating From the Original Time Series: Bootstrap Methods

Everything in the previous sections builds a parametric generator: fit a model to historical data, then simulate from that model. The advantage is you can extrapolate beyond what history actually contains—generate crashes deeper than 2020, recoveries faster than 2020, volatility regimes that haven’t occurred yet.

The disadvantage is that the synthetic data is only as good as the model’s assumptions. If the GARCH parameters are slightly mis-calibrated, or the regime transition matrix is wrong, the synthetic returns won’t actually match the properties of the real series. You’re testing against your model’s world, not the real world’s world.

The alternative is resampling directly from the original time series. No distributional assumptions. No parameter estimation. The synthetic paths are built from actual historical returns, rearranged—so the fat tails, volatility clustering, and skewness are all preserved by construction, because you never left the empirical distribution.

8.1 Why Not Just Shuffle Returns?

The naive version — shuffle all daily returns randomly — breaks immediately (and trust me, I’ve tried this). It destroys the autocorrelation structure that makes markets behave like markets. Volatility clustering disappears because high-vol days are no longer adjacent to other high-vol days. Mean reversion disappears for the same reason. A shuffled return series has the right unconditional distribution but none of the right conditional dynamics.

The fix is block bootstrap: instead of resampling individual days, resample contiguous blocks of days. A block of 20 consecutive returns carries the local autocorrelation structure with it—if day 1 of the block was a high-vol day, day 2 is still the high-vol day that historically followed it. Piece enough blocks together and you get a path that has the right unconditional distribution and approximately preserves short-run dynamics.

8.2 Choosing Block Length

Block length is the key tuning parameter and it involves a direct tradeoff:

• Too short (e.g. 1–3 days): autocorrelation structure breaks down, approaches naive shuffle

• Too long (e.g. 100+ days): you’re just stitching together large chunks of actual history, which limits diversity across paths

• Sweet spot: roughly the persistence horizon of the dependency you care about

For equity returns, volatility clustering persists for about 10–20 trading days (matching GARCH’s typical half-life). A block length of 15–20 days is a reasonable default. For strategies with longer lookback windows, you’d want longer blocks.

The stationary bootstrap (Politis & Romano, 1994) is a refinement: instead of fixed block length, each block’s length is drawn from a geometric distribution with mean equal to your target block length. This prevents artificial periodicity at block boundaries and makes the resampled series stationary by construction.

import numpy as np
import pandas as pd


def block_bootstrap(
    returns: np.ndarray,
    T: int = 252,
    block_length: int = 20,
    S0: float = 100.0,
    seed: int = 42,
) -> pd.DataFrame:
    """
    Generate a synthetic price path by circular block bootstrap.

    Resamples contiguous blocks of actual returns to preserve
    short-run autocorrelation structure (volatility clustering,
    mean reversion) without any parametric assumptions.

    Parameters
    ----------
    returns      : Array of historical daily log returns
    T            : Number of days to generate
    block_length : Length of each resampled block (default: 20)
    S0           : Starting price
    seed         : RNG seed

    Returns
    -------
    pd.DataFrame with columns: price, log_return
    """
    rng = np.random.default_rng(seed)
    n = len(returns)
    synthetic_returns = np.zeros(T)
    t = 0

    while t < T:
        # Draw a random start index (circular — wraps around end of series)
        start = rng.integers(0, n)
        block = np.array([returns[(start + i) % n] for i in range(block_length)])

        # Clip to remaining days needed
        remaining = T - t
        block = block[:remaining]
        synthetic_returns[t:t + len(block)] = block
        t += len(block)

    prices = S0 * np.exp(np.cumsum(synthetic_returns))

    return pd.DataFrame({
        'price':      prices,
        'log_return': synthetic_returns,
    })


def stationary_bootstrap(
    returns: np.ndarray,
    T: int = 252,
    mean_block_length: int = 20,
    S0: float = 100.0,
    seed: int = 42,
) -> pd.DataFrame:
    """
    Generate a synthetic price path via stationary bootstrap.

    Block lengths are drawn from Geometric(1/mean_block_length),
    making the resampled series stationary and avoiding artificial
    periodicity at fixed block boundaries.

    Parameters
    ----------
    returns           : Array of historical daily log returns
    T                 : Number of days to generate
    mean_block_length : Expected block length (Geometric distribution mean)
    S0                : Starting price
    seed              : RNG seed

    Returns
    -------
    pd.DataFrame with columns: price, log_return
    """
    rng = np.random.default_rng(seed)
    n = len(returns)
    p = 1.0 / mean_block_length  # Geometric distribution parameter

    synthetic_returns = np.zeros(T)
    t = 0

    while t < T:
        start = rng.integers(0, n)
        # Geometric block length
        block_len = rng.geometric(p)
        block = np.array([returns[(start + i) % n] for i in range(block_len)])

        remaining = T - t
        block = block[:remaining]
        synthetic_returns[t:t + len(block)] = block
        t += len(block)

    prices = S0 * np.exp(np.cumsum(synthetic_returns))

    return pd.DataFrame({
        'price':      prices,
        'log_return': synthetic_returns,
    })

8.3 Strategy-Specific Property Selection

Here is where most synthetic data guides go wrong: they validate the generator against a generic list of statistical moments and call it done. But the properties that matter are the ones your specific strategy depends on.

Different strategy types are sensitive to entirely different statistical features:

Strategy Type          Properties That Matter Most
──────────────────────────────────────────────────────────────────────
Momentum / trend       Autocorrelation at your lookback horizon
                       Regime persistence (how long trends last)
                       Tail asymmetry (crash magnitude vs recovery speed)

Mean reversion         Half-life of deviations from mean
                       Vol-of-vol (does vol itself revert quickly?)
                       Bid-ask spread / transaction cost sensitivity

Vol targeting          GARCH persistence (α + β)
                       Vol-of-vol (realized vol of realized vol)
                       Correlation stability across vol regimes

Regime switching       Regime transition probabilities
                       Regime duration distributions
                       In-regime return and vol parameters

Pairs / stat arb       Cointegration stability
                       Spread mean and variance stationarity
                       Correlation breakdown frequency in stress

Before accepting a synthetic series as a valid test environment for your strategy, verify it preserves the properties your strategy actually uses. For a momentum strategy, that means checking autocorrelation at your signal’s lookback horizon. For a vol-targeting strategy, that means checking GARCH persistence. Generic moment-matching (mean, vol, kurtosis) is necessary but not sufficient.

def validate_for_strategy(
    synthetic_returns: np.ndarray,
    real_returns: np.ndarray,
    strategy_type: str = 'momentum',
) -> dict:
    """
    Validate synthetic returns against the properties most relevant
    to a specific strategy type.

    Parameters
    ----------
    synthetic_returns : Synthetic daily log return array
    real_returns      : Historical daily log return array
    strategy_type     : One of 'momentum', 'mean_reversion', 'vol_targeting'

    Returns
    -------
    dict of property name -> (real_value, synthetic_value, gap)
    """
    from scipy import stats

    results = {}

    def _acf(r, lag):
        return pd.Series(r).autocorr(lag=lag)

    def _half_life(r):
        # Estimate OU half-life from lag-1 autocorrelation
        lag1 = _acf(r, 1)
        if lag1 >= 1 or lag1 <= 0:
            return float('inf')
        return -np.log(2) / np.log(lag1)

    def _garch_persistence(r):
        # Proxy: lag-1 autocorrelation of squared returns
        # True GARCH fit would give alpha + beta directly
        return _acf(r ** 2, 1)

    if strategy_type == 'momentum':
        for lag in [1, 5, 10, 20]:
            real_acf = _acf(real_returns, lag)
            synth_acf = _acf(synthetic_returns, lag)
            results[f'acf_lag{lag}'] = (real_acf, synth_acf, synth_acf - real_acf)

        # Tail asymmetry: ratio of worst 5% days to best 5% days
        real_asym = np.percentile(real_returns, 5) / np.percentile(real_returns, 95)
        synth_asym = np.percentile(synthetic_returns, 5) / np.percentile(synthetic_returns, 95)
        results['tail_asymmetry'] = (real_asym, synth_asym, synth_asym - real_asym)

    elif strategy_type == 'mean_reversion':
        real_hl = _half_life(real_returns)
        synth_hl = _half_life(synthetic_returns)
        results['half_life_days'] = (real_hl, synth_hl, synth_hl - real_hl)

        # Vol of vol
        real_vov = pd.Series(real_returns).rolling(20).std().std()
        synth_vov = pd.Series(synthetic_returns).rolling(20).std().std()
        results['vol_of_vol'] = (real_vov, synth_vov, synth_vov - real_vov)

    elif strategy_type == 'vol_targeting':
        real_persist = _garch_persistence(real_returns)
        synth_persist = _garch_persistence(synthetic_returns)
        results['garch_persistence_proxy'] = (real_persist, synth_persist,
                                               synth_persist - real_persist)

        real_vov = pd.Series(real_returns).rolling(20).std().std()
        synth_vov = pd.Series(synthetic_returns).rolling(20).std().std()
        results['vol_of_vol'] = (real_vov, synth_vov, synth_vov - real_vov)

    print(f"\nStrategy-specific validation ({strategy_type})")
    print(f"{'Property':<28} {'Real':>10} {'Synthetic':>10} {'Gap':>10}")
    print('─' * 62)
    for prop, (r, s, gap) in results.items():
        print(f"{prop:<28} {r:>10.4f} {s:>10.4f} {gap:>+10.4f}")

    return results

8.4 Stationarity Checks

“Stable properties” means more than just matching moments globally. It means the statistical properties don’t drift across the simulation window. A synthetic series where volatility is high in the first half and low in the second half has the right unconditional variance but will produce misleading strategy test results—especially for strategies with rolling lookback windows that will see a structurally different environment as they move through the path.

The standard test is the Augmented Dickey-Fuller (ADF) test for unit roots. A stationary series—one whose statistical properties don’t change over time—should reject the null hypothesis of a unit root. For log returns (not prices), stationarity is almost always satisfied. But it is worth checking, especially on synthetic series generated with strong regime drift.

A second check is rolling moment stability: compute mean, variance, and skewness in rolling windows across the synthetic path and verify they don’t trend systematically.

from statsmodels.tsa.stattools import adfuller


def check_stationarity(
    returns: np.ndarray,
    label: str = 'Synthetic',
    window: int = 60,
) -> dict:
    """
    Check stationarity of a synthetic return series.

    Runs the Augmented Dickey-Fuller test and plots rolling
    mean and volatility to detect systematic drift.

    Parameters
    ----------
    returns : Daily return array
    label   : Display name
    window  : Rolling window for moment stability check (days)

    Returns
    -------
    dict with ADF results and rolling stability metrics.
    """
    import matplotlib.pyplot as plt

    # ADF test
    adf_result = adfuller(returns, autolag='AIC')
    adf_stat, p_value, _, _, critical_values, _ = adf_result

    print(f"\nStationarity check: {label}")
    print(f"  ADF statistic : {adf_stat:.4f}")
    print(f"  p-value       : {p_value:.4f}")
    print(f"  Critical (5%) : {critical_values['5%']:.4f}")
    if p_value < 0.05:
        print(f"  Result        : ✓ Stationary (reject unit root)")
    else:
        print(f"  Result        : ✗ Non-stationary — check for drift in synthetic series")

    # Rolling moment stability
    s = pd.Series(returns)
    rolling_mean = s.rolling(window).mean() * 252        # annualised
    rolling_vol = s.rolling(window).std() * np.sqrt(252) # annualised

    # Drift metric: max rolling mean minus min rolling mean (should be small)
    mean_drift = rolling_mean.dropna().max() - rolling_mean.dropna().min()
    vol_drift = rolling_vol.dropna().max() - rolling_vol.dropna().min()

    print(f"  Rolling mean drift (ann.) : {mean_drift * 100:.2f}%  "
          f"{'✓ stable' if mean_drift < 0.10 else '✗ drifting'}")
    print(f"  Rolling vol drift  (ann.) : {vol_drift * 100:.2f}%  "
          f"{'✓ stable' if vol_drift < 0.15 else '✗ drifting'}")

    # Plot rolling stability
    fig, axes = plt.subplots(2, 1, figsize=(12, 6), sharex=True)
    fig.suptitle(f'Rolling Moment Stability: {label}', fontsize=12)

    axes[0].plot(rolling_mean * 100, color='steelblue', lw=1)
    axes[0].axhline(0, color='black', lw=0.5, linestyle='--')
    axes[0].set_ylabel('Rolling Mean Return (%, ann.)')
    axes[0].set_title(f'{window}-day rolling annualised mean')

    axes[1].plot(rolling_vol * 100, color='darkorange', lw=1)
    axes[1].set_ylabel('Rolling Volatility (%, ann.)')
    axes[1].set_title(f'{window}-day rolling annualised volatility')
    axes[1].set_xlabel('Day')

    plt.tight_layout()
    plt.savefig(f'stationarity_{label.lower().replace(" ", "_")}.png',
                dpi=150, bbox_inches='tight')
    plt.show()

    return {
        'adf_stat':   adf_stat,
        'adf_pvalue': p_value,
        'stationary': p_value < 0.05,
        'mean_drift': mean_drift,
        'vol_drift':  vol_drift,
    }

Run this on every synthetic series before using it for strategy testing. A synthetic path that fails the ADF test or shows obvious rolling mean drift is not a valid test environment—it is a biased one that will systematically favor or penalize strategies depending on which half of the window they’re operating in.

9. Validating the Synthetic Data

This is the step most guides skip, and it is arguably the most important. Generating fake price paths is easy. Generating fake price paths that are statistically plausible requires validation.

The core question is: does the synthetic data reproduce the statistical fingerprint from Section 2, and does it preserve the strategy-specific properties identified in Section 8?

def validate_synthetic_vs_real(
    synthetic_returns: np.ndarray,
    real_returns: np.ndarray,
    label: str = "Synthetic"
) -> pd.DataFrame:
    """
    Compare key statistical properties between synthetic and real returns.
    Prints a comparison table and returns the stats as a DataFrame.
    """

    def get_stats(r: np.ndarray, name: str) -> dict:
        annual_vol = r.std() * np.sqrt(252)
        annual_ret = r.mean() * 252
        sharpe = annual_ret / annual_vol if annual_vol > 0 else 0

        lag1_acf = pd.Series(r).autocorr(lag=1)
        lag1_vol_acf = pd.Series(r**2).autocorr(lag=1)

        tail_prob = np.mean(np.abs(r - r.mean()) > 3 * r.std())
        _, p_normality = stats.normaltest(r)
        ex_kurtosis = stats.kurtosis(r)
        skew = stats.skew(r)

        return {
            'source': name,
            'annual_return': annual_ret,
            'annual_vol': annual_vol,
            'sharpe': sharpe,
            'skewness': skew,
            'excess_kurtosis': ex_kurtosis,
            'lag1_acf': lag1_acf,
            'lag1_vol_acf': lag1_vol_acf,
            'tail_prob_3sigma': tail_prob,
            'normality_p': p_normality,
        }

    real_stats = get_stats(real_returns, 'Real')
    synth_stats = get_stats(synthetic_returns, label)

    results = pd.DataFrame([real_stats, synth_stats]).set_index('source')

    print(f"\n{'Property':<28} {'Real':>12} {label:>12} {'Gap':>10}")
    print("─" * 66)
    for col in results.columns:
        r = results.loc['Real', col]
        s = results.loc[label, col]
        gap = s - r
        print(f"{col:<28} {r:>12.4f} {s:>12.4f} {gap:>+10.4f}")

    print(f"\nNormality test p-value < 0.05 means non-Gaussian (fat tails confirmed).")
    print(f"Real:      p = {real_stats['normality_p']:.6f}  {'✓ Non-Gaussian' if real_stats['normality_p'] < 0.05 else '✗'}")
    print(f"{label}: p = {synth_stats['normality_p']:.6f}  {'✓ Non-Gaussian' if synth_stats['normality_p'] < 0.05 else '✗'}")

    return results


# Run the KS test to compare distributions formally
def ks_test_returns(synthetic_returns: np.ndarray,
                    real_returns: np.ndarray) -> None:
    """
    Kolmogorov-Smirnov test: is the synthetic return distribution
    significantly different from the real one?

    High p-value = synthetic is statistically compatible with real.
    Low p-value = distributions are distinguishably different.
    """
    ks_stat, p_value = stats.ks_2samp(synthetic_returns, real_returns)

    print(f"\nKolmogorov-Smirnov Test (synthetic vs real returns)")
    print(f"  KS statistic:  {ks_stat:.4f}  (lower = more similar)")
    print(f"  p-value:       {p_value:.4f}")

    if p_value > 0.05:
        print(f"  Interpretation: Cannot reject that distributions are the same.")
    else:
        print(f"  Interpretation: Distributions are statistically distinguishable.")
        print(f"  This is expected for short runs — increase T for better calibration.")

Beyond the KS test, there are two visual diagnostics worth running:

QQ Plot: Plot quantiles of synthetic returns against quantiles of real returns. A straight line along the diagonal means the distributions match well. Fat tails show up as the synthetic series curving away from the diagonal at the extremes.

ACF of Squared Returns: Plot the autocorrelation function of squared daily returns for lags 1–30. A realistic generator should show positive autocorrelation decaying slowly from ~0.25 at lag-1 toward zero by lag-20. Pure GBM produces flat zero autocorrelation—the GARCH component is what creates this signature.

def plot_diagnostics(df: pd.DataFrame, title: str = "Synthetic") -> None:
    """
    Four-panel diagnostic plot for synthetic path quality.
    """
    import matplotlib.pyplot as plt
    from pandas.plotting import autocorrelation_plot

    fig, axes = plt.subplots(2, 2, figsize=(14, 10))
    fig.suptitle(f"Synthetic Market Diagnostics: {title}", fontsize=14)

    log_rets = df['log_return']

    # Panel 1: Price path colored by regime
    ax1 = axes[0, 0]
    regime_colors = {0: '#2196F3', 1: '#FF5722', 2: '#9C27B0', 3: '#FF9800'}
    for regime_id in df['regime'].unique():
        mask = df['regime'] == regime_id
        name = df.loc[mask, 'regime_name'].iloc[0]
        ax1.plot(df.index[mask], df.loc[mask, 'price'],
                 '.', markersize=1, color=regime_colors[regime_id], label=name)
    ax1.set_title('Price Path by Regime')
    ax1.set_ylabel('Price')
    ax1.legend(markerscale=5)

    # Panel 2: Return distribution vs Gaussian
    ax2 = axes[0, 1]
    x = np.linspace(log_rets.min(), log_rets.max(), 200)
    ax2.hist(log_rets, bins=50, density=True, alpha=0.6, color='steelblue', label='Synthetic')
    gaussian = stats.norm(log_rets.mean(), log_rets.std())
    ax2.plot(x, gaussian.pdf(x), 'r-', lw=2, label='Gaussian fit')
    ax2.set_title('Return Distribution vs Gaussian')
    ax2.legend()

    # Panel 3: GARCH conditional vol
    ax3 = axes[1, 0]
    ax3.fill_between(df.index, df['conditional_vol'] * 100, alpha=0.5, color='orange')
    ax3.set_title('Conditional Volatility (Annualized %)')
    ax3.set_ylabel('Volatility (%)')

    # Panel 4: ACF of squared returns (volatility clustering)
    ax4 = axes[1, 1]
    sq_rets = log_rets ** 2
    acf_vals = [sq_rets.autocorr(lag=k) for k in range(1, 31)]
    ax4.bar(range(1, 31), acf_vals, color='steelblue', alpha=0.7)
    ax4.axhline(0, color='black', lw=0.5)
    ax4.axhline(1.96 / np.sqrt(len(log_rets)), color='red', lw=1, linestyle='--',
                label='95% CI')
    ax4.axhline(-1.96 / np.sqrt(len(log_rets)), color='red', lw=1, linestyle='--')
    ax4.set_title('ACF of Squared Returns (Volatility Clustering)')
    ax4.set_xlabel('Lag (days)')
    ax4.legend()

    plt.tight_layout()
    plt.savefig('synthetic_diagnostics.png', dpi=150, bbox_inches='tight')
    plt.show()

10. Running Multiple Paths: Monte Carlo

A single synthetic path is an anecdote. A thousand paths is a distribution.

Monte Carlo simulation runs the same generator with different seeds and collects the resulting strategy performance metrics. The output is a distribution over possible outcomes—not a single number—which is far more useful for risk assessment.

def monte_carlo_paths(
    n_paths: int = 1000,
    T: int = 252,
    S0: float = 100.0,
    generator_kwargs: dict = None,
    seed_start: int = 0,
) -> pd.DataFrame:
    """
    Run n_paths independent synthetic simulations.
    Returns DataFrame with one column per path (prices, indexed by day).
    """
    if generator_kwargs is None:
        generator_kwargs = {}

    gen = SyntheticMarketGenerator(**generator_kwargs)
    all_prices = np.zeros((T, n_paths))

    for i in range(n_paths):
        df = gen.generate(T=T, S0=S0, seed=seed_start + i)
        all_prices[:, i] = df['price'].values

    return pd.DataFrame(all_prices, columns=[f'path_{i}' for i in range(n_paths)])


def summarize_monte_carlo(paths_df: pd.DataFrame, S0: float = 100.0) -> dict:
    """
    Compute summary statistics across all Monte Carlo paths.
    """
    final_prices = paths_df.iloc[-1]
    final_returns = final_prices / S0 - 1

    # Per-path max drawdown
    max_drawdowns = []
    for col in paths_df.columns:
        prices = paths_df[col]
        dd = ((prices / prices.cummax()) - 1).min()
        max_drawdowns.append(dd)
    max_drawdowns = np.array(max_drawdowns)

    results = {
        'mean_return':       final_returns.mean(),
        'median_return':     final_returns.median(),
        'std_return':        final_returns.std(),
        'pct5_return':       final_returns.quantile(0.05),
        'pct95_return':      final_returns.quantile(0.95),
        'prob_positive':     (final_returns > 0).mean(),
        'mean_max_dd':       max_drawdowns.mean(),
        'median_max_dd':     np.median(max_drawdowns),
        'worst_max_dd':      max_drawdowns.min(),
        'pct5_max_dd':       np.percentile(max_drawdowns, 5),
    }

    print(f"\nMonte Carlo Summary ({len(paths_df.columns)} paths, {len(paths_df)} days)")
    print("─" * 50)
    print(f"{'Mean annual return':<28} {results['mean_return'] * 100:>8.1f}%")
    print(f"{'5th pct return':<28} {results['pct5_return'] * 100:>8.1f}%")
    print(f"{'95th pct return':<28} {results['pct95_return'] * 100:>8.1f}%")
    print(f"{'Prob(positive return)':<28} {results['prob_positive'] * 100:>8.1f}%")
    print(f"{'Mean max drawdown':<28} {results['mean_max_dd'] * 100:>8.1f}%")
    print(f"{'Worst max drawdown':<28} {results['worst_max_dd'] * 100:>8.1f}%")
    print(f"{'5th pct max drawdown':<28} {results['pct5_max_dd'] * 100:>8.1f}%")

    return results


# Example: run 500 paths and summarize
paths = monte_carlo_paths(n_paths=500, T=252, S0=450.0, seed_start=1000)
summary = summarize_monte_carlo(paths, S0=450.0)

11. What Synthetic Data Cannot Do

This is the part that deserves more space than it usually gets.

It doesn’t model economic causation. Real markets move on Fed announcements, earnings surprises, geopolitical shocks, credit events. A stochastic model has no knowledge of any of this. It can reproduce the statistical shape of volatility clustering without knowing why volatility clusters. This matters because the causal mechanisms that create market dynamics in the future may differ systematically from those that created them in the past.

Correlation structure breaks in crises. During normal markets, equity-bond correlations are reliably negative (~-0.4). During some crisis periods, both equities and bonds sell off simultaneously (as in 2022). A fixed correlation matrix misses this regime-dependence entirely. You can partially address this with regime-conditional correlations, but it adds significant complexity.

The tails are still underestimated. Even with Student’s t at ν = 4.51, the synthetic model underestimates the frequency of truly extreme events. The 1987 Black Monday crash was a 22% single-day decline—an event that would occur essentially never even under our fat-tailed model. The generator calibrates to the statistical distribution of historical returns, but the most extreme events are by definition underrepresented in any finite historical sample.

It is dangerous as training data. I covered this in the old post, but it bears repeating. Training a model on synthetic data risks teaching it to recognize patterns that exist in the simulator’s assumptions rather than in real market structure. The model can overfit to your stochastic model’s quirks. Use synthetic data to test robustness; be extremely cautious about using it to train signal models.

It says nothing about structural regime changes. The Markov transition matrix was estimated from 1993–2025 data. If we enter a genuinely new macroeconomic regime—say, structurally higher inflation, or a period of elevated geopolitical fragmentation that disconnects global markets—those transition probabilities are wrong. The model doesn’t know what it doesn’t know.

Despite all of this, the tool is genuinely useful. The failure modes it surfaces are real failure modes—they just tend to be different failure modes than the ones historical data surfaces. That difference is exactly the point.

12. A Complete Worked Example

Putting it all together in a single callable pipeline:

import numpy as np
import pandas as pd
from scipy import stats


def run_full_synthetic_analysis(
    scenarios: list[str] = None,
    n_monte_carlo: int = 500,
    T: int = 252,
    S0: float = 100.0,
    seed: int = 42,
) -> dict:
    """
    Run the full synthetic market analysis pipeline:
    1. Generate base calibration path
    2. Run Monte Carlo over base parameters
    3. Generate all requested scenarios
    4. Report summary statistics for each

    Returns dict of scenario_name -> summary stats.
    """
    if scenarios is None:
        scenarios = ['crash', 'choppy', 'stagflation', 'bull', 'slow_bleed',
                     'flash_crash', 'sideways', 'bear']

    results = {}

    print("=" * 70)
    print("Synthetic Market Analysis")
    print("=" * 70)

    # Base calibration
    gen = SyntheticMarketGenerator()
    base_df = gen.generate(T=T, S0=S0, seed=seed)
    base_return = base_df['price'].iloc[-1] / S0 - 1
    base_mdd = ((base_df['price'] / base_df['price'].cummax()) - 1).min()

    print(f"\nBase generator (1 path, {T} days):")
    print(f"  Return:       {base_return * 100:>7.1f}%")
    print(f"  Max Drawdown: {base_mdd * 100:>7.1f}%")

    # Monte Carlo on base
    print(f"\nRunning {n_monte_carlo} Monte Carlo paths...")
    paths = monte_carlo_paths(n_paths=n_monte_carlo, T=T, S0=S0, seed_start=1000)
    mc_summary = summarize_monte_carlo(paths, S0=S0)
    results['base_monte_carlo'] = mc_summary

    # Scenario runs
    print("\n" + "─" * 70)
    print(f"\n{'Scenario':<16} {'Return':>10} {'Sharpe':>10} {'Max DD':>10}")
    print("─" * 50)

    for scenario_name in scenarios:
        try:
            df = generate_scenario(scenario_name, T=T, S0=S0, seed=seed)
            log_rets = np.log(df['price'] / df['price'].shift(1)).dropna()
            ret = df['price'].iloc[-1] / S0 - 1
            mdd = ((df['price'] / df['price'].cummax()) - 1).min()
            ann_vol = log_rets.std() * np.sqrt(252)
            ann_ret = log_rets.mean() * 252
            sharpe = ann_ret / ann_vol if ann_vol > 0 else 0

            print(f"{scenario_name:<16} {ret * 100:>9.1f}% {sharpe:>10.2f} {mdd * 100:>9.1f}%")
            results[scenario_name] = {
                'return': ret, 'sharpe': sharpe, 'max_drawdown': mdd,
                'annual_vol': ann_vol,
            }
        except Exception as e:
            print(f"{scenario_name:<16} ERROR: {e}")

    print("\n" + "=" * 70)
    return results


# Run it
if __name__ == '__main__':
    all_results = run_full_synthetic_analysis(
        n_monte_carlo=500,
        T=252,
        S0=450.0,
        seed=42,
    )

13. Key Takeaways

The goal of a synthetic market generator is not to predict the future. It is to build a machine for asking what if—with enough statistical discipline that the answers carry real information.

A few things I’ve learned from building and using this:

Layer complexity incrementally. GBM first, then GARCH, then fat tails, then regime switching. Each layer should be validated against data before adding the next. If you add everything at once, you have no idea which component is producing which effect. This becomes super duper important, especially when you are trying to figure out whether the Sharpe of 3.0 is real with ROI of 8000%.

Match moments, not prices. The generator should reproduce annualized return, volatility, skewness, excess kurtosis, lag-1 autocorrelation, and volatility clustering. These are the moments that matter for strategy behavior. The specific prices are irrelevant.

Regime transitions are the most important calibration input. Getting the transition matrix right — especially the crisis regime’s diagonal (how sticky crashes are)—has more impact on strategy stress-test results than almost any other parameter choice.

Run enough paths. Single synthetic paths are anecdotes. The minimum useful Monte Carlo run is probably 250–500 paths for stable statistics on tail metrics. Max drawdown in particular has high variance in finite samples.

Validate against the source. Always run KS tests and visual diagnostics comparing synthetic returns to the historical returns used for calibration. If the synthetic distribution is statistically distinguishable from the real distribution (p < 0.05), the calibration is wrong.

And finally: synthetic markets surface failure modes that history cannot. I mean that is the entire point. The scenario that breaks your strategy is almost never the scenario you expected, and almost always the one you didn’t have data for.

14. Python Code

I’ve shared all the code on this GitHub repo here:

https://github.com/kniyer/stochastic-data-generation

Generates realistic synthetic financial price paths by combining five statistical models:

• GBM — geometric Brownian motion baseline

• GARCH(1,1) — volatility clustering (persistence after shocks)

• Student’s t — fat tails (4× more extreme events than Gaussian)

• Ornstein-Uhlenbeck — mean reversion in returns and VIX

• Markov regime switching — Bull / Bear / Crisis / Choppy regimes with calibrated transition probabilities

Calibrated to 30+ years of SPY data (1993–2025).

All code is open source — use it, modify it, build on it. No guarantees about correctness or performance. Test everything yourself before deploying with real capital.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise. And could be a liar even with fake synthetic data.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice. This analysis probably contains errors — if you find them, let me know.

This is part 86 of my series — Building & Scaling Algorithmic Trading Strategies

Final part of the Microstructure Edge series. Part 1: Order flow toxicity. Part 2: Auction mechanics. Part 3: Dealer gamma exposure.

From Three Posts of Theory to One System

Over the last three posts, I’ve covered order flow toxicity and spread dynamics (Part 1), intraday auction mechanics and execution timing (Part 2), and dealer gamma exposure (Part 3). Each post ended with a variation of “this matters for V6.”

Now it’s time to actually build the thing.

The goal is a microstructure signal layer that sits on top of V6’s existing allocator and does two things: (1) scale position size based on microstructure conditions, and (2) optimize execution timing. No changes to V6’s underlying logic. No new instruments. Just a filter that says “conditions are favorable, run at full size” or “conditions are deteriorating, scale down.”

Four layers. Data comes from Polygon, CBOE options, Squeezemetrics, and FRED. Signals are computed daily: spread z-score, OFI proxy, GEX regime, and execution timing. The decision layer converts signals into a position scalar (0.3× to 1.0×) and an execution window. The V6 allocator runs as before, just with the scalar applied.

The Three Signals

Signal 1: Spread Z-Score

From Part 1: wide spreads predict negative returns and elevated adverse selection. We track the effective spread relative to its own history.

def spread_z_score(spread_series, fast=21, slow=63):
    """
    Compute z-score of current spread vs. recent history.
    Positive z-score = unusually wide spread = bad.
    """
    ma = spread_series.rolling(fast).mean()
    std = spread_series.rolling(slow).std()
    return (spread_series - ma) / std.replace(0, 1)

If you don’t have tick-level spread data, approximate the effective spread from daily OHLCV using the Corwin-Schultz (2012) estimator:

def corwin_schultz_spread(high, low):
    """
    Estimate effective spread from daily high/low prices.
    """
    beta = (np.log(high / low)) ** 2
    beta_sum = beta.rolling(2).sum()
    gamma = (np.log(high.rolling(2).max() / low.rolling(2).min())) ** 2
    alpha = (np.sqrt(2 * beta_sum) - np.sqrt(beta_sum)) / \
            (3 - 2 * np.sqrt(2)) - np.sqrt(gamma / (3 - 2 * np.sqrt(2)))
    spread = 2 * (np.exp(alpha) - 1) / (1 + np.exp(alpha))
    return spread.clip(lower=0)

This gives you a daily spread estimate from freely available data. The z-score tells you whether today’s spread is unusually wide (positive z) or narrow (negative z) relative to the last few months.

Signal inversion: we invert the z-score for the composite because high spread = bad microstructure.

Signal 2: OFI Proxy

From Part 1: order flow imbalance predicts short-term returns. Without tick data, we approximate net buying/selling pressure using daily price action:

def ofi_proxy(close, open_price, high, low):
    """
    Proxy for order flow imbalance from daily OHLCV.
    Positive = buying pressure, negative = selling pressure.
    """
    # Where did the close land relative to the day's range?
    range_position = (close - open_price) / (high - low).replace(0, 1)
    return range_position.rolling(5).mean()

The logic: if the stock closes near its high, buyers dominated. If it closes near its low, sellers dominated. The 5-day smoothing reduces noise. This is a rough proxy — nothing like actual trade-level OFI — but it captures the directional pressure signal at the daily frequency where V6 operates.

Signal 3: GEX Regime

From Part 3: positive dealer gamma exposure suppresses volatility, negative GEX amplifies it. We use a binary regime indicator:

def gex_regime(gex_series):
    """
    Binary GEX regime from Squeezemetrics or SpotGamma data.
    Returns +0.5 for positive GEX, -0.5 for negative.
    """
    return np.where(gex_series > 0, 0.5, -0.5)

If you’re using Squeezemetrics’ free (delayed) GEX, you get yesterday’s reading. For a daily allocator, one day of lag is acceptable — GEX regimes persist for days to weeks, not hours.

The Composite Score

The three signals combine into a single composite microstructure score:

def composite_score(spread_z, ofi, gex, 
                     w_spread=0.35, w_ofi=0.30, w_gex=0.35):
    """
    Composite microstructure score in [-1, 1].
    Positive = favorable conditions.
    Negative = unfavorable conditions.
    """
    spread_signal = np.clip(-spread_z / 3, -1, 1)  # inverted
    ofi_signal = np.clip(ofi / 3, -1, 1)
    gex_signal = gex  # already in [-0.5, 0.5]
    
    composite = (w_spread * spread_signal + 
                 w_ofi * ofi_signal + 
                 w_gex * gex_signal)
    return pd.Series(composite).rolling(5).mean()

The weights (35/30/35) reflect my judgment about relative signal strength. GEX and spread carry the most weight because they have the clearest theoretical mechanisms. OFI is noisier at the daily frequency.

Five panels showing the pipeline. Price (top), then the three input signals (spread z-score, OFI proxy, GEX regime), then the composite score (bottom). The composite turns negative during stress periods (red shading) and positive during calm markets. Decision thresholds at ±0.2 define the transition zone.

The Position Scaling Function

The composite score maps to a position scalar through a sigmoid function:

def position_scalar(score, s_min=0.3, s_max=1.0):
    """
    Maps composite score to position size.
    score >> 0 → 1.0× (full allocation)
    score << 0 → 0.3× (minimum allocation)
    """
    k = 10  # steepness
    sigmoid = 1 / (1 + np.exp(-k * score))
    return s_min + (s_max - s_min) * sigmoid

The sigmoid mapping. At a composite score of +0.4 (good microstructure), position is near 100%. At -0.5 (bad microstructure), position drops to 30%. The transition zone between -0.2 and +0.2 is where most of the scaling action happens.

Why 0.3× minimum instead of 0× (full exit)?

Three reasons. First, you never want to be fully out — you’d miss the snap-back rally that often follows the worst microstructure conditions. Second, the composite score is noisy; going to zero on a false signal is expensive. Third, V6’s TLT hedge component should already be working during stress periods, so some exposure is warranted.

Walkforward Backtest

I ran this over 5 years of simulated data with the same walkforward structure from Part 82 (Strategy Decay series — quarterly refit of composite weights):

Top: equity curves. V6 base (gray) reaches 113% cumulative but suffers -45% max drawdown. V6 with the microstructure layer (green) reaches 72% but with only -26% max drawdown. Bottom: drawdown comparison. The microstructure layer consistently limits drawdowns, especially during the five stress periods (red shading).

The headline numbers:

                V6 Base     V6 + Microstructure
Ann. Return:    22.4%       14.4%
Ann. Vol:       23.0%       15.5%
Sharpe:         0.97        0.93
Max Drawdown:   -45%        -26%

The Sharpe ratios are similar because the microstructure layer reduces both returns and volatility proportionally. But the max drawdown improvement — from -45% to -26% — is the real value. A -45% drawdown requires a +82% recovery. A -26% drawdown requires a +35% recovery. That’s the difference between 2 years underwater and 8 months.

Monthly Returns Comparison

Side-by-side monthly return heatmaps. The microstructure version (right) has fewer deeply red months — the worst months are compressed from -5% or -6% to -3% or -4%. The trade-off: some of the best months are also compressed, because the filter was partially engaged during the recovery.

What Each Component Contributes

Not all signals are equally valuable. Here’s the breakdown:

GEX regime contributes the most to both Sharpe improvement (+0.18) and drawdown reduction (-9pp). Spread conditioning is second (+0.12 Sharpe, -4pp DD). Execution timing is a steady +0.08 Sharpe from reduced transaction costs. OFI proxy is the weakest standalone signal (+0.06) but adds diversification to the composite. The full composite is slightly better than the sum of parts due to signal diversification.

If you could only implement one thing from this series: implement GEX conditioning. It’s the highest-impact single signal, it requires only free (delayed) data from Squeezemetrics, and the implementation is a three-line if-else statement.

The Full Implementation

Here’s the complete signal layer in one block:

import pandas as pd
import numpy as np

class MicrostructureLayer:
    def __init__(self, w_spread=0.35, w_ofi=0.30, w_gex=0.35,
                 min_position=0.3, max_position=1.0):
        self.w_spread = w_spread
        self.w_ofi = w_ofi
        self.w_gex = w_gex
        self.min_pos = min_position
        self.max_pos = max_position
    
    def compute_signals(self, df):
        """
        df must have columns: 
          open, high, low, close, spread (or estimated), gex
        """
        # Signal 1: Spread z-score
        spread_ma = df['spread'].rolling(21).mean()
        spread_std = df['spread'].rolling(63).std()
        spread_z = (df['spread'] - spread_ma) / spread_std
        
        # Signal 2: OFI proxy
        ofi = ((df['close'] - df['open']) / 
               (df['high'] - df['low']).replace(0, 1))
        ofi = ofi.rolling(5).mean()
        
        # Signal 3: GEX regime
        gex = np.where(df['gex'] > 0, 0.5, -0.5)
        
        # Composite
        composite = (
            self.w_spread * np.clip(-spread_z / 3, -1, 1) +
            self.w_ofi * np.clip(ofi / 3, -1, 1) +
            self.w_gex * gex
        )
        composite = pd.Series(composite, 
                              index=df.index).rolling(5).mean()
        
        return composite
    
    def get_position_scalar(self, composite_score):
        """Sigmoid mapping from score to position size."""
        sigmoid = 1 / (1 + np.exp(-10 * composite_score))
        return self.min_pos + (self.max_pos - self.min_pos) * sigmoid
    
    def get_execution_window(self, is_month_end=False, 
                              signal_urgent=False):
        """Returns recommended execution time."""
        if signal_urgent:
            return "ASAP (next liquid window)"
        if is_month_end:
            return "14:00-15:00 ET (pre-MOC)"
        return "10:30-11:00 ET (default)"

That’s the entire layer. Fifty-odd lines of Python. It can run on a single day’s data — no ML, no GPU, no streaming infrastructure.

The Implementation Checklist

Everything in this table is achievable with a free Polygon.io plan (for spread data), Squeezemetrics (for GEX), and Yahoo Finance or any OHLCV source (for the OFI proxy). Total data cost: $0 to $30/month.

TL;DR

This series covered a lot of ground — VPIN, Kyle’s lambda, spread decomposition, intraday patterns, MOC imbalances, the overnight premium, dealer gamma exposure, GEX, and now a composite signal layer. The theory is robust and well-supported by academic research.

But I need to flag the elephant in the room: all the backtests in this series are simulated, not historical.

I used realistic parameters calibrated to published research, but I did not run these signals on real-time market data with actual V6 returns. I’ve started with the simplest, most robust signals (execution timing, GEX binary) and currently building toward the full composite only after out-of-sample validation.

This is the building-in-public deal: I show you the framework, the math, and the plan. Then I’ll report back in future posts with actual results. If the microstructure layer works, you’ll see it in V6’s equity curve. If it doesn’t, you’ll read about that too.

This concludes the 4-part Microstructure Edge series.

Next up: The Carry Trade Across Everything — a 3-part series on how carry (FX, bonds, vol, equity) creates the oldest and most persistent risk premium in markets.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 85 of my series — Building & Scaling Algorithmic Trading Strategies

Part 3 of the Microstructure Edge series. Part 1: Order flow toxicity. Part 2: Auction mechanics of open and close.

The Equity & Options Market Myth

Here’s the uncomfortable reality about modern equity markets: options market makers — not fundamental investors, not momentum traders, not retail — are often the largest marginal buyers and sellers of stock on any given day.

They don’t want to be but they’re forced to be — by math.

When you buy a call option, the market maker on the other side sells it to you. That creates directional exposure the dealer doesn’t want. To neutralize it, they buy shares of the underlying stock. When the stock moves, their hedge is wrong, so they must adjust — buying more shares if the stock rises (for a short call), selling if it falls.

This forced, mechanical hedging flow is large enough to move prices. And it creates a feedback loop that either suppresses or amplifies volatility depending on the sign of the dealers’ aggregate gamma exposure.

The Feedback Loop

Left: when aggregate dealer gamma is positive, hedging dampens price moves. The stock ticks up → dealers sell to rehedge → selling pressure pushes price back. The result: lower realized volatility, mean reversion, and prices pinned near high-gamma strikes. Right: when gamma is negative, hedging amplifies moves. The stock drops → dealers sell to rehedge → selling pressure pushes price further down. The result: higher realized volatility, momentum, and gap-prone markets.

This isn’t theoretical hand-waving — in fact, the mechanism is mechanical:

Dealer sells you a call option at strike K
  → Dealer is short Γ contracts of gamma
  → When stock rises $1, dealer's delta changes by Γ
  → Dealer must buy Γ shares to stay hedged
  → This buying pushes the stock up slightly more
  → Which changes delta again... (feedback loop)

When dealers are long gamma (positive GEX), the loop is stabilizing — they buy dips and sell rips. When they’re short gamma (negative GEX), the loop is destabilizing — they sell into declines and buy into rallies, amplifying both.

Greeks Refresher: Why Gamma Matters

If you need a refresher on option Greeks, here’s the 60-second version:

Left: delta measures how much the option price moves per $1 of underlying movement. It ranges from 0 (far OTM) to 1 (deep ITM), with ATM options at roughly 0.5. Right: gamma is the rate of change of delta — it measures how much hedging adjustment is needed per $1 move. Gamma peaks at the strike price and falls off on both sides.

The key insight: gamma is highest for at-the-money options near expiration. A 0DTE SPX option with the strike right at the current price has enormous gamma — its delta changes massively with every tick. The dealer hedging against it generates correspondingly massive stock flow.

This is why the 0DTE boom matters for market structure. In 2018, 0DTE options were a rounding error. By 2024 they represented over 50% of SPX options volume. That’s a massive increase in near-expiry, high-gamma options — and therefore a massive increase in the mechanical hedging flow that suppresses or amplifies stock movements.

GEX: Gamma Exposure Across Strikes

GEX aggregates gamma across all open option positions to estimate the net gamma exposure of dealers at each strike price:

GEX(K) = Γ(K) × OI_calls(K) × 100 × S × 0.01
        - Γ(K) × OI_puts(K)  × 100 × S × 0.01

Where Γ(K) is gamma at strike K, OI is open interest, S is the spot price, and the sign convention assumes dealers are short options (long for market participants, short for dealers).

The sign tells you the hedging direction:

GEX(K) > 0: Dealers are long gamma at this strike
  → They'll buy dips and sell rips near K
  → Price tends to pin toward K

GEX(K) < 0: Dealers are short gamma at this strike  
  → They'll sell into declines near K
  → Price accelerates through K

A simulated GEX profile for SPX at 5500. Green bars above zero: positive GEX strikes where dealer hedging creates resistance (dampens upward moves) and support (dampens downward moves). Red bars below zero: negative GEX strikes where hedging accelerates price through these levels. The max positive GEX strike at 5550 acts as a magnet; the max negative GEX strike at 5400 acts as an accelerant.

Reading the GEX Profile

Three things to look for:

1. The gamma flip level. The price where aggregate GEX crosses from positive to negative. Above this level, dealers dampen moves. Below it, they amplify. Think of it as the market’s “stability boundary.”

2. High positive GEX strikes. These act as magnets — price tends to pin toward them, especially near expiration. Options traders call this “max pain” or “pinning.”

3. High negative GEX strikes. These act as accelerants — if price reaches them, dealer hedging pushes it further in the same direction, creating cascading moves.

GEX Predicts Volatility

The empirical relationship is strong: higher GEX correlates with lower realized volatility.

Left: scatter plot of standardized GEX vs. realized volatility with quintile means and a regression line. The relationship is clearly negative. Right: bar chart showing ~6 percentage points of difference in realized vol between the most negative and most positive GEX quintiles.

Luo and Zhang (2024) formalize this in their study of 0DTE options: they find that dealer delta-hedging of short-dated options contributes to intraday mean reversion and compressed realized vol. When 0DTE volume is high and GEX is positive, the market becomes “pinned” — large moves are mechanically bought or sold back toward the high-gamma strike.

The practical takeaway: GEX is a volatility regime indicator. High positive GEX = low vol, mean-reverting market. Negative GEX = high vol, trending/gapping market. This is directly useful for a strategy like V6 that needs to choose between momentum and mean-reversion positioning.

The Gamma Flip

The most important single number in the GEX framework is the gamma flip level — the price where aggregate dealer gamma goes from positive to negative.

Top: dealer GEX over time. Green shading = positive gamma (dealers dampen moves). Red shading = negative gamma (dealers amplify moves). Orange vertical lines mark gamma flips. Bottom: the price path. In green zones, the price action is calm with low volatility. In red zones, volatility explodes — drawdowns are deeper, moves are more erratic. The simulated vol ratio is 2.3× between negative and positive GEX regimes.

When the market breaks below the gamma flip level, a cascade can begin:

Price drops below gamma flip
  → Aggregate dealer gamma turns negative
  → Dealers start selling into further declines
  → Selling pressure pushes price lower
  → More put gamma enters the money
  → Dealer gamma becomes more negative
  → More forced selling
  → Volatility explodes

This is one mechanism behind the “orderly decline → sudden crash” pattern. The market can decline slowly while still in the positive GEX zone, with dealer hedging providing a cushion. Then it crosses the gamma flip level and the decline accelerates mechanically.

Think of the gamma flip as a phase transition, like water turning to ice. Conditions change gradually until they hit a critical threshold, then the behavior of the system changes discontinuously.

Where to Get GEX Data

For research and backtesting, the most rigorous approach is to compute GEX yourself from options chain data. You need open interest by strike (CBOE or your broker’s API), a Black-Scholes calculator for gamma, and an assumption about which side of the trade the dealer is on (typically: dealers are short options that customers buy).

For daily monitoring, Squeezemetrics publishes a free (delayed) aggregate GEX index alongside their Dark Index (DIX). SpotGamma is the best-known retail provider, publishing daily GEX levels, the gamma flip, and key strike levels.

What This Means for V6

V6 operates at the daily frequency. It doesn’t trade options. But GEX conditioning can improve its risk-adjusted returns by telling V6 what kind of market it’s trading in.

Top: GEX regime over 3 years. Bottom: V6 equity curve with and without GEX conditioning. The conditioned version scales to 30% position size when GEX is negative. It sacrifices raw returns (74% vs. 138%) but improves the Sharpe from 1.89 to 2.14 and cuts max drawdown from -27% to -9%.

The trade-off is stark: GEX conditioning costs you nearly half the total return but cuts the worst drawdown by two-thirds. For a strategy designed to compound over years, the max drawdown reduction is more valuable than the raw return — because a -27% drawdown requires a +37% recovery, while a -9% drawdown only requires +10%.

The V6 GEX Integration Rules

GEX Positive (aggregate GEX above zero):
  → Full V6 allocation
  → Expect mean-reverting market
  → Momentum signals may lag (price pinned)
  → Use tighter rebalance triggers

GEX Negative (aggregate GEX below zero):
  → Reduce to 30% allocation
  → Expect trending/gapping market
  → Momentum signals may be too slow
  → Widen stop-loss, reduce frequency

Gamma Flip (crossing from positive to negative):
  → Immediate risk reduction
  → This is not a drawdown to ride
  → Review TLT hedge allocation

The Honest Limitations

GEX is powerful but it’s not magic. Some caveats:

1. The dealer assumption is uncertain. We assume dealers are short the options customers buy. But sometimes dealers initiate positions, and sometimes customers are sellers. The sign can be wrong.

2. GEX explains vol, not direction. Positive GEX tells you vol will be low, but it doesn’t tell you if the market goes up or down (slowly). Negative GEX tells you moves will be large, but not which way.

3. The feedback is reflexive. As more traders watch GEX, they front-run the gamma flip, which changes the dynamics. The signal may decay (recall Part 80 on strategy decay half-lives).

4. 0DTE complicates everything. Most GEX models use end-of-day open interest. But 0DTE options open and close within the day, creating intraday GEX fluctuations that the end-of-day snapshot misses.

Despite these limitations, GEX as a volatility regime indicator is robust — the mechanical relationship between dealer hedging and vol is structural, not statistical. Dealers must hedge. That flow must affect prices. The direction and magnitude are uncertain, but the existence of the effect is not.

Up Next

Part 4: Building the Microstructure Signal Layer for V6 — Putting it all together. We’ll take VPIN (Part 1), spread dynamics (Part 1), execution timing (Part 2), and GEX conditioning (Part 3) and build a concrete signal layer on top of V6. Python code, data sources, and a walkforward backtest.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 84 of my series — Building & Scaling Algorithmic Trading Strategies

Part 2 of the Microstructure Edge series. Part 1: Order flow toxicity and the signals beneath the price.

Two Different Markets Wearing the Same Ticker

Here’s something most retail traders never realize: the market at 9:30 AM and the market at 3:50 PM are fundamentally different systems. They have different participants, different liquidity dynamics, different statistical properties, and different alpha opportunities.

The open is an auction where overnight information gets incorporated into price — violently, imprecisely, and with systematic overshoot. The close is a different auction where index funds, ETFs, and institutional rebalancers dump mechanical flow into the market. The midday session is a low-information random walk.

If you trade a daily strategy and your execution ignores this, you’re leaving money on the table.

Volatility is 3-4x higher at the open and 2-3x higher at the close compared to midday. This isn’t noise — it’s structural. The open processes overnight information. The close processes end-of-day rebalance flow. Midday has neither, so it’s quiet.

This U-shape was first documented by Wood, McInish, and Ord (1985) and has been robust across decades and markets. It’s one of the most replicated findings in finance. And it has direct implications for when V6 should execute its trades.

The Opening Auction: Systematic Overshoot

The opening auction is a mechanism for incorporating overnight information. Between 4:00 PM yesterday and 9:30 AM today, the world happened — earnings, macro data, war, geopolitics, analyst upgrades, overseas market moves. All of this needs to get into the price.

The problem: the opening auction is a one-shot process. All the accumulated overnight orders cross at a single price. This price tends to overshoot the equilibrium because:

1. Order clustering. Retail market-on-open orders pile up in the same direction. If Apple reported good earnings, there’s a wall of buy orders at the open. The opening print reflects this temporary demand spike.

2. Stale limit orders. Some limit orders were placed yesterday and are now mispriced. They get picked off at the open, contributing to the overshoot before being corrected.

3. Bid-ask bounce. The opening spread is wide (2-3x normal), so the first few prints oscillate wildly between bid and ask before the book stabilizes.

The result: large overnight gaps tend to partially reverse in the first 15-30 minutes.

Left: mean first-30-minute return conditioned on overnight gap size. Large down gaps are followed by +0.28% rallies. Large up gaps are followed by -0.24% declines. The pattern is monotonic and statistically significant. Right: average minute-by-minute price path after different gap sizes. Large gaps (red and green) reverse, while small gaps (gray) continue.

Can You Trade This?

In theory, yes — fade the gap. In practice, it’s harder:

Execution cost. The spread at 9:30 is 2-3x wider than normal. A 5 bps edge net of a 3-4 bps spread leaves maybe 1-2 bps. Enough for a market maker, marginal for retail.

Tail risk. Sometimes the gap isn’t an overshoot — it’s the start of a genuine trend. The 2020 COVID crash opened with massive gaps that didn’t reverse. Fading those would have been catastrophic.

Speed. The reversal happens in 10-20 minutes. You need to identify the gap, evaluate whether it’s reversible, and execute — all before the opportunity closes.

For V6, the practical takeaway isn’t “trade the gap” — it’s “don’t execute at the open.” We’ll quantify this in Chart 7.

The Close: Where the Money Machine Lives

If the open is chaotic, the close is mechanical.

Every day at the close, a set of large, predictable flows converge on the market:

Index funds and ETFs need to track their benchmarks at the closing price. If $500M flowed into SPY today, the fund must buy the underlying stocks at or near the close.

Mutual funds calculate NAV at the closing price. Subscriptions and redemptions execute at the close.

Futures basis traders roll positions and hedge against the closing price.

Corporate buybacks often execute using VWAP algorithms weighted toward the close.

These flows are not information — they’re plumbing. But they’re large enough to move prices, creating the MOC (market-on-close) imbalance effect.

MOC Imbalances

NYSE publishes the indicative MOC imbalance at 3:45 PM ET — 15 minutes before the close. This tells you the net buy/sell imbalance of orders that will execute at the closing auction.

Top: MOC net imbalance ($M) over a year. The pattern is noisy daily but has structure: month-end and quarter-end (purple lines) show systematic buy imbalances from passive fund inflows. Bottom: MOC imbalance is correlated with the last-15-minute return. Larger buy imbalances predict positive closing moves.

The Month-End Effect

Pension funds and target-date funds rebalance monthly. If stocks outperformed bonds this month, they sell stocks and buy bonds to return to target weights — and vice versa. This creates a predictable MOC flow pattern:

Month where stocks outperformed bonds:
  → Pension rebalance sells stocks at close
  → MOC imbalance is sell-heavy last 1-2 days of month
  → Closing price pressured down

Month where stocks underperformed bonds:
  → Pension rebalance buys stocks at close
  → MOC imbalance is buy-heavy last 1-2 days of month
  → Closing price pushed up

This is well-documented — Etula, Rinne, Suominen, and Vaittinen (2020) estimate the pension rebalance effect at roughly $5-10B per month-end. It’s large enough to move SPY by 5-15 bps on rebalancing days.

The Daily Rhythm: Volume and Spread

The volume and spread profiles follow predictable intraday patterns that directly affect execution quality:

Top: volume is U-shaped — high at open, lunch dip, high at close. Bottom: spreads are also U-shaped but mirror-image to volume — widest when volume is thinnest (open) and during lunch. Two green zones mark the optimal execution windows: 10:30-11:30 and 14:00-15:00, where volume is adequate and spreads are narrow.

The key insight here is that volume and spreads are negatively correlated intraday, but in a specific pattern. At the open, both are elevated — high volume with wide spreads (poor execution despite liquidity). At midday, both are low — narrow spreads but thin order books. The sweet spot is mid-morning and mid-afternoon, where volume is reasonable and spreads have tightened from the opening chaos.

Where the Returns Actually Come From

The most surprising finding in microstructure research is the overnight premium: most of the equity risk premium accrues between the close and the next open, not during trading hours.

Left: cumulative returns decomposed into overnight (close-to-open) and intraday (open-to-close) components over 20 simulated years. The overnight component drives the vast majority of total returns. Right: annualized statistics. Overnight returns have a higher Sharpe ratio than intraday returns despite lower volatility.

This pattern was documented by Cliff, Cooper, and Gulen (2008) and has been robust across markets and decades. The likely mechanism: overnight, investors demand compensation for holding risk when they can’t trade. This “holding premium” is collected by anyone who is long at the close and sells at the open.

Implications for V6

V6 is a daily allocator that holds positions overnight. This means V6 is already collecting the overnight premium — which is good news. But it also means that V6’s execution timing matters:

If you buy at the open and sell at the close, you’re paying the overnight premium (buying when others sell overnight holdings, selling before the premium accrues).

If you buy at the close and hold overnight, you’re collecting the premium.

For a strategy that rebalances daily, this argues for executing rebalance trades near the close, not at the open. Every day you execute at 9:30 instead of 15:00, you’re giving away a few basis points.

The Patterns Playbook

Here’s a summary of exploitable intraday patterns:

The “Tradeable?” column is the honest assessment. Most of these patterns exist but are marginal after costs. The two that matter most for a daily allocator like V6 are the overnight premium (which V6 already captures by holding overnight) and execution timing (which V6 can improve).

V6 Execution Timing: When You Trade Matters

Putting it all together: if V6 needs to rebalance, when should it execute?

I modeled the total execution cost across different times of day, accounting for spread, market impact, and timing alpha:

Total execution cost by time of day. The open (9:30) costs ~8.5 bps — wide spreads, high impact, and adverse timing. The optimal window is around 11:00 AM at ~0.8 bps, where spreads have tightened, the opening reversal is complete, and the close-induced flow hasn’t started. The close (15:45 MOC) costs ~6 bps due to the MOC-driven spread widening.

The difference between worst and best timing is 7.7 bps per trade. For a strategy that rebalances weekly, that’s roughly 7.7 × 52 = 400 bps per year of unnecessary cost. That’s the difference between a 0.8 Sharpe and a 1.0 Sharpe — just from execution timing.

The V6 Execution Rule

Based on this analysis:

Default: Execute rebalances at 10:30-11:00 AM ET
  - Spreads have tightened from the open
  - Opening reversal is complete
  - Volume is adequate

Exception 1: If signal is time-sensitive (CUSUM alarm),
  execute at next available window, don't wait

Exception 2: Month/quarter end
  - If rebalancing into stocks: execute before 3:00 PM
    to avoid pension sell-imbalance at close
  - If rebalancing out of stocks: execute at/after 3:00 PM
    to ride the pension flow

Up Next

Part 3: Dealer Gamma Exposure — How options market makers become involuntary directional traders and create mechanical support/resistance levels. GEX, the delta-hedging feedback loop, and why the options market is quietly driving the stock market.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 83 of my series — Building & Scaling Algorithmic Trading Strategies

This begins a 4-part series on market microstructure — the physics of how prices actually form. Part 1: the signals // Part 2: open/close auction mechanics // Part 3: dealer gamma exposure // Part 4: building a microstructure signal layer for V6.

The Signal You’re Not Looking At

Almost every stock chart we read is effectively a culmination — in some ways, it’s a lie by omission.

Not because the price is wrong — it’s accurate — but rather, the price is the last thing to move. Before the price drops, the order flow shifts. Before the order flow shifts, the spread widens. Before the spread widens, the limit order book thins out on one side.

If price is the smoke, microstructure is the fire.

Top: the price chart. At tick 200-ish, it drops. If you’re watching the chart, you see this after it happens. Bottom: the order flow imbalance. Sell pressure starts building around tick 180 — roughly 20 ticks before the price move is visible. The smoothed OFI (yellow) turns sharply negative while the price chart still looks normal.

This is not theoretical. Market makers have been trading on this information for decades. The academic literature — Kyle (1985), Glosten and Milgrom (1985), Easley and O’Hara (1992) — formalized what floor traders already knew: you can hear the market shift before the price moves.

The question for us: can a retail quant use any of this?

Kyle’s Lambda: The Price of Information

In 1985, Albert Kyle published a model that became the foundation of market microstructure theory. The core insight is elegantly simple:

Price impact is proportional to order flow.

ΔP = λ · (Buys - Sells) + noise

Where λ (lambda) is the “price impact coefficient” — the market’s sensitivity to net order flow. Lambda captures something deep: it’s the market maker’s estimate of how much information is embedded in the flow.

When λ is high, the market is saying: “I think some of you know something I don’t, so I’m going to move the price more aggressively in response to orders.”

When λ is low, the market is saying: “This looks like random noise, I’ll absorb it without moving much.”

Left: the linear price impact model. In liquid markets (SPY), you can trade 1000 shares with ~1 bps of impact. In illiquid or stressed markets, the same order moves the price 15x more. Right: lambda is not constant — it spikes during stress events. This is why execution costs blow up precisely when you need to trade most urgently.

Why Lambda Matters for You

Even if you never estimate λ directly, it explains two phenomena you’ve experienced:

1. Slippage is worst when you need to trade. During the exact periods where your strategy signals a regime change and you want to rebalance, λ has spiked. The cost of executing the trade partially offsets the benefit of the signal.

2. Backtest transaction cost assumptions are wrong. Most backtests use a fixed transaction cost (say 5 bps). But λ varies by 10-20x between calm and stressed markets. Your worst trades — the ones in crises — have the highest costs, and your backtest understates them.

Estimating Lambda

The simplest estimator, following Hasbrouck (2009):

λ̂ = Cov(ΔPₜ, sign(Qₜ)) / Var(sign(Qₜ))

Where ΔP is the price change and sign(Q) is the direction of the trade (+1 for buyer-initiated, -1 for seller-initiated). You can compute this with daily data from WRDS or even from publicly available trade-and-quote data.

For SPY, λ typically runs around 0.5-2 bps per standard deviation of net flow in calm markets, rising to 5-15 bps during stress. I’ll show the exact computation in Part 4 when we build the signal layer.

VPIN: The Flash Crash’s Early Warning System

VPIN — Volume-Synchronized Probability of Informed Trading — was introduced by Easley, López de Prado, and O’Hara in 2012. It gained attention because it rose dramatically in the hours before the May 6, 2010 Flash Crash.

The Intuition

VPIN measures the imbalance between buy-initiated and sell-initiated volume, normalized by total volume, computed over volume buckets rather than time buckets. The volume-synchronization is the key innovation: instead of sampling every minute or every day, you sample every time a fixed amount of volume has traded.

Why? Because informed traders cluster their activity in high-volume periods. If you sample by time, you might miss 80% of the information in a burst of trading. If you sample by volume, each observation carries roughly the same amount of information.

The Math

Divide the trading day into N volume buckets, each containing V_bucket shares. For each bucket n:

V_buy(n) = volume classified as buyer-initiated
V_sell(n) = volume classified as seller-initiated

VPIN = (1/N) · Σₙ |V_buy(n) - V_sell(n)| / V_bucket

VPIN ∈ [0, 1]. When VPIN = 0, buying and selling are perfectly balanced (uninformed flow dominates). When VPIN approaches 1, flow is completely one-sided (likely informed).

VPIN in Action

Three panels showing a simulated flash crash at day 150. The price drops suddenly (top). Volume spikes (middle). VPIN crosses the alert threshold at 0.4 as the crash hits (bottom). In the original Easley et al. study, VPIN on real E-mini S&P 500 futures rose to elevated levels hours before the 2010 Flash Crash — offering a warning that price alone did not.

The Controversy

VPIN is not without critics. Andersen and Bondarenko (2014) argued that VPIN’s predictive power comes largely from its correlation with volatility, and that simpler vol measures perform comparably. The debate continues, but the core insight is robust: volume-weighted flow imbalance contains information that time-weighted measures miss.

For our purposes, the practical question isn’t “is VPIN the best metric?” — it’s “does flow imbalance predict anything useful at the daily/weekly horizon where V6 operates?” I’ll test this in Part 4.

The Bid-Ask Spread: Three Stories in One Number

The spread is the most visible microstructure signal — you see it every time you place a trade. But most people think of it as a transaction cost. It’s actually a compressed information signal.

The spread decomposes into three components, each telling a different story:

Green (bottom): order processing cost — the fixed cost of running a market-making operation. Relatively constant. Blue (middle): inventory risk — the market maker’s cost of holding a position. Rises with volatility. Red (top): adverse selection — the market maker’s cost of trading against informed participants. This is where the information lives.

The Three Components

1. Order processing cost (~0.3-0.5 bps for SPY). The infrastructure cost. Declines over time with technology.

2. Inventory risk (~1-4 bps, vol-dependent). The market maker holds a position and faces price risk. This scales with volatility — when VIX doubles, the inventory component roughly doubles.

3. Adverse selection (~0.3-1.5 bps, information-dependent). This is the market maker’s compensation for getting picked off by informed traders. When a market maker sells to a buyer who has material non-public information, the market maker loses. The adverse selection component of the spread reflects the market maker’s estimate of how likely this is.

Why You Should Care

The adverse selection component is the market’s real-time estimate of information asymmetry. When it spikes, someone knows something.

Formal decomposition requires TAQ (trade and quote) data and methods like Huang and Stoll (1997) or the more recent LOBSTER dataset. But even the total spread — which you can get from any broker — carries signal.

The Spread Predicts Returns

Wide spreads predict negative future returns. This is one of the most robust findings in microstructure research, documented by Amihud and Mendelson (1986), Pastor and Stambaugh (2003), and Acharya and Pedersen (2005).

The mechanism: when spreads widen, it means market makers are uncertain. They’re either (a) seeing informed flow, (b) facing high inventory risk, or (c) both. In either case, the expected near-term price path is negative — either because informed sellers are right, or because the liquidity withdrawal itself pushes prices down.

Left: scatter plot of spread vs. next-day return with quintile means (yellow dots). The relationship is negative and statistically significant. Right: bar chart of mean next-day returns by spread quintile. The widest spread quintile has the most negative expected return.

The effect size is small — a few basis points per day — which means it’s not directly tradeable for most retail accounts after costs. But it’s useful as a conditioning variable: scale your existing strategy’s position size based on the current spread regime.

The Signal Hierarchy

Not all microstructure signals are created equal, and not all are accessible to retail traders:

The accessibility column is the key constraint. Bid-ask spreads and basic OFI (order flow imbalance) are available from any decent data vendor. VPIN requires intraday trade data. Kyle’s lambda and depth imbalance require institutional-grade data feeds.

For the V6 signal layer in Part 4, I’ll focus on what we can actually compute: spread-based signals, daily OFI approximations, and VPIN using publicly available tick data.

Why This Matters for V6

V6 is a daily allocator. It doesn’t trade at the tick level. So why does microstructure matter?

Because microstructure conditions change the profitability of the strategy.

Top: the spread regime over 2 years, with wide-spread periods shaded red and narrow-spread periods shaded green. Bottom: V6 equity curve. The Sharpe ratio is 1.33 during narrow-spread periods versus 0.30 during wide-spread periods. Same strategy, same parameters — just different microstructure conditions.

This 4x Sharpe difference across spread regimes is the core insight. If V6 knew when microstructure conditions were unfavorable and scaled down, it would improve risk-adjusted returns without changing the underlying strategy.

This is what we’ll build in Part 4.

Up Next

Part 2: The Auction Mechanics of Open and Close — Why the first 15 minutes and last 15 minutes of the trading day follow different statistical rules than the rest. MOC (market-on-close) imbalances, the opening auction, and exploitable patterns around daily rotations.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 82 of my series — Building & Scaling Algorithmic Trading Strategies

The Obvious Solution That Isn’t

In Part 1, we discussed how almost all strategies decay — momentum signals with a half-life of 18-24 months, VIX signals with 12-18 months, short-term reversal with just 6-12 months. In Part 2, we talked through building a detection toolkit: CUSUM, Bayesian changepoint detection, rolling Sharpe with null bands.

So the natural next question for us is if the optimal parameters drift over time, why not just... update them?

Refit the model. Roll the window. Use online learning. Let the strategy adapt. Sounds pretty cool right?

I mean, it sounds so obvious but realistically it’s also a trap. The same mechanism that allows a strategy to adapt to changing markets also allows it to overfit to noise in real-time. And overfitting isn’t obvious at first, because we all like to think it’s just our great instincts, which is what makes it really hard.

Besides, adaptation and overfitting are the same mathematical operation — the difference is whether you’re tracking a real shift or chasing randomness.

This post is about finding the line between them.

The Adaptation Spectrum

There’s a continuum of approaches, from fully static to fully adaptive:

White line: the “true” optimal parameter drifting over time. Red dashed: static strategy that never updates (increasingly wrong). Orange steps: periodic annual refit (catches broad trends, misses details). Blue: 126-day rolling window. Green: exponentially weighted online learning. Each approach trades off responsiveness against stability.

Five approaches, ranked by adaptiveness:

Approach           Responsiveness    Overfit Risk    Complexity
──────────────────────────────────────────────────────────────
Static             None              None            Zero
Periodic refit     Low               Low             Low
Rolling window     Medium            Medium          Low
Online learning    High              High            Medium
Full ML retraining Very high         Very high       High

The key insight from the chart: no approach tracks the true optimal perfectly. The rolling window (blue) and online learning (green) are closest, but they also oscillate more — tracking noise, not just signal.

After the structural shift at day 800, the static strategy is permanently wrong. The periodic refit catches it a year late. The rolling window and online learner adjust within 2-3 months. But that speed comes with a cost visible in the rest of the chart: they also “adapt” to random fluctuations that mean nothing.

The Bias-Variance Trade-off (Again)

If you’ve taken an ML course, you’ve seen this. But it applies differently in finance because the data is non-stationary.

In classical statistics, the bias-variance trade-off is about model complexity: too simple = high bias (underfitting), too complex = high variance (overfitting).

In strategy adaptation, the trade-off is about window length:

Left: shorter windows are more responsive but noisier. The 21-day estimate (red) oscillates wildly. The 252-day estimate (green) is smooth but systematically wrong — it can’t track the declining trend. Right: the mean squared error has a minimum around 60-80 days. Shorter than that, you’re overfitting to noise. Longer, you’re missing real drift.

The math is clean. For a parameter μ that drifts at rate δ per day, estimated using a rolling window of length w:

Bias²    = (δ · w / 2)²          ← grows with window length
Variance = σ² / w                 ← shrinks with window length
MSE      = Bias² + Variance
         = (δw/2)² + σ²/w

Taking the derivative and setting to zero:

dMSE/dw = δ²w/2 - σ²/w² = 0
w* = (2σ² / δ²)^(1/3)

For a strategy with daily return noise σ = 1% and parameter drift δ = 0.01% per day:

w* = (2 × 0.01² / 0.0001²)^(1/3)
   = (2,000,000)^(1/3)
   ≈ 126 days

Approximately 6 months. Which is suspiciously close to the “quarterly refit” heuristic that many quant shops use. Sometimes the rules of thumb are right.

The Goldilocks Problem: How Often Should You Refit?

Theory says ~126 days. But that assumes you know the drift rate δ, which you don’t. And it assumes the drift is constant, which it isn’t.

I simulated this across 200 Monte Carlo paths with drifting parameters:

Violin plots show the distribution of out-of-sample Sharpe ratios across simulations. The mean (yellow line) peaks around 42 days. The dashed curves decompose the error: overfitting cost (red) decreases with longer windows while staleness cost (purple) increases. The crossing point defines the optimum.

The takeaways:

Too frequent (5-10 days): The mean Sharpe is lower and the variance is higher. You’re getting whipsawed. The parameter estimate is dominated by the last week of noise.

Too infrequent (252-504 days): Lower mean Sharpe because you’re systematically stale. You miss regime changes by 6-12 months.

The sweet spot (21-63 days): Highest mean Sharpe with acceptable variance. Monthly to quarterly refitting balances responsiveness against noise.

One detail that matters: the variance is widest on the left side of the chart. High-frequency refitting doesn’t just give you worse average performance — it gives you more unpredictable performance. You might get lucky with a 5-day refit window, but you’re more likely to get unlucky.

Walkforward Optimization: The Right Way to Refit

If you’re going to refit, the methodology matters enormously. The wrong way: re-optimize on the full historical dataset every time. That’s just expanding-window overfitting.

The right way: walkforward optimization. Train on a fixed-length window, test on the next period, slide forward, repeat.

Each segment uses only prior data for optimization and tests on future data. The test periods, stitched together, form a continuous out-of-sample equity curve.

The rules are as follows…

1. Training window ≥ 2× test window. If you test on 63 days, train on at least 126 — less than that, and your parameter estimates are too noisy.

2. No peeking. The test period parameters are locked before you see test period data. Any modification during the test period invalidates the out-of-sample claim.

3. Stitch the test periods. Your “real” equity curve is the concatenation of all test periods. This is the only performance number that matters.

4. Track parameter stability across segments. If the optimal momentum threshold is 0.1% in segment 1, 0.3% in segment 2, 0.05% in segment 3, and 0.4% in segment 4, you don’t have a stable signal — you have noise.

For V6, the walkforward structure I use:

Training:  126 trading days (~6 months)
Testing:    63 trading days (~3 months)
Slide:      63 days (non-overlapping test periods)
Parameters: momentum threshold, VIX boundaries

Forgetting Factors: How Fast Should the Strategy Learn?

An alternative to discrete refitting is continuous learning via exponential weighting. Instead of a hard window, old data fades gradually:

μ̂ₜ = (1 - λ) · μ̂ₜ₋₁ + λ · xₜ

Where λ is the “learning rate” or “forgetting factor.” Higher λ = faster adaptation.

The effective window length is approximately 1/λ. So λ = 0.02 corresponds to roughly a 50-day window, and λ = 0.005 corresponds to roughly 200 days.

Top: how different learning rates track the true alpha. The slow learner (green, λ=0.005) barely notices the regime change. The fast learner (red, λ=0.15) oscillates wildly in both regimes. The moderate learner (blue, λ=0.02) finds the balance. Bottom: cumulative P&L when using each estimate to make allocation decisions.

The regime change happens around day 300-400: alpha drops from ~15% to ~7.5% annualized. Watch how each learning rate handles it:

Slow (λ = 0.005): Still thinks alpha is 15% at day 500. Overallocated for 200+ days.

Fast (λ = 0.15): Detects the shift quickly, but oscillates so much that its allocation decisions are essentially random. The P&L shows the damage: constant whipsaw.

Moderate (λ = 0.02): Adjusts within ~60 days. Not perfect, but the P&L shows the benefit: it captures most of the upside in the first regime and limits damage in the transition.

Choosing λ

The optimal λ depends on the same parameters as the optimal window length:

λ* ≈ √(δ² / (2σ²))

For σ = 1% daily noise and δ = 0.01% daily drift:

λ* ≈ √(0.0001² / (2 × 0.01²)) ≈ 0.007

This corresponds to an effective window of ~140 days. Consistent with the walkforward analysis.

In practice, I’d recommend starting with λ = 0.01 (effective 100-day window) and adjusting based on your CUSUM monitoring from Part 2. If CUSUM is alarming frequently, your λ is too low (you’re adapting too slowly). If your P&L shows whipsaw, your λ is too high.

The Adaptation Paradox

Here’s the punchline of this series: moderate adaptation beats both extremes.

I simulated three approaches over 10 years of synthetic data with regime changes:

Green (moderate adaptation) wins on both Sharpe ratio and max drawdown. Static (gray) takes the full force of every bear market. Hyper-adaptive (red) gets whipsawed during choppy periods, missing recoveries while sitting in cash.

The numbers tell the story clearly:

The static approach takes every drawdown on the chin — 74% max drawdown over 10 years. It ends slightly positive but the ride is brutal.

The hyper-adaptive approach tries to dodge drawdowns by switching to cash whenever the 21-day Sharpe turns negative. The problem: it also switches to cash during normal pullbacks within bull markets, then misses the recovery. Every chop zone becomes a series of buys near the top and sells near the bottom.

The moderate approach scales down gradually when evidence accumulates over 63 days. It doesn’t dodge every drawdown — it still takes hits. But it avoids the worst of them while maintaining enough exposure to capture recoveries.

What Not to Adapt

Not everything should be adaptive. Some components of a strategy are better left fixed:

Adapt: Position sizing, momentum threshold, allocation weights between assets.

Don’t adapt: The fundamental structure of the strategy (e.g., “momentum works”), the risk management rules (e.g., max drawdown tolerance), the instruments traded.

Why? Adapting the structure is equivalent to strategy selection, which requires an order of magnitude more data. You can estimate an optimal momentum threshold with 126 days of data. You cannot determine whether momentum still works as a concept with 126 days of data.

For V6 specifically:

The key design choice: VIX regime boundaries are fixed and never adapted. These aren’t fitted parameters — they’re based on structural reasoning about what VIX levels correspond to different market states. Adapting them would mean the strategy is learning “when to be scared,” which is exactly the kind of thing that overfits.

Meanwhile, the momentum threshold is adapted quarterly via walkforward optimization, because the speed of trends genuinely changes over time. And position sizing uses daily volatility targeting, which is the most adaptive component — but with hard guardrails (max 100% TQQQ, no leverage).

The Hierarchy of Adaptation

If I had to distill this entire series into a single framework, it’s this:

Level 1: Monitor (always)
  → CUSUM, rolling Sharpe, Bayesian CPD
  → Cost: zero (just watching)
  → Benefit: early warning

Level 2: Scale positions (daily)
  → Volatility targeting, risk budgeting
  → Cost: some tracking error
  → Benefit: smooth equity curve

Level 3: Refit parameters (quarterly)
  → Walkforward optimization on tactical parameters
  → Cost: risk of overfitting to recent regime
  → Benefit: adapts to drift

Level 4: Review structure (annually)
  → Does the fundamental thesis still hold?
  → Cost: high (may require new strategy entirely)
  → Benefit: avoids being stuck in a dead paradigm

Level 5: Replace strategy (when evidence is overwhelming)
  → Multiple tests alarming, structural thesis broken
  → Cost: very high (starting over)
  → Benefit: survival

Most days, you’re at Level 1 and 2. Quarterly, you do Level 3. Annually, you do Level 4. Level 5 happens maybe once every 3-5 years.

The mistake most people make is oscillating between Level 0 (ignore everything) and Level 5 (blow it up and start over) with nothing in between. The whole point of this series is to fill in Levels 1-4.

What’s Next for V6

Concretely, here’s what I’m implementing based on this series:

Daily: CUSUM monitoring on excess returns (from Part 2). Volatility-targeted position sizing with 15% annualized vol target. Auto-email if CUSUM crosses 70% of alarm threshold.

Monthly: Update the EWM alpha estimate (λ = 0.01). Compare to the 63-day rolling Sharpe. If both suggest alpha < 5% annualized, scale to 50% allocation.

Quarterly: Walkforward refit of the momentum threshold. Training on prior 126 days, testing on next 63 days. Guardrails: threshold must stay between 0.05% and 0.20%.

Annually: Full structural review. Does the TLT hedge still work? Is stock/bond correlation still negative in stress? Does momentum still persist?

The system isn’t complicated but the hard part isn’t the implementation — it’s the discipline to follow it, especially when the CUSUM is flashing yellow and every instinct says to override the system.

This concludes the 3-part series on strategy decay. If you want to go deeper on any of these topics — especially the implementation details of CUSUM monitoring or walkforward optimization — let me know in the comments.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 81 of my series — Building & Scaling Algorithmic Trading Strategies

The Hardest Question in Quant Trading

You deploy a strategy and it makes money for a few months. Then it draws down for three weeks, where you lose a lot more than you made. It’s not just that you lose money but also that you don’t see it coming.

Is it a drawdown — the normal cost of doing business — or is the strategy broken?

The solid blue line is identical in both panels. At day 220, you see the same equity curve, the same drawdown, the same P&L. But the future diverges completely. Scenario A recovers. Scenario B flatlines. You’re standing at the yellow dot. Which scenario are you in?

This is the fundamental problem in strategy monitoring. Drawdowns and structural decay are indistinguishable by eye during the early stages. By the time the difference is obvious, you’ve already lost months of returns.

In Part 1, I covered why strategies decay — crowding, regime change, and overfitting. This post is about detecting it before the damage is done.

The Toolkit: Four Tests, Different Trade-offs

There’s no single test that solves this problem perfectly. Every approach trades off detection speed against false alarm rate. Here’s the lineup:

I’ll walk through each one with worked examples, starting with the most practical.

Test 1: CUSUM (Cumulative Sum Control Chart)

CUSUM is the workhorse. It was invented by E.S. Page in 1954 for industrial quality control — detecting when a manufacturing process has shifted — and it translates directly to strategy monitoring.

The idea: accumulate evidence that the process has shifted away from its expected value. Small daily deviations accumulate into a large statistic if the shift is real, but random noise resets back toward zero.

The Math

Define the one-sided CUSUM statistic for detecting a negative shift in alpha:

S⁺ₜ = max(0, S⁺ₜ₋₁ + (μ₀ - xₜ) - k)

Where:

• μ₀ is the expected daily return under H₀ (”strategy is working”)

• xₜ is the observed daily return

• k is the “allowance” or slack parameter (typically 0.5σ)

• S⁺₀ = 0

When the strategy is working, daily returns are close to μ₀, so (μ₀ - xₜ) fluctuates around zero, and the max(0, ...) keeps the statistic near zero. When the strategy decays, returns fall below μ₀, the difference accumulates, and S⁺ drifts upward.

You signal an alarm when S⁺ₜ exceeds a threshold h (typically 4–5σ).

CUSUM in Action

Here’s a simulated strategy with a true changepoint at day 200 — alpha drops from ~15% annualized to approximately -5%:

Top: cumulative returns. The curve slopes upward before day 200, then flattens and declines. Bottom: the CUSUM statistic. It stays near zero while the strategy is healthy, then climbs after the changepoint and triggers an alarm at day 250 — 50 trading days after the true change.

Fifty days of detection delay. Is that good? For a strategy running with a Sharpe of ~1, that’s about 2 months of potentially degraded returns. Not instant, but far better than waiting 6+ months for a rolling Sharpe to clearly trend negative.

Tuning the Parameters

The two knobs are k (allowance) and h (threshold):

k controls sensitivity:
  k = 0       → detects any shift, many false alarms
  k = 0.5σ    → standard setting (recommended)
  k = 1.0σ    → only detects large shifts

h controls the trade-off:
  h = 3σ      → fast detection, ~1-2 false alarms per year
  h = 4σ      → balanced (~0.5 false alarms per year)  
  h = 5σ      → slow detection, very few false alarms

For strategy monitoring, I’d start with k = 0.5σ and h = 4σ. You can calibrate from there based on your tolerance for false alarms versus detection delay.

The Key Limitation

CUSUM requires you to specify μ₀ — the “expected” return under normal conditions. This is harder than it sounds. If you use the full backtest mean, you’re including the regime you’re trying to detect. If you use a short calibration window, you’re sensitive to the choice of window.

My approach: use the mean return from a clean, out-of-sample validation period. For V6, that’s the walkforward validation period we discussed in Part 32.

Test 2: Page-Hinkley Test

Page-Hinkley is CUSUM’s cousin. Well, second cousin maybe. Instead of resetting to zero, it tracks the cumulative deviation from the running mean:

mₜ = (1/t) · Σᵢ xᵢ                    (running mean)
Uₜ = Σᵢ (xᵢ - mₜ - δ)                 (cumulative deviation)
PHₜ = Uₜ - min(U₁, U₂, ..., Uₜ)       (test statistic)

Where δ is a tolerance parameter (similar to k in CUSUM).

Alarm when PHₜ > λ (the threshold).

The practical difference from CUSUM: Page-Hinkley adapts its reference point over time (via the running mean), which makes it better at detecting gradual drift versus sudden shifts. CUSUM is better for clean breaks; Page-Hinkley handles slow erosion.

For strategy monitoring, I run both in parallel — CUSUM for sudden breakdowns (regime changes) and Page-Hinkley for slow decay (crowding, signal erosion).

Test 3: Rolling Sharpe with Confidence Bands

This is the least sophisticated but most intuitive approach. Calculate the Sharpe ratio over a rolling window and watch for it to trend toward zero.

Adding Statistical Rigor

The raw rolling Sharpe is noisy. A 63-day window produces Sharpe estimates with a standard error of approximately:

SE(SR) ≈ √((1 + 0.5·SR²) / n)

For a true Sharpe of 1.0 and n = 63 days:

SE ≈ √((1 + 0.5) / 63) ≈ 0.154

Annualized, that’s SE × √252 ≈ 2.45. Your 95% confidence interval around a “true” annualized Sharpe of 1.0 is roughly (-3.9, 5.9).

This is absurdly wide. Which is the core problem with rolling Sharpe: by the time you have enough data to be statistically confident the Sharpe has changed, you’ve already lost a lot of money.

Blue: raw 63-day rolling Sharpe. Yellow: smoothed trend. Gray band: 95% confidence interval under the null hypothesis (Sharpe = 0). The rolling Sharpe can wander well outside the null band while alpha is positive, then gradually drift back toward zero as the strategy decays.

When Rolling Sharpe Is Useful

Despite the wide confidence intervals, rolling Sharpe is good for two things:

Trend detection. Apply a moving average to the rolling Sharpe. If the smoothed trend is declining over 3-6 months, that’s a signal — even if the point estimate hasn’t crossed zero yet.

Dashboard visualization. It’s the easiest metric for a quick glance. Is the Sharpe trending up, flat, or down? You don’t need to explain CUSUM to a non-quant collaborator, but everyone understands “the risk-adjusted returns are declining.”

Test 4: Bayesian Online Changepoint Detection

This is the most powerful and most complex approach. Instead of testing “has a change occurred?” (binary), it estimates “what is the probability that a change occurred at each point in time?” (continuous).

The Framework

The Bayesian approach maintains a probability distribution over the “run length” — how long since the last changepoint. At each new observation, it updates two things:

The growth probability: the data is consistent with the current regime continuing.

The changepoint probability: the data suggests a new regime has started.

P(rₜ = 0 | x₁:ₜ) ∝ Σᵣ P(xₜ | rₜ₋₁) · P(rₜ₋₁ | x₁:ₜ₋₁) · H(rₜ₋₁)

Where H(r) is the hazard function — the prior probability of a changepoint at any given time.

In practice, you’re running a recursive Bayesian filter that maintains a distribution over “how long has the current regime been running?” and updates it with each new return observation.

What It Looks Like

Top: daily returns with a rolling mean. Middle: Bayesian posterior estimate of annualized alpha with 95% credible interval. The interval shrinks as evidence accumulates, then widens around changepoints. Bottom: estimated probability that a changepoint has occurred. Spikes above the 95% threshold indicate detected regime changes.

Why Bayesian Is Better (and Worse)

Better: it gives you probabilities, not binary alarms. “There’s a 73% chance alpha shifted negative last week” is more useful than “ALARM” or “NO ALARM.”

Better: it naturally handles multiple changepoints. CUSUM needs to be reset after an alarm. The Bayesian filter adapts continuously.

Worse: it’s computationally heavier and requires specifying prior distributions. The choice of hazard function (how often do you expect changepoints?) and the likelihood model (are returns normally distributed?) affect results significantly.

Worse: it can be harder to calibrate. With CUSUM, you have two knobs (k, h). With Bayesian CPD, you have a prior over run lengths, a likelihood model, and hyperparameters for both.

For V6 monitoring, I’d implement Bayesian CPD for weekly analysis and retrospective study, while running CUSUM for real-time alerts.

The Fundamental Trade-off

Every test lives on the same curve: faster detection means more false alarms. Slower detection means larger losses before you act.

Each test traces a curve in detection-delay / false-alarm space. The “sweet spot” for most strategies is 0.5–2 false alarms per year with 20–40 days of detection delay.

The practical implication: don’t rely on a single test. Stack them.

Alert Level    Trigger                              Action
──────────────────────────────────────────────────────────────
GREEN          All tests normal                     Continue trading
YELLOW         Rolling Sharpe trend declining OR     Reduce position size 50%
               CUSUM > 70% of threshold             
ORANGE         CUSUM alarm fired OR                  Halt new trades,
               Bayesian P(change) > 80%              review fundamentals
RED            Multiple tests alarming,              Full stop. Paper trade
               confirmed by Bayesian analysis        until edge reconfirmed

Building the Dashboard

Here’s what a monitoring dashboard looks like in practice, applied to simulated V6 returns:

Four panels showing the key metrics at a glance. The equity curve provides context. Rolling Sharpe shows the trend. CUSUM gives the formal alarm status. The alpha posterior shows the current best estimate with uncertainty.

This dashboard takes maybe 50 lines of Python to build with matplotlib and runs in under a second. No fancy infrastructure needed. I update it daily as part of the morning trading workflow.

The key design principle: every panel answers a different question.

The equity curve answers: “am I making money?” The rolling Sharpe answers: “is my risk-adjusted performance trending?” The CUSUM answers: “has something structurally changed?” The alpha posterior answers: “what’s my best estimate of current edge, with uncertainty?”

The Real-World Complications

Everything above assumes stationary noise and clean changepoints. Reality is messier:

Non-normal returns. Strategy returns have fat tails, which inflate both the CUSUM statistic and the Sharpe estimate. Use robust estimators (median, MAD) or fit a Student-t likelihood in the Bayesian framework.

Multiple strategies. If you’re monitoring five strategies simultaneously, you’ll get a false alarm from at least one of them every few months. Apply a Bonferroni or FDR correction: if monitoring N strategies, use a threshold of h + σ·√(2·ln(N)).

Correlated returns. If your strategy is correlated with the market, a broad drawdown can trigger false alarms. Monitor alpha (returns minus benchmark) rather than raw returns.

Survivorship in your own analysis. You’ll naturally pay more attention to tests that alarm at convenient times. Be honest with yourself about how you’d act on each alert level before the alarm fires.

Implications for V6

I’m considering the following as a concrete plan for V6:

Daily. Run CUSUM with k = 0.5σ, h = 4σ on V6’s excess returns (over SPY). Auto-email if CUSUM crosses 70% of threshold.

Weekly. Update rolling 42-day and 63-day Sharpe ratios. Compare to the null bands. Run Page-Hinkley in parallel.

Monthly. Full Bayesian changepoint analysis on the trailing 6 months. Estimate posterior probability of alpha shift. Review whether the fundamental assumptions (negative stock/bond correlation, momentum persistence) still hold.

Quarterly. Structural break test (Chow test) against the most recent quarter to compare with the validation period from Part 32. Comprehensive review.

Up Next

Part 3: Adaptive Strategies — If decay is inevitable, can a strategy evolve to stay ahead of it? We’ll look at parameter drift, online learning, and the tension between adaptation and overfitting. Spoiler: this is harder than it sounds.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 80 of my series — Building & Scaling Algorithmic Trading Strategies

The Problem I Keep Running Into

Over the past 79 posts, I’ve built, tested, and killed more strategies than I care to count. A VIX term structure arbitrage that looked beautiful until I fixed the lookahead bias. An ML model that “learned” to exploit a data bug and produced imaginary 6,000% returns. A Recession Anxiety Index that turned out to be buy-and-hold with extra steps.

Every one of these followed the same arc: discover a signal, get excited about the backtest, dig deeper, discover it doesn’t work — or works far less well than it appeared.

I used to think this was my problem. Bad code, sloppy methodology, insufficient rigor. And some of it was (hello, lookahead bias). But the pattern is too consistent and too universal to be purely personal. There’s a deeper dynamic at work.

Strategies decay — all of them. The question isn’t whether your edge will erode — it’s how fast and why. And unless you have a framework for thinking about this, you’re flying blind.

This is Part 1 of a 3-part series on the mathematics of strategy decay:

• Part 1 (this post): Half-lives of alpha — the empirical evidence and mathematical framework

• Part 2: Monitoring a live strategy — statistical tests for detecting structural breaks

• Part 3: Adaptive strategies — can a strategy evolve, or is that just overfitting in real-time?

My Own Strategy Graveyard

Before we get to the theory, let me show you the data I’ve accumulated involuntarily.

Here’s what happened across six versions of my dual allocator strategy:

V1 through V4: the backtest always looked better than reality. V5: the ML model found the bug, not the signal. V6: the first version where backtest and reality converge.

The pattern across V1–V5 is textbook:

Version    What Happened                              Backtest   Reality    Gap
─────────────────────────────────────────────────────────────────────────────────
V1         Initial momentum strategy                     35%       12%     -66%
V2         "Optimized" parameters                        55%       18%     -67%
V3         Added volatility enhancement                  82%       15%     -82%
V4         Multi-factor, more complexity                 45%       10%     -78%
V5         ML model (found the bug, not alpha)        6,000%      -28%    -100%
V6         Rebuilt from scratch, no lookahead            25%       25%       0%

Notice something? The gap between backtest and reality got worse as the strategies got more complex. V1’s 66% haircut became V3’s 82% haircut became V5’s complete inversion.

This isn’t coincidence. Complexity creates more surface area for overfitting. More parameters means more opportunities for the model to memorize noise instead of learning signal. And ML models are particularly good at this — V5 didn’t find alpha, it found the data leak.

V6 is the only version where the numbers converge. Not because V6 is brilliant (24.8% CAGR is good but not extraordinary), but because it’s simple enough that there’s nowhere for false alpha to hide.

The Decay Model: Exponential with a Floor

The simplest useful model for strategy decay is exponential:

α(t) = α₀ · e^(-λt)

Where:

• α₀ is the initial alpha at time of discovery

• λ is the decay rate (higher = faster decay)

• t is time since discovery (in months)

The half-life — the time for alpha to drop by 50% — follows directly:

t½ = ln(2) / λ = 0.693 / λ

This is the same mathematics that governs radioactive decay. Which turns out to be a surprisingly apt metaphor: like unstable isotopes, unstable alpha sources shed their excess energy (returns) at a rate determined by their fundamental structure.

The critical insight is the transaction cost floor. Alpha doesn’t just decay toward zero — it becomes unprofitable once it drops below the cost of trading. For a typical retail strategy with 1-2% round-trip costs (spread + slippage + commissions), any signal with less than ~1.5% annualized alpha is actually a drag on returns.

This means the useful half-life is shorter than the mathematical half-life. A signal with α₀ = 5% and λ = 0.035 (t½ ≈ 20 months) hits the cost floor at:

1.5 = 5.0 · e^(-0.035·t)
ln(1.5/5.0) = -0.035·t
t = -ln(0.3) / 0.035
t ≈ 34.4 months

So the signal is mathematically “alive” for 20 months at half-strength, but only tradeable for about 34 months. After that, you’re paying more in costs than you’re earning in alpha.

Empirical Evidence: What the Research Shows

This isn’t just theory. There’s a growing body of academic evidence on post-publication decay.

The landmark study is McLean & Pontiff (2016), which examined 97 signals from academic finance journals. They found that post-publication returns declined by roughly 32% on average. But the averages mask huge variation — some signals lost nearly all their edge, while others appeared more resilient.

Here’s my synthesis of the evidence across signal types, pulling from McLean & Pontiff, Jacobs & Müller (2020), and AQR’s research:

A few things jump out:

Short-term signals decay fastest. Short-term reversal lost 63% of its pre-publication Sharpe, with an estimated half-life of 6-12 months. This makes sense: short-term signals are easier to implement, so capital floods in quickly.

Structural risk premia are more resilient. Low volatility only lost 27% of its Sharpe. Why? Because the low-vol premium is partly driven by institutional constraints (benchmarking, tracking error budgets) that don’t go away just because you published a paper about them.

VIX term structure signals decayed fast. This one hit home for me. The VIX contango/backwardation trade was printing money in 2012-2017, but post-publication and post-”Volmageddon” (XIV blow-up in February 2018), the easy money evaporated. Estimated half-life: 12-18 months.

Why Strategies Decay: Three Mechanisms

The exponential decay model tells you that alpha erodes but not why. There are three distinct mechanisms, and understanding which one is killing your strategy determines what (if anything) you can do about it.

1. Crowding

The most intuitive mechanism. You find an edge. You trade it. Others find it too. Capital flows in. The signal gets arbitraged away.

The math of crowding follows a logistic diffusion model combined with a capacity constraint:

Participants(t) = P_max / (1 + e^(-k(t - t₀)))
α(t) = α₀ · (1 - Participants(t)/P_max)^γ

Where γ > 1 captures the nonlinear nature of crowding — the first few participants barely dent the alpha, but beyond some tipping point, returns collapse rapidly.

The speed of diffusion (k) depends on how the signal is discovered:

Channel                      Diffusion Speed (k)    Time to 50% Awareness
──────────────────────────────────────────────────────────────────────────
Academic journal                   0.08                  ~30 months
Quant blog / Substack              0.15                  ~18 months
FinTwit / Reddit                   0.35                  ~8 months
Retail brokerage feature            0.80                  ~3 months

Left: how fast different channels spread awareness. Right: how the corresponding alpha decays. Faster diffusion → faster death.

This has an uncomfortable implication for what I’m doing with this Substack. Every time I write about a strategy, I’m contributing to its diffusion. The Recession Anxiety Index post probably didn’t move markets, but the principle applies: publicity accelerates decay.

The saving grace for retail quants is capacity. Large-cap momentum can absorb billions before degrading significantly. A micro-cap mean reversion strategy might only support $10M. If your strategy operates in a capacity-constrained niche, crowding is less of a threat — but so is your ability to scale.

Different strategies have different capacity ceilings. VIX term structure arb hits zero alpha at ~$5B deployed. Large-cap momentum can absorb 20x that before breaking.

2. Regime Change

The market’s statistical properties shift. Correlations that were negative become positive. Volatility regimes flip. Interest rate environments change.

This is different from crowding because it’s not caused by other participants — it’s the world changing underneath your model.

I lived through this with the TLT hedge in V6. The strategy assumes negative stock/bond correlation during stress events. This relationship held beautifully from 2000 to 2021. Then 2022 happened: stocks and bonds fell together, and the hedge became a second source of losses.

Regime change produces a discontinuous decay rather than exponential. Alpha doesn’t gently decline — it drops off a cliff when the underlying relationship breaks.

The mathematical signature looks like:

α(t) = α_base + Δα · 𝟙[regime = favorable]

Where 𝟙 is an indicator function that equals 1 when you’re in the right regime and 0 otherwise. The strategy doesn’t decay smoothly — it alternates between working and not working.

The danger is that regime changes look like drawdowns in real-time. You can’t distinguish “the strategy is temporarily losing money” from “the world has changed and the strategy is broken” without statistical tests, which is what Part 2 will cover.

3. Overfitting (The Signal Was Never Real)

The most humbling mechanism. The alpha wasn’t real in the first place. Your backtest found a pattern that existed in the historical data but was never a true market relationship.

This is what happened with my V5 ML model. It didn’t decay — it was never alive. The 6,000% return was an artifact of a data leak. When I fixed the leak, the “alpha” became a -28% loss.

Overfitting doesn’t follow an exponential decay curve. It follows a revelation curve: alpha appears stable in-sample, then drops abruptly when tested out of sample or deployed live.

The mathematical model:

In-sample:   α̂ = α_true + α_noise
Out-of-sample: α̂ = α_true

If α_true ≈ 0, then all the in-sample alpha was noise. Your backtest was measuring the noise component, which vanishes out of sample.

The number of parameters in your model is the best predictor of how much noise you’re capturing:

Expected noise alpha ≈ σ · √(k/n)

Where k is the number of parameters, n is the number of observations, and σ is the noise level. V5 had hundreds of ML features. V6 has three parameters (momentum threshold, VIX thresholds, allocation weights). The expected noise alpha in V5 was orders of magnitude higher.

Putting It Together: The Half-Life Spectrum

Different signal types have inherently different half-lives, driven by their susceptibility to each of the three mechanisms:

HFT signals decay in weeks. Structural risk premia last years. Everything else is somewhere in between.

The key takeaway for my own work:

V6 is a momentum + volatility regime strategy. Cross-sectional momentum has an estimated half-life of 18-24 months. The VIX regime component is harder to classify — it’s partly a structural risk premium (volatility sells off during calm periods) and partly a timing signal (which decays faster).

My best estimate for V6’s alpha half-life: 18-36 months, assuming the stock/bond correlation relationship holds and the momentum signal continues to work.

That’s not forever. It means I need to be actively monitoring for decay and preparing to adapt — which is exactly what Parts 2 and 3 of this series will cover.

What This Means for Retail Quants

A few practical implications:

1. Simplicity extends half-life. Complex strategies overfit more and decay faster. V6 works partly because it’s simple. Three inputs, one decision tree, one hedge. There’s less to break.

2. Structural edges decay slower than statistical edges. The TLT flight-to-quality trade works because of institutional behavior (pension funds, central banks) not because of a statistical anomaly. That’s harder to arbitrage away. Look for strategies where the reason the edge exists is durable.

3. Publication accelerates decay. Every time someone writes about a strategy on Substack or FinTwit, the clock speeds up. This creates an ironic tension for people like me who build in public. The act of sharing knowledge erodes the edge being shared. On the other hand, you keep testing and building new strategies, so that’s an advantage. But you’ll notice I don’t share everything (e.g., my Dual Allocator code) — and that’s with good reason.

4. You need a monitoring framework. If alpha has a half-life, you need instruments to measure it in real-time. “Trust the backtest” is not a monitoring framework. Part 2 will look at building one.

Up Next

Part 2: Detecting Decay in Real-Time — CUSUM tests, Page’s test, Bayesian changepoint detection, and building a dashboard that tells you when your strategy is dying.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

These posts are about methodology, not recommendations. If you find errors in my math, let me know — I’ve built an entire series around discovering my own mistakes, so one more won’t hurt.

The material presented in Math & Markets is for informational purposes only. It does not constitute investment or financial advice.

This is part 79 of my series — Building & Scaling Algorithmic Trading Strategies

I ran two experiments on the Dual Allocator over the past few weeks. The first tested whether an HMM-based ensemble could improve V6.2’s regime detection — but alas! It failed badly. The second tested whether targeted enhancements to V6.2’s allocation logic — hysteresis, dynamic leverage, and volatility targeting — could improve risk-adjusted returns. The initial results were overfitted, but after walk-forward validation, V6.4 delivered a real 39% return improvement over V6.2.

V6.2 Baseline

For reference, V6.2 uses two VIX thresholds (15 and 25) for regime detection, with TQQQ allocations up to 130% in calm markets and TLT in high-volatility regimes. Out-of-sample performance (2021–2026): 158% return, 0.697 Sharpe, −41.1% max drawdown.

Experiment 1: HMM Ensemble (V6.3) — Failed

In the regime detection series (Parts 69–71), I covered ensemble methods that combine HMMs with changepoint detection. The literature suggests blending fast signals with smooth ones should reduce false switches and improve risk-adjusted returns. I built V6.3 to test this.

V6.3 replaces V6.2’s threshold logic with a three-signal ensemble:

Ensemble = 0.40 × HMM + 0.35 × VIX_Threshold + 0.25 × CUSUM

The HMM is a 4-state Gaussian model trained on VIX, returns, realized volatility, and momentum. The CUSUM detector tracks deviations in realized volatility to signal sudden regime shifts. The ensemble score maps to three regimes: below 0.3 is aggressive, 0.3–0.6 is cautious, above 0.6 is defensive.

V6.3 captured 9% of V6.2’s out-of-sample gains. It traded 50% more often (42 vs 28 trades/year) and had a deeper max drawdown (−47.8% vs −41.1%). The only metric where V6.3 performed better was raw volatility (30.2% vs 37.0%), which resulted from sitting in cash and TLT while missing rallies.

Full period comparison. Panel 5 shows the ensemble signal components — the ensemble mean sits at 0.476, rarely dropping below the 0.3 aggressive threshold.

Out-of-sample comparison. Rolling 3-month Sharpe (Panel 4) shows V6.2 recovering quickly while V6.3 spends extended periods negative.

Why the Ensemble Failed

Six root causes, in order of impact:

Ensemble averaging dampened signal. The three signals averaged to a permanently moderate reading (ensemble mean 0.476). The HMM ran consistently elevated (0.551 ± 0.419), the VIX threshold was the most accurate (0.388 ± 0.332), and CUSUM was near-random noise (0.480 ± 0.496). The ensemble rarely dropped below 0.3 to trigger aggressive allocations, which is where V6.2 makes its returns.

HMM detection lag. The Baum-Welch algorithm needs evidence to accumulate before updating state estimates. Regime detection lagged by 5–10 trading days. For 3x leveraged TQQQ, that translates to 5–10% of portfolio value. During the March 2023 banking crisis, V6.2 went defensive immediately when VIX crossed 25. The HMM was still in bull-regime estimation five days after SVB collapsed.

CUSUM added noise. With a coefficient of variation over 100% and ~0.3 correlation with actual regime changes, CUSUM injected noise into 25% of the ensemble signal.

Wrong ensemble weights. 40% weight on the slowest signal (HMM) and only 35% on the most accurate (VIX) was backwards. A 60/20/20 VIX-dominant weighting would have been better, but at that point the ensemble is mostly just the VIX threshold with overhead.

Regime thresholds miscalibrated. The 0.3/0.6 thresholds resulted in only ~30% of time in the aggressive regime. Markets are calm approximately 60% of the time. Adjusting post-hoc introduces curve-fitting risk.

Training data mismatch. The HMM trained on 2010–2020 (low volatility, steady uptrend) classified the elevated-VIX 2021–2026 environment as perpetually moderate-to-high risk. V6.2’s thresholds are regime-agnostic — no training, no drift.

Conclusion: Abandoned HMM ensemble. V6.2’s threshold approach is already near-optimal for regime detection in this strategy.

Experiment 2: Conservative Enhancements (V6.4) — Overfitted, Then Fixed

Since regime detection couldn’t be improved through complexity, I focused on three targeted enhancements to V6.2’s allocation logic.

The Three Enhancements

Hysteresis (threshold buffers). VIX oscillating around 15 or 25 causes whipsaw trades. Instead of a flat threshold, V6.4 uses a buffer: VIX must exceed 15 + buffer to switch from low-vol regime, or drop below 15 − buffer to switch back. This reduces unnecessary trades around the threshold boundaries.

if current_regime == 'low_vol':
    switch_threshold = 15 + hysteresis_buffer  # Must exceed to switch
else:
    switch_threshold = 15 - hysteresis_buffer  # Must drop below to switch back

Dynamic leverage. Fixed leverage doesn’t adapt to realized conditions. V6.4 scales leverage based on 20-day realized volatility: boost allocation (1.2–1.3x) when realized vol is low, reduce allocation (0.8x) when realized vol is high.

Volatility targeting. Portfolio volatility varies from 20% to 60% across regimes. V6.4 targets a constant annual volatility by scaling allocation:

allocation = base_allocation × (target_vol / current_vol).

Initial Results: Too Good to Be True

I ran V6.4 with these enhancements optimized over the full 2010–2026 period.

Full period return: 16,994%. Sharpe: 1.071. Max drawdown: −50.85%.

That result was immediately suspicious. A 170x return over 16 years, with a deeper max drawdown than V6.2, pointed to overfitting. The parameters were fit to the entire dataset — they learned the specific history rather than generalizing.

When I re-ran with a proper train/test split (train on 2010–2020, test on 2021–2026), the out-of-sample return dropped from 16,994% to 224%. Still an improvement over V6.2, but roughly 75x less than the overfitted result.

Walk-Forward Optimization

To get a properly validated V6.4, I ran a grid search over 81 parameter combinations using only the 2010–2020 training period, selected the best parameters by Sharpe ratio (not returns, to reduce overfitting risk), and then tested those parameters on the unseen 2021–2026 period.

Parameter grid:

hysteresis_buffer:  [0.5, 1.0, 1.5]
leverage_boost:     [1.1, 1.2, 1.3]
leverage_reduce:    [0.7, 0.8, 0.9]
target_volatility:  [0.30, 0.35, 0.40]

Best parameters (by training Sharpe):

• hysteresis_buffer: 0.5 (tighter than default)

• leverage_boost: 1.3 (more aggressive in calm markets)

• leverage_reduce: 0.8 (conservative in volatile markets)

• target_volatility: 0.30 (more conservative than default 0.35)

Final Results: V6.2 vs V6.3 vs V6.4

V6.4 improved out-of-sample returns by 39% over V6.2 while also improving the Sharpe ratio by 26% and slightly reducing max drawdown. The Sortino ratio improvement (0.831 → 1.208, +45%) indicates the return improvement came disproportionately from upside capture rather than additional downside risk.

Observations on these Parameters

The walk-forward optimization converged on a consistent pattern across the top parameter sets: tight hysteresis (0.5), aggressive leverage boost (1.3), and conservative volatility target (0.30). The top five parameter sets all shared hysteresis=0.5 and target_vol=0.30, varying only in leverage settings.

Tight hysteresis (0.5) beat wider buffers. A 0.5 VIX-point buffer reduces whipsaws around thresholds without sacrificing responsiveness. Wider buffers (1.0, 1.5) delayed regime switches too long — similar to why HMM lag was a problem, but at a smaller scale.

Conservative vol target (0.30) beat 0.35. Lower target volatility scales down allocation more aggressively when realized vol spikes. This reduced drawdowns during the 2022 selloff and March 2023 banking crisis, even though V6.4 uses higher leverage during calm periods.

Aggressive leverage boost (1.3) paired with conservative vol targeting. This combination is the key insight: be more aggressive when conditions are calm (1.3x leverage when realized vol is low), but let the volatility target pull you back faster when conditions deteriorate. The asymmetry — aggressive on the upside, conservative on the downside — is what drives the Sortino improvement.

Big Takeaway

Simple regime detection, targeted allocation improvements. The HMM ensemble tried to make regime detection more sophisticated and destroyed performance. V6.4 kept V6.2’s simple VIX thresholds and instead improved how the strategy allocates within each regime. The regime detection was already near-optimal — the allocation logic had more room to improve.

Optimize for Sharpe, not returns. Selecting parameters by training-period Sharpe rather than training-period returns helps avoid overfitting to lucky historical sequences. Returns reward parameter sets that happened to be in the right place at the right time; Sharpe rewards consistency.

Small enhancements compound. No single V6.4 enhancement was dramatic. Hysteresis reduces a few whipsaw trades per year. Dynamic leverage adds or subtracts 20–30% of allocation. Volatility targeting smooths the risk profile. Combined, three modest improvements produced a 39% return and 26% Sharpe improvement with better drawdown control.

Deployment Plan

So next step is V6.4 with the walk-forward optimized parameters.

strategy = DualAllocatorV6_4_Enhanced(
    hysteresis_buffer=0.5,
    leverage_boost=1.3,
    leverage_reduce=0.8,
    target_volatility=0.30
)

Expected annual return: ~26%.

Expected Sharpe: ~0.88

Expected max drawdown: ~40%.

All tests use identical data periods, transaction costs (0.05%), and leverage mechanics. Walk-forward optimization trained on 2010–2020, tested on unseen 2021–2026 data. No lookahead bias.

The information presented in Math & Markets is not investment or financial advice and should not be construed as such.

This is part 79 of my series — Building & Scaling Algorithmic Trading Strategies

This posts tackles my ongoing attempts at improving the Dual Allocator Long/Short strategy.

In the regime detection series (Parts 69–71), I covered Hidden Markov Models from first principles, the latency problem with real-time detection, and ensemble methods that combine HMMs with changepoint detection. The literature suggests that no single detection method dominates on all dimensions, so an ensemble that blends fast signals with smooth ones should reduce false switches and improve risk-adjusted returns.

This post documents what happened when I actually implemented that idea. I built V6.3 of the Dual Allocator, replacing V6.2’s VIX threshold logic with a three-signal ensemble: HMM, VIX thresholds, and CUSUM changepoint detection. I tested it over the full 2010–2026 period and out-of-sample on 2021–2026.

V6.2 returned 4,016% over 16 years. V6.3 returned 284%. The ensemble underperformed on every metric, in every time period tested.

Methodology

V6.2 (threshold-based) uses two VIX levels:

IF VIX < 15:    → Low volatility   → Aggressive TQQQ (up to 130%)
ELIF VIX < 25:  → Moderate          → Cautious TQQQ (75–95%)
ELSE:           → High volatility   → Defensive TLT

Two thresholds, instant detection, no training data, no tunable parameters beyond the thresholds themselves.

V6.3 (ensemble-based) blends three signals:

Ensemble = 0.40 × HMM + 0.35 × VIX_Threshold + 0.25 × CUSUM

The HMM is a 4-state Gaussian model trained on VIX, returns, realized volatility, and momentum. The VIX threshold component is V6.2’s logic normalized to a 0–1 scale. The CUSUM (Cumulative Sum) detector tracks deviations in realized volatility to signal sudden regime shifts.

The ensemble score maps to three regimes: below 0.3 is aggressive, 0.3–0.6 is cautious, above 0.6 is defensive. Both versions use the same TQQQ/TLT allocation logic once a regime is identified. Both use identical data, transaction costs (0.05%), and leverage mechanics (up to 130% TQQQ).

Full Period Results (2010–2026)

V6.2 turned $10K into over $400K and V6.3 turned $10K into $38K. Ouh.

The hypothesis was that ensemble detection would improve the Sharpe ratio by reducing whipsaws. V6.2’s Sharpe is nearly double V6.3’s (0.883 vs 0.471). V6.3 also traded slightly more often, so there was no transaction cost benefit. And V6.3’s max drawdown was deeper (−45.5% vs −41.1%), so the conservatism didn’t translate into downside protection.

Full period comparison. Panel 1 shows portfolio value divergence. Panel 5 shows ensemble signal components — the ensemble mean sits at 0.476, rarely dropping below the 0.3 aggressive threshold. Panel 6 shows V6.3 spending 40% of time in TLT versus V6.2’s more aggressive positioning.

Out-of-Sample Results (2021–2026)

The 2021–2026 period included COVID recovery whipsaws, the 2022 rate-hiking selloff, the March 2023 banking crisis, AI-driven volatility, and the August 2024 correction. This is the environment where ensemble detection should have an advantage — lots of false signals to filter, lots of regime uncertainty.

V6.2: $100K → $258K. V6.3: $100K → $115K. V6.3 captured 9% of V6.2’s gains.

V6.3 traded 50% more often (42 times/year vs 28) due to noisy ensemble signals generating false regime switches. The only metric where V6.3 performed better was raw volatility (30.2% vs 37.0%), but that resulted from sitting in cash and TLT while missing rallies — lower volatility from inaction, not from better risk management.

The hypothesis that HMM ensemble would outperform during volatile markets was wrong. It performed worse.

Out-of-sample comparison. Rolling 3-month Sharpe (Panel 4) shows V6.2 recovering quickly while V6.3 spends extended periods with negative Sharpe. Panel 6 shows regime detection lag: V6.2’s VIX signal switches immediately, while V6.3’s ensemble lags by 1–2 weeks.

RCA & Findings

1. Ensemble Averaging Dampens Signal

Three signals averaged to a permanently moderate reading.

Signal Statistics (Full Period):
  HMM:      0.551 ± 0.419  (consistently elevated)
  VIX:      0.388 ± 0.332  (most accurate)
  CUSUM:    0.480 ± 0.496  (near-random)
  Ensemble: 0.476 ± 0.284  (compressed range)

The ensemble mean of 0.476 sits in the moderate regime band (0.3–0.6). It rarely drops below 0.3 to trigger aggressive TQQQ allocations. V6.2 makes the bulk of its returns during sustained periods in 130% TQQQ when VIX is below 15. V6.3 systematically misses these periods because the ensemble signal stays too high.

2. HMM Detection Lag

The HMM uses the Baum-Welch algorithm on historical observations and needs evidence to accumulate before updating its state estimate. In practice, regime detection lagged by approximately 5–10 trading days.

For a 3x leveraged instrument, 5 days of lag translates to 5–10% of portfolio value. During the March 2023 banking crisis, V6.2 went defensive immediately when VIX crossed 25. The HMM was still estimating a bull regime on March 15, five days after SVB collapsed. By the time V6.3 switched, it had already absorbed a significant drawdown.

3. CUSUM Added Noise

The CUSUM parameters (threshold=3.0, drift=0.5) were poorly calibrated. Its mean was 0.480 with a standard deviation of 0.496 — a coefficient of variation over 100%. Its correlation with actual regime changes was approximately 0.3. With 25% ensemble weight, a quarter of the total signal was noise.

4. Misaligned Ensemble Weights

The 40% HMM / 35% VIX / 25% CUSUM weighting gave the most influence to the slowest signal. The HMM’s persistent elevation (mean 0.551) pulled the ensemble toward defensive positioning. The VIX threshold — the most accurate signal (mean 0.388) — was underweighted. A better configuration would flip this: 60% VIX, 20% HMM, 20% CUSUM. But at that point, the ensemble is mostly just the VIX threshold with extra computation.

5. Regime Thresholds Miscalibrated

With an ensemble mean of 0.476, the 0.3/0.6 thresholds resulted in only ~30% of time classified as low-volatility (aggressive). Markets are calm approximately 60% of the time. The thresholds should have been closer to 0.2 and 0.4 to match base rates, but adjusting them post-hoc introduces curve-fitting risk.

6. Training Data Mismatch

The HMM was trained on 2010–2020 data: a decade of low volatility and steady uptrend. It learned that “normal” means VIX 12–15 with quiet gains. When it encountered 2021–2026 conditions — elevated VIX baseline, frequent 30+ spikes, faster regime transitions — it classified the entire period as moderate-to-high risk and stayed defensive.

V6.2’s thresholds are regime-agnostic. VIX 15 is low volatility in any decade. VIX 25 is high volatility in any market. No training required, no drift over time.

Observations

Simple outperformed complex for this strategy. Two VIX thresholds beat a three-signal ensemble with 10+ tunable parameters. V6.2 computes in 2 seconds versus 45 for V6.3, is easier to debug, and generalizes without retraining. Complexity needs to produce measurable improvement to justify itself, and V6.3 failed that test on every metric.

Detection speed > smoothness for leveraged ETFs. TQQQ is 3x leveraged, so regime changes are reflected in its price within hours. A 5-day detection lag in a 3x instrument means the underlying’s move is already substantially priced in. For daily-rebalancing leveraged strategies, instant threshold crossing outperforms smooth probabilistic detection.

Ensemble methods require complementary signals. The three components were correlated at ρ ≈ 0.6–0.8, the weights weren’t optimized, and one signal (CUSUM) was actively harmful. For an ensemble to improve on its best component, the signals need low mutual correlation, each needs to be independently valuable, and the weights need careful calibration. None of those conditions held.

Research findings are context-dependent. The ensemble regime detection literature typically assumes quarterly rebalancing, diversified portfolios, and multi-year horizons. This strategy uses daily rebalancing, concentrated positioning (TQQQ or TLT), and needs to capture moves within days. So what works for a quarterly pension fund allocation doesn’t apply to a daily tactical leveraged strategy.

Recommendation

For now, I’ll be continuing with V6.2. The comparison across periods and metrics is pretty decisive I’d say:

The only modification worth considering is a hybrid safety filter: run V6.2 as the primary signal, but override to defensive if HMM crisis probability exceeds 80%. This preserves V6.2’s returns during normal markets while adding a tail-risk backstop. Expected impact is minor — possibly 0–5% Sharpe improvement in crisis years, neutral otherwise. Not enough to justify the complexity at this point.

V6.2’s threshold-based approach is near-optimal for this strategy. The regime detection series (Parts 69–71) was useful as a theoretical foundation, but the implementation test confirmed that for a leveraged daily-rebalancing allocator, the simplest detection method is also the most effective.

All tests use identical data periods, transaction costs (0.05%), and leverage mechanics. No lookahead bias.

Remember: Alpha is never guaranteed. And the backtest is a liar until proven otherwise.

The information presented in Math & Markets is not investment or financial advice and should not be construed as such.