Hull Tactical: Chaos & Entropy Market Forecasting (v7.1)

6-Month Live Paper Trading Evaluation

This repository contains Hull Tactical v7.1, a regime-aware "grey box" strategy for the Hull Tactical Market Prediction competition. The model combines chaos/entropy features (Econophysics) with a LightGBM forecaster and a volatility-targeting servo to output a daily portfolio weight constrained to [0.0, 2.0].

Core innovation in v7.1: Smart Noise. Walk-forward telemetry shows that due to the strict entropy filtering we implemented (see predictability score formula below), the S&P 500 market is classified as "Unpredictable" (noise-dominated) for 97% of days. In these regimes, the strategy closet-indexes (pegs weight near 1.0) to avoid the competition's return gap penalty, reserving active risk for the rare 3% of days showing clear fractal structure.

---

Competition Overview

The competition challenges the Efficient Market Hypothesis by asking participants to predict S&P 500 returns using a constrained "grey box" approach. Hull Tactical Market Prediction Competition

• Task: Output a daily portfolio weight $w_t \in [0, 2]$ applied to the market return.
• Constraint: Strategy volatility must remain within 120% of market volatility (or be penalized).
• Data Scale: The training set covers date_id 0 to 8989, representing approximately 35 years of market history. This requires the strategy to generalize across vastly different economic cycles (e.g., the 1990s boom, 2008 GFC, and 2020 COVID crash).
• Evaluation: Adjusted Sharpe with penalties for:
  1. Excess Volatility (linear penalty if vol ratio exceeds 1.2×).
  2. Return Gap (quadratic penalty if strategy geometric excess return falls behind market).

"In the age of machine learning, is it irresponsible to NOT try and time the market?" — Blair Hull

---

1. Philosophy: Efficient or Fractal?

The Competition Challenge:

"Is the EMH an extreme oversimplification at best and possibly just…false?"

Our Hypothesis (The "97/3" Rule): Telemetry from our Walk-Forward Validation suggests the EMH is not false, but conditional. While weak patterns may exist daily, actionable fractal structures emerge in only ~3% of trading days (crises and strong trends). For the remaining 97%, the market is effectively random relative to transaction costs and risk penalties.

Why Chaos?

Many financial models implicitly assume a "random walk" (Gaussian noise). Econophysics literature instead models markets as fractal: they alternate between:

• High entropy / weak structure (noise-dominated)
• Lower entropy / stronger structure (trend/mean-reversion regimes)

Hull Tactical v7.1 is explicitly built around the idea that regime detection should happen before sizing risk.

Key insight: different regimes require different sizing logic.

• High entropy (random): index / mean-reversion / defensive behavior
• Low entropy (structured): trend following (larger sizing)

---

2. Feature Set (What the Code Actually Computes)

This implementation is intentionally self-contained: it derives its main features from a rolling buffer of lagged market returns. We do not rely on standard financial features (Macro E*, Sentiment S*, Momentum MOM*, etc.).

Feature (code name)	Concept	Purpose
pe_log_60d	Permutation Entropy (Bandt–Pompe) on log(1+ret)	High PE ≈ noise; Low PE ≈ structure
wpe_log_60d	Weighted Permutation Entropy	Adds amplitude sensitivity vs PE
hurst_90d	Hurst exponent via DFA	H > 0.5 persistence (trend); H < 0.5 anti-persistence (mean-revert)
net_spec_entropy	Spectral entropy of lag-embedded correlation matrix	Low entropy implies "synchronization" / fragility proxy
net_avg_abs_corr	Avg absolute corr in lag-embedded matrix	Co-movement proxy
vix_term_structure (proxy)	Realized vol ratio = std(ret[-5:]) / std(ret[-60:])	Stress proxy (short-term vol spike)
predictability_score	Weighted mix of entropy + hurst + network	Low score → "UNPREDICTABLE" regime
ret_1d, ret_5d, ret_10d, ret_20d, vol_20d	Simple rolling stats	Used for trend/stress checks and sizing

Feature Importance (Verified in Logs)

Global split importance from the final LightGBM model:

1. vix_term_structure (1113.0): Primary stress detector.
2. ret_1d (1046.0): Immediate momentum.
3. vol_20d (1029.0): Volatility regime.
4. wpe_log_60d (964.0): Weighted Permutation Entropy (Chaos).
5. hurst_90d (959.0): Fractal Market trend persistence.
6. predictability_score (904.0): Composite regime signal.

Predictability Score (Exact Formula)

Definition (as implemented):

pred_score = 0.30·(1 − PE) + 0.30·(1 − WPE) + 0.20·min(2·|H − 0.5|, 1) + 0.20·(1 − net_entropy)

Where:

• (1 − PE) and (1 − WPE): higher when the market is more structured (lower entropy).
• min(2·|H − 0.5|, 1): higher when Hurst deviates from a random walk (either persistent trending or anti-persistent mean reversion).
• (1 − net_entropy): higher when the lag-embedded correlation structure is more synchronized (lower spectral entropy).

When this score falls below 0.25, the regime becomes UNPREDICTABLE.

---

3. Architecture

flowchart TD
    A[Lagged Market Returns + Risk-Free Rate] --> B[Feature Builder<br/>Rolling Buffer 252d]
    B --> C[Chaos & Info Features]
    C --> D{Regime Detection<br/>Priority Order}

    D -->|"1. pred_score < 0.25"| E[UNPREDICTABLE]
    D -->|"2. PE > p90 AND vol > p85"| F[CRISIS]
    D -->|"3. vol_ratio > 1.3"| G[WARNING]
    D -->|"4. Hurst < p25"| H[CHOPPY]
    D -->|"5. Hurst > p75"| I[TRENDING]
    D -->|"6. else"| J[NORMAL]

    C --> K[LightGBM Return Forecast]
    
    E --> M[Smart Noise Logic]
    M --> L
    F --> L[Return-to-Weight Mapping]
    G --> L
    H --> L
    I --> L
    J --> L
    K --> L

    L --> N[Volatility Targeting Servo]
    N --> O[Final Weight<br/>clipped to 0..2]

Loading

Important nuances:

1. Regime classification is rule-based with strict priority order — earlier conditions take precedence.
2. LightGBM forecasts the next return; position sizing is regime-dependent.
3. In UNPREDICTABLE regime, the ML forecast is ignored in favor of Smart Noise logic.

---

4. Regime Definitions (v7.1)

Calibrated Thresholds

Thresholds are calibrated from training feature distributions:

Threshold	Percentile	Default (if no history)
pe_threshold_crisis	90th percentile of PE	0.88
vol_threshold_high	85th percentile of vol_20d	0.020
hurst_threshold_choppy	25th percentile of Hurst	0.42
hurst_threshold_trending	75th percentile of Hurst	0.58
pred_threshold_unpredictable	Fixed	0.25

Regime Detection Priority

The regime check follows a strict priority order — the first matching condition wins:

1. UNPREDICTABLE: predictability_score < 0.25
2. CRISIS: pe_log_60d > pe_threshold_crisis AND vol_20d > vol_threshold_high
3. WARNING: vix_term_structure > 1.3
4. CHOPPY: hurst_90d < hurst_threshold_choppy
5. TRENDING: hurst_90d > hurst_threshold_trending
6. NORMAL: Default (none of the above)

Regime Logic Table

Regime	Trigger Condition	Gain Mult	Weight Caps (floor, cap)	Behavior
CRISIS	High PE (p90) AND high vol (p85)	0.40×	(0.00, 0.50)	Defensive sizing
WARNING	vol(5d)/vol(60d) > 1.3	0.70×	(0.20, 1.00)	Risk reduction
CHOPPY	Hurst < p25	0.50×	(0.20, 1.00)	Smaller, mean-reversion friendly
TRENDING	Hurst > p75	1.35×	(0.50, 2.00)	Larger sizing
NORMAL	Default	1.15×	(0.50, 1.90)	Slight leverage
UNPREDICTABLE	predictability_score < 0.25	Dynamic	(0.30, 1.30)	Smart Noise (below)

---

5. The "Smart Noise" Innovation (v7.1)

Problem: In strong bull markets, price action can be high-entropy (unpredictable) even while trending upward. Many defensive models reduce exposure in "unpredictable" regimes and get penalized for lagging the market.

Key competition reality: The return gap penalty is quadratic. Even small persistent underperformance can dominate the score.

Solution: Directional Noise Logic

When regime == UNPREDICTABLE, v7.1 ignores the ML forecast and uses a directional rule:

Condition	Action	Signal	Multiplier
ret_20d > 0 (bullish drift)	Closet Indexing	signal = 0	mult = 0
ret_20d ≤ 0 (bearish drift)	Defensive Mean Reversion	See formula below	mult = 0.40

Bearish Noise Signal (exact formula):

signal = -0.5 * np.sign(ret_1d) * min(abs(ret_1d), 0.02)

This fades yesterday's return (contrarian), capped at ±2% magnitude, scaled by 0.5.

Why this works:

• Bullish noise: Setting signal=0 and mult=0 yields w_raw = 1.0 exactly (closet indexing). This preserves upside participation during noisy bull runs.
• Bearish noise: The contrarian signal with mult=0.40 provides defensive positioning while allowing some mean-reversion alpha.

Why this matters: Walk-forward logs show the market is classified as UNPREDICTABLE for ~97% of trading days (8,402 days out of 8,640). Therefore, the "Smart Noise" logic is not an edge case—it is the primary driver of the strategy's long-term performance, successfully splitting these days into Bullish (5,541 days) and Bearish (2,861 days) sub-regimes.

---

6. Position Sizing (Return → Weight)

Outside UNPREDICTABLE, the pipeline is:

1. Predict next return with LightGBM: pred_ret

2. Compute volatility scale (target = 16% annualized):

vol_scale = clip((0.16 / sqrt(252)) / risk_ewma, 0.5, 2.0)

3. Compute effective gain:

effective_gain = gain * mult(regime)

4. Raw weight:

w_raw = 1.0 + effective_gain * pred_ret * vol_scale

5. Clip to regime caps:

w_final = clip(w_raw, floor, cap)

Key Clipping Bounds

Parameter	Clipping Range	Purpose
vol_scale	[0.5, 2.0]	Prevents extreme leverage swings
gain	[50, 100]	Stabilizes position sizing
final_weight	[0.0, 2.0]	Competition constraint

Gain Calibration

Gain is auto-calibrated from the training prediction distribution:

1. Compute $p_{95}$ = 95th percentile of $|\hat{r}|$
2. Target weight at $p_{95}$: $w_{target} = 1.0 + 0.85 \times (1.5 - 1.0) = 1.425$
3. Solve: $gain = \frac{w_{target} - 1.0}{p_{95}}$
4. Clip to [50, 100]

This ensures that extreme predictions don't produce weights outside the useful range.

---

7. Volatility Targeting Servo (Risk Control)

A PID-like servo adjusts leverage using the rolling realized volatility ratio.

Parameters

Parameter	Value	Purpose
SERVO_WINDOW	60	Rolling window for vol calculation
SERVO_LO	0.85	Below this ratio → nudge leverage up
SERVO_HI	1.15	Above this ratio → nudge leverage down
SERVO_STEP	0.03	Adjustment step size
SERVO_MIN_OBS	30	Minimum observations before activation
TARGET_VOL_RATIO	1.20	Hard cap (competition constraint)

Logic

1. Maintain rolling windows (60 days) for market and strategy returns.
2. Compute ratio: $ratio = \sigma_{strat} / \sigma_{mkt}$
3. Adjust scale:
   • If $ratio < 0.85$: scale *= 1.03 (increase leverage)
   • If $ratio > 1.15$: scale *= 0.97 (decrease leverage)
4. Extreme vol gate: If $ratio > 1.20$, force scale = min(scale, 1.20/ratio)
5. Clip cumulative scale to [0.5, 1.8]
6. Apply: $w_{scaled} = 1.0 + (w_{raw} - 1.0) \times scale$

This helps stabilize realized risk without constantly hard-clipping weights.

---

8. Scoring Metric (As Implemented)

The strategy return is computed from the market return and risk-free rate:

$$r^{strat}_t = r^{rf}_t(1-w_t) + w_t \cdot r^{mkt}_t$$

$$r^{excess}_t = r^{strat}_t - r^{rf}_t$$

Geometric mean excess return over the evaluation window $T$:

$$\mu^{strat}_{ex} = \left(\prod_{t=1}^{T}(1+r^{excess}_t)\right)^{1/T} - 1$$

Sharpe (annualized):

$$Sharpe = \frac{\mu^{strat}_{ex}}{\sigma(r^{strat})}\sqrt{252}$$

Penalties:

• Volatility penalty (linear):

$$vol_pen = 1 + \max\left(0, \frac{\sigma(r^{strat})}{\sigma(r^{mkt})} - 1.2\right)$$

• Return gap penalty (quadratic):

$$gap = \max\left(0, (\mu^{mkt}_{ex} - \mu^{strat}_{ex}) \cdot 100 \cdot 252\right)$$

$$ret_pen = 1 + \frac{gap^2}{100}$$

Final score:

$$Score = \frac{Sharpe}{vol_pen \cdot ret_pen}$$

This repository's internal telemetry focuses heavily on minimizing the annualized geometric return gap, because the penalty is quadratic.

---

9. Understanding Logs (Telemetry)

Example line:

[Fold 1] BEHIND MARKET by 2.08% (Geo)

Interpretation:

• This is an annualized geometric excess-return gap between strategy and market.
• It does not necessarily mean the strategy lost money; it means that if performance persisted, the strategy's compounded excess return would trail the market by that amount.
• Because the penalty is quadratic, a 2.08% gap implies:

$$ret_pen \approx 1 + \frac{2.08^2}{100} \approx 1.043$$

Minimizing this gap was a primary design objective for v7.1. The log shows that the strategy performs best in Crisis/High-Vol recovery or strong trends (e.g., Fold 7 had a Sharpe of 3.235 ) but struggles in choppy/transition periods (e.g., Fold 17 had a Sharpe of -1.525 ). The strategy excels in distinct trends (Best Fold Sharpe: 3.235) but can lag during rapid regime transitions, relying on the 'Smart Noise' closet-indexing to minimize penalties during these periods.

---

10. Usage

1. Kaggle (Recommended)

This solution is a self-contained notebook designed to run on Kaggle's infrastructure.

-Open the Hull Tactical Chaos Notebook on Kaggle.

Hull Tactical Market Prediction Competition

-Click "Copy and Edit".

-Attach the competition dataset.

-Run all cells. The notebook will:

Train the LightGBM model.

Run the Walk-Forward Validation (if enabled).

Generate the submission file.

2. Local Verification

2. Local Verification To reproduce the Walk-Forward CV results locally: Download the train.csv from Kaggle and place it in the working directory. Run the notebook in Jupyter or convert to a script:

   jupyter nbconvert --to script hull-tactical-chaos.ipynb
   python hull-tactical-chaos.py
   

---

11. Project Structure

This repository follows a flat structure designed for direct deployment to Kaggle.

.
├── LICENSE                     # Apache License
├── README.md                   # Strategy documentation & results
├── hull-tactical-chaos.ipynb   # Complete strategy code (Training, Inference, & Walk-Forward CV)
└── hull-tactical-chaos.log     # 48-Fold Walk-Forward CV verification logs (0.806 Sharpe)

---

12. References

1. Hull, B., et al. (2025). Hull Tactical - Market Prediction. Kaggle. Retrieved from https://kaggle.com/competitions/hull-tactical-market-prediction

2. Bandt, C., & Pompe, B. (2002). Permutation entropy: A natural complexity measure for time series. Physical Review Letters, 88(17), 174102.

3. Fadlallah, B., et al. (2013). Weighted-permutation entropy: A complexity measure for time series incorporating amplitude information. Physical Review E, 87(2), 022911.

4. Peng, C.K., et al. (1994). Mosaic organization of DNA nucleotides. Physical Review E, 49(2), 1685. [DFA method for Hurst exponent]

5. Peters, E.E. (1994). Fractal Market Analysis: Applying Chaos Theory to Investment and Economics. Wiley.

6. Mantegna, R.N. (1999). Hierarchical structure in financial markets. European Physical Journal B, 11(1), 193-197.

7. Moreira, A., & Muir, T. (2017). Volatility-managed portfolios. The Journal of Finance, 72(4), 1611-1644.

8. Hsieh, D.A. (1991). Chaos and Nonlinear Dynamics: Application to Financial Markets. Journal of Finance, 46(5), 1839-1877.

---

Disclaimer

This project is for research and competition purposes only and does not constitute investment advice. Past results (including internal CV) do not guarantee future performance.

---