chore: persist Mac Mini Docker deployment

- Click any metric card to see historical periods where it was at a similar level
- Purple dot highlighting on chart shows matching periods
- Metric overlay line plotted on chart (dashed purple)
- Metric Context panel shows percentile, comparable days, avg forward returns,
  and historical examples from different market cycles
- New /api/metric-context endpoint for per-metric similarity analysis
- Backtest chart_data now includes per-metric raw values
- score_day() returns raw metric values alongside scores
- Fixed JS SyntaxError from broken inline onclick escaping (uses addEventListener)

Co-Authored-By: Claude Opus 4.6 <<EMAIL>>

.dockerignore

@@ -0,0 +1,11 @@
+.git
+.gitea
+.github
+.venv
+.playwright
+__pycache__
+*.py[cod]
+*.log
+.env
+results
+screenshots

.gitea/workflows/ci.yml

@@ -0,0 +1,28 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          version: "0.11.6"
+          enable-cache: true
+
+      - name: Validate lockfile and install dependencies
+        run: uv sync --locked --group runtime --group ml --group dev
+
+      - name: Compile Python sources
+        run: uv run --frozen python -m compileall -q dashboard scrapers scoring backtesting ml ml_engine llm_client scripts orchestrator.py
+
+      - name: Run tests
+        run: uv run --frozen pytest

.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.pyc
+.venv/
+.playwright/
+.pytest_cache/
+data/cache.json
+data/history.json
+data/score_history.jsonl
+data/jobs.json
+data/*.lock
+config/llm_settings.json
+results/
+*.log
+.env
+node_modules/

ARCHITECTURE.md

@@ -0,0 +1,262 @@
+# Bitcoin Accumulation Zone Monitor — Architecture & Logic
+
+## Overview
+
+This is **NOT** a trading bot or ML predictor. It monitors proven Bitcoin on-chain metrics that have historically signaled optimal accumulation (buying) zones for long-term holders. Each metric scores 0-10 points, producing a composite score of 0-100.
+
+**Philosophy:** Every signal is transparent and traceable. No black box. The metrics used have correctly identified every major Bitcoin cycle bottom since 2010.
+
+## How It Works
+
+### Data Pipeline
+
+```
+LookIntoBitcoin.com ──┐
+  (Playwright scraper) │
+                       ├──> data/cache.json (current values, refreshed every 15min)
+alternative.me API ────┤    data/history.json (full history back to 2010, refreshed weekly)
+                       │
+CoinGecko API ─────────┘
+         │
+         ▼
+   Scoring Engine (scoring/engine.py)
+         │
+         ▼
+   Composite Score 0-100
+         │
+    ┌────┴────┐
+    ▼         ▼
+ Dashboard   Backtest Engine
+ (live)      (historical validation)
+```
+
+### Data Sources
+
+All data is scraped or fetched from free sources — **no API keys required**.
+
+| Source | Method | Data |
+|--------|--------|------|
+| LookIntoBitcoin / BitcoinMagazinePro | Playwright browser scraping of Plotly Dash charts | Puell Multiple, MVRV Z-Score, Reserve Risk, RHODL Ratio, NUPL, 200W SMA, LTH Realized Price, LTH Supply, Hash Ribbons, Pi Cycle |
+| alternative.me | Free REST API | Fear & Greed Index (daily, back to Feb 2018) |
+| CoinGecko | Free REST API | BTC price, market cap, 24h change |
+
+#### Scraping Method (LookIntoBitcoin)
+
+The site uses Plotly Dash charts. We intercept the `_dash-update-component` XHR response which contains the full chart data as JSON:
+
+```python
+page.on("response", handler)  # Intercept XHR
+page.goto("https://www.lookintobitcoin.com/charts/puell-multiple/")
+# Response contains: response['chart']['figure']['data'] → list of trace objects
+# Each trace: {name: str, x: [dates], y: [values]}
+```
+
+This gives us the **complete historical time series** (5000+ data points per metric going back to 2010) without needing any API key.
+
+## Scoring System
+
+### Individual Metrics (0-10 each)
+
+#### 1. Fear & Greed Index (source: alternative.me)
+Measures market sentiment from social media, surveys, and momentum.
+
+| F&G Value | Classification | Score |
+|-----------|---------------|-------|
+| 0-10 | Extreme Fear | 10 |
+| 11-25 | Fear | 7 |
+| 26-45 | Neutral-low | 4 |
+| 46-55 | Neutral | 2 |
+| 56-75 | Greed | 1 |
+| 76-100 | Extreme Greed | 0 |
+
+**Logic:** "Be fearful when others are greedy, be greedy when others are fearful." — Buffett. Extreme Fear has historically coincided with cycle bottoms.
+
+#### 2. Puell Multiple (source: LookIntoBitcoin)
+Measures miner revenue relative to 365-day average. When miners earn very little (low Puell), they're capitulating — historically a bottom signal.
+
+| Puell Value | Meaning | Score |
+|-------------|---------|-------|
+| < 0.3 | Deep miner capitulation | 10 |
+| 0.3-0.5 | Miner stress | 8 |
+| 0.5-0.8 | Below average revenue | 5 |
+| 0.8-1.2 | Normal | 3 |
+| 1.2-2.0 | Above average | 1 |
+| > 2.0 | Miner euphoria | 0 |
+
+**Historical accuracy:** Puell < 0.5 identified the Dec 2018, Mar 2020, and Jun 2022 bottoms.
+
+#### 3. MVRV Z-Score (source: LookIntoBitcoin)
+Compares market value to realized value. Negative Z-Score means the market is valued below what everyone paid — extreme undervaluation.
+
+| Z-Score | Meaning | Score |
+|---------|---------|-------|
+| < 0 | Below realized value | 10 |
+| 0-0.5 | Undervalued | 8 |
+| 0.5-1.5 | Fair value | 5 |
+| 1.5-3.0 | Overvalued | 2 |
+| 3.0-5.0 | Very overvalued | 1 |
+| > 5.0 | Extreme overvaluation | 0 |
+
+**Historical accuracy:** Every time MVRV Z-Score went below 0, buying led to >200% returns within 2 years (100% hit rate across all cycles).
+
+#### 4. Drawdown from ATH (calculated from price)
+How far BTC has fallen from its all-time high. Larger drawdowns = better buying opportunity historically.
+
+| Drawdown | Score |
+|----------|-------|
+| > 70% | 10 |
+| 50-70% | 8 |
+| 30-50% | 6 |
+| 20-30% | 4 |
+| 10-20% | 2 |
+| < 10% | 0 |
+
+#### 5. Price vs 200-Week SMA (source: LookIntoBitcoin)
+The 200-week moving average has historically acted as the absolute floor in bear markets.
+
+| Position | Score |
+|----------|-------|
+| Below 200W SMA | 10 |
+| 0-20% above | 6 |
+| 20-50% above | 3 |
+| 50-100% above | 1 |
+| > 100% above | 0 |
+
+#### 6. Reserve Risk (source: LookIntoBitcoin)
+Measures the confidence of long-term holders relative to the price. Low Reserve Risk = high confidence among HODLers + low price = excellent time to buy.
+
+| Reserve Risk | Score |
+|--------------|-------|
+| < 0.002 | 10 |
+| 0.002-0.005 | 7 |
+| 0.005-0.01 | 4 |
+| 0.01-0.02 | 2 |
+| > 0.02 | 0 |
+
+#### 7. RHODL Ratio (source: LookIntoBitcoin)
+Ratio of 1-week old coins to 1-2 year old coins. Low ratio = long-term holders dominating (accumulation). High ratio = short-term speculation (distribution).
+
+| RHODL | Score |
+|-------|-------|
+| < 100 | 10 |
+| 100-500 | 7 |
+| 500-2000 | 4 |
+| 2000-10000 | 1 |
+| > 10000 | 0 |
+
+#### 8. NUPL — Net Unrealized Profit/Loss (source: LookIntoBitcoin)
+Shows what fraction of market cap is unrealized profit. Negative = market is at a loss (capitulation). Above 0.75 = euphoria.
+
+| NUPL | Phase | Score |
+|------|-------|-------|
+| < 0 | Capitulation | 10 |
+| 0-0.25 | Hope/Fear | 7 |
+| 0.25-0.5 | Optimism | 4 |
+| 0.5-0.75 | Belief/Greed | 1 |
+| > 0.75 | Euphoria | 0 |
+
+#### 9. LTH Realized Price vs Spot (source: LookIntoBitcoin)
+Long-Term Holder Realized Price = average cost basis of coins held >155 days. When spot price drops below this, even diamond hands are underwater — extreme value.
+
+| Position | Score |
+|----------|-------|
+| Price below LTH RP | 10 |
+| 0-20% above | 6 |
+| 20-50% above | 3 |
+| > 50% above | 1 |
+
+#### 10. Hash Ribbons / Miner Capitulation (source: LookIntoBitcoin)
+When miners capitulate (hash rate declining), it signals maximum pain. The recovery signal (hash rate resuming growth) has been a reliable buy signal.
+
+| Signal | Score |
+|--------|-------|
+| Active buy signal | 10 |
+| Recent recovery | 6 |
+| Normal | 3 |
+| Miner euphoria | 0 |
+
+### Composite Score
+
+```
+Total Score = Sum of all individual metric scores (0-100)
+```
+
+| Score Range | Assessment | Action |
+|-------------|------------|--------|
+| 85-100 | Extreme Accumulation Zone | Strong buy — historically rare, ~4x per decade |
+| 70-84 | Strong Accumulation | Buy — excellent long-term entry |
+| 55-69 | Moderate Opportunity | Consider buying — decent entry |
+| 40-54 | Neutral | Hold — not compelling either way |
+| 25-39 | Caution | Reduce or wait — market heating up |
+| 0-24 | Extreme Caution | Do NOT buy — historically the worst times |
+
+## Backtest Engine
+
+### Purpose
+Reconstruct the composite score historically and compare against actual BTC forward returns to validate the scoring system's accuracy.
+
+### Methodology
+
+1. **Historical Reconstruction:** Using scraped historical data (2010-present), calculate what each metric's score would have been on every day
+2. **Forward Returns:** For each historical day, calculate what BTC actually returned over the next 30, 90, 180, and 365 days
+3. **Score Bracket Analysis:** Group days by score bracket and calculate average forward returns, win rates, max drawdowns
+4. **Recency Weighting:** More recent cycles weighted higher because BTC's cycle-over-cycle returns diminish as it matures:
+   - 2022-present: 4x weight
+   - 2020-2021: 3x weight
+   - 2018-2019: 2x weight
+   - Before 2018: 1x weight
+5. **Cycle-Separated Results:** Returns shown per cycle (Cycle 3: 2016-2019, Cycle 4: 2020-2023, Cycle 5: 2024+)
+
+### Diminishing Returns Adjustment
+
+Bitcoin's gains decrease every cycle. A score of 90 in 2018 led to different outcomes than a score of 90 in 2022:
+- The backtest separates results by cycle
+- Current expectations are based on the 2 most recent comparable cycles
+- Adaptive thresholds recalculate based on rolling 2-year windows
+
+## Architecture
+
+```
+/opt/apps/btc-ml-optimizer/
+├── dashboard/
+│   └── server.py              # FastAPI + inline HTML/JS dashboard
+├── scrapers/
+│   ├── __init__.py
+│   ├── lookintobitcoin.py     # Playwright scraper for on-chain charts
+│   ├── history_collector.py   # Full historical data collection
+│   ├── fear_greed.py          # alternative.me Fear & Greed API
+│   └── price.py               # CoinGecko BTC price API
+├── scoring/
+│   ├── __init__.py
+│   └── engine.py              # Scoring logic and thresholds
+├── backtesting/
+│   ├── __init__.py
+│   └── engine.py              # Historical backtest calculations
+├── data/
+│   ├── cache.json             # Current metric values (refreshed every 15min)
+│   └── history.json           # Full historical data (refreshed weekly)
+├── config/
+│   ├── thresholds.json        # Configurable scoring thresholds
+│   └── llm_settings.json      # Optional LLM provider config for AI commentary
+├── llm_client/
+│   └── analyzer.py            # Optional LLM integration for signal analysis
+└── README.md
+```
+
+## Infrastructure
+
+- **Server:** Main VPS (Hostinger), Tailscale IP 100.94.106.120
+- **Port:** 3088
+- **Process Manager:** pm2 (`btc-ml-optimizer`)
+- **Dashboard URL:** http://100.94.106.120:3088
+- **Backtest URL:** http://100.94.106.120:3088/backtest
+- **Git Repo:** https://git.bizzle.lol/bizzle/btc-accumulation-monitor
+
+## Dependencies
+
+- Python 3.13
+- FastAPI + uvicorn
+- Playwright (Chromium, headless)
+- requests
+- No ML libraries required
+- No paid API keys required

Dockerfile

@@ -0,0 +1,31 @@
+# syntax=docker/dockerfile:1.7
+FROM ghcr.io/astral-sh/uv:0.11.6 AS uv
+FROM python:3.13.5-slim-bookworm
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \
+    UV_COMPILE_BYTECODE=1 \
+    UV_LINK_MODE=copy
+
+COPY --from=uv /uv /uvx /bin/
+
+WORKDIR /app
+
+COPY pyproject.toml uv.lock ./
+RUN uv sync --frozen --no-install-project --no-dev --group runtime --group ml \
+    && uv run --frozen --no-dev --group runtime --group ml \
+       playwright install --with-deps chromium \
+    && chmod -R a+rX /ms-playwright
+
+COPY --chown=10001:10001 . .
+RUN mkdir -p /app/data /app/config \
+    && chown -R 10001:10001 /app/data /app/config
+
+USER 10001:10001
+EXPOSE 3088
+
+HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
+  CMD ["/app/.venv/bin/python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3088/health/live', timeout=3)"]
+
+CMD ["/app/.venv/bin/python", "-m", "uvicorn", "dashboard.server:app", "--host", "0.0.0.0", "--port", "3088"]

README.md

@@ -1,160 +1,269 @@
-# BTC ML Trading Strategy Optimizer
+# Bitcoin Accumulation Zone Monitor
 
-An automated optimization loop that trains ML models on BTC/USDT data, backtests trading strategies, and uses an LLM to iteratively improve the configuration.
+> Bitcoin on-chain metrics dashboard with classic equal-weight scoring, ML-optimized scoring, historical backtesting, and click-to-select metric context for long-term BTC accumulation decisions.
+
+![Dashboard](screenshots/dashboard-main.png)
+
+## What It Does
+
+Monitors Bitcoin accumulation conditions using 16 scored market/on-chain indicators plus optional informational cycle metrics. Each scored metric receives a 0-10 score and rolls into a 0-100 accumulation score.
+
+The dashboard now supports two scoring modes:
+
+- **Classic** — transparent equal-weight scoring across every active metric.
+- **ML** — feature-importance weights trained against historical 365-day forward returns, with displayed per-metric weights and point contributions.
+
+Historical backtests show score-vs-price behavior, score bracket performance, major signal events, and current-score context. Metric cards are clickable: selecting a metric overlays its historical series on the score chart and shows comparable historical periods with forward returns.
+
+## Screenshots
+
+### Main Dashboard — ML mode + metric context
+![Main Dashboard](screenshots/dashboard-main.png)
+*Live BTC price, Classic/ML scoring toggle, 16 active scored metrics, ML weights/contributions, metric sparklines, and click-to-select historical context.*
+
+### Historical Backtest
+![Backtest](screenshots/dashboard-backtest.png)
+*Current signal percentile, comparable historical periods by cycle, score-vs-BTC chart, bracket performance, and major signal events.*
+
+### Settings
+![Settings](screenshots/dashboard-settings.png)
+*LLM provider configuration for optional AI-powered signal commentary and local/cloud model selection.*
+
+## Feature Highlights
+
+- **16 scored metrics** from market sentiment, miner stress, valuation, holder behavior, network activity, and velocity signals.
+- **Classic vs ML scoring toggle** on the dashboard and backtest API.
+- **ML score explainability**: metric cards show learned weight and contribution in points.
+- **Leakage-resistant ML validation**: training uses purged time-series splits so 365-day forward-return labels do not overlap validation windows.
+- **Historical context panel**: compares the current composite score against historical periods and forward returns.
+- **Clickable metric cards**: select any metric to see percentile, similar historical levels, forward returns, example dates by market cycle, and highlighted chart periods.
+- **Score history chart** with BTC price overlay, range controls, and selected-metric overlay.
+- **Backtest dashboard** with current signal context, score bracket performance, and signal-crossing events.
+- **Quick vs full refresh**: quick refresh updates BTC price and Fear & Greed; full refresh re-scrapes on-chain sources.
+- **LLM settings UI** for Ollama, LM Studio, OpenAI, Anthropic, and OpenRouter.
+
+## Metrics
+
+| # | Metric | Source | Accumulation Signal |
+|---|--------|--------|-------------------|
+| 1 | Fear & Greed Index | alternative.me API | Extreme fear / capitulation sentiment |
+| 2 | Puell Multiple | LookIntoBitcoin | Miner revenue stress |
+| 3 | MVRV Z-Score | LookIntoBitcoin | Market near/below realized value |
+| 4 | Drawdown from ATH | Calculated from BTC price | Deep correction from cycle high |
+| 5 | Price vs 200W SMA | LookIntoBitcoin + BTC price | Price near/below long-term trend |
+| 6 | Reserve Risk | LookIntoBitcoin | High holder confidence relative to price |
+| 7 | RHODL Ratio | LookIntoBitcoin | Long-term holder dominance |
+| 8 | Net Unrealized Profit/Loss (NUPL) | LookIntoBitcoin | Capitulation / early recovery zones |
+| 9 | LTH Realized Price | LookIntoBitcoin | Price near long-term holder cost basis |
+| 10 | Hash Ribbons | LookIntoBitcoin | Miner capitulation/recovery signal |
+| 11 | SOPR | CheckOnChain | Spent outputs near loss / reset territory |
+| 12 | Sell-side Risk Ratio | CheckOnChain | Low realized profit/loss pressure |
+| 13 | Active Address Momentum | CheckOnChain | Network activity momentum extremes |
+| 14 | Transaction Count Momentum | CheckOnChain | Transaction activity momentum extremes |
+| 15 | NVT Price | CheckOnChain | Network-value valuation discount/premium |
+| 16 | VDD Multiple | CheckOnChain | Coin-days/velocity reset conditions |
+
+Informational cards may also appear when data is available, such as **Long-Term Holder Supply** and **Pi Cycle Bottom**. These are displayed for context and are not included in the composite score.
+
+## Score Interpretation
+
+| Score | Assessment | Interpretation |
+|-------|-----------|----------------|
+| 80-100 | 🟢 Extreme Accumulation Zone | Broad capitulation/value conditions across active metrics |
+| 65-79 | 🟢 Strong Accumulation Zone | Historically attractive long-term entry territory |
+| 50-64 | 🟡 Moderate Opportunity | DCA-friendly, but not maximum-signal conditions |
+| 35-49 | 🟡 Neutral | Mixed signals; not compelling either direction |
+| 20-34 | 🔴 Caution — Overheated | Market conditions becoming less favorable |
+| 0-19 | 🔴 Extreme Caution | Historically poor accumulation setup |
+
+Backtest tables provide actual historical forward-return statistics per score bracket, including 30d/90d/180d/1yr averages, win rate, max gain/loss, and average max drawdown.
+
+## ML-Optimized Scoring
+
+The ML mode uses a `GradientBoostingClassifier` trained on historical feature rows to predict whether a day was a good long-term buy based on 365-day forward return. Training features include:
+
+- Classic metric scores.
+- Raw metric values.
+- 30-day metric deltas.
+- Interaction features such as MVRV × NUPL and Puell × Reserve Risk.
+- Cycle-position context such as days since ATH.
+
+The resulting feature importances are aggregated back into transparent metric weights stored in `config/ml_weights.json`. The UI displays normalized weight and contribution for each active metric.
+
+Validation uses purged expanding time-series splits: because each label uses a 365-day forward-return window, training rows whose label windows overlap validation are removed before scoring validation folds.
+
+## Tech Stack
+
+| Component | Technology |
+|-----------|-----------|
+| Backend | Python 3.13 + FastAPI |
+| Frontend | Inline HTML/CSS/JS dark trading-terminal UI |
+| Charts | Chart.js |
+| Scraping | requests + Playwright-style browser scraping where needed |
+| Data APIs | alternative.me, CoinGecko, LookIntoBitcoin, CheckOnChain |
+| ML | NumPy + pandas + scikit-learn GradientBoostingClassifier |
+| Process Manager | pm2 or uvicorn |
+| Default Port | 3088 |
+
+## How Data Is Collected
+
+Data is collected from free/public sources and cached locally under `data/`.
+
+- Fast live refreshes update BTC price, ATH/drawdown, 200D SMA/Mayer where possible, and Fear & Greed.
+- On-chain metrics are cached and reused because they update slowly.
+- Full refresh re-scrapes on-chain metrics from LookIntoBitcoin/CheckOnChain.
+- Historical backtest data lives in `data/history.json` and supports charting, backtests, and metric-context lookups.
+- Score history appends to `data/score_history.jsonl`.
+
+## Project Structure
+
+```
+├── dashboard/
+│   └── server.py                  # FastAPI server + inline dashboard/backtest/settings UI
+├── scrapers/
+│   ├── lookintobitcoin.py         # LookIntoBitcoin metric scraping
+│   ├── checkonchain.py            # CheckOnChain metric scraping
+│   ├── history_collector.py       # Full historical data collection
+│   ├── history_updater.py         # Incremental historical updates
+│   ├── fear_greed.py              # Fear & Greed Index API
+│   └── price.py                   # BTC price, ATH, drawdown, SMA helpers
+├── scoring/
+│   └── engine.py                  # Classic + ML-weighted scoring logic
+├── backtesting/
+│   └── engine.py                  # Historical backtest engine
+├── ml/
+│   └── optimizer.py               # ML training, purged CV, weight export
+├── tests/
+│   ├── test_ml_optimizer_validation.py
+│   └── test_scoring_engine_ml.py
+├── data/
+│   ├── cache.json                 # Live metric cache
+│   ├── history.json               # Historical metric/time-series data
+│   └── score_history.jsonl        # Live score history
+├── config/
+│   ├── thresholds.json            # Classic scoring thresholds
+│   ├── ml_weights.json            # Learned ML metric weights
+│   └── llm_settings.json          # Optional AI commentary provider config
+├── screenshots/                   # README screenshots
+├── scripts/run.sh                 # Locked local launcher with Playwright path
+├── .gitea/workflows/ci.yml        # Gitea Actions test/compile gates
+├── Dockerfile                     # Non-root Chromium-enabled image
+├── docker-compose.yml             # Port, healthcheck, restart, persistent volumes
+├── pyproject.toml                 # Runtime, ML, and development dependency groups
+├── uv.lock                        # Exact reproducible dependency resolution
+├── ARCHITECTURE.md
+└── README.md
+```
+
+## Reproducible Setup
+
+Install [uv](https://docs.astral.sh/uv/) and use Python 3.11-3.13. Dependencies are declared in explicit `runtime`, `ml`, and `dev` groups in `pyproject.toml`; exact cross-platform resolutions are committed in `uv.lock`.
+
+```bash
+git clone <repository-url>
+cd btc-accumulation-monitor
+uv sync --locked --group runtime --group ml --group dev
+```
+
+Install the Chromium binary once for full on-chain refreshes. Keep its path explicit so installation and runtime use the same browser cache:
+
+```bash
+export PLAYWRIGHT_BROWSERS_PATH="$PWD/.playwright"
+uv run --frozen playwright install chromium
+```
+
+`requirements_vps.txt` is a lock-derived, hash-pinned compatibility export for pip-based hosts. `pyproject.toml` and `uv.lock` remain authoritative; regenerate the compatibility file after dependency changes with:
+
+```bash
+uv export --frozen --no-dev --group runtime --group ml \
+  --no-emit-project --no-header --output-file requirements_vps.txt
+```
+
+## Running
+
+The executable launcher fixes `PYTHONPATH`, preserves an explicitly supplied `PLAYWRIGHT_BROWSERS_PATH`, and starts port 3088 from the locked environment:
+
+```bash
+./scripts/run.sh
+```
+
+Equivalent exact command:
+
+```bash
+PLAYWRIGHT_BROWSERS_PATH="$PWD/.playwright" PYTHONPATH=. \
+  uv run --frozen --no-dev --group runtime --group ml \
+  python -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088
+```
+
+Then visit `http://localhost:3088`.
+
+## Container Deployment
+
+The image uses a multi-architecture Python base, installs Playwright Chromium and its OS libraries during the build, and runs the application as non-root UID `10001`. Compose publishes port 3088, restarts unless stopped, and persists `/app/data` and `/app/config` in named volumes.
+
+```bash
+docker compose build
+docker compose up -d
+```
+
+The Docker and Compose healthchecks probe `GET /health/live`. The deployment must include the reliability revision that supplies that endpoint; without it, Docker correctly reports the container unhealthy even if the older application server is accepting requests.
+
+Named volumes are initialized from the image on first use. Back up both before replacing or deleting them:
+
+```bash
+docker volume inspect btc-accumulation-monitor_btc-monitor-data
+docker volume inspect btc-accumulation-monitor_btc-monitor-config
+```
+
+For bind-mounted deployments, ensure the host directories are writable by UID `10001` and do not replace `config/` with an empty directory.
+
+## First Run and Data Freshness
+
+1. Visit `http://localhost:3088` for the dashboard.
+2. Use **Quick Refresh** for price and Fear & Greed updates while retaining cached slow-moving on-chain metrics.
+3. Use **Full Refresh** when on-chain source data must be re-scraped; this requires the installed Playwright Chromium browser and external source availability.
+4. Visit `http://localhost:3088/backtest` for historical analysis.
+5. If historical data is missing, populate `data/history.json` through the existing collection flow.
+
+Freshness is metric-specific. Price and sentiment APIs can update frequently, while public on-chain chart sources commonly update daily and may be reused from cache. A successful refresh is not proof that every upstream metric has a new observation. Check source timestamps/status exposed by the running revision, and treat missing, stale, or scrape-failed metrics as unavailable rather than silently current. `data/` is operational state and should be persisted and backed up.
+
+## ML and Backtest Caveats
+
+ML weights and backtest output are research artifacts, not investment advice or evidence of future performance. Any reported ML result must retain its provenance: source-data snapshot/range, feature and label definitions, training window, purge/embargo policy, code revision, dependency lock, random seed (when applicable), and generated weight/config artifact.
+
+Model selection and threshold tuning must use training/validation data only. Report final performance on a genuinely untouched out-of-sample (OOS) period; do not describe in-sample fit, cross-validation used for selection, or the best result from repeated experiments as OOS. Forward-return labels require purging overlapping label horizons, but purged cross-validation alone does not create an untouched final test set. Results without reproducible provenance and a reserved OOS evaluation should be labeled exploratory.
+
+## Useful API Endpoints
+
+| Endpoint | Description |
+|----------|-------------|
+| `GET /api/data?mode=classic` | Current metrics using equal-weight scoring |
+| `GET /api/data?mode=ml` | Current metrics using ML-optimized weights |
+| `GET /api/history` | Recent live score history |
+| `POST /api/refresh` | Quick refresh |
+| `POST /api/refresh?full=true` | Full on-chain refresh |
+| `GET /api/backtest?mode=classic` | Historical backtest with classic scoring |
+| `GET /api/backtest?mode=ml` | Historical backtest with ML scoring |
+| `GET /api/metric-context?metric=mvrv_zscore&mode=ml` | Similar historical levels and forward returns for one metric |
+| `GET /api/settings` | Safe LLM settings payload |
+
+## Testing and CI
+
+Run the committed test suite and the same static compilation gate used by Gitea Actions:
+
+```bash
+uv sync --locked --group runtime --group ml --group dev
+uv run --frozen python -m compileall -q \
+  dashboard scrapers scoring backtesting ml ml_engine llm_client scripts orchestrator.py
+uv run --frozen pytest
+```
+
+`.gitea/workflows/ci.yml` runs lock validation/install, static compilation, and tests for pull requests and pushes to `main`.
 
 ## Architecture
 
-```
+See [ARCHITECTURE.md](ARCHITECTURE.md) for deeper implementation details on scoring, data collection, and backtesting.
-┌─────────────────────────────────────────────────────────────────┐
-│                        Optimization Loop                        │
-│                                                                 │
-│  ┌──────────┐    ┌───────────────┐    ┌──────────────────────┐  │
-│  │   VPS    │───>│  Windows PC   │───>│     Mac Mini         │  │
-│  │ (Orch.)  │<───│  (GPU/ML)     │    │     (LLM)           │  │
-│  │          │<───────────────────────>│                      │  │
-│  │ - Fetch  │    │ - XGBoost     │    │ - Ollama             │  │
-│  │   data   │    │ - LightGBM    │    │ - qwen3.5:27b        │  │
-│  │ - Coord  │    │ - CatBoost    │    │ - Analyze results    │  │
-│  │ - Store  │    │ - RTX 4070 Ti │    │ - Suggest changes    │  │
-│  └──────────┘    └───────────────┘    └──────────────────────┘  │
-│       ▲                                        │                │
-│       └────────────────────────────────────────┘                │
-│                    Modified config                              │
-└─────────────────────────────────────────────────────────────────┘
-```
 
-### Machines (Tailscale)
+## License
 
-| Machine    | Role         | Address           | Key Resources        |
+Private — not for public distribution.
-|------------|-------------|-------------------|---------------------|
-| VPS        | Orchestrator | localhost          | Coordination, data   |
-| Windows PC | ML Engine    | 100.76.218.38     | RTX 4070 Ti GPU      |
-| Mac Mini   | LLM         | 100.100.242.21    | Ollama, qwen3.5:27b  |
-
-## Directory Structure
-
-```
-btc-ml-optimizer/
-├── orchestrator.py              # Main loop — coordinates everything
-├── ml_engine/
-│   └── train_and_backtest.py    # Self-contained ML script (runs on Windows)
-├── llm_client/
-│   └── analyzer.py              # LLM strategy analyzer (calls Mac Mini)
-├── scripts/
-│   ├── fetch_data.py            # BTC/USDT data fetcher (ccxt)
-│   └── setup_windows.sh         # Install deps on Windows PC
-├── config/
-│   └── initial_config.json      # Starting configuration
-├── data/                        # OHLCV CSV files
-├── results/                     # Iteration results + logs
-├── requirements_vps.txt         # VPS Python dependencies
-└── requirements_windows.txt     # Windows PC Python dependencies
-```
-
-## Setup
-
-### 1. VPS (this machine)
-
-```bash
-pip install -r requirements_vps.txt
-```
-
-### 2. Windows PC
-
-```bash
-# From VPS — installs all ML deps on Windows via SSH
-bash scripts/setup_windows.sh
-```
-
-Or manually on Windows:
-```bash
-pip install -r requirements_windows.txt
-```
-
-### 3. Mac Mini
-
-Ensure Ollama is running with the qwen3.5:27b model:
-```bash
-ollama pull qwen3.5:27b
-ollama serve  # should already be running
-```
-
-## Usage
-
-### Fetch Data
-
-```bash
-python3 scripts/fetch_data.py
-```
-
-Downloads 2 years of BTC/USDT 1h and 4h OHLCV data from Binance.
-
-### Run the Optimizer
-
-```bash
-python3 orchestrator.py
-```
-
-The optimizer will:
-1. Ensure data is fetched
-2. Upload ML engine + data to Windows PC
-3. Train model and backtest on GPU
-4. Send results to LLM for analysis
-5. Apply LLM-suggested config changes
-6. Repeat until convergence (or 50 iterations)
-
-### Run ML Engine Standalone (on Windows)
-
-```bash
-python train_and_backtest.py --config config.json --data btc_4h.csv --output results.json
-```
-
-## Configuration Reference
-
-### `model_type`
-- `xgboost` — XGBoost with GPU (default, generally best)
-- `lightgbm` — LightGBM with GPU (faster training)
-- `catboost` — CatBoost with GPU (handles interactions well)
-- `ensemble` — Soft voting of all three
-
-### `features`
-- `technical_indicators` — List of indicators to compute
-- `lookback_periods` — Windows for return/volatility features
-- `use_volume_features` — Include volume-derived features
-- `use_volatility_features` — Include volatility features
-- `use_candle_patterns` — Include candlestick pattern features
-- `use_lag_features` — Include lagged feature values
-- `lag_periods` — Specific lag periods to use
-
-### `target`
-- `direction` — `"long"` or `"both"`
-- `horizon_candles` — Forward-looking prediction window
-- `threshold_pct` — Minimum % move to label as positive
-
-### `hyperparameters`
-Standard gradient boosting params: `learning_rate`, `max_depth`, `n_estimators`, `subsample`, `colsample_bytree`, `min_child_weight`, `gamma`, `reg_alpha`, `reg_lambda`
-
-### `strategy`
-- `entry_threshold` — Min probability to enter trade (0.5-0.8)
-- `stop_loss_pct` — Stop loss percentage
-- `take_profit_pct` — Take profit percentage
-- `trailing_stop_pct` — Trailing stop distance
-- `position_sizing` — `"confidence_scaled"` or `"fixed"`
-- `min_confidence_to_trade` — Absolute minimum confidence
-
-### `training`
-- `walk_forward_windows` — Number of walk-forward splits (3-10)
-- `train_pct` / `validation_pct` / `test_pct` — Data split ratios
-
-## Convergence Criteria
-
-The optimizer stops when:
-- Sharpe ratio exceeds 3.0
-- Sharpe improvement < 1% over 5 consecutive iterations
-- Maximum 50 iterations reached
-
-## Output
-
-- `config/best_config.json` — Best configuration found
-- `results/iterations.jsonl` — Full log of every iteration
-- `results/results_iter_N.json` — Detailed results per iteration

backtesting/engine.py

@@ -0,0 +1,772 @@
+"""Historical backtest engine for Bitcoin Accumulation Zone scoring."""
+
+import copy
+import json
+import logging
+import os
+import sys
+import threading
+from collections import defaultdict
+from datetime import datetime, timedelta
+
+from scoring.policy import SCORE_BRACKETS, SCORE_VERSION, score_in_bracket
+from ml.artifacts import validate_ml_artifact
+from backtesting.statistics import summarize_returns
+
+log = logging.getLogger(__name__)
+
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, BASE_DIR)
+
+HISTORY_PATH = os.path.join(BASE_DIR, "data", "history.json")
+CACHE_PATH = os.path.join(BASE_DIR, "data", "cache.json")
+ML_WEIGHTS_PATH = os.path.join(BASE_DIR, "config", "ml_weights.json")
+
+_BACKTEST_CACHE = {}
+_BACKTEST_CACHE_LOCK = threading.Lock()
+_BACKTEST_CACHE_LIMIT = 4
+
+# Score brackets matching the dashboard assessment levels
+BRACKETS = SCORE_BRACKETS
+
+# Scoring thresholds — load from config/thresholds.json (single source of truth)
+import os as _os
+import json as _json
+
+_THRESH_PATH = _os.path.join(_os.path.dirname(_os.path.dirname(_os.path.abspath(__file__))), "config", "thresholds.json")
+try:
+    with open(_THRESH_PATH) as _f:
+        _THRESH = _json.load(_f)
+except Exception:
+    _THRESH = {}
+
+METRIC_SCORERS = {
+    "fear_greed": {"ranges": _THRESH.get("fear_greed", {}).get("ranges", [[0, 15, 10], [15, 30, 8], [30, 45, 5], [45, 55, 3], [55, 75, 1], [75, None, 0]])},
+    "puell_multiple": {"ranges": _THRESH.get("puell_multiple", {}).get("ranges", [[None, 0.4, 10], [0.4, 0.7, 8], [0.7, 1.0, 5], [1.0, 1.5, 3], [1.5, 2.0, 1], [2.0, None, 0]])},
+    "mvrv_zscore": {"ranges": _THRESH.get("mvrv_zscore", {}).get("ranges", [[None, 0, 10], [0, 1.0, 8], [1.0, 2.0, 5], [2.0, 3.0, 3], [3.0, 5.0, 1], [5.0, None, 0]])},
+    "reserve_risk": {"ranges": _THRESH.get("reserve_risk", {}).get("ranges", [[None, 0.002, 10], [0.002, 0.005, 7], [0.005, 0.01, 4], [0.01, 0.02, 2], [0.02, None, 0]])},
+    "rhodl_ratio": {"ranges": _THRESH.get("rhodl_ratio", {}).get("ranges", [[None, 200, 10], [200, 1000, 7], [1000, 5000, 4], [5000, 20000, 1], [20000, None, 0]])},
+    "nupl": {"ranges": _THRESH.get("nupl", {}).get("ranges", [[None, 0, 10], [0, 0.3, 8], [0.3, 0.5, 4], [0.5, 0.75, 1], [0.75, None, 0]])},
+}
+
+RATIO_SCORERS = {
+    "price_vs_200w_sma": {
+        "ranges": _THRESH.get("price_vs_200w_sma", {}).get("ranges", [[None, 0, 10], [0, 30, 7], [30, 60, 5], [60, 100, 2], [100, None, 0]]),
+        "price_key": "btc_price",
+        "ref_key": "200w_sma",
+    },
+    "lth_realized_price": {
+        "ranges": _THRESH.get("lth_realized_price", {}).get("ranges", [[None, 0, 10], [0, 30, 7], [30, 80, 5], [80, 150, 3], [150, None, 1]]),
+        "price_key": "btc_price",
+        "ref_key": "lth_realized_price",
+    },
+}
+
+BACKTEST_METRIC_PANEL = tuple(METRIC_SCORERS) + tuple(RATIO_SCORERS) + ("drawdown",)
+METRIC_MAX_AGE_DAYS = {
+    "fear_greed": 2,
+    "puell_multiple": 7,
+    "mvrv_zscore": 7,
+    "reserve_risk": 7,
+    "rhodl_ratio": 7,
+    "nupl": 7,
+    "btc_price": 3,
+    "btc_price_coingecko": 3,
+    "btc_price_sma": 3,
+    "btc_price_lth": 3,
+    "200w_sma": 7,
+    "lth_realized_price": 7,
+}
+
+DRAWDOWN_RANGES = _THRESH.get("drawdown", {}).get("ranges", [[60, None, 10], [40, 60, 8], [25, 40, 6], [15, 25, 4], [5, 15, 2], [None, 5, 0]])
+
+
+def _score_range(value, ranges):
+    """Score a value using range-based thresholds."""
+    if value is None:
+        return None
+    for low, high, score in ranges:
+        low_ok = low is None or value >= low
+        high_ok = high is None or value < high
+        if low_ok and high_ok:
+            return score
+    return 0
+
+
+def _build_daily_index(history):
+    """Build a dict mapping metric_key -> {date_str: value} for fast lookup."""
+    index = {}
+    for key, data in history.items():
+        if key.startswith("_") or not isinstance(data, dict) or "dates" not in data:
+            continue
+        lookup = {}
+        for d, v in zip(data["dates"], data["values"]):
+            lookup[d] = v
+        index[key] = lookup
+    return index
+
+
+def _get_all_dates(index):
+    """Get sorted union of all dates across all metrics."""
+    all_dates = set()
+    for lookup in index.values():
+        all_dates.update(lookup.keys())
+    return sorted(all_dates)
+
+
+def _last_known_value(lookup, date, max_lookback=0):
+    """Get value for date, or a prior value within an explicit lookback."""
+    if date in lookup:
+        return lookup[date]
+    d = datetime.strptime(date, "%Y-%m-%d")
+    for i in range(1, max_lookback + 1):
+        prev = (d - timedelta(days=i)).strftime("%Y-%m-%d")
+        if prev in lookup:
+            return lookup[prev]
+    return None
+
+
+def _metric_observation(lookup, date, metric_key):
+    """Return value, source date, and age under a metric-specific freshness rule."""
+    max_age = METRIC_MAX_AGE_DAYS.get(metric_key, 0)
+    target = datetime.strptime(date, "%Y-%m-%d")
+    for age in range(max_age + 1):
+        source_date = (target - timedelta(days=age)).strftime("%Y-%m-%d")
+        if source_date in lookup:
+            return lookup[source_date], source_date, age
+    return None, None, None
+
+
+def _compute_ath_series(price_lookup, dates):
+    """Compute running ATH and drawdown for each date."""
+    ath = 0
+    drawdowns = {}
+    for d in dates:
+        p = price_lookup.get(d)
+        if p is None:
+            continue
+        if p > ath:
+            ath = p
+        if ath > 0:
+            drawdowns[d] = ((ath - p) / ath) * 100
+    return drawdowns
+
+
+def _load_ml_artifact():
+    """Load an ML artifact and return it with validation status."""
+    ml_path = _os.path.join(_os.path.dirname(_os.path.dirname(_os.path.abspath(__file__))), "config", "ml_weights.json")
+    try:
+        with open(ml_path) as f:
+            data = _json.load(f)
+        status = validate_ml_artifact(data)
+        if not status["valid"]:
+            log.error("Rejected invalid ML artifact: %s", ", ".join(status["errors"]))
+            return None, status
+        return data, status
+    except Exception as exc:
+        return None, {"valid": False, "errors": [f"load_error:{exc}"]}
+
+
+def _build_ml_backtest_plan(artifact):
+    """Choose OOS fold weights when available; otherwise mark final weights in-sample."""
+    status = validate_ml_artifact(artifact)
+    if not status["valid"]:
+        raise ValueError("invalid ML artifact: " + ", ".join(status["errors"]))
+
+    if status["has_oos_fold_weights"]:
+        folds = []
+        for fold in artifact["cv_results"]["folds"]:
+            start, separator, end = fold["date_ranges"]["validation"].partition(" to ")
+            if not separator:
+                raise ValueError("invalid validation date range")
+            folds.append({
+                "fold": fold.get("fold"),
+                "start": start,
+                "end": end,
+                "weights": fold["weights"],
+            })
+        return {
+            "evaluation_scope": "out_of_sample_validation_folds",
+            "is_out_of_sample": True,
+            "weighting_source": "fold_specific_weights",
+            "folds": folds,
+            "weights": None,
+        }
+
+    return {
+        "evaluation_scope": "in_sample_full_history_weights",
+        "is_out_of_sample": False,
+        "weighting_source": "final_full_history_weights",
+        "folds": [],
+        "weights": artifact["weights"],
+    }
+
+
+def _weights_for_backtest_date(date, plan):
+    """Return date-appropriate weights and fold number for an ML plan."""
+    if plan["is_out_of_sample"]:
+        for fold in plan["folds"]:
+            if fold["start"] <= date <= fold["end"]:
+                return fold["weights"], fold["fold"]
+        return None, None
+    return plan["weights"], None
+
+
+def _load_ml_weights():
+    """Compatibility helper returning valid final weights only."""
+    artifact, _ = _load_ml_artifact()
+    return artifact.get("weights", {}) if artifact else {}
+
+# ML weight key mapping (backtest metric keys -> ML weight keys)
+_BT_ML_KEY_MAP = {
+    "fear_greed": "fear_greed",
+    "puell_multiple": "puell_multiple",
+    "mvrv_zscore": "mvrv_zscore",
+    "reserve_risk": "reserve_risk",
+    "rhodl_ratio": "rhodl_ratio",
+    "nupl": "nupl",
+    "price_vs_200w_sma": "pct_above_200w_sma",
+    "lth_realized_price": "pct_above_lth_rp",
+    "drawdown": "drawdown",
+}
+
+
+def _common_panel_current_score(scored, ml_weights=None):
+    """Recompute the current score using only metrics present historically."""
+    by_key = {
+        metric.get("key"): metric.get("score")
+        for metric in scored.get("metrics", [])
+        if metric.get("key") in BACKTEST_METRIC_PANEL and metric.get("score") is not None
+    }
+    available_keys = [key for key in BACKTEST_METRIC_PANEL if key in by_key]
+    coverage = {
+        "available_count": len(available_keys),
+        "panel_count": len(BACKTEST_METRIC_PANEL),
+        "available_keys": available_keys,
+    }
+    if not available_keys:
+        return None, coverage
+
+    if ml_weights:
+        weighted = [
+            (by_key[key], ml_weights.get(_BT_ML_KEY_MAP[key], 0.0))
+            for key in available_keys
+        ]
+        weight_total = sum(weight for _, weight in weighted)
+        if weight_total > 0:
+            return round(sum(score * weight for score, weight in weighted) / weight_total * 10, 1), coverage
+
+    return round(sum(by_key[key] for key in available_keys) / len(available_keys) * 10, 1), coverage
+
+
+def _backtest_data_quality_metadata(metric_counts):
+    """Describe historical panel, coverage, and freshness assumptions."""
+    coverage = {
+        "minimum_metrics": min(metric_counts),
+        "maximum_metrics": max(metric_counts),
+        "average_metrics": round(sum(metric_counts) / len(metric_counts), 1),
+        "panel_count": len(BACKTEST_METRIC_PANEL),
+    } if metric_counts else {
+        "minimum_metrics": 0,
+        "maximum_metrics": 0,
+        "average_metrics": 0,
+        "panel_count": len(BACKTEST_METRIC_PANEL),
+    }
+    return {
+        "metric_panel": {
+            "id": "historical-common-v1",
+            "keys": list(BACKTEST_METRIC_PANEL),
+            "count": len(BACKTEST_METRIC_PANEL),
+        },
+        "coverage": coverage,
+        "staleness_days": dict(METRIC_MAX_AGE_DAYS),
+    }
+
+
+def score_day(date, index, drawdowns, ml_weights=None):
+    """Score a single day using all available metrics. Returns (composite_score, details, n_metrics).
+
+    If ml_weights is provided, uses ML-optimized weighting instead of equal weights.
+    details includes both "score" and "raw" (the actual metric value before scoring).
+    """
+    scores = []
+    details = {}
+
+    # Simple range-based metrics
+    for metric_key, cfg in METRIC_SCORERS.items():
+        val, observed_date, age_days = _metric_observation(
+            index.get(metric_key, {}), date, metric_key
+        )
+        if val is not None:
+            s = _score_range(val, cfg["ranges"])
+            if s is not None:
+                scores.append(s)
+                details[metric_key] = {
+                    "value": val,
+                    "score": s,
+                    "raw": val,
+                    "observed_date": observed_date,
+                    "age_days": age_days,
+                }
+
+    # Ratio-based metrics (price vs reference)
+    for metric_key, cfg in RATIO_SCORERS.items():
+        price_val, price_date, price_age = _metric_observation(
+            index.get(cfg["price_key"], {}), date, cfg["price_key"]
+        )
+        # Try alternate price keys, each with an explicit freshness rule.
+        if price_val is None:
+            for pk in ["btc_price_coingecko", "btc_price_sma", "btc_price_lth"]:
+                price_val, price_date, price_age = _metric_observation(index.get(pk, {}), date, pk)
+                if price_val is not None:
+                    break
+        ref_val, ref_date, ref_age = _metric_observation(
+            index.get(cfg["ref_key"], {}), date, cfg["ref_key"]
+        )
+        if price_val is not None and ref_val is not None and ref_val > 0:
+            pct_above = ((price_val - ref_val) / ref_val) * 100
+            s = _score_range(pct_above, cfg["ranges"])
+            if s is not None:
+                scores.append(s)
+                details[metric_key] = {
+                    "value": pct_above,
+                    "score": s,
+                    "raw": pct_above,
+                    "observed_date": min(price_date, ref_date),
+                    "age_days": max(price_age, ref_age),
+                }
+
+    # Drawdown
+    dd = drawdowns.get(date)
+    if dd is not None:
+        s = _score_range(dd, DRAWDOWN_RANGES)
+        if s is not None:
+            scores.append(s)
+            details["drawdown"] = {"value": dd, "score": s, "raw": dd}
+
+    if not scores:
+        return None, details, 0
+
+    if ml_weights:
+        # ML-weighted composite
+        weighted_sum = 0.0
+        weight_total = 0.0
+        for metric_key, info in details.items():
+            ml_key = _BT_ML_KEY_MAP.get(metric_key, metric_key)
+            w = ml_weights.get(ml_key, 0.0)
+            weighted_sum += info["score"] * w
+            weight_total += w
+        if weight_total > 0:
+            composite = weighted_sum / weight_total * 10
+        else:
+            composite = sum(scores) / len(scores) * 10
+    else:
+        composite = sum(scores) / len(scores) * 10
+    return round(composite, 1), details, len(scores)
+
+
+def compute_forward_returns(price_lookup, dates_sorted):
+    """Precompute forward returns for all dates."""
+    periods = [30, 90, 180, 365]
+    returns = {}
+    for d in dates_sorted:
+        p0 = price_lookup.get(d)
+        if p0 is None or p0 <= 0:
+            continue
+        r = {}
+        dt = datetime.strptime(d, "%Y-%m-%d")
+        for days in periods:
+            future = (dt + timedelta(days=days)).strftime("%Y-%m-%d")
+            pf = price_lookup.get(future)
+            if pf is not None:
+                r[f"{days}d"] = round(((pf - p0) / p0) * 100, 2)
+        if r:
+            returns[d] = r
+    return returns
+
+
+def compute_max_drawdown_forward(price_lookup, date, window=90):
+    """Compute max drawdown within N days after a given date."""
+    dt = datetime.strptime(date, "%Y-%m-%d")
+    p0 = price_lookup.get(date)
+    if p0 is None or p0 <= 0:
+        return None
+    peak = p0
+    max_dd = 0
+    for i in range(1, window + 1):
+        future = (dt + timedelta(days=i)).strftime("%Y-%m-%d")
+        pf = price_lookup.get(future)
+        if pf is None:
+            continue
+        if pf > peak:
+            peak = pf
+        dd = ((peak - pf) / peak) * 100
+        if dd > max_dd:
+            max_dd = dd
+    return round(max_dd, 2) if max_dd > 0 else 0
+
+
+def _file_signature(path):
+    """Return a cheap signature that invalidates when an input file changes."""
+    try:
+        stat = os.stat(path)
+        return path, stat.st_mtime_ns, stat.st_size
+    except OSError:
+        return path, None, None
+
+
+def clear_backtest_cache():
+    """Clear memoized backtest results (primarily for explicit refreshes/tests)."""
+    with _BACKTEST_CACHE_LOCK:
+        _BACKTEST_CACHE.clear()
+
+
+def _add_return_statistics(stats, period, returns):
+    """Add return summaries and a moving-block-bootstrap mean interval."""
+    horizon_days = int(period.removesuffix("d"))
+    summary = summarize_returns(
+        returns,
+        block_size=min(horizon_days, len(returns)),
+        n_resamples=400,
+    )
+    stats[f"avg_{period}"] = summary["mean"]
+    stats[f"median_{period}"] = summary["median"]
+    stats[f"win_rate_{period}"] = summary["win_rate"]
+    stats[f"avg_{period}_ci_low"] = summary["mean_ci_low"]
+    stats[f"avg_{period}_ci_high"] = summary["mean_ci_high"]
+    stats[f"max_gain_{period}"] = round(max(returns), 2)
+    stats[f"max_loss_{period}"] = round(min(returns), 2)
+    stats[f"n_{period}"] = summary["n"]
+
+
+def run_backtest(ml_mode=False):
+    """Return an isolated cached result keyed by all material input files."""
+    signature = (
+        bool(ml_mode),
+        _file_signature(HISTORY_PATH),
+        _file_signature(_THRESH_PATH),
+        _file_signature(ML_WEIGHTS_PATH),
+        _file_signature(CACHE_PATH),
+    )
+    with _BACKTEST_CACHE_LOCK:
+        cached = _BACKTEST_CACHE.get(signature)
+    if cached is not None:
+        return copy.deepcopy(cached)
+
+    result = _compute_backtest(ml_mode=ml_mode)
+    with _BACKTEST_CACHE_LOCK:
+        _BACKTEST_CACHE[signature] = copy.deepcopy(result)
+        while len(_BACKTEST_CACHE) > _BACKTEST_CACHE_LIMIT:
+            _BACKTEST_CACHE.pop(next(iter(_BACKTEST_CACHE)))
+    return copy.deepcopy(result)
+
+
+def _compute_backtest(ml_mode=False):
+    """Run the full backtest and return comprehensive results.
+
+    If ml_mode=True, uses ML-optimized metric weights instead of equal weights.
+    """
+    log.info("Loading historical data... (ml_mode=%s)", ml_mode)
+    if not os.path.exists(HISTORY_PATH):
+        return {"error": "No historical data found. Run history collector first."}
+
+    with open(HISTORY_PATH) as f:
+        history = json.load(f)
+
+    index = _build_daily_index(history)
+
+    # Build price lookup (prefer coingecko for completeness)
+    price_lookup = {}
+    for pk in ["btc_price_coingecko", "btc_price", "btc_price_sma", "btc_price_lth"]:
+        if pk in index:
+            for d, v in index[pk].items():
+                if d not in price_lookup:
+                    price_lookup[d] = v
+
+    all_dates = _get_all_dates(index)
+    if not all_dates:
+        return {"error": "No date data available."}
+
+    log.info("Date range: %s to %s (%d days)", all_dates[0], all_dates[-1], len(all_dates))
+
+    # Compute drawdowns
+    drawdowns = _compute_ath_series(price_lookup, all_dates)
+
+    # Precompute forward returns
+    log.info("Computing forward returns...")
+    fwd_returns = compute_forward_returns(price_lookup, all_dates)
+
+    # Build an explicit evaluation plan. Fold-specific validation weights are OOS;
+    # final weights fitted on full history are never represented as OOS.
+    ml_plan = None
+    ml_artifact = None
+    ml_artifact_status = None
+    if ml_mode:
+        ml_artifact, ml_artifact_status = _load_ml_artifact()
+        if ml_artifact:
+            ml_plan = _build_ml_backtest_plan(ml_artifact)
+        else:
+            log.warning("ML mode requested with invalid artifact — falling back to equal weights")
+
+    # Score each day
+    log.info("Scoring %d days...", len(all_dates))
+    daily_scores = []
+    for d in all_dates:
+        ml_weights = None
+        ml_fold = None
+        if ml_plan:
+            ml_weights, ml_fold = _weights_for_backtest_date(d, ml_plan)
+            if ml_plan["is_out_of_sample"] and ml_weights is None:
+                continue
+        composite, details, n_metrics = score_day(d, index, drawdowns, ml_weights=ml_weights)
+        if composite is not None and n_metrics >= 3:  # Require at least 3 metrics
+            price = price_lookup.get(d)
+            # Collect raw metric values for per-metric historical exploration
+            metric_values = {}
+            for mk, info in details.items():
+                raw = info.get("raw")
+                if raw is not None:
+                    metric_values[mk] = round(raw, 6) if isinstance(raw, float) else raw
+            entry = {
+                "date": d,
+                "score": composite,
+                "n_metrics": n_metrics,
+                "price": price,
+                "forward_returns": fwd_returns.get(d, {}),
+                "metric_values": metric_values,
+            }
+            if ml_fold is not None:
+                entry["ml_fold"] = ml_fold
+            daily_scores.append(entry)
+
+    if not daily_scores:
+        return {"error": "No scored days (insufficient metric overlap)."}
+
+    log.info("Scored %d days with 3+ metrics", len(daily_scores))
+
+    # --- Bracket statistics ---
+    bracket_stats = []
+    for low, high, label in BRACKETS:
+        days_in = [d for d in daily_scores if score_in_bracket(d["score"], (low, high, label))]
+        if not days_in:
+            bracket_stats.append({
+                "range": f"{low}-{high}", "label": label, "days": 0,
+            })
+            continue
+
+        stats = {"range": f"{low}-{high}", "label": label, "days": len(days_in)}
+        for period in ["30d", "90d", "180d", "365d"]:
+            returns = [d["forward_returns"][period] for d in days_in if period in d["forward_returns"]]
+            if returns:
+                _add_return_statistics(stats, period, returns)
+
+        # Average max drawdown within 90 days
+        dd_list = []
+        for d in days_in:
+            dd = compute_max_drawdown_forward(price_lookup, d["date"], 90)
+            if dd is not None:
+                dd_list.append(dd)
+        if dd_list:
+            stats["avg_max_drawdown_90d"] = round(sum(dd_list) / len(dd_list), 2)
+
+        bracket_stats.append(stats)
+
+    # --- Peak signal events ---
+    signal_events = []
+    thresholds = [90, 80, 70]
+    for thresh in thresholds:
+        prev_score = 0
+        for d in daily_scores:
+            if d["score"] >= thresh and prev_score < thresh:
+                event = {
+                    "date": d["date"],
+                    "score": d["score"],
+                    "threshold": thresh,
+                    "price": d["price"],
+                    "forward_returns": d["forward_returns"],
+                }
+                # Add future prices
+                if d["price"]:
+                    dt = datetime.strptime(d["date"], "%Y-%m-%d")
+                    for days_ahead in [30, 90, 365]:
+                        future = (dt + timedelta(days=days_ahead)).strftime("%Y-%m-%d")
+                        fp = price_lookup.get(future)
+                        if fp:
+                            event[f"price_{days_ahead}d"] = round(fp, 2)
+                signal_events.append(event)
+            prev_score = d["score"]
+
+    signal_events.sort(key=lambda e: e["date"])
+
+    # --- Current signal context ---
+    all_scores_list = [d["score"] for d in daily_scores]
+    all_scores_list.sort()
+
+    # Get current score from cache, recomputed on the common historical panel.
+    current_score = None
+    current_price = None
+    current_coverage = None
+    if os.path.exists(CACHE_PATH):
+        try:
+            with open(CACHE_PATH) as f:
+                cache = json.load(f)
+            scored = cache.get("_scored", {})
+            current_ml_weights = ml_artifact.get("weights") if ml_mode and ml_artifact else None
+            current_score, current_coverage = _common_panel_current_score(scored, current_ml_weights)
+            current_price = cache.get("price", {}).get("price")
+        except Exception:
+            pass
+
+    # If no comparable cache panel is available, use latest historical score.
+    if current_score is None and daily_scores:
+        current_score = daily_scores[-1]["score"]
+        current_price = daily_scores[-1].get("price")
+        current_coverage = {
+            "available_count": daily_scores[-1]["n_metrics"],
+            "panel_count": len(BACKTEST_METRIC_PANEL),
+            "available_keys": list(daily_scores[-1].get("metric_values", {})),
+        }
+
+    current_context = None
+    if current_score is not None:
+        # Percentile
+        below = len([s for s in all_scores_list if s <= current_score])
+        percentile = round(below / len(all_scores_list) * 100, 1)
+
+        # Find comparable historical periods
+        comparable = []
+        margin = 5
+        for d in daily_scores:
+            if abs(d["score"] - current_score) <= margin and d["forward_returns"]:
+                comparable.append(d)
+
+        avg_returns = {}
+        if comparable:
+            for period in ["30d", "90d", "180d", "365d"]:
+                vals = [d["forward_returns"][period] for d in comparable if period in d["forward_returns"]]
+                if vals:
+                    avg_returns[period] = round(sum(vals) / len(vals), 2)
+        avg_1yr = avg_returns.get("365d")
+
+        # Best comparable examples — one per market cycle for diversity
+        # Cycles: pre-2016, 2016-2017 bull, 2018-2019 bear, 2020-2021 bull, 2022-2023 bear, 2024+
+        cycle_bins = [
+            ("pre-2016", "2010-01-01", "2015-12-31"),
+            ("2016-17 Bull", "2016-01-01", "2017-12-31"),
+            ("2018-19 Bear", "2018-01-01", "2019-12-31"),
+            ("2020-21 Bull", "2020-01-01", "2021-12-31"),
+            ("2022-23 Bear", "2022-01-01", "2023-12-31"),
+            ("2024+", "2024-01-01", "2099-12-31"),
+        ]
+        examples = []
+        used_cycles = set()
+        # Sort comparable by closest score first, then pick one per cycle
+        sorted_comp = sorted(comparable, key=lambda d: abs(d["score"] - current_score))
+        for d in sorted_comp:
+            cycle_label = None
+            for label, start, end in cycle_bins:
+                if start <= d["date"] <= end:
+                    cycle_label = label
+                    break
+            if cycle_label and cycle_label not in used_cycles:
+                used_cycles.add(cycle_label)
+                examples.append({
+                    "date": d["date"],
+                    "score": d["score"],
+                    "price": d["price"],
+                    "forward_returns": d["forward_returns"],
+                    "cycle": cycle_label,
+                })
+            if len(examples) >= 6:
+                break
+        # Sort examples chronologically
+        examples.sort(key=lambda d: d["date"])
+
+        current_context = {
+            "current_score": current_score,
+            "current_price": current_price,
+            "score_version": SCORE_VERSION,
+            "metric_panel_id": "historical-common-v1",
+            "coverage": current_coverage,
+            "current_weighting_source": (
+                "final_full_history_weights" if ml_mode and ml_artifact else "equal_weight"
+            ),
+            "percentile": percentile,
+            "comparable_days": len(comparable),
+            "avg_1yr_return": avg_1yr,
+            "avg_30d_return": avg_returns.get("30d"),
+            "avg_90d_return": avg_returns.get("90d"),
+            "avg_180d_return": avg_returns.get("180d"),
+            "examples": examples,
+        }
+
+    # --- Build time series for charting ---
+    # Smart downsampling: daily for last 2 years, weekly before that
+    # Include per-metric values so the frontend can plot any metric.
+    chart_data = []
+    import datetime as _dt
+    try:
+        last_date = _dt.datetime.strptime(daily_scores[-1]["date"], "%Y-%m-%d")
+        cutoff_date = (last_date - _dt.timedelta(days=730)).strftime("%Y-%m-%d")
+    except Exception:
+        cutoff_date = "2024-01-01"
+
+    # Collect all metric keys that were ever scored (for per-metric series)
+    all_metric_keys = set()
+    for d in daily_scores:
+        all_metric_keys.update(d.get("metric_values", {}).keys())
+
+    for i, d in enumerate(daily_scores):
+        is_recent = d["date"] >= cutoff_date
+        if is_recent or i % 7 == 0 or i == len(daily_scores) - 1:
+            entry = {
+                "date": d["date"],
+                "score": d["score"],
+                "price": d["price"],
+            }
+            # Include per-metric values (raw metric value, not score)
+            metric_vals = d.get("metric_values", {})
+            if metric_vals:
+                entry["metric_values"] = metric_vals
+            chart_data.append(entry)
+
+    if not ml_mode:
+        ml_evaluation = {"requested": False, "is_out_of_sample": False}
+    elif ml_plan:
+        ml_evaluation = {
+            "requested": True,
+            "evaluation_scope": ml_plan["evaluation_scope"],
+            "is_out_of_sample": ml_plan["is_out_of_sample"],
+            "weighting_source": ml_plan["weighting_source"],
+            "folds": len(ml_plan["folds"]),
+            "artifact": ml_artifact_status,
+        }
+    else:
+        ml_evaluation = {
+            "requested": True,
+            "evaluation_scope": "equal_weight_fallback",
+            "is_out_of_sample": False,
+            "weighting_source": "none_invalid_artifact",
+            "folds": 0,
+            "artifact": ml_artifact_status,
+        }
+
+    data_quality = _backtest_data_quality_metadata([day["n_metrics"] for day in daily_scores])
+    result = {
+        "date_range": {"start": daily_scores[0]["date"], "end": daily_scores[-1]["date"]},
+        "total_days_scored": len(daily_scores),
+        "metric_panel": data_quality["metric_panel"],
+        "coverage": data_quality["coverage"],
+        "staleness_days": data_quality["staleness_days"],
+        "bracket_stats": bracket_stats,
+        "signal_events": signal_events,
+        "current_context": current_context,
+        "chart_data": chart_data,
+        "ml_mode": ml_mode,
+        "ml_evaluation": ml_evaluation,
+        "score_version": SCORE_VERSION,
+        "computed_at": datetime.utcnow().isoformat() + "Z",
+    }
+
+    log.info("Backtest complete: %d days, %d signal events", len(daily_scores), len(signal_events))
+    return result

backtesting/statistics.py

@@ -0,0 +1,85 @@
+"""Statistical helpers for honest time-series backtest reporting."""
+
+from __future__ import annotations
+
+import math
+import random
+import statistics as stdlib_statistics
+from collections.abc import Iterable
+
+
+def _quantile(sorted_values: list[float], probability: float) -> float:
+    position = (len(sorted_values) - 1) * probability
+    lower = math.floor(position)
+    upper = math.ceil(position)
+    if lower == upper:
+        return sorted_values[lower]
+    fraction = position - lower
+    return sorted_values[lower] * (1 - fraction) + sorted_values[upper] * fraction
+
+
+def moving_block_bootstrap_ci(
+    values: Iterable[float],
+    *,
+    block_size: int = 30,
+    n_resamples: int = 1_000,
+    confidence: float = 0.95,
+    seed: int = 42,
+) -> dict[str, float | int]:
+    """Estimate a mean and CI while preserving local serial dependence."""
+    series = [float(value) for value in values]
+    if not series:
+        raise ValueError("values must not be empty")
+    if block_size < 1 or block_size > len(series):
+        raise ValueError("block_size must be between 1 and the number of values")
+    if n_resamples < 2:
+        raise ValueError("n_resamples must be at least 2")
+    if not 0 < confidence < 1:
+        raise ValueError("confidence must be between 0 and 1")
+
+    rng = random.Random(seed)
+    sample_means: list[float] = []
+    final_start = len(series) - block_size
+    for _ in range(n_resamples):
+        sample: list[float] = []
+        while len(sample) < len(series):
+            start = rng.randint(0, final_start)
+            sample.extend(series[start:start + block_size])
+        sample = sample[:len(series)]
+        sample_means.append(sum(sample) / len(sample))
+
+    sample_means.sort()
+    tail = (1 - confidence) / 2
+    return {
+        "estimate": sum(series) / len(series),
+        "ci_low": _quantile(sample_means, tail),
+        "ci_high": _quantile(sample_means, 1 - tail),
+        "n": len(series),
+    }
+
+
+def summarize_returns(
+    values: Iterable[float],
+    *,
+    block_size: int = 30,
+    n_resamples: int = 1_000,
+    confidence: float = 0.95,
+    seed: int = 42,
+) -> dict[str, float | int]:
+    """Summarize realized returns with an autocorrelation-aware mean CI."""
+    series = [float(value) for value in values]
+    interval = moving_block_bootstrap_ci(
+        series,
+        block_size=min(block_size, len(series)),
+        n_resamples=n_resamples,
+        confidence=confidence,
+        seed=seed,
+    )
+    return {
+        "n": len(series),
+        "mean": round(float(interval["estimate"]), 2),
+        "median": round(stdlib_statistics.median(series), 2),
+        "win_rate": round(sum(value > 0 for value in series) / len(series) * 100, 1),
+        "mean_ci_low": round(float(interval["ci_low"]), 2),
+        "mean_ci_high": round(float(interval["ci_high"]), 2),
+    }

config/best_config.json

@@ -0,0 +1,73 @@
+{
+  "model_type": "xgboost",
+  "features": {
+    "use_price_position": true,
+    "use_momentum": true,
+    "use_volatility": true,
+    "use_volume": true,
+    "use_cycle": true,
+    "use_pca": false,
+    "pca_variance": 0.95,
+    "use_scaler": true
+  },
+  "target": {
+    "type": "regression",
+    "forward_periods_1h": [
+      168,
+      720,
+      2160
+    ],
+    "forward_periods_4h": [
+      42,
+      180,
+      540
+    ],
+    "weights": [
+      0.2,
+      0.3,
+      0.5
+    ],
+    "return_scales_pct": [
+      10.0,
+      30.0,
+      60.0
+    ],
+    "score_range": [
+      0,
+      100
+    ]
+  },
+  "hyperparameters": {
+    "learning_rate": 0.01,
+    "max_depth": 4,
+    "n_estimators": 300,
+    "subsample": 0.8,
+    "colsample_bytree": 0.8,
+    "min_child_weight": 20,
+    "gamma": 0.3,
+    "reg_alpha": 0.5,
+    "reg_lambda": 3.0,
+    "lstm_hidden_size": 128,
+    "lstm_num_layers": 2,
+    "lstm_dropout": 0.3,
+    "lstm_epochs": 100,
+    "lstm_batch_size": 64,
+    "lstm_sequence_length": 30,
+    "lstm_patience": 10
+  },
+  "strategy": {
+    "strong_buy_threshold": 65,
+    "good_buy_threshold": 55,
+    "poor_threshold": 35
+  },
+  "training": {
+    "rolling_window": true,
+    "rolling_train_size": 2500,
+    "rolling_test_size": 300,
+    "walk_forward_windows": 5,
+    "train_pct": 0.7,
+    "validation_pct": 0.3,
+    "test_pct": 0.15
+  },
+  "timeframe": "4h"
+}

config/current_config.json

@@ -0,0 +1,73 @@
+{
+  "model_type": "xgboost",
+  "features": {
+    "use_price_position": true,
+    "use_momentum": true,
+    "use_volatility": true,
+    "use_volume": true,
+    "use_cycle": true,
+    "use_pca": false,
+    "pca_variance": 0.85,
+    "use_scaler": true
+  },
+  "target": {
+    "type": "regression",
+    "forward_periods_1h": [
+      168,
+      720,
+      2160
+    ],
+    "forward_periods_4h": [
+      42,
+      180,
+      540
+    ],
+    "weights": [
+      0.2,
+      0.3,
+      0.5
+    ],
+    "return_scales_pct": [
+      10.0,
+      30.0,
+      60.0
+    ],
+    "score_range": [
+      0,
+      100
+    ]
+  },
+  "hyperparameters": {
+    "learning_rate": 0.005,
+    "max_depth": 5,
+    "n_estimators": 800,
+    "subsample": 0.7,
+    "colsample_bytree": 0.7,
+    "min_child_weight": 15,
+    "gamma": 0.5,
+    "reg_alpha": 0.3,
+    "reg_lambda": 1.0,
+    "lstm_hidden_size": 64,
+    "lstm_num_layers": 2,
+    "lstm_dropout": 0.4,
+    "lstm_epochs": 80,
+    "lstm_batch_size": 64,
+    "lstm_sequence_length": 30,
+    "lstm_patience": 15
+  },
+  "strategy": {
+    "strong_buy_threshold": 55,
+    "good_buy_threshold": 35,
+    "poor_threshold": 20
+  },
+  "training": {
+    "rolling_window": true,
+    "rolling_train_size": 3500,
+    "rolling_test_size": 300,
+    "walk_forward_windows": 5,
+    "train_pct": 0.7,
+    "validation_pct": 0.3,
+    "test_pct": 0.15
+  },
+  "timeframe": "4h"
+}

config/initial_config.json

@@ -1,59 +1,73 @@
 {
   "model_type": "xgboost",
   "features": {
-    "technical_indicators": [
+    "use_price_position": true,
-      "RSI_14", "RSI_7", "RSI_21",
+    "use_momentum": true,
-      "MACD_line", "MACD_signal", "MACD_hist",
+    "use_volatility": true,
-      "BB_upper", "BB_lower", "BB_width",
+    "use_volume": true,
-      "ATR_14",
+    "use_cycle": true,
-      "SMA_5", "SMA_10", "SMA_20", "SMA_50", "SMA_200",
+    "use_pca": false,
-      "EMA_5", "EMA_10", "EMA_20", "EMA_50",
+    "pca_variance": 0.95,
-      "OBV",
+    "use_scaler": true
-      "stoch_k", "stoch_d",
-      "williams_r",
-      "CCI_20",
-      "ROC_10",
-      "keltner_upper", "keltner_lower"
-    ],
-    "lookback_periods": [3, 5, 10, 20],
-    "use_volume_features": true,
-    "use_volatility_features": true,
-    "use_candle_patterns": true,
-    "use_lag_features": true,
-    "lag_periods": [1, 2, 3, 5]
   },
   "target": {
-    "type": "classification",
+    "type": "regression",
-    "direction": "long",
+    "forward_periods_1h": [
-    "horizon_candles": 6,
+      168,
-    "threshold_pct": 1.0
+      720,
+      2160
+    ],
+    "forward_periods_4h": [
+      42,
+      180,
+      540
+    ],
+    "weights": [
+      0.2,
+      0.3,
+      0.5
+    ],
+    "return_scales_pct": [
+      10.0,
+      30.0,
+      60.0
+    ],
+    "score_range": [
+      0,
+      100
+    ]
   },
   "hyperparameters": {
-    "learning_rate": 0.05,
+    "learning_rate": 0.01,
-    "max_depth": 6,
+    "max_depth": 4,
-    "n_estimators": 500,
+    "n_estimators": 300,
     "subsample": 0.8,
     "colsample_bytree": 0.8,
-    "min_child_weight": 5,
+    "min_child_weight": 20,
-    "gamma": 0.1,
+    "gamma": 0.3,
-    "reg_alpha": 0.1,
+    "reg_alpha": 0.5,
-    "reg_lambda": 1.0
+    "reg_lambda": 3.0,
+    "lstm_hidden_size": 128,
+    "lstm_num_layers": 2,
+    "lstm_dropout": 0.3,
+    "lstm_epochs": 100,
+    "lstm_batch_size": 64,
+    "lstm_sequence_length": 30,
+    "lstm_patience": 10
   },
   "strategy": {
-    "entry_threshold": 0.60,
+    "strong_buy_threshold": 65,
-    "exit_type": "trailing_stop",
+    "good_buy_threshold": 55,
-    "stop_loss_pct": 2.0,
+    "poor_threshold": 35
-    "take_profit_pct": 4.0,
-    "trailing_stop_pct": 1.5,
-    "position_sizing": "confidence_scaled",
-    "max_position_pct": 100,
-    "min_confidence_to_trade": 0.55
   },
   "training": {
+    "rolling_window": true,
+    "rolling_train_size": 2500,
+    "rolling_test_size": 300,
     "walk_forward_windows": 5,
     "train_pct": 0.7,
-    "validation_pct": 0.15,
+    "validation_pct": 0.3,
     "test_pct": 0.15
   },
   "timeframe": "4h"
-}
+}

config/llm_settings.example.json

@@ -0,0 +1,21 @@
+{
+  "provider": "ollama",
+  "model": "qwen3.5:27b",
+  "providers": {
+    "ollama": {
+      "base_url": "http://127.0.0.1:11434"
+    },
+    "lmstudio": {
+      "base_url": "http://127.0.0.1:1234"
+    },
+    "openai": {
+      "api_key": ""
+    },
+    "anthropic": {
+      "api_key": ""
+    },
+    "openrouter": {
+      "api_key": ""
+    }
+  }
+}

config/ml_weights.json

@@ -0,0 +1,383 @@
+{
+  "artifact_schema_version": 2,
+  "score_version": "accumulation-score-v2",
+  "weights": {
+    "pct_above_200w_sma": 0.5075,
+    "drawdown": 0.1716,
+    "pct_above_lth_rp": 0.0934,
+    "rhodl_ratio": 0.08,
+    "fear_greed": 0.0459,
+    "puell_multiple": 0.0384,
+    "reserve_risk": 0.0335,
+    "mvrv_zscore": 0.018,
+    "nupl": 0.0116
+  },
+  "feature_importances": {
+    "raw_pct_above_200w_sma": 0.448165,
+    "days_since_ath": 0.138053,
+    "raw_pct_above_lth_rp": 0.093426,
+    "raw_rhodl_ratio": 0.080038,
+    "score_pct_above_200w_sma": 0.059336,
+    "raw_fear_greed": 0.04573,
+    "puell_x_reserve": 0.034997,
+    "raw_drawdown": 0.033552,
+    "raw_puell_multiple": 0.02023,
+    "raw_reserve_risk": 0.012477,
+    "raw_mvrv_zscore": 0.011366,
+    "mvrv_x_nupl": 0.008204,
+    "raw_nupl": 0.003843,
+    "delta_30d_nupl": 0.003624,
+    "delta_30d_reserve_risk": 0.003541,
+    "delta_30d_mvrv_zscore": 0.002548,
+    "delta_30d_puell_multiple": 0.000677,
+    "score_fear_greed": 0.000182,
+    "score_rhodl_ratio": 1.1e-05,
+    "score_puell_multiple": 0.0,
+    "score_mvrv_zscore": 0.0,
+    "score_reserve_risk": 0.0,
+    "score_nupl": 0.0,
+    "score_drawdown": 0.0,
+    "score_pct_above_lth_rp": 0.0
+  },
+  "cv_results": {
+    "mean_auc": 0.7667,
+    "std_auc": 0.1734,
+    "mean_f1": 0.4085,
+    "mean_precision": 0.3708,
+    "mean_recall": 0.4898,
+    "validation_method": "purged_expanding_window",
+    "label_horizon_days": 365,
+    "folds": [
+      {
+        "fold": 1,
+        "weights": {
+          "pct_above_lth_rp": 0.7418,
+          "drawdown": 0.1523,
+          "reserve_risk": 0.0339,
+          "mvrv_zscore": 0.0211,
+          "puell_multiple": 0.0164,
+          "rhodl_ratio": 0.0141,
+          "nupl": 0.0128,
+          "pct_above_200w_sma": 0.0043,
+          "fear_greed": 0.0033
+        },
+        "metrics": {
+          "auc": 0.7583,
+          "f1": 0.0,
+          "precision": 0.0,
+          "recall": 0.0
+        },
+        "date_ranges": {
+          "train": "2018-02-01 to 2019-08-05",
+          "validation": "2020-08-04 to 2021-10-31"
+        },
+        "n_train": 548,
+        "n_validation": 454
+      },
+      {
+        "fold": 2,
+        "weights": {
+          "reserve_risk": 0.4125,
+          "puell_multiple": 0.3942,
+          "drawdown": 0.085,
+          "pct_above_lth_rp": 0.0429,
+          "pct_above_200w_sma": 0.0324,
+          "rhodl_ratio": 0.0135,
+          "nupl": 0.0131,
+          "mvrv_zscore": 0.0053,
+          "fear_greed": 0.001
+        },
+        "metrics": {
+          "auc": 0.8346,
+          "f1": 0.6582,
+          "precision": 0.4906,
+          "recall": 1.0
+        },
+        "date_ranges": {
+          "train": "2018-02-01 to 2020-11-01",
+          "validation": "2021-11-01 to 2023-01-28"
+        },
+        "n_train": 1002,
+        "n_validation": 454
+      },
+      {
+        "fold": 3,
+        "weights": {
+          "reserve_risk": 0.2919,
+          "puell_multiple": 0.2495,
+          "drawdown": 0.1619,
+          "pct_above_lth_rp": 0.1028,
+          "pct_above_200w_sma": 0.089,
+          "fear_greed": 0.0754,
+          "nupl": 0.0103,
+          "mvrv_zscore": 0.0099,
+          "rhodl_ratio": 0.0091
+        },
+        "metrics": {
+          "auc": 0.9755,
+          "f1": 0.9757,
+          "precision": 0.9926,
+          "recall": 0.9593
+        },
+        "date_ranges": {
+          "train": "2018-02-01 to 2022-01-29",
+          "validation": "2023-01-29 to 2024-04-26"
+        },
+        "n_train": 1456,
+        "n_validation": 454
+      },
+      {
+        "fold": 4,
+        "weights": {
+          "drawdown": 0.6216,
+          "reserve_risk": 0.1302,
+          "puell_multiple": 0.1161,
+          "fear_greed": 0.0554,
+          "rhodl_ratio": 0.0322,
+          "pct_above_lth_rp": 0.0163,
+          "mvrv_zscore": 0.0117,
+          "pct_above_200w_sma": 0.0094,
+          "nupl": 0.0071
+        },
+        "metrics": {
+          "auc": 0.4983,
+          "f1": 0.0,
+          "precision": 0.0,
+          "recall": 0.0
+        },
+        "date_ranges": {
+          "train": "2018-02-01 to 2023-04-28",
+          "validation": "2024-04-27 to 2025-07-26"
+        },
+        "n_train": 1910,
+        "n_validation": 454
+      }
+    ]
+  },
+  "training_info": {
+    "n_samples": 2728,
+    "n_positive": 1554,
+    "positive_rate": 0.5696,
+    "n_features": 25,
+    "target_threshold": 30.0,
+    "date_range": "2018-02-01 to 2025-07-26",
+    "model": "GradientBoostingClassifier"
+  },
+  "provenance": {
+    "validation_method": "purged_expanding_window",
+    "label_horizon_days": 365,
+    "weight_scope": "full_history_fit",
+    "training_date_range": {
+      "start": "2018-02-01",
+      "end": "2025-07-26"
+    },
+    "trained_at": "2026-07-26T23:19:01.231759+00:00"
+  },
+  "comparison": {
+    "equal_weight": [
+      {
+        "range": "0-20",
+        "label": "EXTREME CAUTION",
+        "days": 286,
+        "avg_365d": -7.25,
+        "median_365d": -13.89,
+        "win_rate_365d": 34.3
+      },
+      {
+        "range": "20-35",
+        "label": "CAUTION \u2014 OVERHEATED",
+        "days": 537,
+        "avg_365d": 8.85,
+        "median_365d": -21.49,
+        "win_rate_365d": 35.6
+      },
+      {
+        "range": "35-50",
+        "label": "NEUTRAL",
+        "days": 660,
+        "avg_365d": 85.33,
+        "median_365d": 16.54,
+        "win_rate_365d": 60.8
+      },
+      {
+        "range": "50-65",
+        "label": "MODERATE OPPORTUNITY",
+        "days": 575,
+        "avg_365d": 113.54,
+        "median_365d": 88.64,
+        "win_rate_365d": 88.5
+      },
+      {
+        "range": "65-80",
+        "label": "STRONG ACCUMULATION ZONE",
+        "days": 339,
+        "avg_365d": 183.9,
+        "median_365d": 128.5,
+        "win_rate_365d": 89.7
+      },
+      {
+        "range": "80-100",
+        "label": "EXTREME ACCUMULATION ZONE",
+        "days": 331,
+        "avg_365d": 120.82,
+        "median_365d": 91.85,
+        "win_rate_365d": 99.7
+      }
+    ],
+    "ml_weighted": [
+      {
+        "range": "0-20",
+        "label": "EXTREME CAUTION",
+        "days": 642,
+        "avg_365d": -7.11,
+        "median_365d": -26.48,
+        "win_rate_365d": 25.1
+      },
+      {
+        "range": "20-35",
+        "label": "CAUTION \u2014 OVERHEATED",
+        "days": 679,
+        "avg_365d": 18.61,
+        "median_365d": 4.67,
+        "win_rate_365d": 53.2
+      },
+      {
+        "range": "35-50",
+        "label": "NEUTRAL",
+        "days": 462,
+        "avg_365d": 138.2,
+        "median_365d": 95.1,
+        "win_rate_365d": 87.0
+      },
+      {
+        "range": "50-65",
+        "label": "MODERATE OPPORTUNITY",
+        "days": 277,
+        "avg_365d": 206.31,
+        "median_365d": 163.37,
+        "win_rate_365d": 87.7
+      },
+      {
+        "range": "65-80",
+        "label": "STRONG ACCUMULATION ZONE",
+        "days": 285,
+        "avg_365d": 182.42,
+        "median_365d": 123.75,
+        "win_rate_365d": 99.3
+      },
+      {
+        "range": "80-100",
+        "label": "EXTREME ACCUMULATION ZONE",
+        "days": 383,
+        "avg_365d": 118.93,
+        "median_365d": 119.03,
+        "win_rate_365d": 100.0
+      }
+    ]
+  },
+  "out_of_sample_comparison": {
+    "folds": 4,
+    "validation_days": 1816,
+    "equal_weight": [
+      {
+        "range": "0-20",
+        "label": "EXTREME CAUTION",
+        "days": 281,
+        "avg_365d": -6.2,
+        "median_365d": -13.66,
+        "win_rate_365d": 34.9
+      },
+      {
+        "range": "20-35",
+        "label": "CAUTION \u2014 OVERHEATED",
+        "days": 432,
+        "avg_365d": 20.09,
+        "median_365d": -14.6,
+        "win_rate_365d": 43.1
+      },
+      {
+        "range": "35-50",
+        "label": "NEUTRAL",
+        "days": 419,
+        "avg_365d": 71.09,
+        "median_365d": -9.19,
+        "win_rate_365d": 48.9
+      },
+      {
+        "range": "50-65",
+        "label": "MODERATE OPPORTUNITY",
+        "days": 255,
+        "avg_365d": 101.47,
+        "median_365d": 103.63,
+        "win_rate_365d": 76.9
+      },
+      {
+        "range": "65-80",
+        "label": "STRONG ACCUMULATION ZONE",
+        "days": 237,
+        "avg_365d": 100.95,
+        "median_365d": 122.53,
+        "win_rate_365d": 85.2
+      },
+      {
+        "range": "80-100",
+        "label": "EXTREME ACCUMULATION ZONE",
+        "days": 192,
+        "avg_365d": 76.91,
+        "median_365d": 51.03,
+        "win_rate_365d": 99.5
+      }
+    ],
+    "ml_weighted": [
+      {
+        "range": "0-20",
+        "label": "EXTREME CAUTION",
+        "days": 371,
+        "avg_365d": 9.72,
+        "median_365d": -19.7,
+        "win_rate_365d": 31.3
+      },
+      {
+        "range": "20-35",
+        "label": "CAUTION \u2014 OVERHEATED",
+        "days": 372,
+        "avg_365d": 13.85,
+        "median_365d": -10.98,
+        "win_rate_365d": 42.2
+      },
+      {
+        "range": "35-50",
+        "label": "NEUTRAL",
+        "days": 263,
+        "avg_365d": 107.81,
+        "median_365d": 76.78,
+        "win_rate_365d": 59.7
+      },
+      {
+        "range": "50-65",
+        "label": "MODERATE OPPORTUNITY",
+        "days": 348,
+        "avg_365d": 64.05,
+        "median_365d": 99.55,
+        "win_rate_365d": 64.9
+      },
+      {
+        "range": "65-80",
+        "label": "STRONG ACCUMULATION ZONE",
+        "days": 223,
+        "avg_365d": 121.71,
+        "median_365d": 126.27,
+        "win_rate_365d": 93.7
+      },
+      {
+        "range": "80-100",
+        "label": "EXTREME ACCUMULATION ZONE",
+        "days": 239,
+        "avg_365d": 61.69,
+        "median_365d": 42.02,
+        "win_rate_365d": 89.1
+      }
+    ]
+  },
+  "trained_at": "2026-07-26T23:19:01.231759+00:00"
+}

config/thresholds.json

@@ -0,0 +1,43 @@
+{
+  "_comment": "Cycle-aware thresholds — widened ranges to account for BTC maturing and diminishing cycle extremes",
+  "fear_greed": {
+    "ranges": [[0, 15, 10], [15, 30, 8], [30, 45, 5], [45, 55, 3], [55, 75, 1], [75, 100, 0]]
+  },
+  "puell_multiple": {
+    "_note": "Post-halving floors rising: 2016=0.15, 2020=0.3, 2024=0.5+",
+    "ranges": [[null, 0.4, 10], [0.4, 0.7, 8], [0.7, 1.0, 5], [1.0, 1.5, 3], [1.5, 2.0, 1], [2.0, null, 0]]
+  },
+  "mvrv_zscore": {
+    "_note": "Bottoms getting shallower: 2015=-0.6, 2018=-0.4, 2022=-0.3, next may be ~0",
+    "ranges": [[null, 0, 10], [0, 1.0, 8], [1.0, 2.0, 5], [2.0, 3.0, 3], [3.0, 5.0, 1], [5.0, null, 0]]
+  },
+  "drawdown": {
+    "_note": "Drawdowns compressing: 2014=86%, 2018=84%, 2022=77%, future may max at 50-60%",
+    "ranges": [[60, null, 10], [40, 60, 8], [25, 40, 6], [15, 25, 4], [5, 15, 2], [null, 5, 0]]
+  },
+  "price_vs_200w_sma": {
+    "_note": "BTC spends more time above 200W SMA as it matures",
+    "ranges": [[null, 0, 10], [0, 30, 7], [30, 60, 5], [60, 100, 2], [100, null, 0]]
+  },
+  "reserve_risk": {
+    "ranges": [[null, 0.002, 10], [0.002, 0.005, 7], [0.005, 0.01, 4], [0.01, 0.02, 2], [0.02, null, 0]]
+  },
+  "rhodl_ratio": {
+    "_note": "RHODL baseline rising with institutional adoption",
+    "ranges": [[null, 200, 10], [200, 1000, 7], [1000, 5000, 4], [5000, 20000, 1], [20000, null, 0]]
+  },
+  "nupl": {
+    "_note": "NUPL bottoms getting shallower as BTC matures",
+    "ranges": [[null, 0, 10], [0, 0.3, 8], [0.3, 0.5, 4], [0.5, 0.75, 1], [0.75, null, 0]]
+  },
+  "lth_realized_price": {
+    "_note": "Price stays further above LTH RP as BTC matures — 60% above is still a good entry in 2024+",
+    "ranges": [[null, 0, 10], [0, 30, 7], [30, 80, 5], [80, 150, 3], [150, null, 1]]
+  },
+  "hash_ribbons": {
+    "buy_signal": 10,
+    "recent_recovery": 6,
+    "normal": 3,
+    "euphoria": 0
+  }
+}

dashboard/jobs.py

@@ -0,0 +1,111 @@
+"""Persistent, thread-safe background job state."""
+
+from __future__ import annotations
+
+import threading
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable
+
+from dashboard.persistence import atomic_write_json, load_json
+
+_ACTIVE = {"queued", "running"}
+
+
+def _now() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+class JobRegistry:
+    """Reserve jobs before spawning and persist their lifecycle."""
+
+    def __init__(self, path: str | Path, *, history_limit: int = 100):
+        self.path = Path(path)
+        self.history_limit = history_limit
+        self._lock = threading.RLock()
+        loaded = load_json(self.path, {"jobs": []}) or {"jobs": []}
+        self._jobs = {
+            job["id"]: dict(job)
+            for job in loaded.get("jobs", [])
+            if isinstance(job, dict) and job.get("id")
+        }
+        changed = False
+        for job in self._jobs.values():
+            if job.get("status") in _ACTIVE:
+                job.update(
+                    status="interrupted",
+                    finished_at=_now(),
+                    error="process restarted before job completed",
+                )
+                changed = True
+        if changed:
+            self._save_locked()
+
+    def _save_locked(self) -> None:
+        jobs = sorted(self._jobs.values(), key=lambda job: job.get("created_at", ""))
+        if len(jobs) > self.history_limit:
+            keep = jobs[-self.history_limit :]
+            self._jobs = {job["id"]: job for job in keep}
+            jobs = keep
+        atomic_write_json(self.path, {"jobs": jobs})
+
+    def reserve(self, kind: str, *, details: dict[str, Any] | None = None) -> dict[str, Any] | None:
+        with self._lock:
+            if any(
+                job.get("kind") == kind and job.get("status") in _ACTIVE
+                for job in self._jobs.values()
+            ):
+                return None
+            job = {
+                "id": uuid.uuid4().hex,
+                "kind": kind,
+                "status": "queued",
+                "created_at": _now(),
+                "started_at": None,
+                "finished_at": None,
+                "progress": {},
+                "details": details or {},
+                "result": None,
+                "error": None,
+            }
+            self._jobs[job["id"]] = job
+            self._save_locked()
+            return dict(job)
+
+    def get(self, job_id: str) -> dict[str, Any] | None:
+        with self._lock:
+            job = self._jobs.get(job_id)
+            return dict(job) if job else None
+
+    def active(self, kind: str) -> dict[str, Any] | None:
+        with self._lock:
+            for job in self._jobs.values():
+                if job.get("kind") == kind and job.get("status") in _ACTIVE:
+                    return dict(job)
+        return None
+
+    def update_progress(self, job_id: str, progress: dict[str, Any]) -> None:
+        with self._lock:
+            job = self._jobs[job_id]
+            job["progress"] = dict(progress)
+            self._save_locked()
+
+    def run(self, job_id: str, operation: Callable[[], Any]) -> Any:
+        with self._lock:
+            job = self._jobs[job_id]
+            if job["status"] != "queued":
+                raise RuntimeError(f"job {job_id} is not queued")
+            job.update(status="running", started_at=_now())
+            self._save_locked()
+        try:
+            result = operation()
+        except Exception as exc:
+            with self._lock:
+                job.update(status="error", error=str(exc), finished_at=_now())
+                self._save_locked()
+            raise
+        with self._lock:
+            job.update(status="complete", result=result, finished_at=_now())
+            self._save_locked()
+        return result

dashboard/persistence.py

@@ -0,0 +1,222 @@
+"""Small, dependency-free persistence primitives for dashboard state."""
+
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+import threading
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Iterator
+
+try:
+    import fcntl
+except ImportError:  # pragma: no cover - Windows fallback uses the process lock
+    fcntl = None
+
+
+_LOCKS: dict[str, threading.RLock] = {}
+_LOCKS_GUARD = threading.Lock()
+_METADATA_KEYS = {"observed_at", "source", "stale", "last_error", "error"}
+
+
+def _thread_lock(path: Path) -> threading.RLock:
+    key = str(path.resolve())
+    with _LOCKS_GUARD:
+        return _LOCKS.setdefault(key, threading.RLock())
+
+
+@contextmanager
+def file_lock(path: str | os.PathLike[str]) -> Iterator[None]:
+    """Serialize readers/writers across threads and, on POSIX, processes."""
+    target = Path(path)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    lock_path = target.with_name(f".{target.name}.lock")
+    with _thread_lock(target):
+        with lock_path.open("a+b") as lock_file:
+            if fcntl is not None:
+                fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX)
+            try:
+                yield
+            finally:
+                if fcntl is not None:
+                    fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
+
+
+def atomic_write_json(path: str | os.PathLike[str], data: Any, *, indent: int = 2) -> None:
+    """Durably replace a JSON file without exposing a partial document."""
+    target = Path(path)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    with file_lock(target):
+        fd, temporary = tempfile.mkstemp(
+            prefix=f".{target.name}.", suffix=".tmp", dir=target.parent
+        )
+        try:
+            with os.fdopen(fd, "w", encoding="utf-8") as handle:
+                json.dump(data, handle, indent=indent, default=str)
+                handle.write("\n")
+                handle.flush()
+                os.fsync(handle.fileno())
+            os.replace(temporary, target)
+            try:
+                directory_fd = os.open(target.parent, os.O_DIRECTORY)
+                try:
+                    os.fsync(directory_fd)
+                finally:
+                    os.close(directory_fd)
+            except (AttributeError, OSError):
+                pass
+        finally:
+            try:
+                os.unlink(temporary)
+            except FileNotFoundError:
+                pass
+
+
+def load_json(path: str | os.PathLike[str], default: Any = None) -> Any:
+    target = Path(path)
+    if not target.exists():
+        return default
+    with file_lock(target):
+        try:
+            with target.open(encoding="utf-8") as handle:
+                return json.load(handle)
+        except (OSError, ValueError):
+            return default
+
+
+def _tail_bytes(target: Path, *, line_hint: int, chunk_size: int) -> bytes:
+    with target.open("rb") as handle:
+        handle.seek(0, os.SEEK_END)
+        position = handle.tell()
+        blocks: list[bytes] = []
+        newlines = 0
+        while position > 0 and newlines <= line_hint:
+            size = min(chunk_size, position)
+            position -= size
+            handle.seek(position)
+            block = handle.read(size)
+            blocks.append(block)
+            newlines += block.count(b"\n")
+        return b"".join(reversed(blocks))
+
+
+def load_jsonl_tail(
+    path: str | os.PathLike[str], *, limit: int = 90, chunk_size: int = 8192
+) -> list[dict[str, Any]]:
+    """Read only enough of a JSONL file to return its last valid entries."""
+    if limit <= 0:
+        return []
+    target = Path(path)
+    if not target.exists():
+        return []
+    with file_lock(target):
+        raw = _tail_bytes(target, line_hint=limit + 8, chunk_size=max(chunk_size, 32))
+    entries: list[dict[str, Any]] = []
+    for line in raw.splitlines():
+        try:
+            value = json.loads(line)
+        except (UnicodeDecodeError, ValueError):
+            continue
+        if isinstance(value, dict):
+            entries.append(value)
+    return entries[-limit:]
+
+
+def _utc_day(timestamp: Any) -> str | None:
+    if not isinstance(timestamp, str):
+        return None
+    try:
+        parsed = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
+        if parsed.tzinfo is None:
+            parsed = parsed.replace(tzinfo=timezone.utc)
+        return parsed.astimezone(timezone.utc).date().isoformat()
+    except ValueError:
+        return None
+
+
+def append_daily_jsonl(path: str | os.PathLike[str], entry: dict[str, Any]) -> bool:
+    """Append at most one record per UTC day, inspecting only the bounded tail."""
+    target = Path(path)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    entry_day = _utc_day(entry.get("timestamp"))
+    if entry_day is None:
+        raise ValueError("entry timestamp must be an ISO-8601 datetime")
+    with file_lock(target):
+        if target.exists():
+            raw = _tail_bytes(target, line_hint=8, chunk_size=4096)
+            for line in reversed(raw.splitlines()):
+                try:
+                    previous = json.loads(line)
+                except (UnicodeDecodeError, ValueError):
+                    continue
+                if _utc_day(previous.get("timestamp")) == entry_day:
+                    return False
+                break
+        payload = (json.dumps(entry, default=str) + "\n").encode("utf-8")
+        fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
+        try:
+            os.write(fd, payload)
+            os.fsync(fd)
+        finally:
+            os.close(fd)
+    return True
+
+
+def has_observation(payload: Any) -> bool:
+    if not isinstance(payload, dict):
+        return payload is not None
+    return any(value is not None for key, value in payload.items() if key not in _METADATA_KEYS)
+
+
+def merge_observation(
+    previous: Any,
+    observed: Any,
+    *,
+    source: str,
+    observed_at: str | None = None,
+    error: str | None = None,
+) -> dict[str, Any]:
+    """Annotate a fresh observation or retain the last-known-good value as stale."""
+    if has_observation(observed):
+        merged = dict(observed) if isinstance(observed, dict) else {"value": observed}
+        merged.update(
+            observed_at=observed_at or datetime.now(timezone.utc).isoformat(),
+            source=source,
+            stale=False,
+            last_error=None,
+        )
+        return merged
+
+    merged = dict(previous) if isinstance(previous, dict) else {}
+    observed_error = observed.get("error") if isinstance(observed, dict) else None
+    merged.update(
+        source=merged.get("source") or source,
+        stale=True,
+        last_error=observed_error or error or "metric was not observed",
+    )
+    merged.setdefault("observed_at", None)
+    return merged
+
+
+def onchain_refresh_due(
+    timestamp: Any,
+    *,
+    now: datetime | None = None,
+    ttl_seconds: int = 6 * 60 * 60,
+) -> bool:
+    """Return whether the last successful on-chain observation exceeded its TTL."""
+    if not isinstance(timestamp, str) or not timestamp:
+        return True
+    try:
+        observed = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
+        if observed.tzinfo is None:
+            observed = observed.replace(tzinfo=timezone.utc)
+    except ValueError:
+        return True
+    current = now or datetime.now(timezone.utc)
+    if current.tzinfo is None:
+        current = current.replace(tzinfo=timezone.utc)
+    return (current.astimezone(timezone.utc) - observed.astimezone(timezone.utc)).total_seconds() >= ttl_seconds

docker-compose.yml

@@ -0,0 +1,27 @@
+services:
+  btc-monitor:
+    build:
+      context: .
+    image: btc-accumulation-monitor:local
+    container_name: btc-accumulation-monitor
+    init: true
+    ports:
+      - "10.0.10.20:3088:3088"
+    restart: unless-stopped
+    environment:
+      PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
+    volumes:
+      # Persist live history/cache and user-editable settings on the Mac Mini.
+      - ./data:/app/data
+      - ./config:/app/config
+    healthcheck:
+      test:
+        - CMD
+        - /app/.venv/bin/python
+        - -c
+        - "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3088/health/live', timeout=3)"
+      interval: 30s
+      timeout: 5s
+      start_period: 30s
+      retries: 3
+    stop_grace_period: 30s

llm_client/analyzer.py

@@ -1,109 +1,274 @@
 #!/usr/bin/env python3
 """
-LLM Strategy Analyzer — Calls Ollama on Mac Mini to analyze results
+LLM Accumulation Signal Analyzer -- Calls LLM to analyze results
 and suggest config modifications for the next iteration.
+Supports multiple providers: Ollama, LM Studio, OpenAI, Anthropic, OpenRouter.
 """
 
 import json
+import os
 import re
 import requests
 
-OLLAMA_URL = "http://100.100.242.21:11434"
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-MODEL = "qwen3.5:27b"
+LLM_SETTINGS_PATH = os.path.join(BASE_DIR, "config", "llm_settings.json")
 
-SYSTEM_PROMPT = """You are a quantitative trading strategy optimizer. You analyze ML model backtesting results for a BTC/USDT trading strategy and suggest precise modifications to improve performance.
+# Fallback defaults
+DEFAULT_OLLAMA_URL = "http://100.100.242.21:11434"
+DEFAULT_MODEL = "qwen3.5:27b"
 
-## Your Task
+
-Given the current configuration and results, suggest 1-3 specific, justified changes to the configuration for the next iteration. Be methodical and scientific — change one thing at a time when possible.
+def load_llm_settings():
+    """Load LLM settings from config file, with fallback to defaults."""
+    if os.path.exists(LLM_SETTINGS_PATH):
+        with open(LLM_SETTINGS_PATH) as f:
+            return json.load(f)
+    return {
+        "provider": "ollama",
+        "model": DEFAULT_MODEL,
+        "providers": {
+            "ollama": {"base_url": DEFAULT_OLLAMA_URL},
+        },
+    }
+
+
+SYSTEM_PROMPT = """You are a quantitative analyst optimizing a BTC ACCUMULATION SIGNAL model. The goal is NOT day-trading -- it is finding statistically optimal times to BUY BTC for long-term holding.
+
+## Core Question
+"Given current market conditions, is NOW a good time to BUY BTC for long-term holding?"
+
+## What the Model Does
+For each candle, the model predicts an Accumulation Score (0-100):
+- 90-100: STRONG BUY -- historically rare, excellent entry point
+- 70-89: GOOD BUY -- better than average entry
+- 50-69: NEUTRAL -- average time to buy
+- 30-49: WAIT -- price likely to come down
+- 0-29: POOR -- historically bad time to buy (near local tops)
+
+The model is trained on ACTUAL forward returns at 7d, 30d, and 90d horizons, weighted 20/30/50. Times when buying led to the best long-term returns get the highest scores.
+
+## Primary Metric: cost_basis_improvement_pct
+This measures how much better the model's average buy price is vs uniform DCA.
+- 10%+ = good
+- 15%+ = excellent
+- 20%+ = exceptional
+Also require strong_buy_signal_count >= 30 for statistical validity.
 
 ## Config Parameters You Can Modify
 
-**model_type**: "xgboost", "lightgbm", "catboost", or "ensemble"
+**model_type**: "xgboost", "lightgbm", "catboost", "lstm", or "hybrid"
-  - xgboost: Generally best for structured data, fast GPU training
+  - hybrid: Average of LSTM + XGBoost regression predictions. Recommended default.
-  - lightgbm: Faster training, good with large feature sets
+  - xgboost: Fast GPU training, good for structured features.
-  - catboost: Handles feature interactions well, less tuning needed
+  - lstm: Captures temporal patterns in price sequences.
-  - ensemble: Combines all three, reduces variance but slower
 
-**hyperparameters**:
+**hyperparameters** (gradient boosting):
-  - learning_rate (0.001-0.3): Lower = more robust but slower. If overfitting, decrease.
+  - learning_rate (0.001-0.1): Lower = more robust. Start conservative.
-  - max_depth (3-10): Controls model complexity. Deeper = more overfitting risk.
+  - max_depth (3-8): Controls complexity. Deeper risks overfitting.
-  - n_estimators (100-2000): More trees = better fit but diminishing returns.
+  - n_estimators (200-1500): More trees = better fit but diminishing returns.
-  - subsample (0.5-1.0): Row sampling. Lower = more regularization.
+  - subsample (0.5-1.0): Row sampling for regularization.
-  - colsample_bytree (0.5-1.0): Feature sampling per tree. Lower = more diversity.
+  - colsample_bytree (0.5-1.0): Feature sampling per tree.
-  - min_child_weight (1-20): Higher = more conservative splits.
+  - min_child_weight (5-30): Higher = more conservative (important for noisy targets).
-  - gamma (0-5): Minimum loss reduction for split. Higher = more pruning.
+  - gamma (0-5): Minimum loss reduction for split.
-  - reg_alpha (0-10): L1 regularization. Encourages sparsity.
+  - reg_alpha (0-10): L1 regularization.
-  - reg_lambda (0-10): L2 regularization. Prevents large weights.
+  - reg_lambda (1-10): L2 regularization. Higher values prevent overfitting.
+
+**hyperparameters** (LSTM):
+  - lstm_hidden_size (32-256): Hidden units.
+  - lstm_num_layers (1-4): Stacked layers. 2 is usually optimal.
+  - lstm_dropout (0.1-0.5): Regularization.
+  - lstm_epochs (50-200): Max training epochs (early stopping usually triggers).
+  - lstm_batch_size (32-128): Smaller = noisier but better generalization.
+  - lstm_sequence_length (15-60): Past candles the LSTM sees. Longer = more context.
+  - lstm_patience (5-20): Early stopping patience.
 
 **target**:
-  - direction: "long" or "both"
+  - forward_periods_4h: List of 3 forward periods in 4h candles [short, medium, long].
-  - horizon_candles (1-20): How far ahead to predict. Longer = smoother but lagging.
+    Defaults: [42, 180, 540] = roughly [7d, 30d, 90d]
-  - threshold_pct (0.3-3.0): Minimum move % to label as positive. Higher = fewer but clearer signals.
+  - weights: Weights for each period. Default [0.2, 0.3, 0.5] (emphasize long-term).
+  - score_range: [0, 100] -- do not change.
 
 **strategy**:
-  - entry_threshold (0.5-0.8): Min prediction probability to enter trade. Higher = fewer trades, higher quality.
+  - strong_buy_threshold (70-95): Score above which = STRONG BUY signal. Higher = fewer but better signals.
-  - stop_loss_pct (0.5-5.0): Max loss before exit. Tighter = more stopped out.
+  - good_buy_threshold (50-80): Score above which = GOOD BUY. Used for cost basis comparison.
-  - take_profit_pct (1.0-10.0): Target profit. Should be > stop_loss for positive expectancy.
+  - poor_threshold (10-40): Score below which = POOR time to buy.
-  - trailing_stop_pct (0.5-3.0): Trailing stop distance. Tighter = locks profit faster but exits early.
-  - min_confidence_to_trade (0.5-0.9): Absolute minimum confidence to consider.
-  - exit_type: "trailing_stop" or "fixed" (just SL/TP)
 
 **features**:
-  - use_volume_features (true/false): Volume features can be noisy in crypto.
+  - use_price_position (true/false): Distance from ATH, 52w high/low, percentile.
-  - use_candle_patterns (true/false): Candle patterns may or may not help.
+  - use_momentum (true/false): RSI, MACD, Stochastic, Williams %R, ROC.
-  - use_lag_features (true/false): Lagged features capture momentum.
+  - use_volatility (true/false): Bollinger Bands, ATR, consecutive red candles, drawdown.
-  - lag_periods: List of lag periods [1,2,3,5,10]
+  - use_volume (true/false): Volume ratio, OBV, red/green volume ratio.
-  - lookback_periods: List of lookback windows [3,5,10,20]
+  - use_cycle (true/false): MA cross regime, candles since major drawdown.
+  - use_pca (true/false): PCA dimensionality reduction.
+  - pca_variance (0.80-0.99): Variance to retain.
+  - use_scaler (true/false): StandardScaler. Critical for LSTM.
 
 **training**:
-  - walk_forward_windows (3-10): More windows = more robust but less data per window.
+  - rolling_window (true/false): Rolling vs static walk-forward.
+  - rolling_train_size (1500-5000): Training window candles.
+  - rolling_test_size (100-500): Test window candles.
 
-## Key Metrics to Optimize (in priority order)
+## Key Metrics to Analyze
-1. **Sharpe Ratio** (target: > 2.0): Risk-adjusted return. Most important metric.
+1. **cost_basis_improvement_pct**: PRIMARY metric. How much better is model buy price vs DCA.
-2. **Profit Factor** (target: > 1.5): Gross profit / gross loss.
+2. **strong_buy_signal_count**: Must be >= 30 for validity. Too few = raise threshold. Too many = lower it.
-3. **Max Drawdown** (target: > -15%): Worst peak-to-trough decline.
+3. **signal_frequency_pct**: Should be 5-15%. If outside, adjust thresholds.
-4. **Win Rate** (target: > 55%): Percentage of winning trades.
+4. **avg_score_at_actual_bottoms**: Should be high (>70). Model should recognize bottoms.
-5. **Trade Count**: Need enough trades for statistical significance (>50).
+5. **avg_score_at_actual_tops**: Should be low (<30). Model should avoid tops.
+6. **model_r2_score**: Regression fit quality. > 0.2 is decent for financial data.
+7. **per_window_cost_improvement**: Consistency across windows. Low variance = robust.
 
 ## Decision Guidelines
-- If Sharpe < 1.0: The strategy is not working well. Consider larger changes.
+- If cost_improvement < 5%: Strategy is barely working. Try: switch model type, enable all features, increase training window, lower good_buy_threshold.
-- If Sharpe 1.0-1.5: Decent. Fine-tune hyperparameters and thresholds.
+- If cost_improvement 5-10%: Decent. Fine-tune thresholds and hyperparameters.
-- If Sharpe 1.5-2.0: Good. Make small, targeted improvements.
+- If cost_improvement 10-15%: Good. Make targeted improvements -- focus on signal consistency.
-- If Sharpe > 2.0: Very good. Be careful not to overfit.
+- If cost_improvement > 15%: Very good. Be careful not to overfit. Check per_window variance.
-- If win_rate < 0.50 but profit_factor > 1.5: Strategy relies on big wins — ok, tighten SL.
+- If signal_count < 30: Not statistically valid. Lower strong_buy_threshold, increase training data.
-- If win_rate > 0.60 but profit_factor < 1.2: Many small wins but losses are too big — widen TP or tighten SL.
+- If signal_frequency > 20%: Too many signals = not selective enough. Raise threshold.
-- If trade_count < 30: Not enough trades. Lower entry_threshold or min_confidence.
+- If signal_frequency < 3%: Too few signals. Lower threshold.
-- If max_drawdown < -20%: Too risky. Increase regularization, tighten stop loss.
+- If score_at_bottoms < 60: Model is missing bottoms. More features, different model type.
-- If per_window_sharpe has high variance: Model is not stable. More regularization or simpler model.
+- If score_at_tops > 40: Model is not avoiding tops. More regularization.
-- Check feature_importances: If top features make financial sense, good. If random features dominate, possible overfitting.
+- If per_window has high variance: Model is unstable. Increase regularization, try hybrid.
+- Check feature_importances: price position features should dominate (distance from ATH, percentile).
 
 ## Response Format
 You MUST respond with ONLY a JSON object (no markdown, no explanation outside the JSON):
 ```
 {
-  "reasoning": "Explanation of what you observed and why you're making these changes",
+  "reasoning": "Explanation of observations and why you are making these changes",
   "changes": ["Change 1 description", "Change 2 description"],
   "config": { <complete modified config JSON> }
 }
 ```
-The "config" field must contain the COMPLETE config (not just changes) so it can be used directly."""
+The "config" field must contain the COMPLETE config so it can be used directly."""
 
 
-def analyze_and_suggest(current_config: dict, results: dict,
+def _call_ollama(settings, messages):
-                        iteration_history: list = None) -> tuple[dict, str]:
+    """Call Ollama API."""
+    provider_cfg = settings.get("providers", {}).get("ollama", {})
+    base_url = provider_cfg.get("base_url", DEFAULT_OLLAMA_URL)
+    model = settings.get("model", DEFAULT_MODEL)
+
+    payload = {
+        "model": model,
+        "messages": messages,
+        "stream": False,
+        "think": False,
+        "options": {"temperature": 0.7, "num_predict": 4096},
+    }
+    print(f"  Calling LLM ({model} via Ollama at {base_url})...")
+    resp = requests.post(f"{base_url}/api/chat", json=payload, timeout=600)
+    resp.raise_for_status()
+    return resp.json()["message"]["content"]
+
+
+def _call_openai_compatible(settings, messages, provider_name):
+    """Call OpenAI-compatible API (LM Studio, OpenAI, OpenRouter)."""
+    provider_cfg = settings.get("providers", {}).get(provider_name, {})
+    model = settings.get("model", "")
+
+    if provider_name == "lmstudio":
+        base_url = provider_cfg.get("base_url", "http://100.100.242.21:1234")
+        url = f"{base_url}/v1/chat/completions"
+        headers = {"Content-Type": "application/json"}
+    elif provider_name == "openai":
+        url = "https://api.openai.com/v1/chat/completions"
+        headers = {
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {provider_cfg.get('api_key', '')}",
+        }
+    elif provider_name == "openrouter":
+        url = "https://openrouter.ai/api/v1/chat/completions"
+        headers = {
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {provider_cfg.get('api_key', '')}",
+        }
+    else:
+        raise ValueError(f"Unknown OpenAI-compatible provider: {provider_name}")
+
+    payload = {
+        "model": model,
+        "messages": messages,
+        "temperature": 0.7,
+        "max_tokens": 4096,
+    }
+    print(f"  Calling LLM ({model} via {provider_name})...")
+    resp = requests.post(url, json=payload, headers=headers, timeout=600)
+    resp.raise_for_status()
+    return resp.json()["choices"][0]["message"]["content"]
+
+
+def _call_anthropic(settings, messages):
+    """Call Anthropic Messages API."""
+    provider_cfg = settings.get("providers", {}).get("anthropic", {})
+    model = settings.get("model", "claude-sonnet-4-20250514")
+    api_key = provider_cfg.get("api_key", "")
+
+    # Anthropic uses system as a top-level param, not in messages
+    system_msg = ""
+    api_messages = []
+    for m in messages:
+        if m["role"] == "system":
+            system_msg = m["content"]
+        else:
+            api_messages.append(m)
+
+    payload = {
+        "model": model,
+        "max_tokens": 4096,
+        "messages": api_messages,
+    }
+    if system_msg:
+        payload["system"] = system_msg
+
+    headers = {
+        "Content-Type": "application/json",
+        "x-api-key": api_key,
+        "anthropic-version": "2023-06-01",
+    }
+    print(f"  Calling LLM ({model} via Anthropic)...")
+    resp = requests.post(
+        "https://api.anthropic.com/v1/messages",
+        json=payload,
+        headers=headers,
+        timeout=600,
+    )
+    resp.raise_for_status()
+    data = resp.json()
+    # Extract text from content blocks
+    return "".join(
+        block["text"] for block in data.get("content", []) if block.get("type") == "text"
+    )
+
+
+def call_llm(messages):
+    """Route LLM call to the configured provider."""
+    settings = load_llm_settings()
+    provider = settings.get("provider", "ollama")
+
+    if provider == "ollama":
+        return _call_ollama(settings, messages)
+    elif provider in ("lmstudio", "openai", "openrouter"):
+        return _call_openai_compatible(settings, messages, provider)
+    elif provider == "anthropic":
+        return _call_anthropic(settings, messages)
+    else:
+        raise ValueError(f"Unknown LLM provider: {provider}")
+
+
+def analyze_and_suggest(current_config, results, iteration_history=None):
     """
     Send current results to LLM and get suggested config modifications.
     Returns (new_config, reasoning).
     """
-    # Build the user prompt with context
     history_text = ""
     if iteration_history:
         history_text = "\n## Previous Iterations (most recent last)\n"
         for h in iteration_history[-5:]:
             history_text += (
-                f"- Iteration {h['iteration']}: Sharpe={h['sharpe']}, "
+                f"- Iteration {h.get('iteration', '?')}: "
-                f"Return={h['return']}%, WinRate={h['win_rate']}, "
+                f"CostImprovement={h.get('cost_improvement', 0):.1f}%, "
-                f"Trades={h['trades']}, Model={h['model_type']}\n"
+                f"Signals={h.get('signal_count', 0)}, "
+                f"R2={h.get('r2_score', 0):.4f}, "
+                f"Model={h.get('model_type', '?')}\n"
             )
 
     user_prompt = f"""## Current Configuration
@@ -112,40 +277,31 @@ def analyze_and_suggest(current_config: dict, results: dict,
 ```
 
 ## Current Results
-- Sharpe Ratio: {results.get('sharpe_ratio', 0)}
+- Cost Basis Improvement: {results.get('cost_basis_improvement_pct', 0):.1f}%
-- Total Return: {results.get('total_return_pct', 0)}%
+- Avg Cost (Model): ${results.get('avg_cost_basis_model', 0):,.2f}
-- Max Drawdown: {results.get('max_drawdown_pct', 0)}%
+- Avg Cost (DCA): ${results.get('avg_cost_basis_dca', 0):,.2f}
-- Win Rate: {results.get('win_rate', 0)}
+- Strong Buy Signals: {results.get('strong_buy_signal_count', 0)}
-- Trade Count: {results.get('trade_count', 0)}
+- Good Buy Signals: {results.get('good_buy_signal_count', 0)}
-- Profit Factor: {results.get('profit_factor', 0)}
+- Signal Frequency: {results.get('signal_frequency_pct', 0):.1f}%
-- Avg Trade Duration: {results.get('avg_trade_duration_candles', 0)} candles
+- Quality of Strong Buys: {results.get('pct_quality_strong_buy', 0):.1%}
-- Per-Window Sharpe: {results.get('per_window_sharpe', [])}
+- Model R2: {results.get('model_r2_score', 0):.4f}
+- Score at Actual Bottoms: {results.get('avg_score_at_actual_bottoms', 0):.1f}
+- Score at Actual Tops: {results.get('avg_score_at_actual_tops', 0):.1f}
+- Per-Window Improvement: {results.get('per_window_cost_improvement', [])}
+- Score Distribution: {results.get('score_distribution', {})}
 
 ## Top Feature Importances
 {json.dumps(dict(list(results.get('feature_importances', {}).items())[:15]), indent=2)}
 {history_text}
 Analyze these results and suggest 1-3 specific modifications to the config. Return ONLY valid JSON."""
 
-    # Call Ollama
+    messages = [
-    payload = {
+        {"role": "system", "content": SYSTEM_PROMPT},
-        "model": MODEL,
+        {"role": "user", "content": user_prompt},
-        "messages": [
+    ]
-            {"role": "system", "content": SYSTEM_PROMPT},
-            {"role": "user", "content": user_prompt},
-        ],
-        "stream": False,
-        "options": {
-            "temperature": 0.7,
-            "num_predict": 4096,
-        },
-    }
 
-    print(f"  Calling LLM ({MODEL} on Mac Mini)...")
+    content = call_llm(messages)
-    resp = requests.post(f"{OLLAMA_URL}/api/chat", json=payload, timeout=300)
-    resp.raise_for_status()
-    content = resp.json()["message"]["content"]
 
-    # Parse JSON from response (handle markdown code blocks)
     # Strip thinking tags if present
     content = re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL).strip()
 
@@ -153,8 +309,6 @@ Analyze these results and suggest 1-3 specific modifications to the config. Retu
     if json_match:
         parsed = json.loads(json_match.group(1))
     else:
-        # Try parsing the whole response as JSON
-        # Find the outermost JSON object
         brace_start = content.find("{")
         if brace_start >= 0:
             depth = 0
@@ -164,7 +318,7 @@ Analyze these results and suggest 1-3 specific modifications to the config. Retu
                 elif content[i] == "}":
                     depth -= 1
                     if depth == 0:
-                        parsed = json.loads(content[brace_start:i + 1])
+                        parsed = json.loads(content[brace_start : i + 1])
                         break
             else:
                 raise ValueError("Could not find complete JSON in LLM response")
@@ -175,8 +329,14 @@ Analyze these results and suggest 1-3 specific modifications to the config. Retu
     changes = parsed.get("changes", [])
     new_config = parsed.get("config", current_config)
 
-    # Validate that config has required fields
+    required_keys = [
-    required_keys = ["model_type", "features", "target", "hyperparameters", "strategy", "training"]
+        "model_type",
+        "features",
+        "target",
+        "hyperparameters",
+        "strategy",
+        "training",
+    ]
     for key in required_keys:
         if key not in new_config:
             new_config[key] = current_config[key]
@@ -186,22 +346,36 @@ Analyze these results and suggest 1-3 specific modifications to the config. Retu
 
 
 if __name__ == "__main__":
-    # Test with dummy data
     import sys
+
     config_path = sys.argv[1] if len(sys.argv) > 1 else "config/initial_config.json"
     with open(config_path) as f:
         config = json.load(f)
 
     dummy_results = {
-        "sharpe_ratio": 1.2,
+        "cost_basis_improvement_pct": 8.5,
-        "total_return_pct": 15.3,
+        "avg_cost_basis_model": 65000,
-        "max_drawdown_pct": -12.5,
+        "avg_cost_basis_dca": 71000,
-        "win_rate": 0.55,
+        "strong_buy_signal_count": 45,
-        "trade_count": 120,
+        "good_buy_signal_count": 120,
-        "profit_factor": 1.4,
+        "signal_frequency_pct": 7.2,
-        "avg_trade_duration_candles": 7.2,
+        "pct_quality_strong_buy": 0.72,
-        "feature_importances": {"RSI_14": 0.15, "MACD_hist": 0.12, "BB_width": 0.10},
+        "model_r2_score": 0.22,
-        "per_window_sharpe": [1.0, 1.3, 1.5, 0.9, 1.1],
+        "avg_score_at_actual_bottoms": 68.5,
+        "avg_score_at_actual_tops": 35.2,
+        "per_window_cost_improvement": [7.1, 9.3, 8.8, 10.2, 7.0],
+        "score_distribution": {
+            "0-20": 80,
+            "20-40": 150,
+            "40-60": 200,
+            "60-80": 130,
+            "80-100": 40,
+        },
+        "feature_importances": {
+            "dist_from_ath_pct": 0.18,
+            "RSI_14": 0.12,
+            "price_percentile_365": 0.10,
+        },
     }
 
     new_config, reasoning = analyze_and_suggest(config, dummy_results)

ml/artifacts.py

@@ -0,0 +1,91 @@
+"""Validation for persisted ML scoring artifacts."""
+
+import math
+
+from scoring.policy import SCORE_VERSION
+
+ML_ARTIFACT_SCHEMA_VERSION = 2
+REQUIRED_WEIGHT_KEYS = frozenset({
+    "puell_multiple",
+    "mvrv_zscore",
+    "reserve_risk",
+    "rhodl_ratio",
+    "nupl",
+    "fear_greed",
+    "drawdown",
+    "pct_above_200w_sma",
+    "pct_above_lth_rp",
+})
+
+
+def _weights_valid(weights):
+    if not isinstance(weights, dict) or not REQUIRED_WEIGHT_KEYS.issubset(weights):
+        return False
+    values = [weights[key] for key in REQUIRED_WEIGHT_KEYS]
+    return all(
+        isinstance(value, (int, float))
+        and not isinstance(value, bool)
+        and math.isfinite(value)
+        and value >= 0
+        for value in values
+    ) and sum(values) > 0
+
+
+def _has_oos_fold_weights(artifact):
+    folds = artifact.get("cv_results", {}).get("folds", [])
+    if not isinstance(folds, list) or not folds:
+        return False
+    for fold in folds:
+        validation_range = fold.get("date_ranges", {}).get("validation")
+        if not validation_range or not _weights_valid(fold.get("weights")):
+            return False
+    return True
+
+
+def validate_ml_artifact(artifact):
+    """Return machine-readable validity and provenance for an ML artifact."""
+    errors = []
+    if not isinstance(artifact, dict):
+        artifact = {}
+        errors.append("artifact_object")
+
+    schema_version = artifact.get("artifact_schema_version")
+    if schema_version != ML_ARTIFACT_SCHEMA_VERSION:
+        errors.append("artifact_schema_version")
+
+    score_version = artifact.get("score_version")
+    if score_version != SCORE_VERSION:
+        errors.append("score_version")
+
+    if not _weights_valid(artifact.get("weights")):
+        errors.append("weights")
+
+    provenance = artifact.get("provenance")
+    if not isinstance(provenance, dict):
+        provenance = {}
+        errors.append("provenance")
+    else:
+        required_provenance = {
+            "validation_method",
+            "label_horizon_days",
+            "weight_scope",
+            "training_date_range",
+            "trained_at",
+        }
+        if not required_provenance.issubset(provenance):
+            errors.append("provenance")
+        if provenance.get("validation_method") != "purged_expanding_window":
+            errors.append("purged_validation")
+        if provenance.get("label_horizon_days") != 365:
+            errors.append("label_horizon_days")
+        if provenance.get("weight_scope") != "full_history_fit":
+            errors.append("weight_scope")
+
+    return {
+        "valid": not errors,
+        "schema_version": schema_version,
+        "score_version": score_version,
+        "weight_scope": provenance.get("weight_scope"),
+        "has_oos_fold_weights": _has_oos_fold_weights(artifact),
+        "errors": list(dict.fromkeys(errors)),
+    }

ml/optimizer.py

@@ -0,0 +1,682 @@
+#!/usr/bin/env python3
+"""
+ML Optimizer for Bitcoin Accumulation Zone Scoring.
+
+Trains a gradient boosted tree model on historical on-chain metrics to find
+optimal metric weights for identifying the best long-term buying opportunities.
+
+Output: config/ml_weights.json with optimized weights and feature importances.
+"""
+
+import json
+import logging
+import os
+import sys
+from datetime import datetime, timedelta
+
+import numpy as np
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.metrics import (
+    classification_report,
+    f1_score,
+    precision_score,
+    recall_score,
+    roc_auc_score,
+)
+from sklearn.model_selection import TimeSeriesSplit
+from sklearn.preprocessing import StandardScaler
+
+from scoring.policy import SCORE_BRACKETS, SCORE_VERSION, score_in_bracket
+from ml.artifacts import ML_ARTIFACT_SCHEMA_VERSION
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(name)s] %(levelname)s: %(message)s",
+)
+log = logging.getLogger("ml-optimizer")
+
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+HISTORY_PATH = os.path.join(BASE_DIR, "data", "history.json")
+OUTPUT_PATH = os.path.join(BASE_DIR, "config", "ml_weights.json")
+THRESHOLDS_PATH = os.path.join(BASE_DIR, "config", "thresholds.json")
+
+# Date range: 2018-02-01 onward (when all 8 metrics + fear_greed available)
+START_DATE = "2018-02-01"
+# Training cutoff: need 1yr forward data for labels
+TRAIN_CUTOFF_DAYS = 365
+# Target: forward 365d return > 30% = "good time to buy"
+GOOD_BUY_THRESHOLD = 30.0
+# Validation embargo/purge horizon: labels use 365-day forward returns.
+LABEL_HORIZON_DAYS = 365
+VALIDATION_SPLITS = 5
+
+# The 8 core metrics we score
+METRIC_KEYS = [
+    "puell_multiple",
+    "mvrv_zscore",
+    "reserve_risk",
+    "rhodl_ratio",
+    "nupl",
+    "fear_greed",
+]
+# Ratio-based metrics (derived from price vs reference)
+RATIO_METRICS = {
+    "pct_above_200w_sma": {"price_key": "btc_price", "ref_key": "200w_sma"},
+    "pct_above_lth_rp": {"price_key": "btc_price", "ref_key": "lth_realized_price"},
+}
+
+
+def load_history():
+    """Load historical data and build date-aligned lookup."""
+    with open(HISTORY_PATH) as f:
+        raw = json.load(f)
+
+    index = {}
+    for key, data in raw.items():
+        if not isinstance(data, dict) or "dates" not in data:
+            continue
+        lookup = {}
+        for d, v in zip(data["dates"], data["values"]):
+            if v is not None:
+                lookup[d] = v
+        index[key] = lookup
+    return index
+
+
+def load_thresholds():
+    """Load scoring thresholds for converting raw values to 0-10 scores."""
+    with open(THRESHOLDS_PATH) as f:
+        return json.load(f)
+
+
+def score_range(value, ranges):
+    """Score a value using range-based thresholds (same logic as scoring/engine.py)."""
+    if value is None:
+        return None
+    for low, high, score in ranges:
+        low_ok = low is None or value >= low
+        high_ok = high is None or value < high
+        if low_ok and high_ok:
+            return score
+    return 0
+
+
+SCORE_KEYS = [
+    "puell_multiple", "mvrv_zscore", "reserve_risk", "rhodl_ratio",
+    "nupl", "fear_greed", "drawdown", "pct_above_200w_sma", "pct_above_lth_rp",
+]
+
+SCORE_FEATURES = [f"score_{k}" for k in SCORE_KEYS]
+RAW_FEATURES = [
+    "raw_puell_multiple", "raw_mvrv_zscore", "raw_reserve_risk",
+    "raw_rhodl_ratio", "raw_nupl", "raw_fear_greed",
+    "raw_pct_above_200w_sma", "raw_pct_above_lth_rp", "raw_drawdown",
+]
+DELTA_FEATURES = [
+    "delta_30d_mvrv_zscore", "delta_30d_nupl",
+    "delta_30d_puell_multiple", "delta_30d_reserve_risk",
+]
+INTERACTION_FEATURES = ["mvrv_x_nupl", "puell_x_reserve"]
+CYCLE_FEATURES = ["days_since_ath"]
+FEATURE_COLS = SCORE_FEATURES + RAW_FEATURES + DELTA_FEATURES + INTERACTION_FEATURES + CYCLE_FEATURES
+
+BRACKETS = SCORE_BRACKETS
+
+
+def _row_date(row):
+    return datetime.strptime(row["date"], "%Y-%m-%d")
+
+
+def purged_time_series_splits(rows, n_splits=VALIDATION_SPLITS,
+                              label_horizon_days=LABEL_HORIZON_DAYS,
+                              embargo_days=0):
+    """Yield expanding-window splits with overlapping forward-label windows removed.
+
+    A row dated T with a 365-day forward-return label consumes information up to
+    T+365. For validation beginning at V, any training row whose label window
+    reaches V is removed. This keeps validation metrics out-of-sample for the
+    forward-return label, not just for features.
+    """
+    base_splitter = TimeSeriesSplit(n_splits=n_splits)
+    row_dates = [_row_date(r) for r in rows]
+    horizon = timedelta(days=label_horizon_days)
+    embargo = timedelta(days=embargo_days)
+
+    for train_idx, val_idx in base_splitter.split(np.arange(len(rows))):
+        val_start = row_dates[val_idx[0]]
+        val_end = row_dates[val_idx[-1]]
+        purged_train = []
+        for idx in train_idx:
+            label_end = row_dates[idx] + horizon
+            before_validation_label_window = label_end <= val_start - embargo
+            after_validation_embargo = row_dates[idx] > val_end + embargo
+            if before_validation_label_window or after_validation_embargo:
+                purged_train.append(idx)
+        if purged_train:
+            yield np.array(purged_train, dtype=int), np.array(val_idx, dtype=int)
+
+
+def viable_classification_splits(y, splits):
+    """Yield only folds whose training window contains both target classes."""
+    for train_idx, val_idx in splits:
+        if len(np.unique(y[train_idx])) < 2:
+            continue
+        yield train_idx, val_idx
+
+
+def artifact_fold_results(fold_results):
+    """Strip training-only row indexes from the persisted ML artifact."""
+    return [
+        {key: value for key, value in fold.items() if key not in {"train_idx", "val_idx"}}
+        for fold in fold_results
+    ]
+
+
+def _build_model():
+    return GradientBoostingClassifier(
+        n_estimators=300,
+        learning_rate=0.05,
+        max_depth=4,
+        subsample=0.8,
+        min_samples_leaf=20,
+        random_state=42,
+    )
+
+
+def derive_metric_weights(feature_cols, importances):
+    """Aggregate feature importances back to transparent score metric weights."""
+    metric_names = list(SCORE_KEYS)
+    feature_to_metric = {}
+    for m in metric_names:
+        feature_to_metric[f"score_{m}"] = m
+        feature_to_metric[f"raw_{m}"] = m
+    feature_to_metric["delta_30d_mvrv_zscore"] = "mvrv_zscore"
+    feature_to_metric["delta_30d_nupl"] = "nupl"
+    feature_to_metric["delta_30d_puell_multiple"] = "puell_multiple"
+    feature_to_metric["delta_30d_reserve_risk"] = "reserve_risk"
+
+    metric_importances = {m: 0.0 for m in metric_names}
+    for name, imp in zip(feature_cols, importances):
+        if name in feature_to_metric:
+            metric_importances[feature_to_metric[name]] += float(imp)
+        elif name == "mvrv_x_nupl":
+            metric_importances["mvrv_zscore"] += float(imp) / 2
+            metric_importances["nupl"] += float(imp) / 2
+        elif name == "puell_x_reserve":
+            metric_importances["puell_multiple"] += float(imp) / 2
+            metric_importances["reserve_risk"] += float(imp) / 2
+        elif name == "days_since_ath":
+            metric_importances["drawdown"] += float(imp)
+
+    total_imp = sum(metric_importances.values())
+    if total_imp > 0:
+        weights = {k: round(v / total_imp, 4) for k, v in metric_importances.items()}
+    else:
+        weights = {k: round(1 / len(metric_importances), 4) for k in metric_importances}
+    return dict(sorted(weights.items(), key=lambda x: x[1], reverse=True))
+
+
+def build_dataset(index, thresholds):
+    """Build aligned training dataset: metric scores + forward returns."""
+    # Get all dates from 2018-02-01 onward
+    all_dates = set()
+    for lookup in index.values():
+        all_dates.update(lookup.keys())
+    dates = sorted(d for d in all_dates if d >= START_DATE)
+
+    # Build price lookup for forward returns
+    price_lookup = {}
+    for pk in ["btc_price", "btc_price_sma", "btc_price_lth"]:
+        if pk in index:
+            for d, v in index[pk].items():
+                if d not in price_lookup:
+                    price_lookup[d] = v
+
+    # Compute ATH series for drawdown
+    all_dates_sorted = sorted(all_dates)
+    ath = 0
+    drawdowns = {}
+    for d in all_dates_sorted:
+        p = price_lookup.get(d)
+        if p is None:
+            continue
+        if p > ath:
+            ath = p
+        if ath > 0:
+            drawdowns[d] = ((ath - p) / ath) * 100
+
+    # Get threshold ranges for scoring raw values
+    metric_ranges = {
+        "puell_multiple": thresholds.get("puell_multiple", {}).get("ranges", []),
+        "mvrv_zscore": thresholds.get("mvrv_zscore", {}).get("ranges", []),
+        "reserve_risk": thresholds.get("reserve_risk", {}).get("ranges", []),
+        "rhodl_ratio": thresholds.get("rhodl_ratio", {}).get("ranges", []),
+        "nupl": thresholds.get("nupl", {}).get("ranges", []),
+        "fear_greed": thresholds.get("fear_greed", {}).get("ranges", []),
+        "drawdown": thresholds.get("drawdown", {}).get("ranges", []),
+        "price_vs_200w_sma": thresholds.get("price_vs_200w_sma", {}).get("ranges", []),
+        "lth_realized_price": thresholds.get("lth_realized_price", {}).get("ranges", []),
+    }
+
+    log.info("Building dataset from %d dates (%s to %s)", len(dates), dates[0], dates[-1])
+
+    rows = []
+    for d in dates:
+        # Get raw metric values
+        vals = {}
+        skip = False
+        for key in METRIC_KEYS:
+            v = index.get(key, {}).get(d)
+            if v is None:
+                skip = True
+                break
+            vals[key] = v
+        if skip:
+            continue
+
+        # Compute ratio metrics
+        price = price_lookup.get(d)
+        sma_200w = index.get("200w_sma", {}).get(d)
+        lth_rp = index.get("lth_realized_price", {}).get(d)
+
+        if price is None or sma_200w is None or lth_rp is None:
+            continue
+        if sma_200w == 0 or lth_rp == 0:
+            continue
+
+        pct_200w = ((price - sma_200w) / sma_200w) * 100
+        pct_lth = ((price - lth_rp) / lth_rp) * 100
+        dd = drawdowns.get(d, 0)
+
+        vals["pct_above_200w_sma"] = pct_200w
+        vals["pct_above_lth_rp"] = pct_lth
+        vals["drawdown"] = dd
+
+        # Score each metric (0-10) using existing thresholds
+        scores = {}
+        scores["puell_multiple"] = score_range(vals["puell_multiple"], metric_ranges["puell_multiple"])
+        scores["mvrv_zscore"] = score_range(vals["mvrv_zscore"], metric_ranges["mvrv_zscore"])
+        scores["reserve_risk"] = score_range(vals["reserve_risk"], metric_ranges["reserve_risk"])
+        scores["rhodl_ratio"] = score_range(vals["rhodl_ratio"], metric_ranges["rhodl_ratio"])
+        scores["nupl"] = score_range(vals["nupl"], metric_ranges["nupl"])
+        scores["fear_greed"] = score_range(vals["fear_greed"], metric_ranges["fear_greed"])
+        scores["drawdown"] = score_range(dd, metric_ranges["drawdown"])
+        scores["pct_above_200w_sma"] = score_range(pct_200w, metric_ranges["price_vs_200w_sma"])
+        scores["pct_above_lth_rp"] = score_range(pct_lth, metric_ranges["lth_realized_price"])
+
+        if any(s is None for s in scores.values()):
+            continue
+
+        # Forward returns
+        dt = datetime.strptime(d, "%Y-%m-%d")
+        fwd = {}
+        for days in [30, 90, 180, 365]:
+            future_d = (dt + timedelta(days=days)).strftime("%Y-%m-%d")
+            fp = price_lookup.get(future_d)
+            if fp is not None and price > 0:
+                fwd[f"fwd_{days}d"] = ((fp - price) / price) * 100
+
+        # Compute rate-of-change features (30d deltas)
+        deltas = {}
+        d_30ago = (dt - timedelta(days=30)).strftime("%Y-%m-%d")
+        for key in ["mvrv_zscore", "nupl", "puell_multiple", "reserve_risk"]:
+            v_now = vals[key]
+            v_prev = index.get(key, {}).get(d_30ago)
+            if v_prev is not None and v_prev != 0:
+                deltas[f"delta_30d_{key}"] = v_now - v_prev
+            else:
+                deltas[f"delta_30d_{key}"] = 0.0
+
+        # Interaction terms
+        interactions = {
+            "mvrv_x_nupl": vals["mvrv_zscore"] * vals["nupl"],
+            "puell_x_reserve": vals["puell_multiple"] * vals["reserve_risk"],
+        }
+
+        # Days since last ATH
+        days_since_ath = 0
+        for i in range(1, 2000):
+            check_d = (dt - timedelta(days=i)).strftime("%Y-%m-%d")
+            check_dd = drawdowns.get(check_d, 100)
+            if check_dd < 0.1:  # essentially at ATH
+                days_since_ath = i
+                break
+        else:
+            days_since_ath = 2000
+
+        row = {
+            "date": d,
+            "price": price,
+            **{f"score_{k}": v for k, v in scores.items()},
+            **{f"raw_{k}": v for k, v in vals.items()},
+            **deltas,
+            **interactions,
+            "days_since_ath": days_since_ath,
+            **fwd,
+        }
+        rows.append(row)
+
+    log.info("Built %d complete data rows", len(rows))
+    return rows
+
+
+def train_model(rows):
+    """Train gradient boosted classifier to identify good buying opportunities."""
+    # Filter to rows that have 365d forward return (for labeling)
+    labeled = [r for r in rows if "fwd_365d" in r]
+    log.info("Rows with 365d forward data: %d", len(labeled))
+
+    if len(labeled) < 100:
+        log.error("Not enough labeled data. Need at least 100 rows, got %d", len(labeled))
+        return None
+
+    # Create binary target: forward 365d return > threshold
+    for r in labeled:
+        r["target"] = 1 if r["fwd_365d"] > GOOD_BUY_THRESHOLD else 0
+
+    positive = sum(r["target"] for r in labeled)
+    log.info("Target distribution: %d positive (%.1f%%), %d negative",
+             positive, positive / len(labeled) * 100, len(labeled) - positive)
+
+    feature_cols = FEATURE_COLS
+
+    X = np.array([[r[f] for f in feature_cols] for r in labeled])
+    y = np.array([r["target"] for r in labeled])
+
+    log.info("Feature matrix: %d samples x %d features", X.shape[0], X.shape[1])
+
+    # Purged time-series cross-validation. Standard TimeSeriesSplit is not
+    # enough here because each label consumes the next 365 days of returns.
+    cv_scores = []
+    cv_f1 = []
+    cv_precision = []
+    cv_recall = []
+    fold_results = []
+
+    splits = list(viable_classification_splits(y, purged_time_series_splits(
+        labeled,
+        n_splits=VALIDATION_SPLITS,
+        label_horizon_days=LABEL_HORIZON_DAYS,
+        embargo_days=0,
+    )))
+    if not splits:
+        log.error("No viable purged validation splits. Need more history for %dd label horizon.",
+                  LABEL_HORIZON_DAYS)
+        return None
+
+    for fold, (train_idx, val_idx) in enumerate(splits):
+        X_train, X_val = X[train_idx], X[val_idx]
+        y_train, y_val = y[train_idx], y[val_idx]
+
+        scaler = StandardScaler()
+        X_train_s = scaler.fit_transform(X_train)
+        X_val_s = scaler.transform(X_val)
+
+        model = _build_model()
+        model.fit(X_train_s, y_train)
+
+        y_pred = model.predict(X_val_s)
+        y_prob = model.predict_proba(X_val_s)[:, 1]
+
+        auc = roc_auc_score(y_val, y_prob) if len(np.unique(y_val)) > 1 else 0
+        f1 = f1_score(y_val, y_pred, zero_division=0)
+        prec = precision_score(y_val, y_pred, zero_division=0)
+        rec = recall_score(y_val, y_pred, zero_division=0)
+
+        cv_scores.append(auc)
+        cv_f1.append(f1)
+        cv_precision.append(prec)
+        cv_recall.append(rec)
+
+        fold_weights = derive_metric_weights(feature_cols, model.feature_importances_)
+        fold_results.append({
+            "fold": fold + 1,
+            "train_idx": train_idx.tolist(),
+            "val_idx": val_idx.tolist(),
+            "weights": fold_weights,
+            "metrics": {
+                "auc": round(float(auc), 4),
+                "f1": round(float(f1), 4),
+                "precision": round(float(prec), 4),
+                "recall": round(float(rec), 4),
+            },
+            "date_ranges": {
+                "train": f"{labeled[train_idx[0]]['date']} to {labeled[train_idx[-1]]['date']}",
+                "validation": f"{labeled[val_idx[0]]['date']} to {labeled[val_idx[-1]]['date']}",
+            },
+            "n_train": len(train_idx),
+            "n_validation": len(val_idx),
+        })
+
+        train_dates = fold_results[-1]["date_ranges"]["train"]
+        val_dates = fold_results[-1]["date_ranges"]["validation"]
+        log.info("Fold %d: Train %s | Val %s | AUC=%.3f F1=%.3f P=%.3f R=%.3f",
+                 fold + 1, train_dates, val_dates, auc, f1, prec, rec)
+
+    log.info("Purged CV Mean AUC: %.3f (+/- %.3f)", np.mean(cv_scores), np.std(cv_scores))
+    log.info("Purged CV Mean F1:  %.3f (+/- %.3f)", np.mean(cv_f1), np.std(cv_f1))
+
+    # Train final model on all labeled data
+    log.info("Training final model on all %d labeled samples...", len(labeled))
+    scaler = StandardScaler()
+    X_scaled = scaler.fit_transform(X)
+
+    final_model = _build_model()
+    final_model.fit(X_scaled, y)
+
+    # Feature importances
+    importances = final_model.feature_importances_
+    feat_imp = sorted(
+        zip(feature_cols, importances),
+        key=lambda x: x[1],
+        reverse=True,
+    )
+
+    log.info("\nFeature Importance Ranking:")
+    log.info("-" * 50)
+    for name, imp in feat_imp:
+        bar = "#" * int(imp * 200)
+        log.info("  %-30s %.4f %s", name, imp, bar)
+
+    weights = derive_metric_weights(feature_cols, importances)
+
+    log.info("\nOptimal Metric Weights:")
+    log.info("-" * 50)
+    equal_weight = round(1 / len(weights), 4)
+    for metric, w in weights.items():
+        change = "+" if w > equal_weight else ""
+        diff = (w - equal_weight) / equal_weight * 100
+        log.info("  %-25s %.4f (%s%.0f%% vs equal)", metric, w, change, diff)
+
+    # Run comparison backtest: ML-weighted vs equal-weight
+    log.info("\n" + "=" * 60)
+    log.info("COMPARISON BACKTEST: ML-Weighted vs Equal-Weight")
+    log.info("=" * 60)
+    comparison = run_comparison(rows, weights)
+    out_of_sample_comparison = run_out_of_sample_comparison(labeled, fold_results)
+
+    # Build output. Final weights are fitted on all labeled history for live use;
+    # only the fold weights below are valid for OOS comparisons.
+    trained_at = datetime.now(tz=__import__('datetime').timezone.utc).isoformat()
+    result = {
+        "artifact_schema_version": ML_ARTIFACT_SCHEMA_VERSION,
+        "score_version": SCORE_VERSION,
+        "weights": weights,
+        "feature_importances": {name: round(float(imp), 6) for name, imp in feat_imp},
+        "cv_results": {
+            "mean_auc": round(float(np.mean(cv_scores)), 4),
+            "std_auc": round(float(np.std(cv_scores)), 4),
+            "mean_f1": round(float(np.mean(cv_f1)), 4),
+            "mean_precision": round(float(np.mean(cv_precision)), 4),
+            "mean_recall": round(float(np.mean(cv_recall)), 4),
+            "validation_method": "purged_expanding_window",
+            "label_horizon_days": LABEL_HORIZON_DAYS,
+            "folds": artifact_fold_results(fold_results),
+        },
+        "training_info": {
+            "n_samples": len(labeled),
+            "n_positive": int(positive),
+            "positive_rate": round(positive / len(labeled), 4),
+            "n_features": len(feature_cols),
+            "target_threshold": GOOD_BUY_THRESHOLD,
+            "date_range": f"{labeled[0]['date']} to {labeled[-1]['date']}",
+            "model": "GradientBoostingClassifier",
+        },
+        "provenance": {
+            "validation_method": "purged_expanding_window",
+            "label_horizon_days": LABEL_HORIZON_DAYS,
+            "weight_scope": "full_history_fit",
+            "training_date_range": {
+                "start": labeled[0]["date"],
+                "end": labeled[-1]["date"],
+            },
+            "trained_at": trained_at,
+        },
+        "comparison": comparison,
+        "out_of_sample_comparison": out_of_sample_comparison,
+        "trained_at": trained_at,
+    }
+
+    return result
+
+
+def _composite_score(row, mode, ml_weights=None):
+    scores = [row[f"score_{k}"] for k in SCORE_KEYS]
+    if mode == "equal_weight" or not ml_weights:
+        return sum(scores) / len(SCORE_KEYS) * 10
+    equal_weight = 1.0 / len(SCORE_KEYS)
+    weighted_sum = sum(row[f"score_{k}"] * ml_weights.get(k, equal_weight) for k in SCORE_KEYS)
+    return weighted_sum * 10
+
+
+def _summarize_brackets(scored_rows, score_key):
+    results = []
+    for low, high, label in BRACKETS:
+        days_in = [r for r in scored_rows if score_in_bracket(r[score_key], (low, high, label))]
+        if not days_in:
+            results.append({
+                "range": f"{low}-{high}", "label": label,
+                "days": 0, "avg_365d": None,
+            })
+            continue
+        returns_365 = [r["fwd_365d"] for r in days_in]
+        returns_sorted = sorted(returns_365)
+        win_rate = len([r for r in returns_365 if r > 0]) / len(returns_365) * 100
+        results.append({
+            "range": f"{low}-{high}",
+            "label": label,
+            "days": len(days_in),
+            "avg_365d": round(sum(returns_365) / len(returns_365), 2),
+            "median_365d": round(returns_sorted[len(returns_sorted) // 2], 2),
+            "win_rate_365d": round(win_rate, 1),
+        })
+    return results
+
+
+def _log_comparison_table(results):
+    log.info("\n%-18s | %-8s %-8s %-8s | %-8s %-8s %-8s",
+             "Bracket", "EQ Avg", "EQ Med", "EQ Win%", "ML Avg", "ML Med", "ML Win%")
+    log.info("-" * 80)
+    for eq, ml in zip(results["equal_weight"], results["ml_weighted"]):
+        eq_avg = f"{eq['avg_365d']:.1f}%" if eq["avg_365d"] is not None else "--"
+        eq_med = f"{eq['median_365d']:.1f}%" if eq.get("median_365d") is not None else "--"
+        eq_win = f"{eq['win_rate_365d']:.0f}%" if eq.get("win_rate_365d") is not None else "--"
+        ml_avg = f"{ml['avg_365d']:.1f}%" if ml["avg_365d"] is not None else "--"
+        ml_med = f"{ml['median_365d']:.1f}%" if ml.get("median_365d") is not None else "--"
+        ml_win = f"{ml['win_rate_365d']:.0f}%" if ml.get("win_rate_365d") is not None else "--"
+        log.info("%-18s | %-8s %-8s %-8s | %-8s %-8s %-8s",
+                 eq["label"], eq_avg, eq_med, eq_win, ml_avg, ml_med, ml_win)
+
+
+def run_comparison(rows, ml_weights):
+    """Compare final ML-weighted scoring vs equal-weight scoring across all labeled rows.
+
+    This is retained for backwards compatibility with existing output. It is an
+    in-sample/full-history comparison; prefer out_of_sample_comparison for model
+    selection decisions.
+    """
+    scored_rows = [dict(r) for r in rows if "fwd_365d" in r]
+    for r in scored_rows:
+        r["composite_equal_weight"] = _composite_score(r, "equal_weight")
+        r["composite_ml_weighted"] = _composite_score(r, "ml_weighted", ml_weights)
+
+    results = {
+        "equal_weight": _summarize_brackets(scored_rows, "composite_equal_weight"),
+        "ml_weighted": _summarize_brackets(scored_rows, "composite_ml_weighted"),
+    }
+    _log_comparison_table(results)
+    return results
+
+
+def run_out_of_sample_comparison(rows, fold_results):
+    """Compare fold-specific ML weights on validation rows only."""
+    validation_rows = []
+    for fold in fold_results:
+        weights = fold.get("weights", {})
+        for idx in fold.get("val_idx", []):
+            if idx >= len(rows) or "fwd_365d" not in rows[idx]:
+                continue
+            r = dict(rows[idx])
+            r["fold"] = fold.get("fold")
+            r["composite_equal_weight"] = _composite_score(r, "equal_weight")
+            r["composite_ml_weighted"] = _composite_score(r, "ml_weighted", weights)
+            validation_rows.append(r)
+
+    results = {
+        "folds": len(fold_results),
+        "validation_days": len(validation_rows),
+        "equal_weight": _summarize_brackets(validation_rows, "composite_equal_weight"),
+        "ml_weighted": _summarize_brackets(validation_rows, "composite_ml_weighted"),
+    }
+    return results
+
+
+def main():
+    log.info("=" * 60)
+    log.info("Bitcoin Accumulation Zone ML Optimizer")
+    log.info("=" * 60)
+
+    if not os.path.exists(HISTORY_PATH):
+        log.error("No historical data at %s. Run history collector first.", HISTORY_PATH)
+        sys.exit(1)
+
+    # Load data
+    log.info("Loading historical data...")
+    index = load_history()
+    thresholds = load_thresholds()
+
+    # Build dataset
+    log.info("Building training dataset...")
+    rows = build_dataset(index, thresholds)
+
+    # Train model
+    log.info("Training ML model...")
+    result = train_model(rows)
+
+    if result is None:
+        log.error("Training failed.")
+        sys.exit(1)
+
+    # Save weights
+    with open(OUTPUT_PATH, "w") as f:
+        json.dump(result, f, indent=2)
+    log.info("\nSaved ML weights to %s", OUTPUT_PATH)
+
+    # Print summary
+    log.info("\n" + "=" * 60)
+    log.info("SUMMARY")
+    log.info("=" * 60)
+    log.info("Model: %s", result["training_info"]["model"])
+    log.info("Samples: %d (%d positive)", result["training_info"]["n_samples"], result["training_info"]["n_positive"])
+    log.info("CV AUC: %.3f (+/- %.3f)", result["cv_results"]["mean_auc"], result["cv_results"]["std_auc"])
+    log.info("CV F1:  %.3f", result["cv_results"]["mean_f1"])
+    log.info("\nTop 5 Feature Importances:")
+    for name, imp in list(result["feature_importances"].items())[:5]:
+        log.info("  %-30s %.4f", name, imp)
+    log.info("\nMetric Weights (ML-Optimized):")
+    for metric, weight in result["weights"].items():
+        log.info("  %-25s %.1f%%", metric, weight * 100)
+
+
+if __name__ == "__main__":
+    main()

orchestrator.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-BTC ML Trading Strategy Optimizer — Orchestrator
+BTC Accumulation Signal Optimizer -- Orchestrator
 Coordinates the optimization loop across VPS, Windows PC (GPU), and Mac Mini (LLM).
 """
 
@@ -28,7 +28,8 @@ MAC_MINI_HOST = "bizzle@bizzles-mac-mini-1"
 MAX_ITERATIONS = 50
 CONVERGENCE_WINDOW = 5
 CONVERGENCE_THRESHOLD = 0.01  # 1% improvement
-TARGET_SHARPE = 3.0
+TARGET_COST_IMPROVEMENT = 20.0  # Backward-compatible name: terminal wealth objective
+MIN_SIGNAL_COUNT = 30  # Minimum strong buy signals for valid results
 ML_TIMEOUT = 600  # 10 minutes
 
 # Colors
@@ -48,6 +49,11 @@ def log(msg, color=""):
     print(f"{C.DIM}[{ts}]{C.RESET} {color}{msg}{C.RESET}")
 
 
+def objective_score(results):
+    """Return the equal-capital portfolio objective used for model selection."""
+    return float(results.get("terminal_wealth_improvement_pct", 0.0))
+
+
 def run_cmd(cmd, timeout=120, check=True):
     """Run a shell command and return stdout."""
     result = subprocess.run(
@@ -98,7 +104,6 @@ def run_ml_training():
     )
     if result.returncode != 0:
         raise RuntimeError(f"ML training failed:\n{result.stderr}\n{result.stdout}")
-    # Print training output
     for line in result.stdout.strip().split("\n"):
         log(f"  {C.DIM}{line}", C.DIM)
     return True
@@ -127,45 +132,54 @@ def check_convergence(history):
     if len(history) < CONVERGENCE_WINDOW + 1:
         return False, "Not enough iterations"
 
-    recent = history[-CONVERGENCE_WINDOW:]
+    # Only consider valid results (enough signals)
-    sharpes = [h["sharpe"] for h in recent]
+    valid = [h for h in history if h.get("signal_count", 0) >= MIN_SIGNAL_COUNT]
 
-    # Check if best sharpe exceeds target
+    if not valid:
-    best_sharpe = max(h["sharpe"] for h in history)
+        return False, "No valid results yet"
-    if best_sharpe >= TARGET_SHARPE:
+
-        return True, f"Target Sharpe reached: {best_sharpe:.3f}"
+    recent = history[-CONVERGENCE_WINDOW:]
+    scores = [h.get("cost_improvement", 0) for h in recent]
+
+    # Check if best score exceeds target
+    best_score = max(h.get("cost_improvement", 0) for h in valid)
+    if best_score >= TARGET_COST_IMPROVEMENT:
+        return True, f"Target cost improvement reached: {best_score:.1f}%"
 
     # Check if improvement has stalled
-    best_recent = max(sharpes)
+    best_recent = max(scores)
-    worst_recent = min(sharpes)
+    worst_recent = min(scores)
     if best_recent > 0 and (best_recent - worst_recent) / best_recent < CONVERGENCE_THRESHOLD:
-        return True, f"Converged: Sharpe variance < {CONVERGENCE_THRESHOLD*100}% over {CONVERGENCE_WINDOW} iterations"
+        return True, f"Converged: variance < {CONVERGENCE_THRESHOLD*100}% over {CONVERGENCE_WINDOW} iterations"
 
     return False, ""
 
 
 def print_header():
     print(f"""
-{C.BOLD}{C.CYAN}╔══════════════════════════════════════════════════╗
+{C.BOLD}{C.CYAN}========================================================
-║       BTC ML Trading Strategy Optimizer          ║
+       BTC Accumulation Signal Optimizer
-║     VPS → Windows GPU → Mac Mini LLM → Loop      ║
+     VPS -> Windows GPU -> Mac Mini LLM -> Loop
-╚══════════════════════════════════════════════════╝{C.RESET}
+========================================================{C.RESET}
 """)
 
 
 def print_results(results, iteration):
-    sharpe = results.get("sharpe_ratio", 0)
+    objective = objective_score(results)
-    sharpe_color = C.GREEN if sharpe > 1.5 else C.YELLOW if sharpe > 1.0 else C.RED
+    color = C.GREEN if objective > 15 else C.YELLOW if objective > 10 else C.RED
     print(f"""
-{C.BOLD}━━━ Iteration {iteration} Results ━━━{C.RESET}
+{C.BOLD}--- Iteration {iteration} Results ---{C.RESET}
-  Sharpe Ratio:    {sharpe_color}{C.BOLD}{sharpe:.3f}{C.RESET}
+  Terminal Wealth vs DCA: {color}{C.BOLD}{objective:.1f}%{C.RESET}
-  Total Return:    {results.get('total_return_pct', 0):.1f}%
+  Legacy Cost Basis Delta: {results.get('cost_basis_improvement_pct', 0):.1f}%
-  Max Drawdown:    {results.get('max_drawdown_pct', 0):.1f}%
+  Avg Cost (Model): ${results.get('avg_cost_basis_model', 0):,.2f}
-  Win Rate:        {results.get('win_rate', 0):.1%}
+  Avg Cost (DCA):   ${results.get('avg_cost_basis_dca', 0):,.2f}
-  Trade Count:     {results.get('trade_count', 0)}
+  Strong Signals:   {results.get('strong_buy_signal_count', 0)}
-  Profit Factor:   {results.get('profit_factor', 0):.3f}
+  Signal Frequency: {results.get('signal_frequency_pct', 0):.1f}%
-  Avg Duration:    {results.get('avg_trade_duration_candles', 0):.1f} candles
+  Quality Score:    {results.get('pct_quality_strong_buy', 0):.1%}
-  Window Sharpes:  {results.get('per_window_sharpe', [])}
+  Model R2:         {results.get('model_r2_score', 0):.4f}
+  Score@Bottoms:    {results.get('avg_score_at_actual_bottoms', 0):.1f}
+  Score@Tops:       {results.get('avg_score_at_actual_tops', 0):.1f}
+  Window Improvements: {results.get('per_window_cost_improvement', [])}
 """)
 
 
@@ -173,14 +187,11 @@ def main():
     print_header()
     os.makedirs(RESULTS_DIR, exist_ok=True)
 
-    # Step 1: Ensure data
     ensure_data()
 
-    # Step 2: Load or create initial config
     config_path = os.path.join(CONFIG_DIR, "initial_config.json")
     best_config_path = os.path.join(CONFIG_DIR, "best_config.json")
 
-    # Resume from best config if it exists
     if os.path.exists(best_config_path):
         log("Resuming from best_config.json", C.GREEN)
         with open(best_config_path) as f:
@@ -191,29 +202,24 @@ def main():
 
     history = load_iteration_history()
     start_iter = len(history) + 1
-    best_sharpe = max((h["sharpe"] for h in history), default=0)
+    best_score = max((h.get("cost_improvement", 0) for h in history), default=0)
 
-    log(f"Starting at iteration {start_iter}, best Sharpe so far: {best_sharpe:.3f}", C.BOLD)
+    log(f"Starting at iteration {start_iter}, best cost improvement so far: {best_score:.1f}%", C.BOLD)
 
-    # Step 3: Setup Windows remote
     setup_windows_remote()
 
-    # SCP the ML engine script (once)
     log("Uploading ML engine to Windows...", C.CYAN)
     scp_to_windows(os.path.join(BASE_DIR, "ml_engine", "train_and_backtest.py"), "train_and_backtest.py")
 
-    # SCP data files (once)
     for tf in ["1h", "4h"]:
         data_file = os.path.join(DATA_DIR, f"btc_{tf}.csv")
         if os.path.exists(data_file):
             log(f"Uploading btc_{tf}.csv to Windows...", C.CYAN)
             scp_to_windows(data_file, f"btc_{tf}.csv")
 
-    # Import LLM analyzer
     sys.path.insert(0, os.path.join(BASE_DIR, "llm_client"))
     from analyzer import analyze_and_suggest
 
-    # Main optimization loop
     for iteration in range(start_iter, MAX_ITERATIONS + 1):
         log(f"\n{'='*50}", C.BOLD)
         log(f"ITERATION {iteration}/{MAX_ITERATIONS}", f"{C.BOLD}{C.CYAN}")
@@ -222,13 +228,11 @@ def main():
             f"Depth: {config.get('hyperparameters', {}).get('max_depth', '?')}", C.DIM)
         log(f"{'='*50}", C.BOLD)
 
-        # Write current config to temp file and SCP
         tmp_config = os.path.join(BASE_DIR, "config", "current_config.json")
         with open(tmp_config, "w") as f:
             json.dump(config, f, indent=2)
         scp_to_windows(tmp_config, "config.json")
 
-        # Run ML training on Windows
         try:
             run_ml_training()
         except (RuntimeError, subprocess.TimeoutExpired) as e:
@@ -238,7 +242,6 @@ def main():
                 config = history[-1].get("config", config)
             continue
 
-        # Fetch results from Windows
         results_local = os.path.join(RESULTS_DIR, f"results_iter_{iteration}.json")
         scp_from_windows("results.json", results_local)
 
@@ -247,34 +250,37 @@ def main():
 
         print_results(results, iteration)
 
-        # Track best
+        current_score = objective_score(results)
-        current_sharpe = results.get("sharpe_ratio", 0)
+        signal_count = results.get("strong_buy_signal_count", 0)
-        is_best = current_sharpe > best_sharpe
+        is_best = current_score > best_score and signal_count >= MIN_SIGNAL_COUNT
 
         if is_best:
-            best_sharpe = current_sharpe
+            best_score = current_score
             with open(best_config_path, "w") as f:
                 json.dump(config, f, indent=2)
-            log(f"NEW BEST! Sharpe: {best_sharpe:.3f}", f"{C.BOLD}{C.GREEN}")
+            log(f"NEW BEST! Terminal Wealth Improvement: {best_score:.1f}%", f"{C.BOLD}{C.GREEN}")
 
-        # Log iteration
         iter_data = {
             "iteration": iteration,
             "timestamp": datetime.now(timezone.utc).isoformat(),
-            "sharpe": current_sharpe,
+            "cost_improvement": current_score,
-            "return": results.get("total_return_pct", 0),
+            "objective_improvement": current_score,
-            "max_drawdown": results.get("max_drawdown_pct", 0),
+            "objective": "equal_periodic_contribution_terminal_wealth",
-            "win_rate": results.get("win_rate", 0),
+            "avg_30d_return": results.get("avg_quality_score_strong_buy", 0),
-            "trades": results.get("trade_count", 0),
+            "avg_90d_return": results.get("pct_quality_strong_buy", 0),
-            "profit_factor": results.get("profit_factor", 0),
+            "signal_count": signal_count,
+            "signal_frequency": results.get("signal_frequency_pct", 0),
+            "r2_score": results.get("model_r2_score", 0),
+            "score_at_bottoms": results.get("avg_score_at_actual_bottoms", 0),
+            "score_at_tops": results.get("avg_score_at_actual_tops", 0),
             "model_type": config.get("model_type", "unknown"),
             "is_best": is_best,
             "config": config,
+            "results": results,
         }
         save_iteration(iter_data)
         history.append(iter_data)
 
-        # Check convergence
         converged, reason = check_convergence(history)
         if converged:
             log(f"\nOptimization converged: {reason}", f"{C.BOLD}{C.GREEN}")
@@ -284,17 +290,15 @@ def main():
             log(f"\nMax iterations ({MAX_ITERATIONS}) reached.", C.YELLOW)
             break
 
-        # Ask LLM for next config
         log("\nConsulting LLM for strategy modifications...", C.MAGENTA)
         try:
             summary_history = [
                 {
                     "iteration": h["iteration"],
-                    "sharpe": h["sharpe"],
+                    "cost_improvement": h.get("cost_improvement", 0),
-                    "return": h["return"],
+                    "signal_count": h.get("signal_count", 0),
-                    "win_rate": h["win_rate"],
+                    "r2_score": h.get("r2_score", 0),
-                    "trades": h["trades"],
+                    "model_type": h.get("model_type", "unknown"),
-                    "model_type": h["model_type"],
                 }
                 for h in history
             ]
@@ -304,37 +308,34 @@ def main():
         except Exception as e:
             log(f"LLM call failed: {e}", C.RED)
             log("Continuing with current config + random perturbation...", C.YELLOW)
-            # Small random perturbation as fallback
             import random
             hp = config.get("hyperparameters", {})
-            hp["learning_rate"] = hp.get("learning_rate", 0.05) * random.uniform(0.8, 1.2)
+            hp["learning_rate"] = hp.get("learning_rate", 0.01) * random.uniform(0.8, 1.2)
-            hp["max_depth"] = max(3, min(10, hp.get("max_depth", 6) + random.choice([-1, 0, 1])))
+            hp["max_depth"] = max(3, min(10, hp.get("max_depth", 5) + random.choice([-1, 0, 1])))
             config["hyperparameters"] = hp
 
-    # Final summary
     print(f"""
-{C.BOLD}{C.GREEN}╔══════════════════════════════════════════════════╗
+{C.BOLD}{C.GREEN}========================================================
-║              Optimization Complete!               ║
+              Optimization Complete!
-╚══════════════════════════════════════════════════╝{C.RESET}
+========================================================{C.RESET}
 
-  Total Iterations: {len(history)}
+  Total Iterations:     {len(history)}
-  Best Sharpe:      {C.BOLD}{best_sharpe:.3f}{C.RESET}
+  Best Terminal Wealth Improvement: {C.BOLD}{best_score:.1f}%{C.RESET}
-  Best Config:      {best_config_path}
+  Best Config:          {best_config_path}
-  Iteration Log:    {ITERATIONS_LOG}
+  Iteration Log:        {ITERATIONS_LOG}
 """)
 
 
 # --- Library API for dashboard integration ---
 
-# Shared state for dashboard
 _stop_event = threading.Event()
 _status = {
-    "state": "idle",  # idle, running, completed, error
+    "state": "idle",
     "iteration": 0,
     "max_iterations": MAX_ITERATIONS,
-    "best_sharpe": 0.0,
+    "best_score": 0.0,
     "error": None,
-    "llm_suggestions": [],  # list of {iteration, reasoning, changes}
+    "llm_suggestions": [],
 }
 _status_lock = threading.Lock()
 
@@ -352,15 +353,9 @@ def update_status(**kwargs):
 
 
 def run_optimization_loop(callback=None, config_override=None):
-    """
+    """Run the optimization loop from a background thread."""
-    Run the optimization loop. Designed to be called from a background thread.
-
-    Args:
-        callback: Called after each iteration with (iteration_number, iter_data_dict).
-        config_override: Optional dict to use instead of loading from disk.
-    """
     _stop_event.clear()
-    update_status(state="running", iteration=0, error=None, best_sharpe=0.0)
+    update_status(state="running", iteration=0, error=None, best_score=0.0)
 
     try:
         os.makedirs(RESULTS_DIR, exist_ok=True)
@@ -380,8 +375,8 @@ def run_optimization_loop(callback=None, config_override=None):
 
         history = load_iteration_history()
         start_iter = len(history) + 1
-        best_sharpe = max((h["sharpe"] for h in history), default=0)
+        best_score = max((h.get("cost_improvement", 0) for h in history), default=0)
-        update_status(best_sharpe=best_sharpe)
+        update_status(best_score=best_score)
 
         setup_windows_remote()
         scp_to_windows(os.path.join(BASE_DIR, "ml_engine", "train_and_backtest.py"), "train_and_backtest.py")
@@ -418,23 +413,28 @@ def run_optimization_loop(callback=None, config_override=None):
             with open(results_local) as f:
                 results = json.load(f)
 
-            current_sharpe = results.get("sharpe_ratio", 0)
+            current_score = objective_score(results)
-            is_best = current_sharpe > best_sharpe
+            signal_count = results.get("strong_buy_signal_count", 0)
+            is_best = current_score > best_score and signal_count >= MIN_SIGNAL_COUNT
+
             if is_best:
-                best_sharpe = current_sharpe
+                best_score = current_score
                 with open(best_config_path, "w") as f:
                     json.dump(config, f, indent=2)
-                update_status(best_sharpe=best_sharpe)
+                update_status(best_score=best_score)
 
             iter_data = {
                 "iteration": iteration,
                 "timestamp": datetime.now(timezone.utc).isoformat(),
-                "sharpe": current_sharpe,
+                "cost_improvement": current_score,
-                "return": results.get("total_return_pct", 0),
+                "objective_improvement": current_score,
-                "max_drawdown": results.get("max_drawdown_pct", 0),
+                "objective": "equal_periodic_contribution_terminal_wealth",
-                "win_rate": results.get("win_rate", 0),
+                "signal_count": signal_count,
-                "trades": results.get("trade_count", 0),
+                "signal_frequency": results.get("signal_frequency_pct", 0),
-                "profit_factor": results.get("profit_factor", 0),
+                "r2_score": results.get("model_r2_score", 0),
+                "score_at_bottoms": results.get("avg_score_at_actual_bottoms", 0),
+                "score_at_tops": results.get("avg_score_at_actual_tops", 0),
+                "quality": results.get("pct_quality_strong_buy", 0),
                 "model_type": config.get("model_type", "unknown"),
                 "is_best": is_best,
                 "config": config,
@@ -459,10 +459,10 @@ def run_optimization_loop(callback=None, config_override=None):
                 update_status(state="completed")
                 return
 
-            # LLM suggestion
             try:
                 summary_history = [
-                    {k: h[k] for k in ("iteration", "sharpe", "return", "win_rate", "trades", "model_type")}
+                    {k: h[k] for k in ("iteration", "cost_improvement", "signal_count", "r2_score", "model_type")
+                     if k in h}
                     for h in history
                 ]
                 new_config, reasoning = analyze_and_suggest(config, results, summary_history)
@@ -471,12 +471,25 @@ def run_optimization_loop(callback=None, config_override=None):
                         "iteration": iteration,
                         "reasoning": reasoning,
                     })
+                # Also persist LLM suggestion to iteration log
+                iter_data["llm_reasoning"] = reasoning
+                iter_data["llm_applied"] = True
                 config = new_config
-            except Exception:
+            except Exception as e:
-                import random
+                import random, traceback
+                err_msg = f"LLM call failed: {type(e).__name__}: {e}"
+                print(f"  WARNING: {err_msg}")
+                traceback.print_exc()
+                with _status_lock:
+                    _status["llm_suggestions"].append({
+                        "iteration": iteration,
+                        "reasoning": f"ERROR: {err_msg} — using random perturbation",
+                    })
+                iter_data["llm_reasoning"] = err_msg
+                iter_data["llm_applied"] = False
                 hp = config.get("hyperparameters", {})
-                hp["learning_rate"] = hp.get("learning_rate", 0.05) * random.uniform(0.8, 1.2)
+                hp["learning_rate"] = hp.get("learning_rate", 0.01) * random.uniform(0.8, 1.2)
-                hp["max_depth"] = max(3, min(10, hp.get("max_depth", 6) + random.choice([-1, 0, 1])))
+                hp["max_depth"] = max(3, min(10, hp.get("max_depth", 5) + random.choice([-1, 0, 1])))
                 config["hyperparameters"] = hp
 
         update_status(state="completed")

pyproject.toml

@@ -0,0 +1,34 @@
+[project]
+name = "btc-accumulation-monitor"
+version = "0.1.0"
+description = "Bitcoin accumulation metrics dashboard and historical scoring tools"
+readme = "README.md"
+requires-python = ">=3.11,<3.14"
+
+[dependency-groups]
+runtime = [
+  "fastapi>=0.116,<1",
+  "playwright>=1.54,<2",
+  "requests>=2.32,<3",
+  "uvicorn[standard]>=0.35,<1",
+]
+ml = [
+  "ccxt>=4.4,<5",
+  "numpy>=2.2,<3",
+  "pandas>=2.2,<3",
+  "scikit-learn>=1.6,<2",
+  "ta>=0.11,<1",
+]
+dev = [
+  "pytest>=8.4,<9",
+]
+
+[tool.uv]
+package = false
+
+default-groups = ["runtime", "ml", "dev"]
+
+[tool.pytest.ini_options]
+addopts = "-q"
+testpaths = ["tests"]
+pythonpath = ["."]

scoring/engine.py

@@ -0,0 +1,685 @@
+"""Scoring engine for Bitcoin accumulation zone metrics."""
+
+import json
+import os
+import logging
+
+from scoring.policy import SCORE_VERSION, assessment_for_score
+from ml.artifacts import validate_ml_artifact
+
+log = logging.getLogger(__name__)
+
+THRESHOLDS_PATH = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+    "config",
+    "thresholds.json",
+)
+
+
+def load_thresholds():
+    """Load scoring thresholds from config."""
+    try:
+        with open(THRESHOLDS_PATH) as f:
+            return json.load(f)
+    except Exception:
+        return {}
+
+
+def _score_range(value, ranges):
+    """Score a value using range-based thresholds.
+    Each range is [low, high, score]. null means unbounded.
+    """
+    if value is None:
+        return None
+    for low, high, score in ranges:
+        low_ok = low is None or value >= low
+        high_ok = high is None or value < high
+        if low_ok and high_ok:
+            return score
+    return 0
+
+
+def _score_range_inverted(value, ranges):
+    """Score where higher value = lower range index (for drawdown)."""
+    if value is None:
+        return None
+    for low, high, score in ranges:
+        low_ok = low is None or value >= low
+        high_ok = high is None or value < high
+        if low_ok and high_ok:
+            return score
+    return 0
+
+
+def score_fear_greed(value, thresholds=None):
+    """Score Fear & Greed index (0-100 input, 0-10 output)."""
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("fear_greed", {})
+    ranges = t.get("ranges", [[0, 10, 10], [11, 25, 7], [26, 45, 4], [46, 55, 2], [56, 75, 1], [76, 100, 0]])
+    score = _score_range(value, ranges)
+
+    if value <= 10:
+        desc = "Extreme Fear — historically excellent buying"
+    elif value <= 25:
+        desc = "Fear — good accumulation territory"
+    elif value <= 45:
+        desc = "Low neutral — moderate opportunity"
+    elif value <= 55:
+        desc = "Neutral"
+    elif value <= 75:
+        desc = "Greed — caution"
+    else:
+        desc = "Extreme Greed — poor time to accumulate"
+    return score, desc
+
+
+def score_puell_multiple(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("puell_multiple", {})
+    # Widened: post-halving Puell floors are rising (2016: 0.15, 2020: 0.3, 2024: 0.5+)
+    ranges = t.get("ranges", [[None, 0.4, 10], [0.4, 0.7, 8], [0.7, 1.0, 5], [1.0, 1.5, 3], [1.5, 2.0, 1], [2.0, None, 0]])
+    score = _score_range(value, ranges)
+
+    if value < 0.3:
+        desc = "Deep value — miners under extreme stress"
+    elif value < 0.5:
+        desc = "Low — miners selling below average"
+    elif value < 0.8:
+        desc = "Below average miner revenue"
+    elif value < 1.2:
+        desc = "Average miner revenue"
+    elif value < 2.0:
+        desc = "Above average — miners profiting well"
+    else:
+        desc = "Elevated — potential top signal"
+    return score, desc
+
+
+def score_mvrv_zscore(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("mvrv_zscore", {})
+    # Widened ranges: BTC cycles compress — Z-Score bottoms are getting shallower
+    # 2015 bottom: -0.6, 2018 bottom: -0.4, 2022 bottom: -0.3, next may be ~0
+    ranges = t.get("ranges", [[None, 0, 10], [0, 1.0, 8], [1.0, 2.0, 5], [2.0, 3, 3], [3, 5, 1], [5, None, 0]])
+    score = _score_range(value, ranges)
+
+    if value < 0:
+        desc = "Below realized value — historically perfect buy zone"
+    elif value < 1.0:
+        desc = "Near realized value — strong accumulation zone"
+    elif value < 2.0:
+        desc = "Fair value — decent entry territory"
+    elif value < 3:
+        desc = "Above fair value"
+    elif value < 5:
+        desc = "Overvalued territory"
+    else:
+        desc = "Extreme overvaluation — cycle top territory"
+    return score, desc
+
+
+def score_drawdown(value, thresholds=None):
+    """Score drawdown from ATH (value is % drawdown, e.g. 50 = 50% below ATH)."""
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("drawdown", {})
+    ranges = t.get("ranges", [[70, None, 10], [50, 70, 8], [30, 50, 6], [20, 30, 4], [10, 20, 2], [None, 10, 0]])
+    score = _score_range(value, ranges)
+
+    if value > 70:
+        desc = f"{value:.0f}% below ATH — extreme capitulation"
+    elif value > 50:
+        desc = f"{value:.0f}% below ATH — deep bear market"
+    elif value > 30:
+        desc = f"{value:.0f}% below ATH — significant correction"
+    elif value > 20:
+        desc = f"{value:.0f}% below ATH — moderate pullback"
+    elif value > 10:
+        desc = f"{value:.0f}% below ATH — minor dip"
+    else:
+        desc = f"{value:.0f}% below ATH — near all-time high"
+    return score, desc
+
+
+def score_price_vs_200w_sma(price, sma_200w, thresholds=None):
+    """Score price relative to 200-week SMA."""
+    if price is None or sma_200w is None or sma_200w == 0:
+        return None, "No data"
+    pct_above = ((price - sma_200w) / sma_200w) * 100
+    t = (thresholds or load_thresholds()).get("price_vs_200w_sma", {})
+    # Widened: BTC increasingly stays above 200W SMA as it matures
+    ranges = t.get("ranges", [[None, 0, 10], [0, 30, 7], [30, 60, 5], [60, 100, 2], [100, None, 0]])
+    score = _score_range(pct_above, ranges)
+
+    if pct_above < 0:
+        desc = f"Below 200W SMA — historically rare buy zone"
+    elif pct_above < 30:
+        desc = f"{pct_above:.0f}% above 200W SMA — strong value"
+    elif pct_above < 60:
+        desc = f"{pct_above:.0f}% above 200W SMA — fair value"
+    elif pct_above < 100:
+        desc = f"{pct_above:.0f}% above 200W SMA — extended"
+    else:
+        desc = f"{pct_above:.0f}% above 200W SMA — extremely overheated"
+    return score, desc
+
+
+def score_reserve_risk(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("reserve_risk", {})
+    ranges = t.get("ranges", [[None, 0.002, 10], [0.002, 0.005, 7], [0.005, 0.01, 4], [0.01, 0.02, 2], [0.02, None, 0]])
+    score = _score_range(value, ranges)
+
+    if value < 0.002:
+        desc = "Very low risk/reward — strong accumulation"
+    elif value < 0.005:
+        desc = "Low risk — good entry"
+    elif value < 0.01:
+        desc = "Moderate risk/reward"
+    elif value < 0.02:
+        desc = "Elevated risk"
+    else:
+        desc = "High risk — cycle top territory"
+    return score, desc
+
+
+def score_rhodl_ratio(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("rhodl_ratio", {})
+    ranges = t.get("ranges", [[None, 100, 10], [100, 500, 7], [500, 2000, 4], [2000, 10000, 1], [10000, None, 0]])
+    score = _score_range(value, ranges)
+
+    if value < 100:
+        desc = "Extreme low — long-term holders dominate"
+    elif value < 500:
+        desc = "Low — mature holder confidence"
+    elif value < 2000:
+        desc = "Moderate rotation"
+    elif value < 10000:
+        desc = "Elevated — new money entering"
+    else:
+        desc = "Extreme — speculative mania"
+    return score, desc
+
+
+def score_nupl(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    t = (thresholds or load_thresholds()).get("nupl", {})
+    # Widened: NUPL bottoms getting shallower as BTC matures
+    # 2015: -0.3, 2018: -0.28, 2022: -0.28, future may only dip to 0-0.1
+    ranges = t.get("ranges", [[None, 0, 10], [0, 0.3, 8], [0.3, 0.5, 4], [0.5, 0.75, 1], [0.75, None, 0]])
+    score = _score_range(value, ranges)
+
+    if value < 0:
+        desc = "Capitulation — holders underwater"
+    elif value < 0.3:
+        desc = "Hope/Fear — early recovery, good accumulation"
+    elif value < 0.5:
+        desc = "Optimism — moderate profit taking"
+    elif value < 0.75:
+        desc = "Belief/Greed — significant unrealized gains"
+    else:
+        desc = "Euphoria — extreme unrealized profit"
+    return score, desc
+
+
+def score_lth_realized_price(price, lth_rp, thresholds=None):
+    """Score price relative to Long-Term Holder realized price."""
+    if price is None or lth_rp is None or lth_rp == 0:
+        return None, "No data"
+    pct_above = ((price - lth_rp) / lth_rp) * 100
+    t = (thresholds or load_thresholds()).get("lth_realized_price", {})
+    # Widened: as BTC matures, price spends more time above LTH RP
+    # In 2024+, even "good" entries are 30-80% above LTH RP
+    ranges = t.get("ranges", [[None, 0, 10], [0, 30, 7], [30, 80, 5], [80, 150, 3], [150, None, 1]])
+    score = _score_range(pct_above, ranges)
+
+    if pct_above < 0:
+        desc = f"Below LTH cost basis — LTHs underwater (extreme value)"
+    elif pct_above < 30:
+        desc = f"{pct_above:.0f}% above LTH cost basis — strong value"
+    elif pct_above < 80:
+        desc = f"{pct_above:.0f}% above LTH cost basis — fair value"
+    elif pct_above < 150:
+        desc = f"{pct_above:.0f}% above LTH cost basis — moderate"
+    else:
+        desc = f"{pct_above:.0f}% above LTH cost basis — extended"
+    return score, desc
+
+
+def score_hash_ribbons(data, thresholds=None):
+    """Score hash ribbons based on buy signal detection."""
+    if not data:
+        return None, "No data"
+    if data.get("buy_signal"):
+        return 10, "Active buy signal — miner capitulation recovery"
+    return 3, "Normal mining activity"
+
+
+def score_sopr(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    if value < 0.98:
+        return 10, "Deep loss realization — capitulation, strong accumulation"
+    if value < 1.0:
+        return 8, "Below breakeven — capitulation, good accumulation"
+    if value < 1.02:
+        return 5, "Near breakeven — neutral"
+    if value < 1.05:
+        return 2, "Moderate profit taking"
+    return 0, "Elevated profit taking — caution"
+
+
+def score_sellside_risk(value, thresholds=None):
+    if value is None:
+        return None, "No data"
+    if value < 0.001:
+        return 10, "Very low sell-side risk — strong accumulation"
+    if value < 0.002:
+        return 8, "Low sell-side risk — good accumulation"
+    if value < 0.005:
+        return 5, "Moderate sell-side risk"
+    if value < 0.01:
+        return 2, "Elevated sell-side risk"
+    return 0, "High sell-side risk"
+
+
+def score_momentum_pct(value):
+    if value is None:
+        return None, "No data"
+    pct = value * 100
+    if pct >= 20:
+        return 10, f"Strong positive momentum (+{pct:.0f}%)"
+    if pct >= 0:
+        return 6, f"Mild positive momentum (+{pct:.0f}%)"
+    if pct >= -10:
+        return 4, f"Slightly negative momentum ({pct:.0f}%)"
+    if pct >= -25:
+        return 2, f"Weak momentum ({pct:.0f}%)"
+    return 1, f"Strong negative momentum ({pct:.0f}%)"
+
+
+def score_nvt_price(nvt_price, spot_price):
+    if nvt_price is None or spot_price is None or spot_price <= 0:
+        return None, "No data"
+    premium = (nvt_price - spot_price) / spot_price * 100
+    if premium < -25:
+        return 10, f"NVT price {abs(premium):.0f}% below spot — deep value"
+    if premium < -10:
+        return 8, f"NVT price {abs(premium):.0f}% below spot — undervalued"
+    if premium < 10:
+        relation = "below" if premium < 0 else "above"
+        return 5, f"NVT price {abs(premium):.0f}% {relation} spot — fair value"
+    if premium < 30:
+        return 2, f"NVT price {premium:.0f}% above spot — extended"
+    return 0, f"NVT price {premium:.0f}% above spot — overheated"
+
+
+def score_all(metrics):
+    """Score all metrics and return individual + composite scores."""
+    thresholds = load_thresholds()
+    results = []
+
+    # Fear & Greed
+    fg = metrics.get("fear_greed", {})
+    fg_score, fg_desc = score_fear_greed(fg.get("value"), thresholds)
+    results.append({
+        "name": "Fear & Greed Index",
+        "key": "fear_greed",
+        "value": fg.get("value"),
+        "display_value": f"{fg.get('value', 'N/A')} — {fg.get('classification', '')}",
+        "score": fg_score,
+        "description": fg_desc,
+        "recent": fg.get("recent", []),
+    })
+
+    # Puell Multiple
+    pm = metrics.get("puell_multiple", {})
+    pm_score, pm_desc = score_puell_multiple(pm.get("value"), thresholds)
+    results.append({
+        "name": "Puell Multiple",
+        "key": "puell_multiple",
+        "value": pm.get("value"),
+        "display_value": f"{pm.get('value', 'N/A'):.4f}" if pm.get("value") is not None else "N/A",
+        "score": pm_score,
+        "description": pm_desc,
+        "recent": pm.get("recent", []),
+    })
+
+    # MVRV Z-Score
+    mz = metrics.get("mvrv_zscore", {})
+    mz_score, mz_desc = score_mvrv_zscore(mz.get("value"), thresholds)
+    results.append({
+        "name": "MVRV Z-Score",
+        "key": "mvrv_zscore",
+        "value": mz.get("value"),
+        "display_value": f"{mz.get('value', 'N/A'):.2f}" if mz.get("value") is not None else "N/A",
+        "score": mz_score,
+        "description": mz_desc,
+        "recent": mz.get("recent", []),
+    })
+
+    # Drawdown from ATH
+    dd = metrics.get("drawdown", {})
+    dd_score, dd_desc = score_drawdown(dd.get("value"), thresholds)
+    results.append({
+        "name": "Drawdown from ATH",
+        "key": "drawdown",
+        "value": dd.get("value"),
+        "display_value": f"{dd.get('value', 0):.1f}%" if dd.get("value") is not None else "N/A",
+        "score": dd_score,
+        "description": dd_desc,
+        "recent": [],
+    })
+
+    # Price vs 200W SMA
+    sma = metrics.get("200w_sma", {})
+    price_data = metrics.get("price", {})
+    current_price = price_data.get("price") or sma.get("btc_price")
+    sma_val = sma.get("value")
+    sma_score, sma_desc = score_price_vs_200w_sma(current_price, sma_val, thresholds)
+    results.append({
+        "name": "Price vs 200W SMA",
+        "key": "price_vs_200w_sma",
+        "value": sma_val,
+        "display_value": f"${sma_val:,.0f}" if sma_val else "N/A",
+        "score": sma_score,
+        "description": sma_desc,
+        "recent": sma.get("recent", []),
+    })
+
+    # Reserve Risk
+    rr = metrics.get("reserve_risk", {})
+    rr_score, rr_desc = score_reserve_risk(rr.get("value"), thresholds)
+    results.append({
+        "name": "Reserve Risk",
+        "key": "reserve_risk",
+        "value": rr.get("value"),
+        "display_value": f"{rr.get('value', 'N/A'):.6f}" if rr.get("value") is not None else "N/A",
+        "score": rr_score,
+        "description": rr_desc,
+        "recent": rr.get("recent", []),
+    })
+
+    # RHODL Ratio
+    rh = metrics.get("rhodl_ratio", {})
+    rh_score, rh_desc = score_rhodl_ratio(rh.get("value"), thresholds)
+    results.append({
+        "name": "RHODL Ratio",
+        "key": "rhodl_ratio",
+        "value": rh.get("value"),
+        "display_value": f"{rh.get('value', 'N/A'):.0f}" if rh.get("value") is not None else "N/A",
+        "score": rh_score,
+        "description": rh_desc,
+        "recent": rh.get("recent", []),
+    })
+
+    # NUPL
+    nu = metrics.get("nupl", {})
+    nu_score, nu_desc = score_nupl(nu.get("value"), thresholds)
+    results.append({
+        "name": "Net Unrealized Profit/Loss",
+        "key": "nupl",
+        "value": nu.get("value"),
+        "display_value": f"{nu.get('value', 'N/A'):.4f}" if nu.get("value") is not None else "N/A",
+        "score": nu_score,
+        "description": nu_desc,
+        "recent": nu.get("recent", []),
+    })
+
+    # LTH Realized Price
+    lth = metrics.get("lth_realized_price", {})
+    lth_price = lth.get("btc_price") or current_price
+    lth_rp = lth.get("value")
+    lth_score, lth_desc = score_lth_realized_price(lth_price, lth_rp, thresholds)
+    results.append({
+        "name": "LTH Realized Price",
+        "key": "lth_realized_price",
+        "value": lth_rp,
+        "display_value": f"${lth_rp:,.0f}" if lth_rp else "N/A",
+        "score": lth_score,
+        "description": lth_desc,
+        "recent": lth.get("recent", []),
+    })
+
+    # Hash Ribbons
+    hr = metrics.get("hash_ribbons", {})
+    hr_score, hr_desc = score_hash_ribbons(hr, thresholds)
+    results.append({
+        "name": "Hash Ribbons",
+        "key": "hash_ribbons",
+        "value": None,
+        "display_value": "Buy Signal" if hr.get("buy_signal") else "Normal",
+        "score": hr_score,
+        "description": hr_desc,
+        "recent": [],
+    })
+
+    # SOPR
+    sopr = metrics.get("sopr", {})
+    sopr_score, sopr_desc = score_sopr(sopr.get("value"), thresholds)
+    results.append({
+        "name": "SOPR",
+        "key": "sopr",
+        "value": sopr.get("value"),
+        "display_value": f"{sopr.get('value', 'N/A'):.4f}" if sopr.get("value") is not None else "N/A",
+        "score": sopr_score,
+        "description": sopr_desc,
+        "recent": sopr.get("recent", []),
+    })
+
+    # Sell-side Risk Ratio
+    ssr = metrics.get("sellside_risk", {})
+    ssr_score, ssr_desc = score_sellside_risk(ssr.get("value"), thresholds)
+    results.append({
+        "name": "Sell-side Risk Ratio",
+        "key": "sellside_risk",
+        "value": ssr.get("value"),
+        "display_value": f"{ssr.get('value', 'N/A'):.6f}" if ssr.get("value") is not None else "N/A",
+        "score": ssr_score,
+        "description": ssr_desc,
+        "recent": ssr.get("recent", []),
+    })
+
+    # Active Address Momentum
+    aam = metrics.get("active_address_momentum", {})
+    aam_score, aam_desc = score_momentum_pct(aam.get("value"))
+    results.append({
+        "name": "Active Address Momentum",
+        "key": "active_address_momentum",
+        "value": aam.get("value"),
+        "display_value": f"{aam.get('value') * 100:.1f}%" if aam.get("value") is not None else "N/A",
+        "score": aam_score,
+        "description": aam_desc,
+        "recent": aam.get("recent", []),
+    })
+
+    # Transaction Count Momentum
+    txm = metrics.get("txcount_momentum", {})
+    txm_score, txm_desc = score_momentum_pct(txm.get("value"))
+    results.append({
+        "name": "Transaction Count Momentum",
+        "key": "txcount_momentum",
+        "value": txm.get("value"),
+        "display_value": f"{txm.get('value') * 100:.1f}%" if txm.get("value") is not None else "N/A",
+        "score": txm_score,
+        "description": txm_desc,
+        "recent": txm.get("recent", []),
+    })
+
+    # NVT Price
+    nvt = metrics.get("nvt_price", {})
+    nvt_score, nvt_desc = score_nvt_price(nvt.get("value"), current_price)
+    results.append({
+        "name": "NVT Price",
+        "key": "nvt_price",
+        "value": nvt.get("value"),
+        "display_value": f"${nvt.get('value'):,.0f}" if nvt.get("value") is not None else "N/A",
+        "score": nvt_score,
+        "description": nvt_desc,
+        "recent": nvt.get("recent", []),
+    })
+
+    # VDD Multiple
+    vdd = metrics.get("vdd_multiple", {})
+    vdd_score, vdd_desc = score_momentum_pct(vdd.get("value"))
+    results.append({
+        "name": "VDD 30-Period Momentum",
+        "key": "vdd_multiple",
+        "transform": "30_period_return",
+        "value": vdd.get("value"),
+        "display_value": f"{vdd.get('value') * 100:.1f}%" if vdd.get("value") is not None else "N/A",
+        "score": vdd_score,
+        "description": vdd_desc,
+        "recent": vdd.get("recent", []),
+    })
+
+    # Compute composite
+    valid_scores = [r["score"] for r in results if r["score"] is not None]
+    if valid_scores:
+        # Scale to 0-100 based on available metrics
+        composite = sum(valid_scores) / len(valid_scores) * 10
+    else:
+        composite = 0
+
+    assessment = assessment_for_score(composite)
+
+    return {
+        "metrics": results,
+        "composite_score": round(composite, 1),
+        "assessment": assessment,
+        "scored_count": len(valid_scores),
+        "total_count": len(results),
+        "score_version": SCORE_VERSION,
+        "metric_panel": {
+            "id": "live-all-v1",
+            "keys": [result["key"] for result in results],
+            "count": len(results),
+        },
+        "coverage": {
+            "available_count": len(valid_scores),
+            "panel_count": len(results),
+            "ratio": len(valid_scores) / len(results),
+        },
+    }
+
+
+# ── ML-Optimized Scoring ──────────────────────────────────────────────
+
+ML_WEIGHTS_PATH = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+    "config",
+    "ml_weights.json",
+)
+
+# Maps scoring engine metric keys to ML weight keys
+_ML_KEY_MAP = {
+    "fear_greed": "fear_greed",
+    "puell_multiple": "puell_multiple",
+    "mvrv_zscore": "mvrv_zscore",
+    "drawdown": "drawdown",
+    "price_vs_200w_sma": "pct_above_200w_sma",
+    "reserve_risk": "reserve_risk",
+    "rhodl_ratio": "rhodl_ratio",
+    "nupl": "nupl",
+    "lth_realized_price": "pct_above_lth_rp",
+}
+
+
+_ml_artifact_status = {"valid": False, "errors": ["not_loaded"]}
+
+
+def load_ml_weights():
+    """Load weights only when their schema and training provenance are valid."""
+    global _ml_artifact_status
+    try:
+        with open(ML_WEIGHTS_PATH) as f:
+            data = json.load(f)
+        _ml_artifact_status = validate_ml_artifact(data)
+        if not _ml_artifact_status["valid"]:
+            log.error("Rejected invalid ML artifact: %s", ", ".join(_ml_artifact_status["errors"]))
+            return {}
+        return data.get("weights", {})
+    except Exception as exc:
+        _ml_artifact_status = {"valid": False, "errors": [f"load_error:{exc}"]}
+        return {}
+
+
+def get_ml_artifact_status():
+    """Return the status from the most recent artifact load attempt."""
+    return dict(_ml_artifact_status)
+
+
+def score_all_ml(metrics):
+    """Score all metrics using ML-optimized weights.
+
+    Same output format as score_all() but uses learned weights
+    instead of equal weighting. Each metric still shows its
+    individual 0-10 score plus the ML weight applied to it.
+    """
+    # Get classic scores first (reuses all individual scoring logic)
+    classic = score_all(metrics)
+    ml_weights = load_ml_weights()
+
+    if not ml_weights:
+        # Fallback to classic if no ML weights available
+        classic["ml_mode"] = False
+        status = get_ml_artifact_status()
+        if status.get("errors") and status["errors"] != ["not_loaded"]:
+            classic["ml_error"] = "ML artifact invalid: " + ", ".join(status["errors"])
+        else:
+            classic["ml_error"] = "ML weights not found — run ml/optimizer.py"
+        classic["ml_artifact"] = status
+        return classic
+
+    results = classic["metrics"]
+
+    # Compute raw ML weights first, then normalize across only the currently
+    # scored metrics. This keeps the dashboard's displayed per-metric weights and
+    # contribution points consistent with the normalized composite score even
+    # when optional metrics are missing or hash ribbons receives its fallback.
+    weighted_metrics = []
+    for m in results:
+        if m["score"] is None:
+            continue
+        ml_key = _ML_KEY_MAP.get(m["key"])
+        if ml_key is None:
+            # Hash ribbons or unknown metric — use small default weight
+            raw_weight = 0.01
+        else:
+            raw_weight = ml_weights.get(ml_key, 0.0)
+        weighted_metrics.append((m, raw_weight))
+
+    weight_total = sum(raw_weight for _, raw_weight in weighted_metrics)
+    if weight_total > 0:
+        composite = sum(m["score"] * raw_weight for m, raw_weight in weighted_metrics) / weight_total * 10
+    else:
+        composite = 0
+
+    for m, raw_weight in weighted_metrics:
+        effective_weight = raw_weight / weight_total if weight_total > 0 else 0.0
+        m["ml_raw_weight"] = round(raw_weight, 4)
+        m["ml_weight"] = round(effective_weight, 4)
+        m["ml_contribution"] = round(m["score"] * effective_weight * 10, 2)
+
+    assessment = assessment_for_score(composite)
+
+    return {
+        "metrics": results,
+        "composite_score": round(composite, 1),
+        "assessment": assessment,
+        "scored_count": classic["scored_count"],
+        "total_count": classic["total_count"],
+        "ml_mode": True,
+        "classic_score": classic["composite_score"],
+        "ml_weight_total": round(weight_total, 4),
+        "score_version": SCORE_VERSION,
+        "metric_panel": classic["metric_panel"],
+        "coverage": classic["coverage"],
+    }

scoring/policy.py

@@ -0,0 +1,35 @@
+"""Canonical score version, brackets, and assessment semantics."""
+
+SCORE_VERSION = "accumulation-score-v2"
+
+# Half-open intervals [low, high), except the final bracket includes 100.
+# Keep labels canonical because they are persisted in live and backtest output.
+SCORE_BRACKETS = [
+    (0, 20, "EXTREME CAUTION"),
+    (20, 35, "CAUTION — OVERHEATED"),
+    (35, 50, "NEUTRAL"),
+    (50, 65, "MODERATE OPPORTUNITY"),
+    (65, 80, "STRONG ACCUMULATION ZONE"),
+    (80, 100, "EXTREME ACCUMULATION ZONE"),
+]
+
+
+def score_in_bracket(score, bracket):
+    """Return whether a 0-100 score belongs to a canonical bracket."""
+    low, high, _ = bracket
+    if not 0 <= score <= 100:
+        return False
+    return low <= score < high or (high == 100 and score == 100)
+
+
+def bracket_for_score(score):
+    """Return the one canonical bracket for a 0-100 score."""
+    for bracket in SCORE_BRACKETS:
+        if score_in_bracket(score, bracket):
+            return bracket
+    raise ValueError(f"score must be between 0 and 100, got {score!r}")
+
+
+def assessment_for_score(score):
+    """Return the canonical assessment label for a score."""
+    return bracket_for_score(score)[2]

scrapers/checkonchain.py

@@ -0,0 +1,170 @@
+"""Scraper for static CheckOnChain Plotly chart HTML pages."""
+
+from __future__ import annotations
+
+import array
+import base64
+import json
+import logging
+import re
+from html import unescape
+
+import requests
+
+log = logging.getLogger(__name__)
+
+CHARTS = {
+    "sopr": {
+        "url": "https://charts.checkonchain.com/btconchain/realised/sopr/sopr_light.html",
+        "traces": ["SOPR"],
+    },
+    "sellside_risk": {
+        "url": "https://charts.checkonchain.com/btconchain/realised/sellsideriskratio_all/sellsideriskratio_all_light.html",
+        "traces": ["Sell-side Risk Ratio", "Sellside Risk Ratio", "SSR"],
+    },
+    "active_address_momentum": {
+        "url": "https://charts.checkonchain.com/btconchain/adoption/actaddress_momentum/actaddress_momentum_light.html",
+        "traces": ["30DMA", "30 Day", "Active Address"],
+    },
+    "txcount_momentum": {
+        "url": "https://charts.checkonchain.com/btconchain/adoption/txcount_momentum/txcount_momentum_light.html",
+        "traces": ["30DMA", "30 Day", "Transaction"],
+    },
+    "nvt_price": {
+        "url": "https://charts.checkonchain.com/btconchain/pricing/pricing_nvtprice/pricing_nvtprice_light.html",
+        "traces": ["NVT Price", "NVT"],
+    },
+    "vdd_multiple": {
+        "url": "https://charts.checkonchain.com/btconchain/lifespan/vddmultiple/vddmultiple_light.html",
+        "traces": ["VDD Multiple", "Value Days Destroyed"],
+    },
+}
+
+
+def _extract_plotly_traces(html_text: str):
+    """Extract first Plotly.newPlot trace array from a static Plotly HTML page."""
+    marker = "Plotly.newPlot("
+    start = html_text.find(marker)
+    if start < 0:
+        return []
+    first_array = html_text.find("[", start)
+    if first_array < 0:
+        return []
+
+    depth = 0
+    in_string = False
+    escape = False
+    quote = ""
+    for idx in range(first_array, len(html_text)):
+        ch = html_text[idx]
+        if in_string:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == quote:
+                in_string = False
+            continue
+        if ch in {'"', "'"}:
+            in_string = True
+            quote = ch
+        elif ch == "[":
+            depth += 1
+        elif ch == "]":
+            depth -= 1
+            if depth == 0:
+                raw = html_text[first_array:idx + 1]
+                return json.loads(raw)
+    return []
+
+
+def scrape_chart(url: str, timeout=30):
+    resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=timeout)
+    resp.raise_for_status()
+    return _extract_plotly_traces(unescape(resp.text))
+
+
+def _find_trace(traces, names):
+    names = [n.lower() for n in names if n]
+    # Prefer non-price traces with the requested terms.
+    for trace in traces:
+        trace_name = str(trace.get("name", "")).lower()
+        if "price" in trace_name and not any("price" in n for n in names):
+            continue
+        if any(n in trace_name for n in names):
+            return trace
+    # Fallback: first numeric non-price trace.
+    for trace in traces:
+        trace_name = str(trace.get("name", "")).lower()
+        if "price" in trace_name:
+            continue
+        y = trace.get("y") or []
+        if any(v is not None for v in y[-30:]):
+            return trace
+    return None
+
+
+def _decode_plotly_array(values):
+    """Decode Plotly typed-array JSON ({dtype, bdata}) or return plain values."""
+    if not isinstance(values, dict) or "bdata" not in values:
+        return values or []
+
+    dtype = values.get("dtype")
+    typecodes = {
+        "f8": "d", "float64": "d",
+        "f4": "f", "float32": "f",
+        "i8": "q", "int64": "q",
+        "i4": "i", "int32": "i",
+        "u8": "Q", "uint64": "Q",
+        "u4": "I", "uint32": "I",
+    }
+    typecode = typecodes.get(dtype)
+    if not typecode:
+        return []
+    decoded = base64.b64decode(values["bdata"])
+    arr = array.array(typecode)
+    arr.frombytes(decoded)
+    if values.get("byteorder") == "big":
+        arr.byteswap()
+    return arr.tolist()
+
+
+def _numeric_values(trace):
+    values = []
+    for value in _decode_plotly_array((trace or {}).get("y", [])):
+        if value is None:
+            continue
+        try:
+            values.append(float(value))
+        except (TypeError, ValueError):
+            pass
+    return values
+
+
+def _latest(values):
+    return values[-1] if values else None
+
+
+def _momentum(values, window=30):
+    if len(values) <= window or values[-window] == 0:
+        return None
+    return (values[-1] - values[-window]) / values[-window]
+
+
+def scrape_all():
+    results = {}
+    for key, cfg in CHARTS.items():
+        log.info("Scraping CheckOnChain %s ...", key)
+        try:
+            traces = scrape_chart(cfg["url"])
+            trace = _find_trace(traces, cfg.get("traces", []))
+            values = _numeric_values(trace)
+            value = _latest(values)
+            if key in {"active_address_momentum", "txcount_momentum", "vdd_multiple"}:
+                # The card value is momentum, while the sparkline shows the raw metric.
+                value = _momentum(values)
+            results[key] = {"value": value, "recent": values[-30:]}
+        except Exception as exc:
+            log.error("CheckOnChain scrape failed for %s: %s", key, exc)
+            results[key] = {"value": None, "error": str(exc)}
+    return results

scrapers/fear_greed.py

@@ -0,0 +1,34 @@
+"""Fear & Greed Index from alternative.me API."""
+
+import logging
+import requests
+
+log = logging.getLogger(__name__)
+
+FNG_URL = "https://api.alternative.me/fng/?limit=30"
+
+
+def fetch():
+    """Fetch Fear & Greed data. Returns dict with value, classification, and recent history."""
+    try:
+        resp = requests.get(FNG_URL, timeout=15)
+        resp.raise_for_status()
+        data = resp.json()
+        entries = data.get("data", [])
+        if not entries:
+            return {"value": None, "error": "No data"}
+
+        current = entries[0]
+        value = int(current["value"])
+        classification = current.get("value_classification", "")
+
+        recent = [int(e["value"]) for e in entries[:30]]
+
+        return {
+            "value": value,
+            "classification": classification,
+            "recent": recent,
+        }
+    except Exception as e:
+        log.error("Fear & Greed fetch error: %s", e)
+        return {"value": None, "error": str(e)}

scrapers/history_collector.py

@@ -0,0 +1,283 @@
+"""Collect full historical time series from LookIntoBitcoin charts, CoinGecko, and Fear & Greed."""
+
+import json
+import logging
+import os
+import time
+from datetime import datetime
+
+import requests
+
+log = logging.getLogger(__name__)
+
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+HISTORY_PATH = os.path.join(BASE_DIR, "data", "history.json")
+
+# Charts to scrape with expected trace names
+CHART_CONFIGS = {
+    "puell_multiple": {
+        "path": "/charts/puell-multiple/",
+        "traces": {"puell_multiple": "Puell Multiple", "btc_price": "Price"},
+    },
+    "mvrv_zscore": {
+        "path": "/charts/mvrv-zscore/",
+        "traces": {"mvrv_zscore": "Z-Score"},
+    },
+    "reserve_risk": {
+        "path": "/charts/reserve-risk/",
+        "traces": {"reserve_risk": "Reserve Risk"},
+    },
+    "rhodl_ratio": {
+        "path": "/charts/rhodl-ratio/",
+        "traces": {"rhodl_ratio": "RHODL Ratio"},
+    },
+    "nupl": {
+        "path": "/charts/relative-unrealized-profit--loss/",
+        "traces": {"nupl": "NUPL"},
+    },
+    "200w_sma": {
+        "path": "/charts/200-week-moving-average-heatmap/",
+        "traces": {"200w_sma": "200 Week Moving Average", "btc_price_sma": "Price"},
+    },
+    "lth_realized_price": {
+        "path": "/charts/long-term-holder-realized-price/",
+        "traces": {"lth_realized_price": "Long-Term Holder Realized Price", "btc_price_lth": "Price"},
+    },
+    "lth_supply": {
+        "path": "/charts/long-term-holder-supply/",
+        "traces": {"lth_supply": None},  # None = grab first numeric trace
+    },
+}
+
+
+def _find_trace(traces, name):
+    """Find a trace by name (case-insensitive partial match)."""
+    if not traces or not name:
+        return None
+    name_lower = name.lower()
+    for t in traces:
+        trace_name = t.get("name", "").lower()
+        if name_lower in trace_name or trace_name in name_lower:
+            return t
+    words = name_lower.split()
+    for t in traces:
+        trace_name = t.get("name", "").lower()
+        if all(w in trace_name for w in words):
+            return t
+    return None
+
+
+def _extract_series(trace):
+    """Extract (dates, values) from a Plotly trace dict."""
+    if not trace:
+        return [], []
+    x = trace.get("x", [])
+    y = trace.get("y", [])
+    dates = []
+    values = []
+    for i, (d, v) in enumerate(zip(x, y)):
+        if v is None:
+            continue
+        try:
+            val = float(v)
+        except (ValueError, TypeError):
+            continue
+        # Normalize date string to YYYY-MM-DD
+        date_str = str(d)[:10]
+        dates.append(date_str)
+        values.append(val)
+    return dates, values
+
+
+def scrape_chart_history(chart_path):
+    """Scrape a chart and return all trace data."""
+    from scrapers.lookintobitcoin import scrape_chart
+    return scrape_chart(chart_path)
+
+
+def collect_onchain_history(progress_cb=None):
+    """Scrape all on-chain charts and return dict of {metric: {dates, values}}."""
+    result = {}
+    total = len(CHART_CONFIGS)
+
+    for idx, (chart_key, cfg) in enumerate(CHART_CONFIGS.items()):
+        label = f"[{idx+1}/{total}] {chart_key}"
+        log.info("Scraping history: %s", label)
+        if progress_cb:
+            progress_cb(chart_key, idx, total)
+
+        try:
+            traces = scrape_chart_history(cfg["path"])
+            if not traces:
+                log.warning("No traces for %s", chart_key)
+                continue
+
+            for metric_key, trace_name in cfg["traces"].items():
+                if trace_name is None:
+                    if metric_key == "lth_supply":
+                        from scrapers.lookintobitcoin import _find_lth_supply_trace
+                        candidates = [_find_lth_supply_trace(traces)]
+                    else:
+                        candidates = traces
+                    # Grab the first validated trace with numeric data.
+                    for candidate in candidates:
+                        if not candidate:
+                            continue
+                        y = candidate.get("y", [])
+                        if y and any(v is not None for v in y[-10:]):
+                            dates, values = _extract_series(candidate)
+                            if dates:
+                                result[metric_key] = {"dates": dates, "values": values}
+                                log.info("  %s: %d data points", metric_key, len(dates))
+                            break
+                else:
+                    t = _find_trace(traces, trace_name)
+                    if not t:
+                        # Fallback: try BTC Price
+                        if "btc_price" in metric_key or "price" in trace_name.lower():
+                            t = _find_trace(traces, "BTC") or _find_trace(traces, "Price")
+                        if not t:
+                            log.warning("  Trace '%s' not found for %s", trace_name, metric_key)
+                            continue
+                    dates, values = _extract_series(t)
+                    if dates:
+                        result[metric_key] = {"dates": dates, "values": values}
+                        log.info("  %s: %d data points (%s to %s)", metric_key, len(dates), dates[0], dates[-1])
+                    else:
+                        log.warning("  %s: no valid data points", metric_key)
+
+        except Exception as e:
+            log.error("Error scraping %s: %s", chart_key, e)
+
+        # Be polite between requests
+        if idx < total - 1:
+            time.sleep(2)
+
+    return result
+
+
+def collect_price_history():
+    """Fetch BTC price history from CoinGecko (max history)."""
+    log.info("Fetching BTC price history from CoinGecko...")
+    try:
+        resp = requests.get(
+            "https://api.coingecko.com/api/v3/coins/bitcoin/market_chart",
+            params={"vs_currency": "usd", "days": "max"},
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+        prices = data.get("prices", [])
+        dates = []
+        values = []
+        seen_dates = set()
+        for ts_ms, price in prices:
+            d = datetime.utcfromtimestamp(ts_ms / 1000).strftime("%Y-%m-%d")
+            if d not in seen_dates:
+                seen_dates.add(d)
+                dates.append(d)
+                values.append(round(price, 2))
+        log.info("CoinGecko BTC price: %d days (%s to %s)", len(dates), dates[0] if dates else "?", dates[-1] if dates else "?")
+        return {"dates": dates, "values": values}
+    except Exception as e:
+        log.error("CoinGecko price fetch failed: %s", e)
+        return None
+
+
+def collect_fear_greed_history():
+    """Fetch full Fear & Greed history from alternative.me."""
+    log.info("Fetching Fear & Greed history...")
+    try:
+        resp = requests.get(
+            "https://api.alternative.me/fng/",
+            params={"limit": "0"},
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json().get("data", [])
+        dates = []
+        values = []
+        for entry in reversed(data):  # API returns newest first
+            ts = int(entry["timestamp"])
+            d = datetime.utcfromtimestamp(ts).strftime("%Y-%m-%d")
+            dates.append(d)
+            values.append(int(entry["value"]))
+        log.info("Fear & Greed: %d days (%s to %s)", len(dates), dates[0] if dates else "?", dates[-1] if dates else "?")
+        return {"dates": dates, "values": values}
+    except Exception as e:
+        log.error("Fear & Greed fetch failed: %s", e)
+        return None
+
+
+def collect_all_history(progress_cb=None):
+    """Collect all historical data and save to history.json."""
+    log.info("=== Starting full historical data collection ===")
+    history = {}
+
+    # 1. On-chain metrics from LookIntoBitcoin
+    onchain = collect_onchain_history(progress_cb=progress_cb)
+    history.update(onchain)
+
+    # 2. BTC price from CoinGecko
+    price = collect_price_history()
+    if price:
+        history["btc_price_coingecko"] = price
+
+    # 3. Fear & Greed
+    fng = collect_fear_greed_history()
+    if fng:
+        history["fear_greed"] = fng
+
+    # Merge BTC price: prefer the LookIntoBitcoin trace (goes to 2010), fill gaps with CoinGecko
+    btc_keys = [k for k in history if "btc_price" in k]
+    if btc_keys:
+        # Use longest series as base
+        best = max(btc_keys, key=lambda k: len(history[k]["dates"]))
+        history["btc_price"] = history[best]
+        log.info("BTC price source: %s (%d days)", best, len(history[best]["dates"]))
+
+    # Add metadata
+    history["_metadata"] = {
+        "collected_at": datetime.utcnow().isoformat() + "Z",
+        "metrics": list(k for k in history if not k.startswith("_")),
+        "metric_counts": {k: len(v["dates"]) for k, v in history.items() if isinstance(v, dict) and "dates" in v},
+    }
+
+    # Save
+    os.makedirs(os.path.dirname(HISTORY_PATH), exist_ok=True)
+    with open(HISTORY_PATH, "w") as f:
+        json.dump(history, f, separators=(",", ":"))
+
+    size_mb = os.path.getsize(HISTORY_PATH) / 1024 / 1024
+    log.info("=== History saved to %s (%.1f MB) ===", HISTORY_PATH, size_mb)
+    log.info("Metrics collected: %s", ", ".join(k for k in history if not k.startswith("_")))
+
+    return history
+
+
+def load_history():
+    """Load history from disk."""
+    if not os.path.exists(HISTORY_PATH):
+        return None
+    with open(HISTORY_PATH) as f:
+        return json.load(f)
+
+
+def history_status():
+    """Check if history exists and return metadata."""
+    if not os.path.exists(HISTORY_PATH):
+        return {"exists": False}
+    try:
+        stat = os.stat(HISTORY_PATH)
+        with open(HISTORY_PATH) as f:
+            data = json.load(f)
+        meta = data.get("_metadata", {})
+        return {
+            "exists": True,
+            "collected_at": meta.get("collected_at"),
+            "metrics": meta.get("metrics", []),
+            "metric_counts": meta.get("metric_counts", {}),
+            "size_mb": round(stat.st_size / 1024 / 1024, 2),
+        }
+    except Exception as e:
+        return {"exists": True, "error": str(e)}

scrapers/history_updater.py

@@ -0,0 +1,112 @@
+"""Incremental history updater — appends new daily data to history.json from cache."""
+
+import json
+import logging
+import os
+from datetime import datetime, timezone
+
+log = logging.getLogger(__name__)
+
+DATA_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "data")
+HISTORY_PATH = os.path.join(DATA_DIR, "history.json")
+CACHE_PATH = os.path.join(DATA_DIR, "cache.json")
+
+
+def update_history():
+    """Append today's values from cache to history.json. Only adds NEW dates."""
+    if not os.path.exists(HISTORY_PATH):
+        log.warning("No history.json found — run full collection first")
+        return False
+
+    if not os.path.exists(CACHE_PATH):
+        log.warning("No cache.json found — run a scrape first")
+        return False
+
+    with open(HISTORY_PATH) as f:
+        history = json.load(f)
+    with open(CACHE_PATH) as f:
+        cache = json.load(f)
+
+    today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+    updated = False
+
+    # Map of cache keys to history keys and how to extract the value
+    mappings = {
+        "puell_multiple": {"history_key": "puell_multiple", "value_key": "value"},
+        "mvrv_zscore": {"history_key": "mvrv_zscore", "value_key": "value"},
+        "reserve_risk": {"history_key": "reserve_risk", "value_key": "value"},
+        "rhodl_ratio": {"history_key": "rhodl_ratio", "value_key": "value"},
+        "nupl": {"history_key": "nupl", "value_key": "value"},
+        "200w_sma": {"history_key": "200w_sma", "value_key": "value"},
+        "lth_realized_price": {"history_key": "lth_realized_price", "value_key": "value"},
+        "lth_supply": {"history_key": "lth_supply", "value_key": "value"},
+    }
+
+    for cache_key, mapping in mappings.items():
+        hkey = mapping["history_key"]
+        if hkey not in history:
+            continue
+
+        h = history[hkey]
+        dates = h.get("dates", [])
+        values = h.get("values", [])
+
+        # Skip if today already in history
+        if dates and dates[-1] >= today:
+            continue
+
+        # Get value from cache
+        cached = cache.get(cache_key, {})
+        val = cached.get(mapping["value_key"])
+        if val is not None:
+            dates.append(today)
+            values.append(val)
+            h["dates"] = dates
+            h["values"] = values
+            updated = True
+            log.info("Appended %s: %s = %s", hkey, today, val)
+
+    # Also update btc_price from cache
+    price_data = cache.get("price", {})
+    btc_price = price_data.get("price")
+    if btc_price and "btc_price" in history:
+        h = history["btc_price"]
+        if h["dates"][-1] < today:
+            h["dates"].append(today)
+            h["values"].append(btc_price)
+            updated = True
+
+    # BTC price for SMA chart
+    if btc_price and "btc_price_sma" in history:
+        h = history["btc_price_sma"]
+        if h["dates"][-1] < today:
+            h["dates"].append(today)
+            h["values"].append(btc_price)
+            updated = True
+
+    # BTC price for LTH chart
+    if btc_price and "btc_price_lth" in history:
+        h = history["btc_price_lth"]
+        if h["dates"][-1] < today:
+            h["dates"].append(today)
+            h["values"].append(btc_price)
+            updated = True
+
+    # Fear & Greed
+    fg = cache.get("fear_greed", {})
+    fg_val = fg.get("value")
+    if fg_val is not None and "fear_greed" in history:
+        h = history["fear_greed"]
+        if h["dates"][-1] < today:
+            h["dates"].append(today)
+            h["values"].append(int(fg_val))
+            updated = True
+
+    if updated:
+        with open(HISTORY_PATH, "w") as f:
+            json.dump(history, f)
+        log.info("History updated with %s data", today)
+    else:
+        log.info("History already up to date (last date >= %s)", today)
+
+    return updated

scrapers/lookintobitcoin.py

@@ -0,0 +1,301 @@
+"""Playwright scraper for LookIntoBitcoin / BitcoinMagazinePro charts."""
+
+import logging
+import traceback
+from contextlib import contextmanager
+
+log = logging.getLogger(__name__)
+
+BASE_URL = "https://www.lookintobitcoin.com"
+
+CHARTS = {
+    "puell_multiple": {
+        "path": "/charts/puell-multiple/",
+        "traces": ["Puell Multiple"],
+    },
+    "mvrv_zscore": {
+        "path": "/charts/mvrv-zscore/",
+        "traces": ["Z-Score"],
+    },
+    "reserve_risk": {
+        "path": "/charts/reserve-risk/",
+        "traces": ["Reserve Risk"],
+    },
+    "rhodl_ratio": {
+        "path": "/charts/rhodl-ratio/",
+        "traces": ["RHODL Ratio"],
+    },
+    "nupl": {
+        "path": "/charts/relative-unrealized-profit--loss/",
+        "traces": ["NUPL"],
+    },
+    "200w_sma": {
+        "path": "/charts/200-week-moving-average-heatmap/",
+        "traces": ["200 Week Moving Average"],
+    },
+    "lth_realized_price": {
+        "path": "/charts/long-term-holder-realized-price/",
+        "traces": ["Long-Term Holder Realized Price", "BTC Price"],
+    },
+    "hash_ribbons": {
+        "path": "/charts/hash-ribbons/",
+        "traces": None,
+    },
+    "pi_cycle_bottom": {
+        "path": "/charts/pi-cycle-top-bottom-indicator/",
+        "traces": None,
+    },
+    "lth_supply": {
+        "path": "/charts/long-term-holder-supply/",
+        "traces": None,
+    },
+}
+
+
+@contextmanager
+def browser_page():
+    """Open one headless browser page for a batch of chart requests."""
+    from playwright.sync_api import sync_playwright
+
+    with sync_playwright() as playwright:
+        browser = playwright.chromium.launch(headless=True)
+        try:
+            yield browser.new_page()
+        finally:
+            browser.close()
+
+
+def scrape_chart(chart_path, timeout=25000, page=None):
+    """Scrape one chart, optionally reusing a caller-owned browser page."""
+    if page is None:
+        with browser_page() as owned_page:
+            return scrape_chart(chart_path, timeout=timeout, page=owned_page)
+
+    store = {"data": None}
+
+    def handle_response(response):
+        if "_dash-update-component" in response.url:
+            try:
+                store["data"] = response.json()
+            except Exception:
+                pass
+
+    page.on("response", handle_response)
+    try:
+        page.goto(f"{BASE_URL}{chart_path}", timeout=timeout)
+        page.wait_for_timeout(6000)
+    except Exception as exc:
+        log.warning("Navigation error for %s: %s", chart_path, exc)
+    finally:
+        page.remove_listener("response", handle_response)
+
+    if store["data"]:
+        try:
+            return store["data"]["response"]["chart"]["figure"]["data"]
+        except (KeyError, TypeError):
+            # Try alternate response structures
+            try:
+                resp = store["data"]
+                if isinstance(resp, dict):
+                    for key in resp:
+                        val = resp[key]
+                        if isinstance(val, dict) and "figure" in val:
+                            return val["figure"]["data"]
+                        if isinstance(val, dict) and "chart" in val:
+                            return val["chart"]["figure"]["data"]
+            except Exception:
+                pass
+    return None
+
+
+def _find_trace(traces, name):
+    """Find a trace by name (case-insensitive partial match)."""
+    if not traces:
+        return None
+    name_lower = name.lower()
+    # First pass: exact or substring match
+    for t in traces:
+        trace_name = t.get("name", "").lower()
+        if name_lower in trace_name or trace_name in name_lower:
+            return t
+    # Second pass: check if all words in name appear in trace name
+    words = name_lower.split()
+    for t in traces:
+        trace_name = t.get("name", "").lower()
+        if all(w in trace_name for w in words):
+            return t
+    return None
+
+
+def _trace_signal_is_active(trace):
+    """Return true only when the signal trace is active at its latest point."""
+    if not trace:
+        return False
+    values = trace.get("y", [])
+    if not values:
+        return False
+    latest = values[-1]
+    try:
+        return latest is not None and float(latest) != 0
+    except (TypeError, ValueError):
+        return bool(latest)
+
+
+def _find_lth_supply_trace(traces):
+    """Select an explicitly named LTH supply series and never a price fallback."""
+    for trace in traces or []:
+        name = str(trace.get("name", "")).lower()
+        is_lth = "long-term holder" in name or "long term holder" in name or "lth" in name
+        if is_lth and "supply" in name and "price" not in name:
+            return trace
+    return None
+
+
+def _get_latest_value(trace):
+    """Get the most recent non-null y value from a trace."""
+    if not trace:
+        return None
+    y = trace.get("y", [])
+    for val in reversed(y):
+        if val is not None:
+            try:
+                return float(val)
+            except (ValueError, TypeError):
+                continue
+    return None
+
+
+def _get_recent_values(trace, n=30):
+    """Get the last n non-null values from a trace."""
+    if not trace:
+        return []
+    y = trace.get("y", [])
+    values = []
+    for val in reversed(y):
+        if val is not None:
+            try:
+                values.append(float(val))
+            except (ValueError, TypeError):
+                continue
+        if len(values) >= n:
+            break
+    values.reverse()
+    return values
+
+
+def scrape_all():
+    """Scrape all charts while reusing one browser process and page."""
+    with browser_page() as page:
+        return _scrape_all_with_page(page)
+
+
+def _scrape_all_with_page(page):
+    results = {}
+
+    for metric_key, chart_info in CHARTS.items():
+        log.info("Scraping %s ...", metric_key)
+        try:
+            traces = scrape_chart(chart_info["path"], page=page)
+            if not traces:
+                log.warning("No data for %s", metric_key)
+                results[metric_key] = {"value": None, "error": "No data returned"}
+                continue
+
+            wanted = chart_info.get("traces")
+
+            if metric_key == "puell_multiple":
+                t = _find_trace(traces, "Puell Multiple")
+                val = _get_latest_value(t)
+                results[metric_key] = {
+                    "value": val,
+                    "recent": _get_recent_values(t),
+                }
+
+            elif metric_key == "mvrv_zscore":
+                t = _find_trace(traces, "Z-Score")
+                val = _get_latest_value(t)
+                results[metric_key] = {
+                    "value": val,
+                    "recent": _get_recent_values(t),
+                }
+
+            elif metric_key == "200w_sma":
+                t = _find_trace(traces, "200 Week Moving Average") or _find_trace(traces, "200 Week MA") or _find_trace(traces, "200W")
+                val = _get_latest_value(t)
+                # Also try to find BTC price trace
+                price_t = _find_trace(traces, "BTC Price") or _find_trace(traces, "Price")
+                price_val = _get_latest_value(price_t)
+                results[metric_key] = {
+                    "value": val,
+                    "btc_price": price_val,
+                    "recent": _get_recent_values(t),
+                }
+
+            elif metric_key == "lth_realized_price":
+                lth_t = _find_trace(traces, "Long-Term Holder Realized Price") or _find_trace(traces, "LTH Realized Price") or _find_trace(traces, "LTH")
+                price_t = _find_trace(traces, "BTC Price") or _find_trace(traces, "Price")
+                lth_val = _get_latest_value(lth_t)
+                price_val = _get_latest_value(price_t)
+                results[metric_key] = {
+                    "value": lth_val,
+                    "btc_price": price_val,
+                    "recent": _get_recent_values(lth_t),
+                }
+
+            elif metric_key == "hash_ribbons":
+                # Look for buy/sell signal traces or MA crossover
+                results[metric_key] = {
+                    "traces": [
+                        {"name": t.get("name", ""), "latest": _get_latest_value(t)}
+                        for t in traces[:6]
+                    ],
+                    "value": None,
+                }
+                # A named signal trace is not itself proof that the signal is active.
+                for t in traces:
+                    name = t.get("name", "").lower()
+                    if ("buy" in name or "signal" in name) and _trace_signal_is_active(t):
+                        results[metric_key]["buy_signal"] = True
+                        break
+
+            elif metric_key == "lth_supply":
+                # Require an explicitly named LTH supply trace; price is not supply.
+                t = _find_lth_supply_trace(traces)
+                recent = _get_recent_values(t, 60)
+                # Determine trend: compare recent avg to older avg
+                trend = None
+                if len(recent) >= 30:
+                    old_avg = sum(recent[:15]) / 15
+                    new_avg = sum(recent[-15:]) / 15
+                    trend = "increasing" if new_avg > old_avg else "decreasing"
+                results[metric_key] = {
+                    "value": _get_latest_value(t),
+                    "trend": trend,
+                    "recent": _get_recent_values(t),
+                }
+
+            else:
+                # Generic: grab first non-layout trace with numeric data
+                t = None
+                if wanted:
+                    for name in wanted:
+                        t = _find_trace(traces, name)
+                        if t:
+                            break
+                if not t:
+                    for candidate in traces:
+                        y = candidate.get("y", [])
+                        if y and any(v is not None for v in y[-10:]):
+                            t = candidate
+                            break
+                val = _get_latest_value(t)
+                results[metric_key] = {
+                    "value": val,
+                    "recent": _get_recent_values(t),
+                }
+
+        except Exception as e:
+            log.error("Error scraping %s: %s\n%s", metric_key, e, traceback.format_exc())
+            results[metric_key] = {"value": None, "error": str(e)}
+
+    return results

scrapers/price.py

@@ -0,0 +1,80 @@
+"""BTC price data from CoinGecko API."""
+
+import logging
+import requests
+
+log = logging.getLogger(__name__)
+
+PRICE_URL = "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_24hr_change=true"
+HISTORY_URL = "https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=365"
+ATH_URL = "https://api.coingecko.com/api/v3/coins/bitcoin?localization=false&tickers=false&market_data=true&community_data=false&developer_data=false"
+
+
+def fetch_current():
+    """Fetch current BTC price and 24h change."""
+    try:
+        resp = requests.get(PRICE_URL, timeout=15)
+        resp.raise_for_status()
+        data = resp.json()
+        btc = data.get("bitcoin", {})
+        return {
+            "price": btc.get("usd"),
+            "change_24h": btc.get("usd_24h_change"),
+        }
+    except Exception as e:
+        log.error("Price fetch error: %s", e)
+        return {"price": None, "error": str(e)}
+
+
+def fetch_historical():
+    """Fetch 365 days of BTC price history. Returns list of [timestamp, price]."""
+    try:
+        resp = requests.get(HISTORY_URL, timeout=30)
+        resp.raise_for_status()
+        data = resp.json()
+        prices = data.get("prices", [])
+        return prices
+    except Exception as e:
+        log.error("Historical price fetch error: %s", e)
+        return []
+
+
+def fetch_ath():
+    """Fetch BTC all-time high from CoinGecko."""
+    try:
+        resp = requests.get(ATH_URL, timeout=15)
+        resp.raise_for_status()
+        data = resp.json()
+        market = data.get("market_data", {})
+        ath = market.get("ath", {}).get("usd")
+        ath_change = market.get("ath_change_percentage", {}).get("usd")
+        return {
+            "ath": ath,
+            "ath_change_pct": ath_change,
+        }
+    except Exception as e:
+        log.error("ATH fetch error: %s", e)
+        return {"ath": None, "error": str(e)}
+
+
+def calculate_200d_sma(prices):
+    """Calculate 200-day SMA from historical price data."""
+    if not prices or len(prices) < 200:
+        return None
+    # prices is [[timestamp, price], ...]
+    recent_200 = [p[1] for p in prices[-200:]]
+    return sum(recent_200) / len(recent_200)
+
+
+def calculate_mayer_multiple(current_price, sma_200d):
+    """Mayer Multiple = current price / 200-day SMA."""
+    if not current_price or not sma_200d or sma_200d == 0:
+        return None
+    return current_price / sma_200d
+
+
+def calculate_drawdown(current_price, ath):
+    """Drawdown from ATH as percentage."""
+    if not current_price or not ath or ath == 0:
+        return None
+    return (ath - current_price) / ath * 100

scripts/run.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+set -eu
+
+ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
+cd "$ROOT"
+
+export PYTHONPATH="${PYTHONPATH:-.}"
+export PLAYWRIGHT_BROWSERS_PATH="${PLAYWRIGHT_BROWSERS_PATH:-$ROOT/.playwright}"
+
+exec uv run --frozen --no-dev --group runtime --group ml \
+  python -m uvicorn dashboard.server:app --host 0.0.0.0 --port "${PORT:-3088}"

tests/test_backtest_cache.py

@@ -0,0 +1,56 @@
+import json
+import os
+
+from backtesting import engine
+
+
+def test_run_backtest_caches_by_input_file_signature(monkeypatch, tmp_path):
+    history = tmp_path / "history.json"
+    thresholds = tmp_path / "thresholds.json"
+    weights = tmp_path / "weights.json"
+    cache = tmp_path / "cache.json"
+    for path in (history, thresholds, weights, cache):
+        path.write_text("{}")
+
+    monkeypatch.setattr(engine, "HISTORY_PATH", str(history))
+    monkeypatch.setattr(engine, "_THRESH_PATH", str(thresholds))
+    monkeypatch.setattr(engine, "ML_WEIGHTS_PATH", str(weights))
+    monkeypatch.setattr(engine, "CACHE_PATH", str(cache))
+    calls = []
+    monkeypatch.setattr(
+        engine, "_compute_backtest",
+        lambda ml_mode=False: calls.append(ml_mode) or {"ml_mode": ml_mode, "calls": len(calls)},
+    )
+    engine.clear_backtest_cache()
+
+    first = engine.run_backtest()
+    second = engine.run_backtest()
+    ml_first = engine.run_backtest(ml_mode=True)
+    ml_second = engine.run_backtest(ml_mode=True)
+    classic_after_ml = engine.run_backtest()
+
+    assert first == second == {"ml_mode": False, "calls": 1}
+    assert ml_first == ml_second == {"ml_mode": True, "calls": 2}
+    assert classic_after_ml == first
+    assert calls == [False, True]
+
+    history.write_text('{"changed": true}')
+    os.utime(history, None)
+    invalidated = engine.run_backtest()
+    assert invalidated == {"ml_mode": False, "calls": 3}
+
+
+def test_cached_backtest_results_are_isolated_from_caller_mutation(monkeypatch, tmp_path):
+    history = tmp_path / "history.json"
+    history.write_text("{}")
+    monkeypatch.setattr(engine, "HISTORY_PATH", str(history))
+    monkeypatch.setattr(engine, "_THRESH_PATH", str(tmp_path / "missing-thresholds.json"))
+    monkeypatch.setattr(engine, "ML_WEIGHTS_PATH", str(tmp_path / "missing-weights.json"))
+    monkeypatch.setattr(engine, "CACHE_PATH", str(tmp_path / "missing-cache.json"))
+    monkeypatch.setattr(engine, "_compute_backtest", lambda ml_mode=False: {"chart_data": [{"score": 10}]})
+    engine.clear_backtest_cache()
+
+    first = engine.run_backtest()
+    first["chart_data"][0]["score"] = 99
+
+    assert engine.run_backtest()["chart_data"][0]["score"] == 10

tests/test_backtest_data_quality.py

@@ -0,0 +1,61 @@
+from backtesting import engine as backtest
+from scoring import engine as scoring
+
+
+def test_metric_specific_staleness_does_not_apply_generic_30_day_fill():
+    lookup = {"2024-01-01": 42}
+
+    assert backtest._metric_observation(lookup, "2024-01-03", "fear_greed") == (42, "2024-01-01", 2)
+    assert backtest._metric_observation(lookup, "2024-01-04", "fear_greed") == (None, None, None)
+    assert backtest._metric_observation(lookup, "2024-01-08", "200w_sma") == (42, "2024-01-01", 7)
+    assert backtest._metric_observation(lookup, "2024-01-02", "unknown_metric") == (None, None, None)
+
+
+def test_current_context_score_uses_only_the_common_backtest_panel():
+    cached_scored = {
+        "composite_score": 100,
+        "metrics": [
+            {"key": "fear_greed", "score": 10},
+            {"key": "puell_multiple", "score": 0},
+            {"key": "sopr", "score": 10},
+            {"key": "vdd_multiple", "score": 10},
+        ],
+    }
+
+    score, coverage = backtest._common_panel_current_score(cached_scored)
+
+    assert score == 50.0
+    assert coverage == {
+        "available_count": 2,
+        "panel_count": len(backtest.BACKTEST_METRIC_PANEL),
+        "available_keys": ["fear_greed", "puell_multiple"],
+    }
+
+
+def test_live_and_backtest_outputs_publish_panel_and_coverage_metadata():
+    live = scoring.score_all({"fear_greed": {"value": 10}})
+
+    assert live["metric_panel"]["id"] == "live-all-v1"
+    assert live["metric_panel"]["count"] == live["total_count"]
+    assert live["coverage"]["available_count"] == live["scored_count"]
+    assert live["coverage"]["ratio"] == live["scored_count"] / live["total_count"]
+    assert "score_version" in live
+
+    metadata = backtest._backtest_data_quality_metadata([3, 5, 9])
+    assert metadata["metric_panel"]["id"] == "historical-common-v1"
+    assert metadata["metric_panel"]["keys"] == list(backtest.BACKTEST_METRIC_PANEL)
+    assert metadata["coverage"] == {
+        "minimum_metrics": 3,
+        "maximum_metrics": 9,
+        "average_metrics": 5.7,
+        "panel_count": 9,
+    }
+    assert metadata["staleness_days"] == backtest.METRIC_MAX_AGE_DAYS
+
+
+def test_chart_data_exposes_metric_values_under_frontend_contract():
+    result = backtest.run_backtest()
+    entries = [entry for entry in result["chart_data"] if entry.get("metric_values")]
+
+    assert entries
+    assert all("metrics" not in entry for entry in entries)

tests/test_backtest_ml_scope.py

@@ -0,0 +1,64 @@
+from backtesting import engine
+from ml.artifacts import ML_ARTIFACT_SCHEMA_VERSION, REQUIRED_WEIGHT_KEYS
+from scoring.policy import SCORE_VERSION
+
+
+def _weights(focus):
+    weights = {key: 0.0 for key in REQUIRED_WEIGHT_KEYS}
+    weights[focus] = 1.0
+    return weights
+
+
+def _artifact(with_folds=True):
+    artifact = {
+        "artifact_schema_version": ML_ARTIFACT_SCHEMA_VERSION,
+        "score_version": SCORE_VERSION,
+        "weights": _weights("fear_greed"),
+        "provenance": {
+            "validation_method": "purged_expanding_window",
+            "label_horizon_days": 365,
+            "weight_scope": "full_history_fit",
+            "training_date_range": {"start": "2018-01-01", "end": "2024-01-01"},
+            "trained_at": "2026-07-01T00:00:00+00:00",
+        },
+        "cv_results": {"folds": []},
+    }
+    if with_folds:
+        artifact["cv_results"]["folds"] = [
+            {
+                "fold": 1,
+                "weights": _weights("drawdown"),
+                "date_ranges": {"validation": "2020-01-01 to 2020-12-31"},
+            },
+            {
+                "fold": 2,
+                "weights": _weights("nupl"),
+                "date_ranges": {"validation": "2021-01-01 to 2021-12-31"},
+            },
+        ]
+    return artifact
+
+
+def test_ml_backtest_plan_prefers_fold_weights_and_marks_them_oos():
+    plan = engine._build_ml_backtest_plan(_artifact(with_folds=True))
+
+    weights, fold = engine._weights_for_backtest_date("2021-06-01", plan)
+
+    assert weights == _weights("nupl")
+    assert fold == 2
+    assert plan["evaluation_scope"] == "out_of_sample_validation_folds"
+    assert plan["is_out_of_sample"] is True
+    assert plan["weighting_source"] == "fold_specific_weights"
+    assert engine._weights_for_backtest_date("2019-12-31", plan) == (None, None)
+
+
+def test_full_history_weights_are_explicitly_not_oos():
+    plan = engine._build_ml_backtest_plan(_artifact(with_folds=False))
+
+    weights, fold = engine._weights_for_backtest_date("2021-06-01", plan)
+
+    assert weights == _weights("fear_greed")
+    assert fold is None
+    assert plan["evaluation_scope"] == "in_sample_full_history_weights"
+    assert plan["is_out_of_sample"] is False
+    assert plan["weighting_source"] == "final_full_history_weights"

tests/test_backtest_statistics.py

@@ -0,0 +1,61 @@
+from backtesting import engine, statistics
+
+
+def test_moving_block_bootstrap_is_deterministic_and_handles_constant_series():
+    first = statistics.moving_block_bootstrap_ci(
+        [12.5] * 120,
+        block_size=15,
+        n_resamples=200,
+        seed=7,
+    )
+    second = statistics.moving_block_bootstrap_ci(
+        [12.5] * 120,
+        block_size=15,
+        n_resamples=200,
+        seed=7,
+    )
+
+    assert first == second
+    assert first == {"estimate": 12.5, "ci_low": 12.5, "ci_high": 12.5, "n": 120}
+
+
+def test_summarize_returns_reports_observations_and_block_bootstrap_interval():
+    summary = statistics.summarize_returns(
+        [10.0, -5.0, 20.0, -10.0],
+        block_size=2,
+        n_resamples=200,
+        seed=3,
+    )
+
+    assert summary["n"] == 4
+    assert summary["mean"] == 3.75
+    assert summary["median"] == 2.5
+    assert summary["win_rate"] == 50.0
+    assert summary["mean_ci_low"] <= summary["mean"] <= summary["mean_ci_high"]
+
+
+def test_backtest_brackets_publish_bootstrap_confidence_intervals():
+    stats = {}
+
+    engine._add_return_statistics(stats, "90d", [10.0, -5.0, 20.0, -10.0])
+
+    assert stats["avg_90d"] == 3.75
+    assert stats["median_90d"] == 2.5
+    assert stats["win_rate_90d"] == 50.0
+    assert stats["avg_90d_ci_low"] <= stats["avg_90d"] <= stats["avg_90d_ci_high"]
+
+
+def test_long_horizon_returns_use_a_matching_dependence_block(monkeypatch):
+    observed = {}
+
+    def fake_summary(values, *, block_size, n_resamples):
+        observed.update(block_size=block_size, n_resamples=n_resamples)
+        return {
+            "mean": 1.0, "median": 1.0, "win_rate": 100.0,
+            "mean_ci_low": 0.5, "mean_ci_high": 1.5, "n": len(values),
+        }
+
+    monkeypatch.setattr(engine, "summarize_returns", fake_summary)
+    engine._add_return_statistics({}, "365d", [1.0] * 500)
+
+    assert observed == {"block_size": 365, "n_resamples": 400}

tests/test_job_registry.py

@@ -0,0 +1,53 @@
+import threading
+
+from dashboard.jobs import JobRegistry
+
+
+def test_job_registry_atomically_reserves_only_one_job_per_kind(tmp_path):
+    registry = JobRegistry(tmp_path / "jobs.json")
+    barrier = threading.Barrier(10)
+    results = []
+
+    def reserve():
+        barrier.wait()
+        results.append(registry.reserve("refresh", details={"full": False}))
+
+    threads = [threading.Thread(target=reserve) for _ in range(10)]
+    for thread in threads:
+        thread.start()
+    for thread in threads:
+        thread.join()
+
+    reserved = [job for job in results if job is not None]
+    assert len(reserved) == 1
+    assert reserved[0]["id"]
+    assert reserved[0]["status"] == "queued"
+    assert registry.active("refresh")["id"] == reserved[0]["id"]
+
+
+def test_job_registry_tracks_completion_and_result(tmp_path):
+    registry = JobRegistry(tmp_path / "jobs.json")
+    job = registry.reserve("history")
+
+    result = registry.run(job["id"], lambda: {"records": 42})
+
+    assert result == {"records": 42}
+    saved = registry.get(job["id"])
+    assert saved["status"] == "complete"
+    assert saved["result"] == {"records": 42}
+    assert saved["started_at"]
+    assert saved["finished_at"]
+    assert registry.active("history") is None
+
+
+def test_job_registry_marks_abandoned_active_jobs_interrupted_on_restart(tmp_path):
+    path = tmp_path / "jobs.json"
+    first = JobRegistry(path)
+    job = first.reserve("refresh")
+
+    restarted = JobRegistry(path)
+
+    recovered = restarted.get(job["id"])
+    assert recovered["status"] == "interrupted"
+    assert recovered["finished_at"]
+    assert restarted.active("refresh") is None

tests/test_legacy_ml_validation.py

@@ -0,0 +1,103 @@
+import numpy as np
+import pandas as pd
+
+import orchestrator
+from ml_engine import train_and_backtest as legacy
+from ml_engine.train_and_backtest import create_accumulation_target
+
+
+def _frame(prices):
+    return pd.DataFrame({"close": prices})
+
+
+def test_accumulation_target_for_existing_row_is_invariant_to_unrelated_future_rows():
+    config = {
+        "timeframe": "4h",
+        "target": {
+            "forward_periods_4h": [1, 2, 3],
+            "weights": [0.2, 0.3, 0.5],
+            "return_scales_pct": [5, 10, 20],
+        },
+    }
+    base = _frame([100, 102, 104, 106, 108, 110, 112, 114])
+    extended = _frame([100, 102, 104, 106, 108, 110, 112, 114, 1000, 1, 2000])
+
+    base_target = create_accumulation_target(base, config)
+    extended_target = create_accumulation_target(extended, config)
+
+    assert np.isclose(base_target.iloc[0], extended_target.iloc[0])
+    assert 0 <= base_target.iloc[0] <= 100
+
+
+def test_rolling_validation_purges_forward_label_horizon_at_train_boundaries(monkeypatch):
+    rows = 200
+    frame = pd.DataFrame({
+        "feature": np.linspace(0, 1, rows),
+        "target": np.arange(rows, dtype=float) % 100,
+        "close": np.linspace(10_000, 20_000, rows),
+    })
+    observed = []
+
+    def fake_train(X_train, y_train, X_val, y_val, X_test, *args):
+        observed.append((len(X_train), len(X_val), len(X_test)))
+        return np.full(len(X_test), 50.0), np.array([1.0])
+
+    monkeypatch.setattr(legacy, "_train_and_predict_window", fake_train)
+    config = {
+        "model_type": "xgboost",
+        "target": {"forward_periods_4h": [1, 2, 3]},
+        "training": {
+            "rolling_train_size": 120,
+            "rolling_test_size": 40,
+            "validation_pct": 0.25,
+        },
+        "features": {"use_scaler": False, "use_pca": False},
+        "strategy": {},
+    }
+
+    legacy.rolling_window_train_test(frame, ["feature"], config)
+
+    assert observed[0] == (87, 27, 40)
+
+
+def test_periodic_accumulation_compares_equal_contributions_and_retains_cash():
+    result = legacy.simulate_periodic_accumulation(
+        predicted_scores=np.array([90, 10, 90, 10], dtype=float),
+        close_prices=np.array([100, 300, 100, 200], dtype=float),
+        buy_threshold=70,
+        contribution=100,
+    )
+
+    assert np.isclose(result["dca_contributed"], 400)
+    assert np.isclose(result["model_contributed"], 400)
+    assert np.isclose(result["model_cash"], 100)
+    assert np.isclose(result["model_btc"], 3)
+    assert result["model_terminal_value"] > result["dca_terminal_value"]
+
+
+def test_compiled_results_publish_equal_capital_terminal_wealth_metric():
+    predictions = [
+        {"predicted": score, "actual": 50.0, "close": price}
+        for score, price in zip([90, 10, 90, 10], [100, 300, 100, 200])
+    ]
+
+    result = legacy.compile_results(
+        predictions,
+        per_window_cost_improvement=[],
+        fi_sum=np.array([1.0]),
+        fi_count=1,
+        feature_cols=["feature"],
+        config={"model_type": "xgboost", "strategy": {"good_buy_threshold": 70}},
+    )
+
+    assert result["terminal_wealth_improvement_pct"] > 0
+    assert result["backtest_objective"] == "equal_periodic_contribution_terminal_wealth"
+
+
+def test_orchestrator_selects_models_by_terminal_wealth_not_cost_basis():
+    results = {
+        "terminal_wealth_improvement_pct": 4.5,
+        "cost_basis_improvement_pct": 99.0,
+    }
+
+    assert orchestrator.objective_score(results) == 4.5

tests/test_ml_artifacts.py

@@ -0,0 +1,60 @@
+import json
+from pathlib import Path
+
+from ml import artifacts
+from scoring import engine
+
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+
+
+def _valid_artifact():
+    return {
+        "artifact_schema_version": artifacts.ML_ARTIFACT_SCHEMA_VERSION,
+        "score_version": artifacts.SCORE_VERSION,
+        "weights": {key: 1 / len(artifacts.REQUIRED_WEIGHT_KEYS) for key in artifacts.REQUIRED_WEIGHT_KEYS},
+        "provenance": {
+            "validation_method": "purged_expanding_window",
+            "label_horizon_days": 365,
+            "weight_scope": "full_history_fit",
+            "training_date_range": {"start": "2018-02-01", "end": "2025-03-21"},
+            "trained_at": "2026-07-01T00:00:00+00:00",
+        },
+    }
+
+
+def test_repository_artifact_has_current_schema_and_purged_provenance():
+    artifact_path = REPO_ROOT / "config" / "ml_weights.json"
+    artifact = json.loads(artifact_path.read_text())
+
+    status = artifacts.validate_ml_artifact(artifact)
+
+    assert status["valid"] is True
+    assert status["schema_version"] == artifacts.ML_ARTIFACT_SCHEMA_VERSION
+    assert status["score_version"] == artifacts.SCORE_VERSION
+    assert status["has_oos_fold_weights"] is True
+    assert status["errors"] == []
+
+
+def test_valid_artifact_requires_schema_score_version_and_purged_provenance():
+    artifact = _valid_artifact()
+
+    status = artifacts.validate_ml_artifact(artifact)
+
+    assert status == {
+        "valid": True,
+        "schema_version": artifacts.ML_ARTIFACT_SCHEMA_VERSION,
+        "score_version": artifacts.SCORE_VERSION,
+        "weight_scope": "full_history_fit",
+        "has_oos_fold_weights": False,
+        "errors": [],
+    }
+
+
+def test_live_scoring_refuses_schema_less_weights(tmp_path, monkeypatch):
+    path = tmp_path / "ml_weights.json"
+    path.write_text(json.dumps({"weights": {"fear_greed": 1.0}}))
+    monkeypatch.setattr(engine, "ML_WEIGHTS_PATH", str(path))
+
+    assert engine.load_ml_weights() == {}
+    assert engine.get_ml_artifact_status()["valid"] is False

tests/test_ml_optimizer_validation.py

@@ -0,0 +1,107 @@
+from datetime import datetime, timedelta
+
+import numpy as np
+
+from ml import optimizer
+
+
+def _row(date, returns=10.0, score_200w=10, score_drawdown=0):
+    row = {
+        "date": date,
+        "price": 100.0,
+        "fwd_365d": returns,
+        "score_puell_multiple": 0,
+        "score_mvrv_zscore": 0,
+        "score_reserve_risk": 0,
+        "score_rhodl_ratio": 0,
+        "score_nupl": 0,
+        "score_fear_greed": 0,
+        "score_drawdown": score_drawdown,
+        "score_pct_above_200w_sma": score_200w,
+        "score_pct_above_lth_rp": 0,
+    }
+    return row
+
+
+def test_purged_time_series_splits_remove_overlapping_forward_label_windows():
+    start = datetime(2020, 1, 1)
+    rows = [_row((start + timedelta(days=i)).strftime("%Y-%m-%d")) for i in range(900)]
+
+    splits = list(
+        optimizer.purged_time_series_splits(
+            rows,
+            n_splits=3,
+            label_horizon_days=365,
+            embargo_days=0,
+        )
+    )
+
+    assert splits, "expected at least one viable split"
+    for train_idx, val_idx in splits:
+        val_start = datetime.strptime(rows[val_idx[0]]["date"], "%Y-%m-%d")
+        latest_allowed_train_date = val_start - timedelta(days=365)
+        assert len(train_idx) > 0, "purging should keep non-overlapping expanding-window training rows"
+        for idx in train_idx:
+            train_date = datetime.strptime(rows[idx]["date"], "%Y-%m-%d")
+            assert train_date <= latest_allowed_train_date
+
+
+def test_run_out_of_sample_comparison_scores_only_validation_rows_with_fold_weights():
+    rows = [
+        _row("2020-01-01", returns=-10, score_200w=0, score_drawdown=10),
+        _row("2020-01-02", returns=-5, score_200w=0, score_drawdown=10),
+        _row("2020-01-03", returns=100, score_200w=10, score_drawdown=10),
+        _row("2020-01-04", returns=120, score_200w=10, score_drawdown=10),
+    ]
+    fold_results = [
+        {
+            "fold": 1,
+            "val_idx": [2, 3],
+            "weights": {"pct_above_200w_sma": 1.0, "drawdown": 0.0},
+        }
+    ]
+
+    comparison = optimizer.run_out_of_sample_comparison(rows, fold_results)
+
+    assert comparison["validation_days"] == 2
+    assert comparison["folds"] == 1
+    assert sum(bucket["days"] for bucket in comparison["ml_weighted"]) == 2
+    assert sum(bucket["days"] for bucket in comparison["equal_weight"]) == 2
+
+    extreme_ml = next(bucket for bucket in comparison["ml_weighted"] if bucket["label"] == "EXTREME ACCUMULATION ZONE")
+    assert extreme_ml["days"] == 2
+    assert extreme_ml["avg_365d"] == 110.0
+
+    caution_equal = next(bucket for bucket in comparison["equal_weight"] if bucket["label"] == "CAUTION — OVERHEATED")
+    assert caution_equal["days"] == 2
+
+
+def test_classification_splits_skip_training_windows_with_one_class():
+    y = np.array([1, 1, 1, 0, 1, 0])
+    splits = [
+        (np.array([0, 1]), np.array([2, 3])),
+        (np.array([0, 1, 3, 4]), np.array([5])),
+    ]
+
+    viable = list(optimizer.viable_classification_splits(y, splits))
+
+    assert len(viable) == 1
+    assert viable[0][0].tolist() == [0, 1, 3, 4]
+
+
+def test_artifact_folds_omit_large_internal_index_arrays():
+    folds = [{
+        "fold": 1,
+        "train_idx": [0, 1],
+        "val_idx": [2, 3],
+        "weights": {"fear_greed": 1.0},
+        "date_ranges": {"validation": "2024-01-01 to 2024-01-02"},
+    }]
+
+    saved = optimizer.artifact_fold_results(folds)
+
+    assert saved == [{
+        "fold": 1,
+        "weights": {"fear_greed": 1.0},
+        "date_ranges": {"validation": "2024-01-01 to 2024-01-02"},
+    }]

tests/test_reliability_persistence.py

@@ -0,0 +1,121 @@
+import json
+import threading
+from datetime import datetime, timedelta, timezone
+
+import pytest
+
+from dashboard.persistence import (
+    append_daily_jsonl,
+    atomic_write_json,
+    load_jsonl_tail,
+    merge_observation,
+    onchain_refresh_due,
+)
+
+
+def test_atomic_write_json_remains_readable_under_concurrent_writers(tmp_path):
+    path = tmp_path / "cache.json"
+
+    threads = [
+        threading.Thread(target=atomic_write_json, args=(path, {"writer": i, "values": list(range(100))}))
+        for i in range(12)
+    ]
+    for thread in threads:
+        thread.start()
+    for thread in threads:
+        thread.join()
+
+    saved = json.loads(path.read_text())
+    assert saved["writer"] in range(12)
+    assert saved["values"] == list(range(100))
+    assert not list(tmp_path.glob(".cache.json.*.tmp"))
+
+
+def test_append_daily_jsonl_writes_at_most_one_entry_per_utc_day(tmp_path):
+    path = tmp_path / "scores.jsonl"
+    first = {"timestamp": "2026-07-26T01:00:00+00:00", "score": 10}
+    duplicate_day = {"timestamp": "2026-07-26T23:59:00+00:00", "score": 20}
+    next_day = {"timestamp": "2026-07-27T00:01:00+00:00", "score": 30}
+
+    assert append_daily_jsonl(path, first) is True
+    assert append_daily_jsonl(path, duplicate_day) is False
+    assert append_daily_jsonl(path, next_day) is True
+
+    assert load_jsonl_tail(path, limit=90) == [first, next_day]
+
+
+def test_load_jsonl_tail_is_bounded_and_ignores_malformed_lines(tmp_path):
+    path = tmp_path / "scores.jsonl"
+    path.write_text("".join(json.dumps({"n": i}) + "\n" for i in range(200)) + "partial{")
+
+    assert load_jsonl_tail(path, limit=3, chunk_size=64) == [{"n": 197}, {"n": 198}, {"n": 199}]
+
+
+def test_merge_observation_preserves_last_known_good_with_stale_metadata():
+    old = {
+        "value": 1.25,
+        "observed_at": "2026-07-25T12:00:00+00:00",
+        "source": "lookintobitcoin",
+        "stale": False,
+        "last_error": None,
+    }
+
+    merged = merge_observation(old, None, source="lookintobitcoin", error="timeout")
+
+    assert merged == {
+        "value": 1.25,
+        "observed_at": "2026-07-25T12:00:00+00:00",
+        "source": "lookintobitcoin",
+        "stale": True,
+        "last_error": "timeout",
+    }
+
+
+def test_merge_observation_records_metadata_for_fresh_value():
+    observed_at = "2026-07-26T12:00:00+00:00"
+
+    merged = merge_observation(
+        {"value": 1.0}, {"value": 2.0, "trend": "up"},
+        source="checkonchain", observed_at=observed_at,
+    )
+
+    assert merged["value"] == 2.0
+    assert merged["trend"] == "up"
+    assert merged["observed_at"] == observed_at
+    assert merged["source"] == "checkonchain"
+    assert merged["stale"] is False
+    assert merged["last_error"] is None
+
+
+def test_merge_observation_rejects_error_only_payload_as_fresh_data():
+    old = {
+        "value": 1.25,
+        "observed_at": "2026-07-25T12:00:00+00:00",
+        "source": "lookintobitcoin",
+        "stale": False,
+        "last_error": None,
+    }
+
+    merged = merge_observation(
+        old,
+        {"value": None, "error": "No data returned"},
+        source="lookintobitcoin",
+        error="metric missing from scrape",
+    )
+
+    assert merged["value"] == 1.25
+    assert merged["observed_at"] == old["observed_at"]
+    assert merged["stale"] is True
+    assert merged["last_error"] == "No data returned"
+
+
+@pytest.mark.parametrize("timestamp", [None, "", "not-a-time"])
+def test_onchain_refresh_due_when_timestamp_is_missing_or_invalid(timestamp):
+    assert onchain_refresh_due(timestamp, now=datetime(2026, 7, 26, tzinfo=timezone.utc)) is True
+
+
+def test_onchain_refresh_due_after_ttl():
+    now = datetime(2026, 7, 26, 12, tzinfo=timezone.utc)
+
+    assert onchain_refresh_due((now - timedelta(hours=5)).isoformat(), now=now, ttl_seconds=21600) is False
+    assert onchain_refresh_due((now - timedelta(hours=7)).isoformat(), now=now, ttl_seconds=21600) is True

tests/test_score_policy.py

@@ -0,0 +1,38 @@
+import pytest
+
+from backtesting import engine as backtest_engine
+from ml import optimizer
+from scoring import engine as scoring_engine
+from scoring import policy
+
+
+def test_score_brackets_are_contiguous_and_shared_by_all_scoring_paths():
+    assert backtest_engine.BRACKETS is policy.SCORE_BRACKETS
+    assert optimizer.BRACKETS is policy.SCORE_BRACKETS
+
+    for left, right in zip(policy.SCORE_BRACKETS, policy.SCORE_BRACKETS[1:]):
+        assert left[1] == right[0]
+
+    for tenth in range(0, 1001):
+        score = tenth / 10
+        matches = [bracket for bracket in policy.SCORE_BRACKETS if policy.score_in_bracket(score, bracket)]
+        assert len(matches) == 1, f"score {score} matched {matches}"
+        assert policy.assessment_for_score(score) == matches[0][2]
+
+
+@pytest.mark.parametrize(
+    ("score", "assessment"),
+    [
+        (0, "EXTREME CAUTION"),
+        (19.999, "EXTREME CAUTION"),
+        (20, "CAUTION — OVERHEATED"),
+        (35, "NEUTRAL"),
+        (50, "MODERATE OPPORTUNITY"),
+        (65, "STRONG ACCUMULATION ZONE"),
+        (80, "EXTREME ACCUMULATION ZONE"),
+        (100, "EXTREME ACCUMULATION ZONE"),
+    ],
+)
+def test_assessment_boundaries_match_live_scoring(score, assessment):
+    assert policy.assessment_for_score(score) == assessment
+    assert scoring_engine.assessment_for_score(score) == assessment

tests/test_scoring_engine_ml.py

@@ -0,0 +1,64 @@
+import math
+
+from scoring import engine
+
+
+def _complete_metrics():
+    return {
+        "fear_greed": {"value": 10, "classification": "Extreme Fear"},
+        "puell_multiple": {"value": 0.3},
+        "mvrv_zscore": {"value": -0.1},
+        "drawdown": {"value": 60.0, "ath": 250.0},
+        "price": {"price": 100.0},
+        "200w_sma": {"value": 120.0},
+        "reserve_risk": {"value": 0.001},
+        "rhodl_ratio": {"value": 50.0},
+        "nupl": {"value": -0.1},
+        "lth_realized_price": {"value": 120.0},
+        "hash_ribbons": {"buy_signal": False},
+    }
+
+
+def test_score_all_ml_normalizes_displayed_weights_and_contributions(monkeypatch):
+    monkeypatch.setattr(
+        engine,
+        "load_ml_weights",
+        lambda: {
+            "fear_greed": 0.40,
+            "puell_multiple": 0.20,
+            "mvrv_zscore": 0.15,
+            "drawdown": 0.10,
+            "pct_above_200w_sma": 0.05,
+            "reserve_risk": 0.04,
+            "rhodl_ratio": 0.03,
+            "nupl": 0.02,
+            "pct_above_lth_rp": 0.01,
+        },
+    )
+
+    scored = engine.score_all_ml(_complete_metrics())
+
+    assert scored["ml_mode"] is True
+    valid_metrics = [m for m in scored["metrics"] if m["score"] is not None]
+    assert scored["ml_weight_total"] == 1.01  # trained weights + small hash-ribbons fallback
+    assert math.isclose(sum(m["ml_weight"] for m in valid_metrics), 1.0, abs_tol=0.001)
+    assert math.isclose(
+        sum(m["ml_contribution"] for m in valid_metrics),
+        scored["composite_score"],
+        abs_tol=0.05,
+    )
+
+    hash_ribbons = next(m for m in valid_metrics if m["key"] == "hash_ribbons")
+    assert hash_ribbons["ml_raw_weight"] == 0.01
+    assert hash_ribbons["ml_weight"] == round(0.01 / 1.01, 4)
+
+
+def test_score_all_ml_preserves_classic_fallback_when_weights_missing(monkeypatch):
+    monkeypatch.setattr(engine, "load_ml_weights", lambda: {})
+
+    scored = engine.score_all_ml(_complete_metrics())
+
+    assert scored["ml_mode"] is False
+    assert scored["ml_error"]
+    assert scored["ml_artifact"]["valid"] is False
+    assert "classic_score" not in scored

tests/test_scraper_semantics.py

@@ -0,0 +1,56 @@
+from contextlib import contextmanager
+
+from scrapers import lookintobitcoin
+from scoring import engine
+
+
+def test_hash_ribbon_signal_requires_a_current_truthy_marker():
+    named_but_inactive = {"name": "Buy Signal", "y": [1, None, None]}
+    active = {"name": "Buy Signal", "y": [None, 0, 1]}
+
+    assert lookintobitcoin._trace_signal_is_active(named_but_inactive) is False
+    assert lookintobitcoin._trace_signal_is_active(active) is True
+
+
+def test_lth_supply_trace_selection_never_falls_back_to_price():
+    traces = [
+        {"name": "BTC Price", "y": [60000, 61000]},
+        {"name": "Long-Term Holder Supply", "y": [14_000_000, 14_100_000]},
+    ]
+
+    assert lookintobitcoin._find_lth_supply_trace(traces)["name"] == "Long-Term Holder Supply"
+    assert lookintobitcoin._find_lth_supply_trace(traces[:1]) is None
+
+
+def test_vdd_derived_return_is_labeled_as_momentum_not_raw_multiple():
+    result = engine.score_all({"vdd_multiple": {"value": 0.12}})
+    vdd = next(metric for metric in result["metrics"] if metric["key"] == "vdd_multiple")
+
+    assert vdd["name"] == "VDD 30-Period Momentum"
+    assert vdd["transform"] == "30_period_return"
+
+
+def test_scrape_all_reuses_one_browser_page(monkeypatch):
+    page = object()
+    seen_pages = []
+
+    @contextmanager
+    def fake_browser_page():
+        yield page
+
+    def fake_scrape_chart(_path, timeout=25000, page=None):
+        seen_pages.append(page)
+        return [{"name": "metric", "y": [1.0]}]
+
+    monkeypatch.setattr(lookintobitcoin, "CHARTS", {
+        "first": {"path": "/first", "traces": ["metric"]},
+        "second": {"path": "/second", "traces": ["metric"]},
+    })
+    monkeypatch.setattr(lookintobitcoin, "browser_page", fake_browser_page)
+    monkeypatch.setattr(lookintobitcoin, "scrape_chart", fake_scrape_chart)
+
+    result = lookintobitcoin.scrape_all()
+
+    assert seen_pages == [page, page]
+    assert result["first"]["value"] == 1.0
+    assert result["second"]["value"] == 1.0

tests/test_server_reliability.py

@@ -0,0 +1,194 @@
+import importlib
+import json
+import sys
+import types
+from datetime import datetime, timedelta, timezone
+
+import pytest
+
+
+@pytest.fixture
+def server(monkeypatch):
+    started = []
+    monkeypatch.setattr("threading.Thread.start", lambda self: started.append(self))
+    sys.modules.pop("dashboard.server", None)
+    module = importlib.import_module("dashboard.server")
+    module._threads_started_during_import = started
+    return module
+
+
+def test_server_import_does_not_start_scheduler_threads(server):
+    assert server._threads_started_during_import == []
+
+
+def test_frontend_uses_backtest_metric_values_contract(server):
+    assert ".filter(d => d.metric_values && d.metric_values[metricKey] != null)" in server.DASHBOARD_HTML
+    assert ".map(d => ({ date: d.date, value: d.metric_values[metricKey]" in server.DASHBOARD_HTML
+
+
+def test_dashboard_does_not_issue_duplicate_initial_backtest_request(server):
+    assert "const br = await fetch('/api/backtest');" not in server.DASHBOARD_HTML
+    assert server.DASHBOARD_HTML.count("fetch('/api/backtest?mode=' + currentMode)") == 1
+
+
+def test_health_endpoints_distinguish_process_liveness_from_data_readiness(server, monkeypatch):
+    assert server.health_live() == {"status": "ok"}
+
+    monkeypatch.setattr(server, "load_cache", lambda: {})
+    unavailable = server.health_ready()
+    assert unavailable.status_code == 503
+
+    monkeypatch.setattr(
+        server,
+        "load_cache",
+        lambda: {"_scored": {"composite_score": 72, "scored_count": 8}},
+    )
+    assert server.health_ready() == {
+        "status": "ready",
+        "score": 72,
+        "scored_metrics": 8,
+    }
+
+
+def test_server_cache_and_history_use_reliable_persistence(server, monkeypatch, tmp_path):
+    monkeypatch.setattr(server, "CACHE_PATH", str(tmp_path / "cache.json"))
+    monkeypatch.setattr(server, "HISTORY_PATH", str(tmp_path / "scores.jsonl"))
+
+    server.save_cache({"metric": {"value": 1}})
+    server.append_history({
+        "composite_score": 50,
+        "scored_count": 1,
+        "metrics": [{"key": "metric", "score": 5, "value": 1}],
+    })
+    server.append_history({
+        "composite_score": 60,
+        "scored_count": 1,
+        "metrics": [{"key": "metric", "score": 6, "value": 2}],
+    })
+
+    assert server.load_cache() == {"metric": {"value": 1}}
+    assert len(server.load_history()) == 1
+
+
+def test_partial_fast_scrape_preserves_last_known_good_metric(server, monkeypatch, tmp_path):
+    cache_path = tmp_path / "cache.json"
+    history_path = tmp_path / "scores.jsonl"
+    now = datetime.now(timezone.utc)
+    cache_path.write_text(json.dumps({
+        "price": {
+            "price": 65000,
+            "observed_at": (now - timedelta(minutes=15)).isoformat(),
+            "source": "coingecko",
+            "stale": False,
+            "last_error": None,
+        },
+        "puell_multiple": {"value": 1.2},
+        "_onchain_timestamp": now.isoformat(),
+    }))
+    monkeypatch.setattr(server, "CACHE_PATH", str(cache_path))
+    monkeypatch.setattr(server, "HISTORY_PATH", str(history_path))
+    monkeypatch.setattr(server.fear_greed, "fetch", lambda: {"value": 25})
+    monkeypatch.setattr(server.price, "fetch_current", lambda: (_ for _ in ()).throw(RuntimeError("price timeout")))
+    monkeypatch.setattr(server.price, "fetch_ath", lambda: (_ for _ in ()).throw(RuntimeError("ATH timeout")))
+    monkeypatch.setattr(server.price, "fetch_historical", lambda: (_ for _ in ()).throw(RuntimeError("history timeout")))
+    monkeypatch.setattr(server.engine, "score_all", lambda metrics: {"composite_score": 50, "scored_count": 1, "metrics": []})
+    monkeypatch.setattr(server.engine, "score_all_ml", lambda metrics: {"composite_score": 50, "scored_count": 1, "metrics": []})
+    fake_updater = types.ModuleType("scrapers.history_updater")
+    fake_updater.update_history = lambda: None
+    monkeypatch.setitem(sys.modules, "scrapers.history_updater", fake_updater)
+
+    server.run_scrape()
+
+    saved = json.loads(cache_path.read_text())
+    assert saved["price"]["price"] == 65000
+    assert saved["price"]["stale"] is True
+    assert "price timeout" in saved["price"]["last_error"]
+    assert saved["fear_greed"]["value"] == 25
+    assert saved["fear_greed"]["stale"] is False
+    assert saved["fear_greed"]["source"] == "alternative.me"
+
+
+def test_onchain_sources_fail_independently(server, monkeypatch):
+    fake_lib = types.ModuleType("scrapers.lookintobitcoin")
+    setattr(fake_lib, "scrape_all", lambda: (_ for _ in ()).throw(RuntimeError("LIB down")))
+    fake_coc = types.ModuleType("scrapers.checkonchain")
+    setattr(fake_coc, "scrape_all", lambda: {"sopr": {"value": 0.99}})
+    monkeypatch.setitem(sys.modules, "scrapers.lookintobitcoin", fake_lib)
+    monkeypatch.setitem(sys.modules, "scrapers.checkonchain", fake_coc)
+
+    observations, errors, successful_sources = server._scrape_onchain_sources()
+
+    assert observations["sopr"]["value"] == 0.99
+    assert successful_sources == 1
+    assert any("LookIntoBitcoin" in error for error in errors)
+
+
+def test_expired_onchain_timestamp_triggers_real_refresh(server, monkeypatch, tmp_path):
+    old = datetime.now(timezone.utc) - timedelta(hours=7)
+    cache_path = tmp_path / "cache.json"
+    cache_path.write_text(json.dumps({
+        "puell_multiple": {"value": 1.2},
+        "_onchain_timestamp": old.isoformat(),
+    }))
+    monkeypatch.setattr(server, "CACHE_PATH", str(cache_path))
+    monkeypatch.setattr(server, "HISTORY_PATH", str(tmp_path / "scores.jsonl"))
+    monkeypatch.setattr(server.fear_greed, "fetch", lambda: {"value": 25})
+    monkeypatch.setattr(server.price, "fetch_current", lambda: {"price": 65000})
+    monkeypatch.setattr(server.price, "fetch_ath", lambda: {"ath": 70000})
+    monkeypatch.setattr(server.price, "fetch_historical", lambda: [])
+    monkeypatch.setattr(server.engine, "score_all", lambda metrics: {"composite_score": 50, "scored_count": 1, "metrics": []})
+    monkeypatch.setattr(server.engine, "score_all_ml", lambda metrics: {"composite_score": 50, "scored_count": 1, "metrics": []})
+
+    calls = []
+    fake_lib = types.ModuleType("scrapers.lookintobitcoin")
+    fake_lib.scrape_all = lambda: calls.append("lib") or {"puell_multiple": {"value": 0.8}}
+    fake_coc = types.ModuleType("scrapers.checkonchain")
+    fake_coc.scrape_all = lambda: calls.append("coc") or {"sopr": {"value": 0.99}}
+    fake_updater = types.ModuleType("scrapers.history_updater")
+    fake_updater.update_history = lambda: None
+    import scrapers
+    monkeypatch.setattr(scrapers, "lookintobitcoin", fake_lib, raising=False)
+    monkeypatch.setattr(scrapers, "checkonchain", fake_coc, raising=False)
+    monkeypatch.setitem(sys.modules, "scrapers.lookintobitcoin", fake_lib)
+    monkeypatch.setitem(sys.modules, "scrapers.checkonchain", fake_coc)
+    monkeypatch.setitem(sys.modules, "scrapers.history_updater", fake_updater)
+
+    server.run_scrape()
+
+    assert calls == ["lib", "coc"]
+    saved = json.loads(cache_path.read_text())
+    assert saved["puell_multiple"]["value"] == 0.8
+    assert saved["puell_multiple"]["source"] == "lookintobitcoin"
+    assert saved["sopr"]["source"] == "checkonchain"
+    assert saved["_onchain_timestamp"] != old.isoformat()
+
+
+def test_refresh_job_is_reserved_before_thread_start(server, monkeypatch, tmp_path):
+    from dashboard.jobs import JobRegistry
+
+    registry = JobRegistry(tmp_path / "jobs.json")
+    monkeypatch.setattr(server, "_jobs", registry, raising=False)
+    monkeypatch.setattr(server, "_scraper_running", False)
+
+    started = server.api_refresh(full=False)
+    duplicate = server.api_refresh(full=False)
+
+    assert started["job_id"]
+    assert started["status"] == "queued"
+    assert registry.get(started["job_id"])["status"] == "queued"
+    assert duplicate.status_code == 409
+
+
+def test_history_collection_has_job_id_and_job_scoped_progress(server, monkeypatch, tmp_path):
+    from dashboard.jobs import JobRegistry
+
+    registry = JobRegistry(tmp_path / "jobs.json")
+    monkeypatch.setattr(server, "_jobs", registry, raising=False)
+
+    started = server.api_backtest_collect()
+    job = registry.get(started["job_id"])
+
+    assert started["status"] == "queued"
+    assert job["kind"] == "history"
+    assert job["progress"] == {"status": "starting", "current": "", "step": 0, "total": 0}
+    assert server.api_job_status(started["job_id"])["id"] == started["job_id"]