docs: update README and dashboard screenshots

README.md

@@ -1,134 +1,221 @@
 # Bitcoin Accumulation Zone Monitor
 
-> On-chain metrics dashboard with historical backtesting for long-term BTC holders. No ML, no black box — pure signal monitoring from proven indicators.
+> Bitcoin on-chain metrics dashboard with classic equal-weight scoring, ML-optimized scoring, historical backtesting, and click-to-select metric context for long-term BTC accumulation decisions.
 
 ![Dashboard](screenshots/dashboard-main.png)
 
 ## What It Does
 
-Monitors 10 proven Bitcoin on-chain metrics that have historically identified optimal buying zones for long-term holders. Each metric scores 0-10, producing a composite accumulation score of 0-100.
+Monitors Bitcoin accumulation conditions using 16 scored market/on-chain indicators plus optional informational cycle metrics. Each scored metric receives a 0-10 score and rolls into a 0-100 accumulation score.
 
-**Current reading example:** Fear & Greed at 11 (Extreme Fear), MVRV Z-Score at 0.52 (undervalued), Puell Multiple at 0.66 — the kind of conditions that preceded every major BTC rally.
+The dashboard now supports two scoring modes:
+
+- **Classic** — transparent equal-weight scoring across every active metric.
+- **ML** — feature-importance weights trained against historical 365-day forward returns, with displayed per-metric weights and point contributions.
+
+Historical backtests show score-vs-price behavior, score bracket performance, major signal events, and current-score context. Metric cards are clickable: selecting a metric overlays its historical series on the score chart and shows comparable historical periods with forward returns.
 
 ## Screenshots
 
-### Main Dashboard
+### Main Dashboard — ML mode + metric context
 ![Main Dashboard](screenshots/dashboard-main.png)
-*Live accumulation score with all 10 metrics, current BTC price, and individual metric breakdowns*
+*Live BTC price, Classic/ML scoring toggle, 16 active scored metrics, ML weights/contributions, metric sparklines, and click-to-select historical context.*
 
 ### Historical Backtest
 ![Backtest](screenshots/dashboard-backtest.png)
-*Historical score vs BTC price overlay, score bracket performance table, and major signal events*
+*Current signal percentile, comparable historical periods by cycle, score-vs-BTC chart, bracket performance, and major signal events.*
 
 ### Settings
 ![Settings](screenshots/dashboard-settings.png)
-*LLM provider configuration for optional AI-powered signal commentary*
+*LLM provider configuration for optional AI-powered signal commentary and local/cloud model selection.*
+
+## Feature Highlights
+
+- **16 scored metrics** from market sentiment, miner stress, valuation, holder behavior, network activity, and velocity signals.
+- **Classic vs ML scoring toggle** on the dashboard and backtest API.
+- **ML score explainability**: metric cards show learned weight and contribution in points.
+- **Leakage-resistant ML validation**: training uses purged time-series splits so 365-day forward-return labels do not overlap validation windows.
+- **Historical context panel**: compares the current composite score against historical periods and forward returns.
+- **Clickable metric cards**: select any metric to see percentile, similar historical levels, forward returns, example dates by market cycle, and highlighted chart periods.
+- **Score history chart** with BTC price overlay, range controls, and selected-metric overlay.
+- **Backtest dashboard** with current signal context, score bracket performance, and signal-crossing events.
+- **Quick vs full refresh**: quick refresh updates BTC price and Fear & Greed; full refresh re-scrapes on-chain sources.
+- **LLM settings UI** for Ollama, LM Studio, OpenAI, Anthropic, and OpenRouter.
 
 ## Metrics
 
 | # | Metric | Source | Accumulation Signal |
 |---|--------|--------|-------------------|
-| 1 | Fear & Greed Index | alternative.me API | Extreme Fear (< 10) |
+| 1 | Fear & Greed Index | alternative.me API | Extreme fear / capitulation sentiment |
-| 2 | Puell Multiple | LookIntoBitcoin (scraped) | Miner capitulation (< 0.5) |
+| 2 | Puell Multiple | LookIntoBitcoin | Miner revenue stress |
-| 3 | MVRV Z-Score | LookIntoBitcoin (scraped) | Below realized value (< 0) |
+| 3 | MVRV Z-Score | LookIntoBitcoin | Market near/below realized value |
-| 4 | Drawdown from ATH | Calculated | Deep correction (> 50%) |
+| 4 | Drawdown from ATH | Calculated from BTC price | Deep correction from cycle high |
-| 5 | Price vs 200W SMA | LookIntoBitcoin (scraped) | Below 200-week average |
+| 5 | Price vs 200W SMA | LookIntoBitcoin + BTC price | Price near/below long-term trend |
-| 6 | Reserve Risk | LookIntoBitcoin (scraped) | High holder confidence (< 0.002) |
+| 6 | Reserve Risk | LookIntoBitcoin | High holder confidence relative to price |
-| 7 | RHODL Ratio | LookIntoBitcoin (scraped) | Long-term holder dominance (< 100) |
+| 7 | RHODL Ratio | LookIntoBitcoin | Long-term holder dominance |
-| 8 | NUPL | LookIntoBitcoin (scraped) | Market capitulation (< 0) |
+| 8 | Net Unrealized Profit/Loss (NUPL) | LookIntoBitcoin | Capitulation / early recovery zones |
-| 9 | LTH Realized Price | LookIntoBitcoin (scraped) | Price below LTH cost basis |
+| 9 | LTH Realized Price | LookIntoBitcoin | Price near long-term holder cost basis |
-| 10 | Hash Ribbons | LookIntoBitcoin (scraped) | Miner capitulation recovery |
+| 10 | Hash Ribbons | LookIntoBitcoin | Miner capitulation/recovery signal |
+| 11 | SOPR | CheckOnChain | Spent outputs near loss / reset territory |
+| 12 | Sell-side Risk Ratio | CheckOnChain | Low realized profit/loss pressure |
+| 13 | Active Address Momentum | CheckOnChain | Network activity momentum extremes |
+| 14 | Transaction Count Momentum | CheckOnChain | Transaction activity momentum extremes |
+| 15 | NVT Price | CheckOnChain | Network-value valuation discount/premium |
+| 16 | VDD Multiple | CheckOnChain | Coin-days/velocity reset conditions |
+
+Informational cards may also appear when data is available, such as **Long-Term Holder Supply** and **Pi Cycle Bottom**. These are displayed for context and are not included in the composite score.
 
 ## Score Interpretation
 
-| Score | Assessment | Historical Outcome |
+| Score | Assessment | Interpretation |
-|-------|-----------|-------------------|
+|-------|-----------|----------------|
-| 85-100 | 🟢 Extreme Accumulation | Rare (~4x per decade). Historically: 200%+ 1yr returns |
+| 80-100 | 🟢 Extreme Accumulation Zone | Broad capitulation/value conditions across active metrics |
-| 70-84 | 🟢 Strong Accumulation | Excellent long-term entry point |
+| 65-79 | 🟢 Strong Accumulation Zone | Historically attractive long-term entry territory |
-| 55-69 | 🟡 Moderate Opportunity | Decent entry, DCA appropriate |
+| 50-64 | 🟡 Moderate Opportunity | DCA-friendly, but not maximum-signal conditions |
-| 40-54 | 🟡 Neutral | Hold — not compelling either way |
+| 35-49 | 🟡 Neutral | Mixed signals; not compelling either direction |
-| 25-39 | 🔴 Caution | Market heating up |
+| 20-34 | 🔴 Caution — Overheated | Market conditions becoming less favorable |
-| 0-24 | 🔴 Extreme Caution | Historically worst times to buy |
+| 0-19 | 🔴 Extreme Caution | Historically poor accumulation setup |
+
+Backtest tables provide actual historical forward-return statistics per score bracket, including 30d/90d/180d/1yr averages, win rate, max gain/loss, and average max drawdown.
+
+## ML-Optimized Scoring
+
+The ML mode uses a `GradientBoostingClassifier` trained on historical feature rows to predict whether a day was a good long-term buy based on 365-day forward return. Training features include:
+
+- Classic metric scores.
+- Raw metric values.
+- 30-day metric deltas.
+- Interaction features such as MVRV × NUPL and Puell × Reserve Risk.
+- Cycle-position context such as days since ATH.
+
+The resulting feature importances are aggregated back into transparent metric weights stored in `config/ml_weights.json`. The UI displays normalized weight and contribution for each active metric.
+
+Validation uses purged expanding time-series splits: because each label uses a 365-day forward-return window, training rows whose label windows overlap validation are removed before scoring validation folds.
 
 ## Tech Stack
 
 | Component | Technology |
 |-----------|-----------|
 | Backend | Python 3.13 + FastAPI |
-| Frontend | Inline HTML/CSS/JS (dark trading terminal theme) |
+| Frontend | Inline HTML/CSS/JS dark trading-terminal UI |
 | Charts | Chart.js |
-| Scraping | Playwright (headless Chromium) |
+| Scraping | requests + Playwright-style browser scraping where needed |
-| Data APIs | alternative.me (F&G), CoinGecko (price) |
+| Data APIs | alternative.me, CoinGecko, LookIntoBitcoin, CheckOnChain |
-| Process Manager | pm2 |
+| ML | NumPy + pandas + scikit-learn GradientBoostingClassifier |
-| Port | 3088 |
+| Process Manager | pm2 or uvicorn |
+| Default Port | 3088 |
 
 ## How Data Is Collected
 
-All data is scraped from free, public sources — **no API keys required**.
+Data is collected from free/public sources and cached locally under `data/`.
 
-LookIntoBitcoin charts use Plotly Dash. We intercept the chart data XHR response which contains full historical time series (5000+ points back to 2010). The scraper runs every 15 minutes for live data and weekly for full historical updates.
+- Fast live refreshes update BTC price, ATH/drawdown, 200D SMA/Mayer where possible, and Fear & Greed.
+- On-chain metrics are cached and reused because they update slowly.
+- Full refresh re-scrapes on-chain metrics from LookIntoBitcoin/CheckOnChain.
+- Historical backtest data lives in `data/history.json` and supports charting, backtests, and metric-context lookups.
+- Score history appends to `data/score_history.jsonl`.
 
 ## Project Structure
 
 ```
 ├── dashboard/
-│   └── server.py              # FastAPI server + inline dashboard HTML
+│   └── server.py                  # FastAPI server + inline dashboard/backtest/settings UI
 ├── scrapers/
-│   ├── lookintobitcoin.py     # Playwright scraper for on-chain charts
+│   ├── lookintobitcoin.py         # LookIntoBitcoin metric scraping
+│   ├── checkonchain.py            # CheckOnChain metric scraping
 │   ├── history_collector.py       # Full historical data collection
+│   ├── history_updater.py         # Incremental historical updates
 │   ├── fear_greed.py              # Fear & Greed Index API
-│   └── price.py               # BTC price API
+│   └── price.py                   # BTC price, ATH, drawdown, SMA helpers
 ├── scoring/
-│   └── engine.py              # Metric scoring logic (0-10 per metric)
+│   └── engine.py                  # Classic + ML-weighted scoring logic
 ├── backtesting/
 │   └── engine.py                  # Historical backtest engine
+├── ml/
+│   └── optimizer.py               # ML training, purged CV, weight export
+├── tests/
+│   ├── test_ml_optimizer_validation.py
+│   └── test_scoring_engine_ml.py
 ├── data/
-│   ├── cache.json             # Live metric cache (auto-generated)
+│   ├── cache.json                 # Live metric cache
-│   └── history.json           # Historical data (auto-generated)
+│   ├── history.json               # Historical metric/time-series data
+│   └── score_history.jsonl        # Live score history
 ├── config/
-│   └── thresholds.json        # Scoring thresholds (customizable)
+│   ├── thresholds.json            # Classic scoring thresholds
-├── screenshots/               # Dashboard screenshots
+│   ├── ml_weights.json            # Learned ML metric weights
-├── ARCHITECTURE.md            # Detailed architecture & scoring logic
+│   └── llm_settings.json          # Optional AI commentary provider config
+├── screenshots/                   # README screenshots
+├── ARCHITECTURE.md
 └── README.md
 ```
 
 ## Running
 
+### Local / ad-hoc with uv
+
 ```bash
-# Install dependencies
+cd /opt/data/btc-accumulation-monitor
-pip install fastapi uvicorn playwright requests
+PYTHONPATH=. uv run \
+  --with fastapi \
+  --with uvicorn \
+  --with requests \
+  --with pandas \
+  --with numpy \
+  --with scikit-learn \
+  python -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088
+```
 
-# Install Playwright browsers (first time only)
+### VPS-style install
-playwright install chromium
 
-# Start the dashboard
+```bash
 cd /opt/apps/btc-ml-optimizer
-python3 -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088
+python3 -m venv .venv
+. .venv/bin/activate
+pip install -r requirements_vps.txt pandas numpy scikit-learn
+python -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088
+```
 
-# Or with pm2
+### pm2
+
+```bash
 pm2 start "python3 -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088" --name btc-ml-optimizer
 ```
 
-### First Run
+## First Run
-1. Visit `http://localhost:3088` — the dashboard will auto-scrape current metrics
-2. Visit `http://localhost:3088/backtest` — triggers historical data collection (takes ~5 min first time)
-3. Data auto-refreshes every 15 minutes after initial scrape
 
-## Backtest Methodology
+1. Visit `http://localhost:3088` for the live dashboard.
+2. Use **Quick Refresh** for fast price/Fear & Greed updates.
+3. Use **Full Refresh** to re-scrape on-chain metrics.
+4. Visit `http://localhost:3088/backtest` to view historical score performance.
+5. If historical data is missing, use the backtest page's collection flow to populate `data/history.json`.
 
-The backtest engine reconstructs the composite score for every historical day and compares against actual BTC forward returns.
+## Useful API Endpoints
 
-**Key feature: Recency weighting** — Bitcoin's cycle returns diminish over time (100x → 30x → 8x → 3-4x). The backtest weights recent cycles more heavily:
+| Endpoint | Description |
-- 2022-present: 4x weight
+|----------|-------------|
-- 2020-2021: 3x weight
+| `GET /api/data?mode=classic` | Current metrics using equal-weight scoring |
-- 2018-2019: 2x weight
+| `GET /api/data?mode=ml` | Current metrics using ML-optimized weights |
-- Pre-2018: 1x weight
+| `GET /api/history` | Recent live score history |
+| `POST /api/refresh` | Quick refresh |
+| `POST /api/refresh?full=true` | Full on-chain refresh |
+| `GET /api/backtest?mode=classic` | Historical backtest with classic scoring |
+| `GET /api/backtest?mode=ml` | Historical backtest with ML scoring |
+| `GET /api/metric-context?metric=mvrv_zscore&mode=ml` | Similar historical levels and forward returns for one metric |
+| `GET /api/settings` | Safe LLM settings payload |
 
-Results are shown per-cycle so you see realistic expectations for the current cycle, not averages inflated by early moonshots.
+## Testing
+
+Focused tests can be run with uv:
+
+```bash
+cd /opt/data/btc-accumulation-monitor
+PYTHONPATH=. uv run --with pytest --with numpy --with scikit-learn --with pandas \
+  pytest -q tests/test_ml_optimizer_validation.py tests/test_scoring_engine_ml.py
+```
 
 ## Architecture
 
-See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation of every metric's scoring logic, data pipeline, and backtest methodology.
+See [ARCHITECTURE.md](ARCHITECTURE.md) for deeper implementation details on scoring, data collection, and backtesting.
 
 ## License