From 63d4b6c86a28f0519199a8f802a2fded24363bc4 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Sun, 26 Jul 2026 22:53:32 +0000 Subject: [PATCH] docs: document locked setup deployment and ML provenance --- README.md | 105 +++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 76 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index a42fcc6..b8bdc47 100644 --- a/README.md +++ b/README.md @@ -145,49 +145,93 @@ Data is collected from free/public sources and cached locally under `data/`. │ ├── ml_weights.json # Learned ML metric weights │ └── llm_settings.json # Optional AI commentary provider config ├── screenshots/ # README screenshots +├── scripts/run.sh # Locked local launcher with Playwright path +├── .gitea/workflows/ci.yml # Gitea Actions test/compile gates +├── Dockerfile # Non-root Chromium-enabled image +├── docker-compose.yml # Port, healthcheck, restart, persistent volumes +├── pyproject.toml # Runtime, ML, and development dependency groups +├── uv.lock # Exact reproducible dependency resolution ├── ARCHITECTURE.md └── README.md ``` -## Running +## Reproducible Setup -### Local / ad-hoc with uv +Install [uv](https://docs.astral.sh/uv/) and use Python 3.11-3.13. Dependencies are declared in explicit `runtime`, `ml`, and `dev` groups in `pyproject.toml`; exact cross-platform resolutions are committed in `uv.lock`. ```bash -cd /opt/data/btc-accumulation-monitor -PYTHONPATH=. uv run \ - --with fastapi \ - --with uvicorn \ - --with requests \ - --with pandas \ - --with numpy \ - --with scikit-learn \ +git clone +cd btc-accumulation-monitor +uv sync --locked --group runtime --group ml --group dev +``` + +Install the Chromium binary once for full on-chain refreshes. Keep its path explicit so installation and runtime use the same browser cache: + +```bash +export PLAYWRIGHT_BROWSERS_PATH="$PWD/.playwright" +uv run --frozen playwright install chromium +``` + +`requirements_vps.txt` is a lock-derived, hash-pinned compatibility export for pip-based hosts. `pyproject.toml` and `uv.lock` remain authoritative; regenerate the compatibility file after dependency changes with: + +```bash +uv export --frozen --no-dev --group runtime --group ml \ + --no-emit-project --no-header --output-file requirements_vps.txt +``` + +## Running + +The executable launcher fixes `PYTHONPATH`, preserves an explicitly supplied `PLAYWRIGHT_BROWSERS_PATH`, and starts port 3088 from the locked environment: + +```bash +./scripts/run.sh +``` + +Equivalent exact command: + +```bash +PLAYWRIGHT_BROWSERS_PATH="$PWD/.playwright" PYTHONPATH=. \ + uv run --frozen --no-dev --group runtime --group ml \ python -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088 ``` -### VPS-style install +Then visit `http://localhost:3088`. + +## Container Deployment + +The image uses a multi-architecture Python base, installs Playwright Chromium and its OS libraries during the build, and runs the application as non-root UID `10001`. Compose publishes port 3088, restarts unless stopped, and persists `/app/data` and `/app/config` in named volumes. ```bash -cd /opt/apps/btc-ml-optimizer -python3 -m venv .venv -. .venv/bin/activate -pip install -r requirements_vps.txt pandas numpy scikit-learn -python -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088 +docker compose build +docker compose up -d ``` -### pm2 +The Docker and Compose healthchecks probe `GET /health/live`. The deployment must include the reliability revision that supplies that endpoint; without it, Docker correctly reports the container unhealthy even if the older application server is accepting requests. + +Named volumes are initialized from the image on first use. Back up both before replacing or deleting them: ```bash -pm2 start "python3 -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088" --name btc-ml-optimizer +docker volume inspect btc-accumulation-monitor_btc-monitor-data +docker volume inspect btc-accumulation-monitor_btc-monitor-config ``` -## First Run +For bind-mounted deployments, ensure the host directories are writable by UID `10001` and do not replace `config/` with an empty directory. -1. Visit `http://localhost:3088` for the live dashboard. -2. Use **Quick Refresh** for fast price/Fear & Greed updates. -3. Use **Full Refresh** to re-scrape on-chain metrics. -4. Visit `http://localhost:3088/backtest` to view historical score performance. -5. If historical data is missing, use the backtest page's collection flow to populate `data/history.json`. +## First Run and Data Freshness + +1. Visit `http://localhost:3088` for the dashboard. +2. Use **Quick Refresh** for price and Fear & Greed updates while retaining cached slow-moving on-chain metrics. +3. Use **Full Refresh** when on-chain source data must be re-scraped; this requires the installed Playwright Chromium browser and external source availability. +4. Visit `http://localhost:3088/backtest` for historical analysis. +5. If historical data is missing, populate `data/history.json` through the existing collection flow. + +Freshness is metric-specific. Price and sentiment APIs can update frequently, while public on-chain chart sources commonly update daily and may be reused from cache. A successful refresh is not proof that every upstream metric has a new observation. Check source timestamps/status exposed by the running revision, and treat missing, stale, or scrape-failed metrics as unavailable rather than silently current. `data/` is operational state and should be persisted and backed up. + +## ML and Backtest Caveats + +ML weights and backtest output are research artifacts, not investment advice or evidence of future performance. Any reported ML result must retain its provenance: source-data snapshot/range, feature and label definitions, training window, purge/embargo policy, code revision, dependency lock, random seed (when applicable), and generated weight/config artifact. + +Model selection and threshold tuning must use training/validation data only. Report final performance on a genuinely untouched out-of-sample (OOS) period; do not describe in-sample fit, cross-validation used for selection, or the best result from repeated experiments as OOS. Forward-return labels require purging overlapping label horizons, but purged cross-validation alone does not create an untouched final test set. Results without reproducible provenance and a reserved OOS evaluation should be labeled exploratory. ## Useful API Endpoints @@ -203,16 +247,19 @@ pm2 start "python3 -m uvicorn dashboard.server:app --host 0.0.0.0 --port 3088" - | `GET /api/metric-context?metric=mvrv_zscore&mode=ml` | Similar historical levels and forward returns for one metric | | `GET /api/settings` | Safe LLM settings payload | -## Testing +## Testing and CI -Focused tests can be run with uv: +Run the committed test suite and the same static compilation gate used by Gitea Actions: ```bash -cd /opt/data/btc-accumulation-monitor -PYTHONPATH=. uv run --with pytest --with numpy --with scikit-learn --with pandas \ - pytest -q tests/test_ml_optimizer_validation.py tests/test_scoring_engine_ml.py +uv sync --locked --group runtime --group ml --group dev +uv run --frozen python -m compileall -q \ + dashboard scrapers scoring backtesting ml ml_engine llm_client scripts orchestrator.py +uv run --frozen pytest ``` +`.gitea/workflows/ci.yml` runs lock validation/install, static compilation, and tests for pull requests and pushes to `main`. + ## Architecture See [ARCHITECTURE.md](ARCHITECTURE.md) for deeper implementation details on scoring, data collection, and backtesting.