Problem
Most RL trading projects train on a fixed dataset and end with a backtest. I wanted to work on the harder part: learning online from streaming market data and showing enough of the system to understand what it is doing. This started as my April 2026 AI/ML bootcamp capstone.
What I built
A paper-trading online-learning system for BTC perpetual futures. A Soft Actor-Critic agent updates as data streams in, so the weights making decisions are the same weights being trained.
It builds a 116-dimensional state vector from 16 feature blocks, including 5Hz L2 order-book microstructure, funding and open interest, cross-exchange premium, liquidations, Deribit options flow, stablecoin supply, and social sentiment. The dashboard tracks freshness for each block.
Four neural encoders turn that state into latents:
- an order-book CNN over the top 15 price levels × 60 ticks of L2 data
- a 2-layer GRU temporal encoder over the recent feature history
- a 116-dim full-state MLP over the raw feature vector
- a cross-block attention encoder treating the 16 blocks as tokens
The continuous SAC head also has an anti-churn temperature integrator, which helps keep it from moving in and out of positions too often, and a hold-triangle reward that rewards holding through favorable moves instead of scalping every tick.
The dashboard is an X-ray of the network as it runs. It shows encoder latent heatmaps, Q-values, actor and critic losses, the trade tape, and the NAV curve over WebSocket.
Outcome
The system runs continuously on a Hetzner box, collecting a dozen market feeds, making paper decisions, and training the SAC agent. The dashboard shows what it is doing, including the P&L. A recorded April 2026 run is also available as a replay.
The useful part of this project is the machinery around online learning: messy multi-source market data, observable model state, and a paper-trading setup that does not hide an unflattering result behind a polished backtest.