A streaming risk pipeline: Kafka market-tick ingestion, Spark Structured Streaming for rolling aggregates, a Delta bronze layer, and a FastAPI service computing historical-simulation VaR behind a Streamlit dashboard.
Traditional risk systems compute Value at Risk (VaR) overnight. A portfolio manager running an intraday position change doesn't know their updated risk exposure until the next morning. In volatile markets, that lag is not a workflow inconvenience - it's a risk management failure.
The secondary problem is architecture: financial data arrives as a firehose of price ticks. Computing portfolio-level VaR from streaming tick data requires joining live prices against position records, maintaining rolling historical windows, and running Monte Carlo or historical simulation on each update. Doing this in a batch system is fundamentally the wrong tool. Streaming with Spark changes the problem into a solvable one.
The pipeline has four distinct layers, with Kafka as the central nervous system.
A Kafka producer simulates realistic equity price ticks: mid-price with configurable
bid-ask spread, volume weighting, and random walk drift. Produces to a
market_data topic partitioned by ticker symbol for ordering guarantees
within each instrument.
Spark Structured Streaming joins live price ticks against a position snapshot loaded at startup. For each 5-second micro-batch, it computes mark-to-market P&L, updates the rolling 30-day return history, and recalculates Historical Simulation VaR at 95% confidence. A Delta writer lands the bronze layer alongside the streaming output.
FastAPI exposes risk query endpoints: current VaR by portfolio or asset class, historical VaR timeseries, and position-level attribution. It reads the most recent processed file and drops back to a seeded sample when the directory is empty, so the dashboard is never blank on a cold start.
Historical Simulation VaR was chosen over parametric (variance-covariance) and Monte Carlo for a deliberate reason: it makes no distributional assumptions. Equity returns are fat-tailed and skewed. A parametric model that assumes normality will systematically underestimate tail risk. Historical simulation uses the actual observed return distribution, so the 2008 crash, the 2020 COVID selloff, and the 2022 rate shock all live in the historical window and inform the VaR calculation.
The implementation maintains a 30-day rolling window of daily returns per instrument. On each Spark micro-batch, it recomputes the P&L distribution by applying historical returns to the current position, sorts the distribution, and reads the 5th percentile as the 95% 1-day VaR. Component VaR is computed per asset, which allows attribution: which positions are contributing most to portfolio tail risk.
The Streamlit surface reads the FastAPI endpoints and renders four panels:
Kafka partitioning by ticker. Partitioning the market data topic by symbol guaranteed ordered delivery per instrument. This made the Spark join against position data deterministic - no race conditions between out-of-order price ticks for the same ticker.
Historical simulation window size. A 30-day window is too short to capture low-frequency stress events. In calm market periods, the VaR estimate will be systematically too low. Needed a secondary stressed VaR calculation using a 2008 or 2020 scenario window - didn't build this.
Loading positions once at startup. Holding the position snapshot in memory rather than re-reading it on every Spark micro-batch kept the join cheap. The caching layer that would back this at scale is part of the unbuilt half.
Designing the dashboard around polling. A 2-second poll against FastAPI adds load and refresh lag for no benefit. Server-sent events pushing on each micro-batch completion is the right shape, and choosing polling first meant designing the API around the wrong access pattern.
Component VaR attribution. Computing per-asset component VaR made the dashboard useful beyond just a total number. Seeing that a single position accounts for 40% of portfolio VaR is actionable - just seeing total VaR isn't.
Spark local mode for development. Running Spark in local mode masked memory and executor issues that appeared in distributed mode. Should have used a Docker Compose Spark cluster from day one - switching modes late in development required reconfiguring checkpoint paths and memory settings.