Chapter 6:Reinforcement Learning and Inverse Reinforcement Learning

What are the best first use cases?Start where state, action, and reward are clear and the feedback cycle is short: adaptive trade execution, dynamic portfolio rebalancing, and cost-aware option hedging. These map cleanly to RL/POMDPs, have measurable baselines (e.g., time-weighted average price/volume-weighted average price [TWAP/VWAP], discrete delta), and abundant historical data for offline training.

Can I train only on historical data, or do I need live exploration?You can (and usually should) start with offline RL using your fills, prices, and positions. Then validate in a high-fidelity simulator with costs/impact/latency, run shadow mode alongside your existing process, and promote gradually with guardrails (caps, kill-switch, rollback).

How do I build risk and costs into the objective?Make risk and costs part of the goal. Define the reward as the money you make after subtracting trading fees/price impact and a penalty for risk. In words:Reward = Profit − Costs − λ × Risk (risk can be tail risk, such as CVaR, drawdown, or mean–variance). Use distributional RL to capture rare big losses (“the tails”). And set hard limits — on exposure, turnover, and market participation — both while training and when the system runs live.

IRL versus imitation learning — when do I use which?Use IRL to infer the underlying objective from behavior (managers, clients, “the market”) when you want portability and the ability to surpass demonstrations. Use imitation to quickly mimic actions when you don’t need a reward function. Ranked data? Consider T-REX. Probabilistic, flexible rewards? MaxEnt/Bayesian (GPIRL).

What metrics should I monitor to know the policy is working?At minimum, track implementation shortfall (IS) for execution quality, risk-adjusted return after costs (e.g., Sharpe or mean–variance utility) for performance, and CVaR/drawdown for tails. Add drift detectors (feature, policy, regime) and compare to baselines (TWAP/VWAP, risk parity, discrete delta).

How do I make the RL/IRL policy compliant and explainable?Log state → action → outcome with immutable audit trails; publish a “policy card” (objective, constraints, data lineage, promotion criteria); add explainability (feature attribution, counterfactuals), runtime guardrails (exposure/participation/loss caps), challenger policies, and human-in-the-loop approvals. These actions turn the model into an accountable decision system, not a black box.