Search papers, labs, and topics across Lattice.
This paper introduces an exact dynamic programming oracle for infinite-shoe casino blackjack, providing a rigorous benchmark for evaluating model-free policy optimization under dynamically masked actions. Three model-free optimizers (REINFORCE, SPSA, and CEM) were trained and evaluated against the oracle, revealing that while REINFORCE was the most sample-efficient, all methods exhibited significant cell-conditional regret, indicating persistent policy errors despite smooth reward convergence. The study also proves and empirically confirms that optimal bet sizing collapses to the table minimum under i.i.d. draws, serving as a negative control.
Even in a seemingly simple tabular environment like Blackjack, model-free RL agents can converge to near-optimal *average* rewards while still making surprisingly poor decisions in specific states.
Infinite-shoe casino blackjack provides a rigorous, exactly verifiable benchmark for discrete stochastic control under dynamically masked actions. Under a fixed Vegas-style ruleset (S17, 3:2 payout, dealer peek, double on any two, double after split, resplit to four), an exact dynamic programming (DP) oracle was derived over 4,600 canonical decision cells. This oracle yielded ground-truth action values, optimal policy labels, and a theoretical expected value (EV) of -0.00161 per hand. To evaluate sample-efficient policy recovery, three model-free optimizers were trained via simulated interaction: masked REINFORCE with a per-cell exponential moving average baseline, simultaneous perturbation stochastic approximation (SPSA), and the cross-entropy method (CEM). REINFORCE was the most sample-efficient, achieving a 46.37% action-match rate and an EV of -0.04688 after 10^6 hands, outperforming CEM (39.46%, 7.5x10^6 evaluations) and SPSA (38.63%, 4.8x10^6 evaluations). However, all methods exhibited substantial cell-conditional regret, indicating persistent policy-level errors despite smooth reward convergence. This gap shows that tabular environments with severe state-visitation sparsity and dynamic action masking remain challenging, while aggregate reward curves can obscure critical local failures. As a negative control, it was proven and empirically confirmed that under i.i.d. draws without counting, optimal bet sizing collapses to the table minimum. In addition, larger wagers strictly increased volatility and ruin without improving expectation. These results highlight the need for exact oracles and negative controls to avoid mistaking stochastic variability for genuine algorithmic performance.
