Search papers, labs, and topics across Lattice.
This paper introduces a transformer-based approach for solving quadratic programs (QPs) by tokenizing matrix variables and emulating iterative optimization algorithms within the transformer architecture. The approach leverages linear attention to solve unconstrained QPs and incorporates MLPs and feedback loops to handle $\ell_1$-penalized and $\ell_1$-constrained QPs, respectively. The proposed "Time2Decide" method, which enhances time series foundation models by explicitly feeding the covariance matrix, outperforms both the base TSFM model and the traditional "Predict-then-Optimize" approach in portfolio optimization tasks.
Transformers can directly solve quadratic programs and leverage covariance matrices for superior decision-making, outperforming traditional "predict-then-optimize" methods in portfolio construction.
We explore the use of transformers for solving quadratic programs and how this capability benefits decision-making problems that involve covariance matrices. We first show that the linear attention mechanism can provably solve unconstrained QPs by tokenizing the matrix variables (e.g.~$A$ of the objective $\frac{1}{2}x^\top Ax+b^\top x$) row-by-row and emulating gradient descent iterations. Furthermore, by incorporating MLPs, a transformer block can solve (i) $\ell_1$-penalized QPs by emulating iterative soft-thresholding and (ii) $\ell_1$-constrained QPs when equipped with an additional feedback loop. Our theory motivates us to introduce Time2Decide: a generic method that enhances a time series foundation model (TSFM) by explicitly feeding the covariance matrix between the variates. We empirically find that Time2Decide uniformly outperforms the base TSFM model for the classical portfolio optimization problem that admits an $\ell_1$-constrained QP formulation. Remarkably, Time2Decide also outperforms the classical "Predict-then-Optimize (PtO)" procedure, where we first forecast the returns and then explicitly solve a constrained QP, in suitable settings. Our results demonstrate that transformers benefit from explicit use of second-order statistics, and this can enable them to effectively solve complex decision-making problems, like portfolio construction, in one forward pass.
