Search papers, labs, and topics across Lattice.
This paper theoretically compares in-context learning (ICL) with fixed queries and agentic learning (AL) with adaptive queries for approximating task families, under both unrestricted and ReLU-realizable settings. They show that while adaptivity never hurts in the unrestricted setting, its advantage can change when restricted to ReLU networks. The authors identify four distinct scenarios where adaptivity's advantage is (a) absent, (b) preserved, (c) emerges, or (d) disappears under ReLU realizability, highlighting the complex interaction between representation and adaptivity.
ReLU network constraints can flip the script on whether adaptive querying helps in-context learning.
We compare in-context learning with fixed queries and agentic learning with adaptive queries for uniform approximation of task families. We consider two settings: an unrestricted regime, where querying and approximation are arbitrary functions, and a realizable regime, where we require these operations to be implemented by ReLU neural networks. In both settings, adaptivity never hinders approximation performance. However, this advantage can change when one passes from the unrestricted regime to the realizable regime. We identify four distinct approximation scenarios, each witnessed by an explicit task family: (a) no advantage of adaptivity; (b) an advantage in the unrestricted regime that persists under ReLU realizability; (c) an advantage that arises only under realizability; and (d) an advantage that disappears under realizability. This demonstrates that representational constraints interact profoundly with the effect of adaptivity.
