Ali Behrouz

Allocating more capacity to earlier layers in language models can significantly enhance performance, challenging the long-held uniform layer design paradigm.

Language models can achieve remarkable improvements in learning and memory retention by mimicking the human processes of sleep and dreaming.

Recurrent models can now achieve Transformer-competitive performance on recall-intensive tasks, thanks to a simple memory caching mechanism that grows memory capacity with sequence length.