Ying Wei

Linear attention's recurrent state can be compressed down to 4 bits while outperforming uniform INT8 once precision is dynamically allocated to long-lived memory tracks and output-sensitive key rows.

OPD transfers reasoning skills rather than answers, revealing a complex interplay between teacher-student origins that can either enhance or hinder model capabilities.