theory

Transformers Are Inherently Succinct: What an Expressivity Result Can and Cannot Tell You

A new paper proves transformers represent certain languages exponentially more succinctly than temporal logic and RNNs, and doubly exponentially more so than automata. It explains scale, it is not an engineering guide.