Paper proposes witness hierarchy theory to explain when attention layers fail
What happened
Researcher Mostafa 'Neo' Mohsenvand has submitted a paper to arXiv and made it public, proposing a "witness hierarchy" theory: each attention layer computes a partition function over a family of witnesses and reads it at a temperature. From this the author explains two kinds of Transformer failure — the inability to learn graph search as graph size grows, and the need to see many different problems before learning the card game SET. The paper gives two laws. The depth law states that L layers of two-step walks cover exactly 2^L − 1 steps, and the author says the corresponding construction has been verified in Lean; the scale law requires that the product of the inverse temperature and the interval be no less than the logarithm of the number of candidates. These are the explanation and criteria the author proposes, and what was checked in Lean is the construction itself.
AI-written from the coverage · updated 12 d ago
Coverage timeline
Follow the coverage to see the story from different sides.
- arXiv Board & Classic Game AIPickAttention is only as good as its witnesses: When higher-order layers help, and when they do not
The authors explain the Transformer's failures — not learning graph search as the graph grows, and needing many different problems before it learns the card game SET — as each layer computing a partition function over a family of witnesses and reading it off at a temperature. Two laws follow: a depth law, in which L layers of two-step walks cover exactly 2^L - 1 steps, with the corresponding construction verified in Lean; and a scale law, which requires inverse temperature times margin to be no less than the log of the number of candidates.