Paper builds controlled Atari and sente evaluation tasks to test the language-based Go model LoGos
Recognizing Atari and Assessing Sente: Controlled Evaluation Tasks for Language-Based Go Agents
The researchers built a controlled evaluation set around the relationship between atari and sente in Go, to test whether language models can recognize these terms and connect them to move evaluation. The subject of the test is LoGos, a Go language model supervised-fine-tuned on terminology and commentary data, and the results show it struggles both to recognize changes in atari relationships and to judge the strategic meaning of such changes.
The authors use controlled local board modifications to separate term recognition from move evaluation, offering a reusable test approach for checking whether language models can ground domain terminology in a specific position.
Source: arXiv Game AI · arxiv.org