arXiv Board & Classic Game AI· Mark Marosi·· 3 d agoAI score26
Language models can recover specific abilities through general progress after training data is filtered
General progress substitutes for specific training data in language models
AI summary
Research shows that when a skill is masked from a language model's training data, the model's general progress on the remaining data can substitute for the masked data. In experiments on chess, Othello, molecular strings, bracket languages, and web text, the delay in the masked skill for larger models can be predicted from smaller models; a chess model never explicitly trained to respond to check ranks almost as highly on legal responses as a specially trained model.
Source: arXiv Board & Classic Game AI · arxiv.org