Skip to content
arXiv Board & Classic Game AI· Mark Marosi·· 3 d agoAI score26

Language models can recover specific abilities through general progress after training data is filtered

General progress substitutes for specific training data in language models

AI summary

Research shows that when a skill is masked from a language model's training data, the model's general progress on the remaining data can substitute for the masked data. In experiments on chess, Othello, molecular strings, bracket languages, and web text, the delay in the masked skill for larger models can be predicted from smaller models; a chess model never explicitly trained to respond to check ranks almost as highly on legal responses as a specially trained model.

Source: arXiv Board & Classic Game AI · arxiv.org