Study: general training progress can substitute for hidden specialized data
What happened
Mark Marosi published research claiming that when a skill is blocked from a language model's training data, its overall progress on the remaining data can substitute for the blocked data. In experiments on chess, Othello, molecular strings, bracket languages, and web text, the research showed that larger models' performance on the blocked skill can be predicted by smaller models; a chess model never explicitly trained to respond to checks ranked legal responses almost on par with a specially trained model. These conclusions come from the above experiments and indicate that models can still acquire related abilities after specialized data is withheld, but the report does not state how far this substitution relationship applies to other tasks or model scales.
AI-written from the coverage · updated 3 d ago
Coverage timeline
Follow the coverage to see the story from different sides.
- arXiv Board & Classic Game AILanguage models can recover specific abilities through general progress after training data is filtered
Research shows that when a skill is masked from a language model's training data, the model's general progress on the remaining data can substitute for the masked data. In experiments on chess, Othello, molecular strings, bracket languages, and web text, the delay in the masked skill for larger models can be predicted from smaller models; a chess model never explicitly trained to respond to check ranks almost as highly on legal responses as a specially trained model.
Heat trend for this story
- Current (comparable)
- 3
- Peak (comparable)
- 7Nov 19 13:00
- 24-hour change
- -50%
The trend only compares participants observed continuously and completely, so it may cover fewer than the current heat count. Hover over or tap the chart to see each hour's heat; with a keyboard, use the left and right arrow keys.