"We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution"
Due to new learning technique the model has achieved better generalization skill without overfitting and memorization. This become possible because of new learning method which made the student model to generate samples for itself. It led to intensive reuse of existing neurons and allowed to encode information in a more dense way
While researchers are calling it a compression, I think it's a retopologization, and Microsoft had tried to do something similar in the past with their Phi model family, which they trained on reduced dictionary and simplified knowledge base first. But it seems like MS' researchers didn't explore this exact way of learning. I believe this should give even better results in the future and this is another small breakthrough moment
Interestingly, the phi team lead then moved to oAI and was (rumoured to be) the lead for gpt-oss, which are still two very strong models in their class, 1 year later. Also trained on mostly synthetic data, and "interesting" close to zero activations for some subjects (erotica, etc) according to the local folks that are into that sort of thing.
"We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution"
Due to new learning technique the model has achieved better generalization skill without overfitting and memorization. This become possible because of new learning method which made the student model to generate samples for itself. It led to intensive reuse of existing neurons and allowed to encode information in a more dense way
While researchers are calling it a compression, I think it's a retopologization, and Microsoft had tried to do something similar in the past with their Phi model family, which they trained on reduced dictionary and simplified knowledge base first. But it seems like MS' researchers didn't explore this exact way of learning. I believe this should give even better results in the future and this is another small breakthrough moment
Paper: https://arxiv.org/html/2607.11883v1
Repository: https://github.com/shikaiqiu/requential-coding
> similar in the past with their Phi model family
Interestingly, the phi team lead then moved to oAI and was (rumoured to be) the lead for gpt-oss, which are still two very strong models in their class, 1 year later. Also trained on mostly synthetic data, and "interesting" close to zero activations for some subjects (erotica, etc) according to the local folks that are into that sort of thing.