The model read about 16 billion characters of web text in 20 languages (about 60% Russian, about 16% English).
It is more of a learning model than a serious industrial one: it was trained on a single laptop GPU, an RTX 4050 (6 GB). During training, without a restart, the model “grew”: we added new body blocks on the fly. After training it was trimmed by removing the least useful half of the neurons and part of the attention. Of 84 M parameters, 47 M remained, and quality barely dropped. After a short round of further training, the model was ready.