Very interesting, now I really wonder about Spanish, and Mandarin Chinese. There is so much written literature in Spanish that I would guess that much of it has been used to train models. For Chinese is a similar case but it has the interesting side effect to be a very different language than English. (English has germanic roots, but most of its words are latin based so not so far away from Spanish).
Finnish is in a totally different language family than Swedish-Norwegian-Danish. However Viking performs the same result. I am wondering the outcome when using Viking for Hungarian, as it is a near relative to Finnish.
Viking was trained on Finnish as well, so that's why it gets good results.
I ran a quick check on Hungarian, Viking didn't perform well at all(about 2x the tokens of English).
Would have been interesting to include German and Dutch. The German training corpus must be much bigger while the language is roughly the same "distance" from English as other Germanic languages except Dutch. Dutch is most similar to English, but likely has a smaller training corpus than German.
Very interesting, now I really wonder about Spanish, and Mandarin Chinese. There is so much written literature in Spanish that I would guess that much of it has been used to train models. For Chinese is a similar case but it has the interesting side effect to be a very different language than English. (English has germanic roots, but most of its words are latin based so not so far away from Spanish).
I’d be curious to see this ran through cohere’s tokenizer - they seem to be the largest lab with a focus on multilingualism.
Finnish is in a totally different language family than Swedish-Norwegian-Danish. However Viking performs the same result. I am wondering the outcome when using Viking for Hungarian, as it is a near relative to Finnish.
Viking was trained on Finnish as well, so that's why it gets good results. I ran a quick check on Hungarian, Viking didn't perform well at all(about 2x the tokens of English).
Would have been interesting to include German and Dutch. The German training corpus must be much bigger while the language is roughly the same "distance" from English as other Germanic languages except Dutch. Dutch is most similar to English, but likely has a smaller training corpus than German.