Turkish language that spaCy doesn’t yet provide pre-trained models

[omer (Omer Yavuz)]

Do you have any model for Turkish language?

I saw that

How is the support for Languages other than English?

To add to @honnibal's comment above, here's a thread that shows an example of using Prodigy with languages that spaCy doesn't yet provide pre-trained models for (in this case, to train a Norwegian text classifier): And this thread discusses using Prodigy to add NER and text classification capabilities to a Chinese spaCy model (which, according to the user, seems to have worked well):

Working with languages not yet supported by Spacy

My first guess was that the spaCy model you’re loading in hasn’t loaded the vectors correctly. If the vectors aren’t loaded, the similarity would be 0.0 for all words. Currently I think this would result in it fruitlessly looping over the vocabulary. However, your script looks correct. I then had another look at the terms.teach recipe. It looks like it checks the vocabulary entries for is_lower and is_alpha. Could you check whether these attributes are set correctly in your vocabulary? You sho…

how to write a model.update() function

My team want to load and re-train some Chinese models through Prodigy, e.g. NER model and text-classification model. Since spaCy doesn’t provide any basic Chinese models, we are trying to implement these recipes. From Here, 'update': model.update, # update model with annotations I am still confused about the input, output and main process logic of this bound method model.update. Is there any paradigm to help me write a correct one?

What is best roadmap to create new model?
What kind of minimum information do I need to create new model?

Thanks for your advice

[ines (Ines Montani)]

If you want to train a model from scratch and you also want to train a tagger and dependency parser, you probably want to start off with an existing treebank. More details here:

• How to go from alpha support to a pre-trained model
• Universal dependencies corpora for Turkish
• Training a spaCy model from a UD corpus via the command line

Using Prodigy, you can then create your own named entity annotations. This is by the way how the named entity recognizer of the new Greek model (currently available for spacy-nightly) was trained. It’s a slightly more involved process, because you want to make sure you have enough examples (minimum a few thousand fully annotated) and also a good evaluation. Because you need gold-standard annotations (all labels annotated and no missing values), you probably want to use the ner.manual recipe and label it by hand.

[leotsiakolou]

In order to train a NER recognizer in this particular context (language like Turkish, Greek without a pre-defined model), do we need to have dependency parser? Or is it enough to annotate only with named entities? What is the recommended number of sentences for like 5 different datatypes?

[ines (Ines Montani)]

@leotsiakolou

In order to train a NER recognizer in this particular context (language like Turkish, Greek without a pre-defined model), do we need to have dependency parser?

spaCy's components can be trained indepdently, so you don't need a parser to train an entity recognizer. (The xx_ent_wiki_sm model is an example of a model that was only trained on NER annotations and that only has an NER component.)

@leotsiakolou

What is the recommended number of sentences for like 5 different datatypes?

That really depends on your data, entity frequencies etc. We typically recommend annotationg at least a few hundred to a few thousand for meaningful results that you can draw conclusions from. If you're using Prodigy, you can periodically run the train-curve recipe to check if your model is improving with more data and to see if you're on the right track.