Spacy NER - tokeniser for camembert-base

[koaning (Vincent D Warmerdam)]

The docs that you mention refer to this custom recipe found in a file called transformers_tokenizers.py on Github. Is this the recipe you were looking for? Let me know if that's not the case. This recipe is meant for users who want to train their own huggingface models without spaCy in the mix.

Note that if you're interested in using spaCy with transformers that you can also just use the standard ner.manual recipe. I wrote a bit more context about that here:

config.cfg for bert.ner.manual

For my understanding, what is your goal? Do you wish to train and update a Huggingface BERT model without spaCy? If so, you'll need to use that library to train a component and you can use the data generated from this recipe. You'd need to take the extra effort here, because Huggingface might use a different tokeniser. If you wish to use BERT as part of a spaCy pipeline, then you can use the normal ner.manual recipe for annotation and you don't need to worry about the tokens. You can just use e…

In case it's relevant: more information on the user of transformer models in spaCy can be found on this Github FAQ.