How to save a custom tokenizer

Sorry about the link restriction – this is just a setting for new accounts, to prevent spam bots.

Could you share the code of your custom tokenizer?

The solution discussed in this thread is definitely an option, especially for more advanced custom tokenizers – but depending on your code, it might not actually be necessary to implement all of the methods from scratch. And while you can package the tokenizer with the model if you want it to be super elegant, you don’t have to, especially not in the beginning while you’re still getting familiar with spaCy’s API :slightly_smiling_face:

So for now, the most important thing is to make the model save out something (anything!) as the tokenizer, so you can save it and load it back in without problems. You can then always re-add your tokenizer afterwards, by writing to nlp.tokenizer of the loaded model:

nlp = spacy.load('/your-custom-model')
nlp.tokenizer = your_custom_tokenizer(nlp)

doc = nlp(u"I will be processed with the custom tokenizer")