Having Issue with Spacy Train with custom tokenizer component which we used to annotate in prodigy

[magdaaniol]

Welcome to the forum @Bhargavi1144

What I think is happening is that spaCy was not explicitly instructed to use the custom tokenizer.
Unfortunately, the sourcing of the tokenizer from the base model is not automated (yet) and we need to instruct spaCy where to source the tokenizer from via the config file.

This thread provides step-by-step instruction on how to properly source a custom tokenizer for training:

Training after annotating with custom tokenizer

Hi @AakankshaP, Apologies for the late reply! Assuming you have already annotated your dataset with your custom tokenizer, the next step would be to generate a config file so that we can control the tokenizer used for data generation and the training. Prodigy provides a utility for generating the base config and we'll use it as a starting point. So, assuming your annotated dataset is named custom_tok_annotations and that your custom pipeline is stored in custom_tokenizer directory, you can o…

Let me know if you need extra support on top of that tutorial