Having Issue with Spacy Train with custom tokenizer component which we used to annotate in prodigy

Welcome to the forum @Bhargavi1144 :wave:

What I think is happening is that spaCy was not explicitly instructed to use the custom tokenizer.
Unfortunately, the sourcing of the tokenizer from the base model is not automated (yet) and we need to instruct spaCy where to source the tokenizer from via the config file.

This thread provides step-by-step instruction on how to properly source a custom tokenizer for training:

Let me know if you need extra support on top of that tutorial :slight_smile: