# Having Issue with Spacy Train with custom tokenizer component which we used to annotate in prodigy

**URL:** <https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054>\
**Category:** Uncategorized\
**Tags:** ner, spacy\
**Created:** [January 25, 2024, 5:00pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054 "2024-01-25T17:00:19Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [January 26, 2024, 3:40pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/2 "2024-01-26T15:40:02Z")

</div>

Welcome to the forum @Bhargavi1144 👋

What I think is happening is that spaCy was not explicitly instructed to use the custom tokenizer.  
Unfortunately, the sourcing of the tokenizer from the base model is not automated (yet) and we need to instruct spaCy where to source the tokenizer from via the config file.

This thread provides step-by-step instruction on how to properly source a custom tokenizer for training:

> [@Training after annotating with custom tokenizer](https://support.prodi.gy/t/training-after-annotating-with-custom-tokenizer/6846/2):
>
> Hi @AakankshaP, Apologies for the late reply! Assuming you have already annotated your dataset with your custom tokenizer, the next step would be to generate a config file so that we can control the tokenizer used for data generation and the training. Prodigy provides a utility for generating the base config and we'll use it as a starting point. So, assuming your annotated dataset is named custom\_tok\_annotations and that your custom pipeline is stored in custom\_tokenizer directory, you can o…

Let me know if you need extra support on top of that tutorial 🙂

---

_[View the full topic](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054)._