# Having Issue with Spacy Train with custom tokenizer component which we used to annotate in prodigy

**URL:** <https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054>\
**Category:** Uncategorized\
**Tags:** ner, spacy\
**Created:** [January 25, 2024, 5:00pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054 "2024-01-25T17:00:19Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhargavi1144](https://avatars.discourse-cdn.com/v4/letter/b/ee7513/32.png) [@Bhargavi1144](https://support.prodi.gy/u/Bhargavi1144)\
**Post date:** [January 25, 2024, 5:00pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/1 "2024-01-25T17:00:19Z")

</div>

nlp.tokenizer.rules = rules  
infixes = list(nlp.Defaults.infixes)  
infixes.extend([r"-", **r"kt",r"mt"** , r"$", r":", r"\€", r"\£", r"\¥", r"+", r"[", r"]"])  
infix\_re = spacy.util.compile\_infix\_regex(infixes)  
nlp.tokenizer.infix\_finditer = infix\_re.finditer  
print(nlp.pipe\_names)  
tok2vec = nlp.add\_pipe("tok2vec")  
nlp.to\_disk("assets/base\_model\_new")

1. I have added some special cases in tokenizer and generated a base model to annotate **100mt as 100 and mt** in prodigy.

2.Used data-to-spacy with the base model I have generated above to get the .spacy files

3.At Last Trained the model using **spacy train --code** command with same custom tokenizer component but the results seems to be not proper.

Can you help me here that am going in a right direction or not?

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [January 26, 2024, 3:40pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/2 "2024-01-26T15:40:02Z")

</div>

Welcome to the forum @Bhargavi1144 👋

What I think is happening is that spaCy was not explicitly instructed to use the custom tokenizer.  
Unfortunately, the sourcing of the tokenizer from the base model is not automated (yet) and we need to instruct spaCy where to source the tokenizer from via the config file.

This thread provides step-by-step instruction on how to properly source a custom tokenizer for training:

> [@Training after annotating with custom tokenizer](https://support.prodi.gy/t/training-after-annotating-with-custom-tokenizer/6846/2):
>
> Hi @AakankshaP, Apologies for the late reply! Assuming you have already annotated your dataset with your custom tokenizer, the next step would be to generate a config file so that we can control the tokenizer used for data generation and the training. Prodigy provides a utility for generating the base config and we'll use it as a starting point. So, assuming your annotated dataset is named custom\_tok\_annotations and that your custom pipeline is stored in custom\_tokenizer directory, you can o…

Let me know if you need extra support on top of that tutorial 🙂

---

<div class="post-metadata">

**Author:** ![Bhargavi1144](https://avatars.discourse-cdn.com/v4/letter/b/ee7513/32.png) [@Bhargavi1144](https://support.prodi.gy/u/Bhargavi1144)\
**Post date:** [January 29, 2024, 10:46am UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/3 "2024-01-29T10:46:07Z")

</div>

Hello,Thanks for the quick response!

I tried the same steps as mentioned in the above documentation.  
I could able to generate the config.cfg and training sets using custom tokenizer.

But when I tried to run the spacy train command ,its giving me the below exception:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/2/2487dfa72cb41ccc9ce2f1e529dab94bdd0ef8b0.png)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/5/50182a3e21f9eba4078223a17fdd51bdaf7a792c.png)

Though I don't see any messages regarding copying the custom tokenizer as well under Initializing pipeline section.

Can you help me with this step if I missing anything here?

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [January 30, 2024, 10:48am UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/4 "2024-01-30T10:48:45Z")

</div>

Hi @Bhargavi1144 ,

The error you are getting is likely related to the component not being correctly initialized.  
And, indeed, in the "Initializing pipeline" section of the console log, there should be information on copying the tokenizer from a pipeline.

Could you share the config file that you use for `train` command?

---

<div class="post-metadata">

**Author:** ![Bhargavi1144](https://avatars.discourse-cdn.com/v4/letter/b/ee7513/32.png) [@Bhargavi1144](https://support.prodi.gy/u/Bhargavi1144)\
**Post date:** [January 30, 2024, 12:31pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/5 "2024-01-30T12:31:18Z")

</div>

[paths]  
train = null  
dev = null  
vectors = null  
init\_tok2vec = null

[system]  
gpu\_allocator = null  
seed = 0

[nlp]  
lang = "en"  
pipeline = ["tok2vec","ner"]  
disabled =   
before\_creation = null  
after\_creation = null  
after\_pipeline\_creation = null  
batch\_size = 1000  
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}  
vectors = {"@vectors":"spacy.Vectors.v1"}

[components]

[components.ner]  
factory = "ner"  
incorrect\_spans\_key = "incorrect\_spans"  
moves = null  
scorer = {"@scorers":"spacy.ner\_scorer.v1"}  
update\_with\_oracle\_cut\_size = 100

[components.ner.model]  
architectures = "spacy.TransitionBasedParser.v2"  
state\_type = "ner"  
extra\_state\_tokens = false  
hidden\_width = 64  
maxout\_pieces = 2  
use\_upper = true  
nO = null

[components.ner.model.tok2vec]  
architectures = "spacy.Tok2VecListener.v1"  
width = 96  
upstream = "\*"

[components.tok2vec]  
source = "assets/base\_model\_new"

[corpora]  
readers = "prodigy.MergedCorpus.v1"  
eval\_split = 0.2  
sample\_size = 1.0

[corpora.ner]  
readers = "prodigy.NERCorpus.v1"  
datasets = ["asia\_steel\_rebar\_volume"]  
eval\_datasets =   
default\_fill = "outside"  
incorrect\_key = "incorrect\_spans"

[training]  
seed = ${system.seed}  
gpu\_allocator = ${system.gpu\_allocator}  
dropout = 0.1  
accumulate\_gradient = 1  
patience = 1600  
max\_epochs = 0  
max\_steps = 20000  
eval\_frequency = 200  
frozen\_components =   
annotating\_components =   
dev\_corpus = "corpora.dev"  
train\_corpus = "corpora.train"  
before\_to\_disk = null  
before\_update = null

[training.batcher]  
batchers = "spacy.batch\_by\_words.v1"  
discard\_oversize = false  
tolerance = 0.2  
get\_length = null

[training.batcher.size]  
schedules = "compounding.v1"  
start = 100  
stop = 1000  
compound = 1.001  
t = 0.0

[training.optimizer]  
optimizers = "Adam.v1"  
beta1 = 0.9  
beta2 = 0.999  
L2\_is\_weight\_decay = true  
L2 = 0.01  
grad\_clip = 1.0  
use\_averages = false  
eps = 0.00000001  
learn\_rate = 0.001

[training.score\_weights]

[pretraining]

[initialize]  
vectors = ${paths.vectors}  
init\_tok2vec = ${paths.init\_tok2vec}  
vocab\_data = null  
lookups = null  
after\_init = null

[initialize.components]

[initialize.before\_init]  
callbacks = "spacy.copy\_from\_base\_model.v1"  
tokenizer = "assets/base\_model\_new"  
vocab = " **assets/base\_model\_new**"

[initialize.tokenizer]

+++++++++++++++++++++++++++++++++

The above one is the config data I have used.

**assets/base\_model\_new --\>** This is the model I have generated using custom tokenization and used the same while generating .spacy files as well.

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [February 1, 2024, 5:05pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/6 "2024-02-01T17:05:55Z")

</div>

Hi @Bhargavi1144,

There seem to be differences in how `"spacy.copy_from_base_model.v1"` works in spaCy v3.6 and v3.7, let me check with the spaCy team and get back to you.

Looking at the config, you are on spaCy 3.7, right?

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [February 5, 2024, 3:39pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/7 "2024-02-05T15:39:54Z")

</div>

Hi @Bhargavi1144 ,

While we are looking into the `copy_from_base_model`, perhaps you could try downgrading spaCy to 3.6 as a workaround. The sourcing should work with this version.

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [February 14, 2024, 3:18pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/8 "2024-02-14T15:18:21Z")

</div>

Hello again @Bhargavi1144 ,

I've looked into the issue with the help of the spaCy team and my first intuition was not confirmed. The sourcing works correctly in 3.7.X.  
I also tried your config substituting the base model and the dataset and it runs without problems.  
The issue is that your `tok2vec` is not intitialized. Looking back at your original snippet:

> tok2vec = nlp.add\_pipe("tok2vec")  
> nlp.to\_disk("assets/base\_model\_new")

you can see that your are actually adding tok2vec and directly save it, without intializing it. And then in the train cfg you source from this model:

> [components.tok2vec]  
> source = "assets/base\_model\_new"

This errors out with this, admittedly, uninformative error because tok2vec is, indeed, not initialized.  
To use the blank tok2vec you should pass a factory instead so:

```python
[components.tok2vec]
factory = "tok2vec"

```

Could you try training w/ the tok2vec factory instead of sourced tok2vec?  
And for the logging message about copying of the tokenizer to appear you should run `spacy train` with `--verbose` flag - sorry I missed that in the original post!

---

<div class="post-metadata">

**Author:** ![Bhargavi1144](https://avatars.discourse-cdn.com/v4/letter/b/ee7513/32.png) [@Bhargavi1144](https://support.prodi.gy/u/Bhargavi1144)\
**Post date:** [February 19, 2024, 12:38pm UTC](https://support.prodi.gy/t/having-issue-with-spacy-train-with-custom-tokenizer-component-which-we-used-to-annotate-in-prodigy/7054/9 "2024-02-19T12:38:38Z")

</div>

> [@Bhargavi1144](#):
>
> source

Hey,

Thanks for checking the issue!

Instead of **copy\_from\_base\_model** approach I did try using below approach.

**Mentioned below property in config.cfg:**  
[initialize.before\_init]  
@callbacks = "customize\_tokenizer"

**Train command I have used while training:**  
python -m spacy train assets/config.cfg --paths.train train.spacy --paths.dev dev.spacy --output model **--code assets/modify\_tokenizer.py**

This resolved the issue and model got generated based on the custom tokenization.

But I will try to check **copy\_from\_base\_model** approach as well with your suggestion as well.

Thanks alot for checking the issues quickly 🙂