# Training after annotating with custom tokenizer

**URL:** https://support.prodi.gy/t/training-after-annotating-with-custom-tokenizer/6846
**Category:** Uncategorized
**Tags:** training, spacy, transformers
**Created:** [October 15, 2023, 3:36am UTC](https://support.prodi.gy/t/training-after-annotating-with-custom-tokenizer/6846 "2023-10-15T03:36:22Z")
**Posts on this page:** 1
**Showing post:** 4

<div class="post-metadata">

### Author: ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)
#### Post date: [November 8, 2023, 9:48am UTC](https://support.prodi.gy/t/training-after-annotating-with-custom-tokenizer/6846/4 "2023-11-08T09:48:18Z")

</div>

Hi @AakankshaP ,

That's right, `tok2vec` is not being trained because none of the components down the pipeline use its predictions so there's no backpropagation of the loss. If you followed the steps we discussed earlier, the NER component (as specified in the config you have used) has its own , internal`tok2vec` so it doesn't use the one at the beginning of the pipeline.  
In fact, this first `tok2vec` should be frozen just like the other components of `en_core_web_sm`. It shouldn't change the performance in your example but that would be a more correct way to do it.  
So there's one more modification to the training config that I missed in my original instruction: list the `tok2vec` under `frozen_components`

```python
...
[training]
"frozen_components": ["tok2vec"]
...

```

Just to explain a bit more:  
There are actually two ways to use the `tok2vec` (embedding) layer: you could make the components share the same `tok2vec` layer or be completely independent and have their own, internal `tok2vec` layer (which is the default setup if you generate the base config using `spacy-config` command - as you could observe in your training).  
Each setup has its own advantages and disadvantages and they are very nicely explained in this spaCy doc: [Embeddings, Transformers and Transfer Learning · spaCy Usage Documentation](https://spacy.io/usage/embeddings-transformers#embedding-layers)  
You can also find there information on how to set shared and independent embedding layer in the config.

For `en_core_web_trf` , please follow this script to generate the initial config (which you can then modify with your custom tokenizer): [https://github.com/explosion/projects/blob/e24a085669b4db6918ffeb2752846089d8dee57a/pipelines/ner\_demo\_update/scripts/create\_config.py](https://github.com/explosion/projects/blob/e24a085669b4db6918ffeb2752846089d8dee57a/pipelines/ner_demo_update/scripts/create_config.py)  
This comes from an example project that you can reuse, but there's also a more generic documentation of creating config for transformer training here: [Embeddings, Transformers and Transfer Learning · spaCy Usage Documentation](https://spacy.io/usage/embeddings-transformers#transformers-training)

---

_[View the full topic](https://support.prodi.gy/t/training-after-annotating-with-custom-tokenizer/6846)._