# Prodigy train fr vectors for Token2Vect

**URL:** <https://support.prodi.gy/t/prodigy-train-fr-vectors-for-token2vect/4695>\
**Category:** Uncategorized\
**Tags:** usage, ner, spacy, solved\
**Created:** [September 17, 2021, 1:30pm UTC](https://support.prodi.gy/t/prodigy-train-fr-vectors-for-token2vect/4695 "2021-09-17T13:30:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jacques](https://avatars.discourse-cdn.com/v4/letter/j/71e660/32.png) [@Jacques](https://support.prodi.gy/u/Jacques)\
**Post date:** [September 17, 2021, 1:30pm UTC](https://support.prodi.gy/t/prodigy-train-fr-vectors-for-token2vect/4695/1 "2021-09-17T13:30:43Z")

</div>

Hi,  
I am replaying the Training a NAMED ENTITY RECOGNITION MODEL with Prodigy and Transfer Learning.  
I am working on french language. I did the ner.manual on my own labels without any issue.  
Right now I want to train the token2vect for ner with my dataset and using the French tok2vec pretain token.  
I am missing information about how to pretrain this token2vec .  
Ines did this pretraining, which took ~8 hours on GPU, and it can be found here: [tok2vec\_cd8\_model289.bin](https://github.com/explosion/projects/releases/download/tok2vec/tok2vec_cd8_model289.bin)  
I would like to learn how to do the same thing for French language.  
Thanks in advance

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [September 20, 2021, 6:27am UTC](https://support.prodi.gy/t/prodigy-train-fr-vectors-for-token2vect/4695/2 "2021-09-20T06:27:45Z")

</div>

Hi! The pretrained tok2vec weights were created using the `spacy pretrain` command with a lot of raw text. You can find the details and documentation here:

- spaCy v2: [https://v2.spacy.io/api/cli/#pretrain](https://v2.spacy.io/api/cli/#pretrain)
- spaCy v3: [https://spacy.io/api/cli/#pretrain](https://spacy.io/api/cli/#pretrain)

The pretraining uses a language modelling objective, similar to how embeddings like BERT are trained. If you have a lot of raw text, this can be a good way to boost your accuracy.

If you're using spaCy v3, you can also initialize your model with existing transformer embeddings, which will have a similar effect. You can use the quickstart widget to generate a transformer-based config for French here: [https://spacy.io/usage/training#quickstart](https://spacy.io/usage/training#quickstart) To export your annotations for use with spaCy, you can use the `data-to-spacy` command. If you're using transformers, you should have a GPU available for training.

---

<div class="post-metadata">

**Author:** ![Jacques](https://avatars.discourse-cdn.com/v4/letter/j/71e660/32.png) [@Jacques](https://support.prodi.gy/u/Jacques)\
**Post date:** [September 21, 2021, 6:59am UTC](https://support.prodi.gy/t/prodigy-train-fr-vectors-for-token2vect/4695/3 "2021-09-21T06:59:06Z")

</div>

Hi Ines,  
Thanks for your replies, I am using Spacy V3 so I will investigate the quickstart you've pointed to me.  
Have a good day and keep going to provide us such great library and useful tools !