# prodigy train using pretrained model **URL:** \ **Category:** Uncategorized\ **Tags:** ner, spacy, solved\ **Created:** [June 1, 2020, 9:56pm UTC](https://support.prodi.gy/t/prodigy-train-using-pretrained-model/2965 "2020-06-01T21:56:54Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![yw2903](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/yw2903/32/1614_2.png) [@yw2903](https://support.prodi.gy/u/yw2903)\ **Post date:** [June 1, 2020, 9:56pm UTC](https://support.prodi.gy/t/prodigy-train-using-pretrained-model/2965/1 "2020-06-01T21:56:54Z")
Hello, I use the spacy pretrain on raw text to get weights model999.bin. However I got the following dimension error when running prodigy train. ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/b/b9d4aaeef72e238e6a2a5ce610a9a062c5eac6b8.png) ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/6/6759255024569f3957a1a755287ca91dfa909cfd.png) 1. Can I train on en\_core\_web\_lg when the weights are trained on en\_vectors\_web\_lg? 2. Where is the shape 480 come from when there are 769 examples in total? 3. Should I pretrain the raw text on the 616 training set or 769 in total? 4. Does the pretrain always stop at 999 if not specified? Thanks. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [June 2, 2020, 1:52pm UTC](https://support.prodi.gy/t/prodigy-train-using-pretrained-model/2965/2 "2020-06-02T13:52:39Z")
> [@yw2903](#): > > Can I train on en\_core\_web\_lg when the weights are trained on en\_vectors\_web\_lg? No, that's likely the problem here. If you've trained the tok2vec weights using the `en_vectors_web_lg` package (by predicting the vector of the next word), you also need to use those vectors during training. > [@yw2903](#): > > Where is the shape 480 come from when there are 769 examples in total? Those are the shapes of the vectors (vectors model vs. tok2vec weights). > [@yw2903](#): > > Should I pretrain the raw text on the 616 training set or 769 in total? Ideally, you should want to pretrain on a very _large sample_ of raw text. Like, billions of words (Reddit, CommonCrawl, Wikipedia etc.). Just training on a few hundred of your examples likely won't be very effective. > [@yw2903](#): > > Does the pretrain always stop at 999 if not specified? The default number of iterations (configurable with `--n-iter`) is 1000. If you're training on a large sample of raw text, you typically wouldn't train for that many iterations – if your corpus is large enough, you can even just stream it in and train on it until the model stops improving. There's not necessarily an advantage in looping over the same data multiple times. ---
**Author:** ![yw2903](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/yw2903/32/1614_2.png) [@yw2903](https://support.prodi.gy/u/yw2903)\ **Post date:** [June 2, 2020, 3:07pm UTC](https://support.prodi.gy/t/prodigy-train-using-pretrained-model/2965/3 "2020-06-02T15:07:56Z")
Thanks Ines. Here is the error message I got by changing the base model to `en_vectors_web_lg` ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/d/db0cf4f1d538eb2a92e5835d4892a293542c2e4b.png) Another thing that confuses me is why is `token_vector_width` 96 and `pretrained_dims` 300? I did not set it up. ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [June 3, 2020, 11:58am UTC](https://support.prodi.gy/t/prodigy-train-using-pretrained-model/2965/4 "2020-06-03T11:58:33Z")
> [@yw2903](#): > > Thanks Ines. Here is the error message I got by changing the base model to `en_vectors_web_lg` I think the problem here is that you didn't pretrain with the setting to use the word vectors as features. If you didn't add the `--use-vectors` flag, the model won't expect to have the word vectors during training. So for the model you've pretrained, try setting `blank:en` as the model rather than `en_vectors_web_lg`. > [@yw2903](#): > > Another thing that confuses me is why is `token_vector_width` 96 and `pretrained_dims` 300? I did not set it up. These are different dimensions in the model. Unfortunately there's no way to talk about this especially clearly, because the neural network model has many layers of activations on the way to finally assigning a vector to each token. There isn't really an elegant way to distinguish those activations from each other terminologically. So, the word vectors you load in, the big static table, are just one feature used to compute the activation for each word individually. The other features are also vectors, representing the word's lower-cased form, prefix, suffix and shape. All of those vectors are mixed together in a feed-forward layer to produce another vector, and then a CNN is run to mix in information from the surrounding context, outputting the thing we call "token vectors". So: - The `pretrained_dims` is the width of the big static table, produced by an algorithm like word2vec or GloVe. It was named that before the `spacy pretrain` command was around. - The `token_vector_width` is the width of the vectors output by the CNN. Again, I regret how confusing this all is, but it's a dilemma that other deep learning models face as well.