I'm not sure whether spacy questions are allowed in this forum. One the one hand there is a separate discussion section on github for that but then again there is a spacy tag?
If this isn't allowed, please forgive me and just remove this thread
I'm a little confused how exactly vectors and tok2vec are working in spacy esp. during training.
In the config I see that there are vectors being added from a pretrained model e.g. en_core_news_lg, but also an option to initialize tok2vec seperately.
Are the vectors from the pretrained model an input to the tok2vec component?
Another thing that's confusing me is that when I initialize tok2vec with weights from pretraining it doesn't seem to have any effect. I get the exact same outputs during training, as if I hadn't. Now I only ran pretraining on a small amount of data for 18 epochs, so I don't expect to improve anything drastically, but it should have some impact on the loss, right?
Hi! For questions like this, the spaCy discussion board is usually a better fit: Discussions · explosion/spaCy · GitHub We try to keep this forum very focused on Prodigy – but of course, questions sometimes cross over to spaCy or spaCy-related features in Prodigy, which is what the spacy tag is typically for.
The pretraining question is a bit difficult to answer from afar – if you've only ran it on a very small sample and not for very long, it could definitely be possible that they just have no impact at all. You can also try enabling verbose logging to and check the logs to make sure that the tok2vec layer is initialized correctly with your pretrained weights.
