# word embeddings from trained NER model?
**URL:** https://support.prodi.gy/t/word-embeddings-from-trained-ner-model/5075
**Category:** Uncategorized
**Tags:** usage, transformers, ner
**Created:** [December 12, 2021, 6:42am UTC](https://support.prodi.gy/t/word-embeddings-from-trained-ner-model/5075 "2021-12-12T06:42:42Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@yihzhang-suncor](https://support.prodi.gy/u/yihzhang-suncor)
#### Post date: [December 12, 2021, 6:42am UTC](https://support.prodi.gy/t/word-embeddings-from-trained-ner-model/5075/1 "2021-12-12T06:42:42Z")
I've trained an NER model using the below command:
```python
python -m prodigy train ./mymodel --ner my_data --base-model en_core_web_trf
```
The model appears to perform adequately for NER and I was hoping to extract the embeddings for some other usage in semantic similarity. However upon extracting the embeddings and calculating cosine similarities with a template like below I'm finding it performs quite poorly when scored against a word similarity dataset that's based on the same data that we used to train the prodigy NER model:
```python
import spacy
from numpy import dot
from numpy.linalg import norm
spacy_nlp=spacy.load('./mymodel/model-best/',exclude='tagger,parser')
nlp_orig = spacy.load("en_core_web_trf")
spacy_nlp.add_pipe("parser", source=nlp_orig, after="transformer")
spacy_nlp.add_pipe("tagger", source=nlp_orig, after="parser")
text1 = "glove"
text2 = "boots"
text1_embeddings = spacy_nlp(text1)._.trf_data.tensors[1][0]
text2_embeddings = spacy_nlp(text2)._.trf_data.tensors[1][0]
similarity = dot(text1_embeddings , text2_embeddings ) / (norm(text1_embeddings ) * norm(text2_embeddings ))
```
The above template code was ran through our similar-word pair dataset, where we have highly similar and dissimilar word pairs that we've prepared. This dataset worked quite well when we scored it with cosine similarities from a separate fasttext trained word embeddings on the same data that we trained our prodigy NER model with.
Did I miss a step or do something wrong here, or does this mean that a model trained in prodigy for NER cannot be used for its embeddings in semantic similarity work?
Thanks a lot.
---
### Author:  [@adriane](https://support.prodi.gy/u/adriane)
#### Post date: [December 13, 2021, 10:48am UTC](https://support.prodi.gy/t/word-embeddings-from-trained-ner-model/5075/2 "2021-12-13T10:48:03Z")
Although it doesn't matter for this short example, this is the pooled output from the transformer for the first span (=the first 128 spacy tokens) in the text, not the whole text:
> [@yihzhang-suncor](#):
>
> `spacy_nlp(text1)._.trf_data.tensors[1][0]`
I'm not sure how well it works for individual words, but you can get better sentence or text similarity results with approaches like [SentenceTransformers](https://sbert.net) and there's a third-party library that incorporates this into spacy: [spaCy - sentence-transformers ยท spaCy Universe](https://spacy.io/universe/project/spacy-sentence-bert). SentenceTransformers (SBERT) was developed in part because researchers had noticed that averaging the transformer embeddings performs worse than averaging static word2vec/glove vectors for sentence similarity.
If you just need vector similarity for individual words, you can add the vectors from `en_core_web_md` or `en_core_web_lg` to a `trf` model like this:
```python
vectors_nlp = spacy.load("en_core_web_md")
nlp = spacy.load("en_core_web_trf")
nlp.vocab.vectors = vectors_nlp.vocab.vectors
nlp.to_disk("/path/to/en_core_web_trf_with_vectors")
```
And then use this path as the base model instead of `en_core_web_trf` in prodigy.
---
### Author:  [@yihzhang-suncor](https://support.prodi.gy/u/yihzhang-suncor)
#### Post date: [December 13, 2021, 4:37pm UTC](https://support.prodi.gy/t/word-embeddings-from-trained-ner-model/5075/3 "2021-12-13T16:37:45Z")
Thank you, regarding the second approach there, after saving the model to disk and then calling it as the --base-model in prodigy train, would the vectors get updated with the training data?
Initially I wasn't sure if sentence transformers was needed since my word similarity usage is almost exclusively single words, I can give that a try as well.
---
### Author:  [@adriane](https://support.prodi.gy/u/adriane)
#### Post date: [December 13, 2021, 4:50pm UTC](https://support.prodi.gy/t/word-embeddings-from-trained-ner-model/5075/4 "2021-12-13T16:50:56Z")
The static vectors don't get updated during training. If your pipeline just includes components that listen to a `transformer`, then the vectors won't be used at all during training. I think if you're comparing single words, static word vectors sound like the right choice.
If you add a large vector table to the `trf` model it will make the overall pipeline package kind of large (1-2G), but if that doesn't cause problems for you, there's no technical reason not to have both.
(Note that if you're fine-tuning `transformer`+`ner` from `en_core_web_trf` with prodigy, you need to be aware that the tagger and parser performance will be degraded afterwards if you just add them back and they listen to the now-modified `transformer`.)