# Add on new name entity incrementally...
**URL:** https://support.prodi.gy/t/add-on-new-name-entity-incrementally/2073
**Category:** Uncategorized
**Tags:** usage, ner
**Created:** [October 3, 2019, 5:29pm UTC](https://support.prodi.gy/t/add-on-new-name-entity-incrementally/2073 "2019-10-03T17:29:59Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@tomlin](https://support.prodi.gy/u/tomlin)
#### Post date: [October 3, 2019, 5:30pm UTC](https://support.prodi.gy/t/add-on-new-name-entity-incrementally/2073/1 "2019-10-03T17:30:00Z")
Hello,
In the great demo video of how to train a new name entity on reddit comment [[https://prodi.gy/docs/video-new-entity-type](https://prodi.gy/docs/video-new-entity-type)], I am curious if at the step of ner.batch-trian, the training process automatically samples and mixes up the annotations from existing entity with the new entity annotations examples (in both train/eval set), so that it doesn't forget the old entity types. In addition, I am assuming that the new model also preserves the old entity, instead of only recognizing the new entity type, correct?
In my case, I will need to add on multiple name entities. Thus, can I train each new entity on previous temporarily saved model iteratively, if binary annotation is the best practice to go for sample collection. Furthermore, how can I reuse the already annotated text, for there are multiple entities in each text? Should I feed in the original text again, or load the saved annotation file for use? Would the annotations present themselves in a list of tuples for multiple entities.
In regard to very specific entity types in language such as Chinese, is it probably better to train from a blank model as I understand? Does it suffice to recognize name entity if the blank model can only segment and tokenize words, meaning that it's not required to have a model trained with pos, dependency parser or word vector?
In aspect of specifying patterns, I am not sure if there are cases for matcher in Chinese, or I should just simply use the pattern of [{}, {'orth': WORD}] to identify possible entity type.
Finally, I notice the annotation content and format from prodigy is quite different from the annotation output I used to know in spacy, which has a pattern like {entity: [(start\_chr, end\_chr, label)]}. I am not sure if I'd like to reuse the annotation samples in spacy model updating, or even export it in IOB/BIOES format for used in other scenario, would that be possible?
Thanks for any clarifications.
---
### Author:  [@ines](https://support.prodi.gy/u/ines)
#### Post date: [October 4, 2019, 2:16pm UTC](https://support.prodi.gy/t/add-on-new-name-entity-incrementally/2073/2 "2019-10-04T14:16:47Z")
Hi! In case you haven't seen it, you might find the NER flowchart useful that describes different scenarios and some general purpose tips and tricks:
> [@Annotation Flowchart: Named Entity Recognition](https://support.prodi.gy/t/annotation-flowchart-named-entity-recognition/1556):
>
> Just shared this on Twitter – if you find flowcharts like this useful, I'm happy to make more smiley Updated on October 18, 2022: New versions of the NER flowcharts! See our [Twitter thread](https://twitter.com/explosion_ai/status/1582389799646023680?s=20&t=JoiOj44gnIExW9DffomOig) for more details.
> [@tomlin](#):
>
> so that it doesn't forget the old entity types. In addition, I am assuming that the new model also preserves the old entity, instead of only recognizing the new entity type, correct?
Thats something that _you_ have to include in your training data if this is how you want the model to behave. The `ner.make-gold` workflow could help you with that: it'll highlight all entities that the model already predicts and lets you correct the predictions and add more labels if needed.
If your new labels potentially overlap with existing types, you probably want to start training from scratch. Otherwise, you're constantly "fighting" the existing predictions and you'll need much more data to teach the model to suddenly predict a label very differently. You can still use a workflow like `ner.make-gold` to create the data semi-automatically, though.
> [@tomlin](#):
>
> In my case, I will need to add on multiple name entities. Thus, can I train each new entity on previous temporarily saved model iteratively, if binary annotation is the best practice to go for sample collection.
While you can use the previously trained model in the loop, it's not really the best stategy if you want to train your final model. It's usually better and cleaner to use the same base model and train it on _all_ annotations.
> [@tomlin](#):
>
> In regard to very specific entity types in language such as Chinese, is it probably better to train from a blank model as I understand? Does it suffice to recognize name entity if the blank model can only segment and tokenize words, meaning that it's not required to have a model trained with pos, dependency parser or word vector?
We don't distribute pretrained spaCy NER models for Chinese, so unless you already have your own pretrained model, you probably need to start from scratch anyways. The model components are completely independent – so in order to train an entity recognizer, you won't need a tagger or parser.
However, having word vectors can improve the training accuracy, because if word vectors are available, those will be used as features in the model and let you provide more information. How well this works in Chinese depends on the tokenization and you typically want to make sure that the vectors you use include embeddings for the same words that the tokenizer produces (and not just invidividual characters).
> [@tomlin](#):
>
> the annotation content and format from prodigy is quite different from the annotation output I used to know in spacy, which has a pattern like {entity: [(start\_chr, end\_chr, label)]}.
It's not _that_ different? Prodigy just uses named keys for the `start`, `end` and `label` – so instead of `(start, end, label)`, an entity is described as `{"start": start, "end": end, "label": label}`.
> [@tomlin](#):
>
> I am not sure if I'd like to reuse the annotation samples in spacy model updating, or even export it in IOB/BIOES format for used in other scenario, would that be possible?
Sure, you can always export the data and use it in spaCy. The values are the same – and if you ever need BILUO tags etc., you can use spaCy's [conversion utilities](https://spacy.io/api/goldparse#biluo_tags_from_offsets).
---
### Author:  [@tomlin](https://support.prodi.gy/u/tomlin)
#### Post date: [October 7, 2019, 7:45am UTC](https://support.prodi.gy/t/add-on-new-name-entity-incrementally/2073/3 "2019-10-07T07:45:23Z")
Hello Ines,
Thanks for your prompt response, and pardon me for a bit more for further clarification.
The NER flowchart absolutely helps massively in a way the we get to have wholistic view of how to practice our NER training. I'd really be appreciated if more of alike are coming.
In the case of `ner.teach`, if we include pre-trained model, say, en\_core\_web\_lg and run the script
```python
ner.teach drug_ner en_core_web_lg --label DRUG, LOCATION --patterns
```
Does it tell the model to identify Drug in the training text based on the patterns we provide, and simultaneously, it also identifies LOCATION for us based on what the pre-trained model has leant.
So that if we'd like to add in a new entity while keep the existing, we would have some dataset that include all entities labeled for us to use in `ner.batch-training`.
Secondly, for the use of `ner.silver-to-gold` and `ner.make-gold`, I assume that these are best applied after we acquire the semi-auto annotated dataset from model suggestions and refine them with manual inspection. So that it's just more efficient than having `ner.manual` from scratch.
Thirdly, I am not use if at the moment, does spacy's matcher is applicable on Chinese words, or normally, I should just stick to `orth` for patter recognition.
In aspect of training our custom word vector vis gensim and applying it in spacy.
From these great threads:
>
I guess if we import our trained vectors via
```python
python -m spacy init-model zh --vectors-loc xxx.word2vec.txt.gz
```
wihtout giving the `--jsonl-loc `. Spacy would still update the vectors of these tokens if they exist in the base model? However, if we'd like to provide the vocabulary\_file, should we fill in all attributes in the entry structure [[https://spacy.io/api/annotation#vocab-jsonl](https://spacy.io/api/annotation#vocab-jsonl)], or we can leave some unfilled.
Lastly, I have read through a couple of threads on how to better design NER. For instance, the case you brought up in Datacamp lesson, saying that `CLOTHES` is better than asking the model to identify `SHIRTS` and `PANTS`. But I am not very clear why it would make the model harder to learn? Is it because these shirts and pants entities appear and surround in similar tokens that it's just harder for model to distinguish the two based on the context. But then what about `PERSON` and `LOCATION`, these two entity types are proper noun, I assume these words will appear in quite similar sentence syntax structure. Why it's applicable in this case?
Sorry again for bombing you all the questions. And thanks again for your contribution in NLP field so much.