# questions on Multi NERs Annotation & Training at Once in a Sentence
**URL:** https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960
**Category:** Uncategorized
**Tags:** usage, spacy, ner
**Created:** [September 22, 2022, 5:58pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960 "2022-09-22T17:58:19Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@ruiyeNLP](https://support.prodi.gy/u/ruiyeNLP)
#### Post date: [September 22, 2022, 5:58pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/1 "2022-09-22T17:58:19Z")
Hi, my task annotates **4 NERs** at once in **a sentence** as shown **below**. After annotation, the 4 NERs were **trained** with **one command** `prodigy train`. Therefore I am wondering:
1 How `prodigy train` actually **trains** the 4 different NERs **simultaneously**?
2 Will the **train-at-once** **influence** the final result? for bad or for good?
3 How to check the **evaluation** for each NER?
4 Do you recommend doing so or training each NER **separately**? If separately, which command to use with the existing annotated dataset (i don't want to annotate one more time ☹ )?
The annotation commond: `prodigy ner.manual data_5_1.0 en_core_web_lg ./input/data_5_ground_truth_1.0.jsonl --label RIGHTV,RIGHTN,ACCESSV,ACCESSN`
The training commond: `prodigy train --ner data_5_1.0 ./tmp_model --eval-split 0.2 --config config.cfg`
The training result is as **follows** :
Since I probably will turn this project into a publication, I need to know what exactly the training work.
Looking forward to discussing this with you.
---
### Author:  [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)
#### Post date: [September 26, 2022, 7:31pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/2 "2022-09-26T19:31:41Z")
hi @ruiye!
Thanks for your message!
> [@ruiyeNLP](#):
>
> Since I probably will turn this project into a publication, I need to know what exactly the training work.
> Looking forward to discussing this with you.
This is excellent. We're happy to help coach you along the way. I'll provide a lot of detailed links that can point you in the right direction. Some you may already know, but hopefully it can help other community members for context. Also, keep searching in the wealth of information in our [spaCy documentation](https://spacy.io/usage/spacy-101), [Prodigy documentation](https://prodi.gy/docs), and in this community and [spaCy GitHub community](https://github.com/explosion/spaCy/discussions).
> [@ruiyeNLP](#):
>
> my task annotates **4 NERs** at once in **a sentence** as shown **below**. After annotation, the 4 NERs were **trained** with **one command** `prodigy train`.
Can I rephrase this to align with Prodigy's terminology?
I would suggest your goal is to **train one NER model** with **four entity types** : `RIGHTV`, `RIGHTN`, `ACCESSV`, `ACCESSN`, not " **4 NERs**". I would recommend looking over [Prodigy's glossary of terms](https://prodi.gy/docs#glossary).
> [@ruiyeNLP](#):
>
> The annotation commond: `prodigy ner.manual data_5_1.0 en_core_web_lg ./input/data_5_ground_truth_1.0.jsonl --label RIGHTV,RIGHTN,ACCESSV,ACCESSN`
`ner.manual` recipe produces **manual annotations** that **highlight the full entity span** , as opposed to using **binary** recipes that create the annotations as Yes/No. Binary recipes like `ner.teach` or `ner.correct` are usually when you already have a model that you want to improve incrementally. This is annotating with a **model-in-the-loop** as the model is affecting the order of annotation or at least being used to predict and the annotator corrects it.
> [@ruiyeNLP](#):
>
> 1 How `prodigy train` actually **trains** the 4 different NERs **simultaneously**?
`prodigy train` is simply a wrapper for [`spacy train`](https://spacy.io/usage/training). `spacy train` is defined by its [config file](https://spacy.io/usage/training#config). To make things easier, `prodigy train` will create a default config file, which is identical to [`spacy init config`](https://spacy.io/api/cli#init-config).
> [@ruiyeNLP](#):
>
> The training commond: `prodigy train --ner data_5_1.0 ./tmp_model --eval-split 0.2 --config config.cfg`
I see you used your own `config.cfg` file. This is excellent. If you're interested in how training is done, see [spaCy training docs](https://spacy.io/usage/training) or look over the [config docs](https://spacy.io/usage/training#config).
Also, you may find the [NER Prodigy docs page's section on training strategies helpful](https://prodi.gy/docs/named-entity-recognition#training). Here's an excerpt:
#### Should I start with a blank model or update an existing one?[¶](https://prodi.gy/docs/named-entity-recognition#tip-blank-vs-update)
spaCy’s NER architecture was designed to support continuous updates with more examples and even adding new labels to existing trained models. **Updating an existing model** makes sense if you want to keep using the same label scheme and only want to fine-tune it on more data or more specific data. However, it can easily lead to inconsistent behavior if you’re adding new entity types and/or annotations that conflict with the data the model was trained on. For instance, if you suddenly want to predict all cities as `CITY` instead of `GPE`. Instead of trying to “fight” the existing weights trained on millions of words, it often makes more sense to **train a new model from scratch**.
Even if you’re training from scratch, you can still use a trained model to help you create training data more efficiently. Prodigy’s [` ner.correct`](https://prodi.gy/docs/recipes#ner-correct) will stream in the model’s predictions for the given labels and lets you manually correct the entity spans. This way, you can let a model label the entity types you want to keep, add your new types on top, and make corrections along the way. This is a very effective method for bootstrapping large gold-standard training corpora without having to do all the labelling from scratch.
Here's another discussion on the differences.
> [@Prodigy ner.batch-train vs Spacy train](https://support.prodi.gy/t/prodigy-ner-batch-train-vs-spacy-train/1665/2):
>
> Hi – this is a totally valid question slightly_smiling_face Since Prodigy focuses a lot on usage as a developer tool, the built-in batch-train commands were also designed with the development aspect in mind. They’re optimised to train from Prodigy-style annotations and smaller datasets, include more complex logic to handle evaluation sets and output more detailed training statistics. Prodigy’s ner.batch-train workflow also supports training from “incomplete” annotations out-of-the-box, e.g. …
Note this was in 2019 when `prodigy train` was called `prodigy batch-train` for `ner`.
Also this post below details some differences between Prodigy and SpaCy. Just note that spaCy 3.0 came out since and has changed a lot. If you want a strong, reproducible project, I would encourage learning [spaCy projects](https://spacy.io/usage/projects). You can find a great template that integrates with Prodigy as part of the spacy projects repo
> [@Prodigy to Spacy Guide](https://support.prodi.gy/t/prodigy-to-spacy-guide/857):
>
> Hi, I admit that I’m entirely new to this world of NLP, NER, spaCy and Prodigy. Earlier it was very overwhelming while I wanted to learn all this stuff. But ever since I discovered Prodigy, I immediately bought the license as I it feels super welcoming to newcomers like me. I have a use-case, where I have to assign labels to either a group or sometime singular english words into something machine readable. For example if the raw text is “My business phone number is 123-234-2323” I want the mod…
> [@ruiyeNLP](#):
>
> 2 Will the **train-at-once** **influence** the final result? for bad or for good?
Ideally, better. See this post:
> [@Training Multiple entities at the Same time?](https://support.prodi.gy/t/training-multiple-entities-at-the-same-time/867/2):
>
> Sure! During annotation, you can always speficy one or more labels via the --label argument – for example, --label PERSON,ORG. We usually recommend focusing on a smaller label set per session and running smaller experiments during development – but once you’re ready to train your final model for production, you should ideally merge your data and include all the labels you’ve annotated, so the model can learn them all at the same time.
> [@ruiyeNLP](#):
>
> 3 How to check the **evaluation** for each NER?
See this post:
> [@Prodigy NER model evaluation and custom evaluation scripts](https://support.prodi.gy/t/prodigy-ner-model-evaluation-and-custom-evaluation-scripts/653/2):
>
> It depends on whether you’re evaluating based on binary annotations, or based on the fully-specified manual annotations. If you’re evaluating the binary annotations, the accuracy score is based on how many of the accepted entities you got right, and how many predicted entities are inconsistent with the annotations (either because they cross a correct entity, or because they match a rejected entity). There will also be some predicted entities that can’t be evaluated. If you’re evaluating the ma…
**Very important** : make sure to create a dedicated hold-out (evaluation) dataset early on if you're experimenting. It's easy to use the `--eval-split` but that will mean your evaluation dataset will change in each run. Without a very large dataset, your model's evaluation may change wildly because of different evaluation sets. This will confuse your results. If you do this, you can specify your evaluation dataset with the `eval:` prefix in `prodigy train` like:
```python
prodigy train --ner train_data,eval:eval_data ...
```
Also, this post explains more:
> [@ner.train number of examples](https://support.prodi.gy/t/ner-train-number-of-examples/641/4):
>
> When looking at the examples, it's important to distinguish between the input hash and the task hash. The input hash is based on only the input data, e.g. the text. So you can easily have lots of annotations on the same input, but with different spans. Prodigy won't replace examples in the dataset – your dataset should always be an exact record of each individual annotation decision. But when you run ner.batch-train, Prodigy will merge all spans on the same input and use the annotation decision…
> [@ruiyeNLP](#):
>
> 4 Do you recommend doing so or training each NER **separately**?
See this post:
> [@Most efficient way to train/label having separate manual and binary data](https://support.prodi.gy/t/most-efficient-way-to-train-label-having-separate-manual-and-binary-data/4493/2):
>
> Hi! These are all good questions and it's always good to consider these questions explicitly for each project slightly_smiling_face The annotations don't necessarily have to be on the same sentences, although it's usually good to have at least some overlap. Otherwise, you can more easily end up with imbalanced data, and you'll also never have examples of texts with multiple different entities, which could mean that there's less useful information for the model to learn from, and more unknown…
But you may want to check out the NER workflow (fyi we're planning to update this very soon with improved names!):
[https://prodi.gy/36f76cffd9cb4ef653a21ee78659d366/prodigy\_flowchart\_ner.pdf](https://prodi.gy/36f76cffd9cb4ef653a21ee78659d366/prodigy_flowchart_ner.pdf)
> [@ruiyeNLP](#):
>
> If separately, which command to use with the existing annotated dataset (i don't want to annotate one more time ☹ )?
Separately isn't needed. However, here's a little background if you want to exclude some examples from annotation.
Be sure to use the `--exclude` argument where you can pass examples to exclude examples from a stream that you don't want to annotate. Typically, Prodigy will [default in its configuration](https://prodi.gy/docs/install#config) exclude by [`task_hash`](https://prodi.gy/docs/api-loaders#hashing), which is automatically done. This creates a unique code (hash) that identifies every record by its input text + annotation task (e.g., `ner.manual`). You can change this to exclude by `input_hash` by changing `exclude_by` on your prodigy.json (config file). There are [50+ Prodigy Support issues](https://support.prodi.gy/search?q=exclude) that tack the problem of `exclude`. See them for examples of workflows and other questions.
Last, an incredibly [powerful design philosophy of Prodigy](https://explosion.ai/blog/prodigy-annotation-tool-active-learning) is that: no one knows what is the **right way** to build your model. Instead, Prodigy is designed to allow you to **rapidly experiment** and **iterate** for your unique problem. In the [Named Entity Documentation page](https://prodi.gy/docs/named-entity-recognition), there's a great section on [How to Choose the right recipe and workflow](https://prodi.gy/docs/named-entity-recognition#workflow).
---
### Author:  [@ruiyeNLP](https://support.prodi.gy/u/ruiyeNLP)
#### Post date: [September 26, 2022, 8:51pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/3 "2022-09-26T20:51:34Z")
Hey, thanks a lot for your detailed reply, the helpful links, and for offering to coach. I would reply **every night** so that you could check my questions and updates on **the second day**. How do you like this style of coaching?
The current training result is as **follows**. It makes sense from my point, since `RIGHTV`,`ACCESSV`,`RIGHTN` are within a dictionary of words, while ACCESSN varies a lot. However, my dataset is around 100 - 150 sentences. How do you think of the NER model trained on **100 sentences**? Do you have any **suggestions** for the training and for a solid project, like data argumentation, or setting a smaller batch size?
---
### Author:  [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)
#### Post date: [September 26, 2022, 9:29pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/4 "2022-09-26T21:29:41Z")
hi @ruiyeNLP!
> [@ruiyeNLP](#):
>
> Hey, thanks a lot for your detailed reply, the helpful links, and for offering to coach. I would reply **every night** so that you could check my questions and updates on **the second day**. How do you like this style of coaching?
I can't guarantee any response turn-around but feel free to keep posting on this chain. We'll answer as we can.
> [@ruiyeNLP](#):
>
> How do you think of the NER model trained on **100 sentences**? Do you have any **suggestions** for the training and for a solid project, like data argumentation, or setting a smaller batch size?
Typically, we'd advocate for more many sentences. For example, our NER workflow recommends to start to with at least 1,000 unlabeled sentences at the very beginning 🙂
While there's no set, we typically recommend models training with about 500 sentences at minimum with evaluation datasets of at least a few hundred. Therefore, if you followed that you wouldn't have enough for training or even data augmentation. Is there anyway to get more sentences?
Hopefully you can as I would be skeptical on how far you can go with only 100 sentences.
If you did want to use data augmentation, [`augmenty`](https://kennethenevoldsen.github.io/augmenty/index.html) could work.
If you can overcome the few sentences, down the road you should consider running [`train-curve`](https://prodi.gy/docs/recipes/#train-curve). This will give you an idea of how your accuracy changes with more annotations. Look for documentation on the recommendations.
> [@what to do if train-curve shows slight decrease in last sample](https://support.prodi.gy/t/what-to-do-if-train-curve-shows-slight-decrease-in-last-sample/4306):
>
> I'm creating a new dataset, and so far I made about 500 annotations. I ran the train-curve command, and the score decreased from 0.3 to 0.29 in the last sample. What should I do at this point to make sure I don't annotate a dataset that won't work? Are there some strategies like going back to make sure that the annotation was more consistent, or troubleshoot and find a root cause if possible? Should I just keep annotating and hope that it improves? Thank you!
---
### Author:  [@ruiyeNLP](https://support.prodi.gy/u/ruiyeNLP)
#### Post date: [October 3, 2022, 8:10pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/5 "2022-10-03T20:10:41Z")
Hi @ryanwesslen , thanks a lot for your detailed reply. Now I have two questions from` prodigy train` to `spacy train`.
1 I used the command to train the model. Do you know what is the same as`--label-stas` in `spacy train`? How to check the **performance for each NER** with `spacy train`?
> prodigy train --ner rights\_training\_1 ./tmp\_model\_all\_1 --eval-split 0.2 --config config.cfg --label-stats
2 In prodigy, i can use `prodigy train-curve`, and it is quite useful. Do you know how can I use `spacy train-curve`? Or how to save the `prodigy train-curve` results as pdf?
Looking forward to your reply.
---
### Author:  [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)
#### Post date: [October 3, 2022, 8:53pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/6 "2022-10-03T20:53:37Z")
hi @ruiyeNLP!
> [@ruiyeNLP](#):
>
> I used the command to train the model. Do you know what is the same as`--label-stas` in `spacy train`? How to check the **performance for each NER** with `spacy train`?
The `spacy evaluate` is [the command](https://spacy.io/api/cli/#evaluate). It can evaluate trained models. Since spaCy is open source, you can see the code for it:
>
What's important is the function `handle_scores_per_type`. This is what is called when using `--label stats`. As you can see, it's called by default for `spacy evaluate`.
>
The one thing you may want to do is create a dedicated hold out (evaluation) dataset. By default, Prodigy can enable the `--eval-split 0.2` which will do the splitting for you. The problem is each run you may get a different set of data. Ideally, you should split the data.
> [@Evaluating a text classification model](https://support.prodi.gy/t/evaluating-a-text-classification-model/2031/2):
>
> How exactly are you checking and calculating this? What do you count as "correct"? And how did you evaluate the model after training – did you use a dedicated evaluation set or just let it hold back some portion of the data? The thing is, when you evaluate a model, the score reflects how accurate the model's predictions are on the given evaluation set. If the evaluation set is good and representative, those results should ideally generalise to other similar datasets. So if you ran the same eva…
If you're going to use `spacy train`, the [`data-to-spacy`](https://prodi.gy/docs/recipes#data-to-spacy) recipe can help. It'll do a partition of your data and then convert it to `.spacy` binary format (which is ideal for using spacy).
> [@ruiyeNLP](#):
>
> Do you know how can I use `spacy train-curve`? Or how to save the `prodigy train-curve` results as pdf?
There isn't a spacy version of `train-curve` unfortunately. You can write your own version. See this for details:
> [@NER train curve with label stats?](https://support.prodi.gy/t/ner-train-curve-with-label-stats/5052/2):
>
> Hi! At the moment, the per-label stats are only available in the regular training, since it'd otherwise get very verbose very quickly. The per-label stats can also be a bit less representative when training with small portions of the data, because you can easily end up with very few instances of a given label. That said, you can take a look at the implementation in recipes/train.py (you can run prodigy stats to find the location of your Prodigy installation) and make a small adjustment to how t…
Also, you can export your `prodigy train-curve results` to a `.txt` file by adding in `"> train_curve.txt"` so:
```python
prodigy train-curve ... > train_curve.txt
```