# questions on Multi NERs Annotation & Training at Once in a Sentence **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, spacy\ **Created:** [September 22, 2022, 5:58pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960 "2022-09-22T17:58:19Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [September 26, 2022, 7:31pm UTC](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960/2 "2022-09-26T19:31:41Z")
hi @ruiye! Thanks for your message! > [@ruiyeNLP](#): > > Since I probably will turn this project into a publication, I need to know what exactly the training work. > Looking forward to discussing this with you. This is excellent. We're happy to help coach you along the way. I'll provide a lot of detailed links that can point you in the right direction. Some you may already know, but hopefully it can help other community members for context. Also, keep searching in the wealth of information in our [spaCy documentation](https://spacy.io/usage/spacy-101), [Prodigy documentation](https://prodi.gy/docs), and in this community and [spaCy GitHub community](https://github.com/explosion/spaCy/discussions). > [@ruiyeNLP](#): > > my task annotates **4 NERs** at once in **a sentence** as shown **below**. After annotation, the 4 NERs were **trained** with **one command** `prodigy train`. Can I rephrase this to align with Prodigy's terminology? I would suggest your goal is to **train one NER model** with **four entity types** : `RIGHTV`, `RIGHTN`, `ACCESSV`, `ACCESSN`, not " **4 NERs**". I would recommend looking over [Prodigy's glossary of terms](https://prodi.gy/docs#glossary). > [@ruiyeNLP](#): > > The annotation commond: `prodigy ner.manual data_5_1.0 en_core_web_lg ./input/data_5_ground_truth_1.0.jsonl --label RIGHTV,RIGHTN,ACCESSV,ACCESSN` `ner.manual` recipe produces **manual annotations** that **highlight the full entity span** , as opposed to using **binary** recipes that create the annotations as Yes/No. Binary recipes like `ner.teach` or `ner.correct` are usually when you already have a model that you want to improve incrementally. This is annotating with a **model-in-the-loop** as the model is affecting the order of annotation or at least being used to predict and the annotator corrects it. > [@ruiyeNLP](#): > > 1 How `prodigy train` actually **trains** the 4 different NERs **simultaneously**? `prodigy train` is simply a wrapper for [`spacy train`](https://spacy.io/usage/training). `spacy train` is defined by its [config file](https://spacy.io/usage/training#config). To make things easier, `prodigy train` will create a default config file, which is identical to [`spacy init config`](https://spacy.io/api/cli#init-config). > [@ruiyeNLP](#): > > The training commond: `prodigy train --ner data_5_1.0 ./tmp_model --eval-split 0.2 --config config.cfg` I see you used your own `config.cfg` file. This is excellent. If you're interested in how training is done, see [spaCy training docs](https://spacy.io/usage/training) or look over the [config docs](https://spacy.io/usage/training#config). Also, you may find the [NER Prodigy docs page's section on training strategies helpful](https://prodi.gy/docs/named-entity-recognition#training). Here's an excerpt: #### Should I start with a blank model or update an existing one?[¶](https://prodi.gy/docs/named-entity-recognition#tip-blank-vs-update) spaCy’s NER architecture was designed to support continuous updates with more examples and even adding new labels to existing trained models. **Updating an existing model** makes sense if you want to keep using the same label scheme and only want to fine-tune it on more data or more specific data. However, it can easily lead to inconsistent behavior if you’re adding new entity types and/or annotations that conflict with the data the model was trained on. For instance, if you suddenly want to predict all cities as `CITY` instead of `GPE`. Instead of trying to “fight” the existing weights trained on millions of words, it often makes more sense to **train a new model from scratch**. Even if you’re training from scratch, you can still use a trained model to help you create training data more efficiently. Prodigy’s [` ner.correct`](https://prodi.gy/docs/recipes#ner-correct) will stream in the model’s predictions for the given labels and lets you manually correct the entity spans. This way, you can let a model label the entity types you want to keep, add your new types on top, and make corrections along the way. This is a very effective method for bootstrapping large gold-standard training corpora without having to do all the labelling from scratch. Here's another discussion on the differences. > [@Prodigy ner.batch-train vs Spacy train](https://support.prodi.gy/t/prodigy-ner-batch-train-vs-spacy-train/1665/2): > > Hi – this is a totally valid question slightly_smiling_face Since Prodigy focuses a lot on usage as a developer tool, the built-in batch-train commands were also designed with the development aspect in mind. They’re optimised to train from Prodigy-style annotations and smaller datasets, include more complex logic to handle evaluation sets and output more detailed training statistics. Prodigy’s ner.batch-train workflow also supports training from “incomplete” annotations out-of-the-box, e.g. … Note this was in 2019 when `prodigy train` was called `prodigy batch-train` for `ner`. Also this post below details some differences between Prodigy and SpaCy. Just note that spaCy 3.0 came out since and has changed a lot. If you want a strong, reproducible project, I would encourage learning [spaCy projects](https://spacy.io/usage/projects). You can find a great template that integrates with Prodigy as part of the spacy projects repo > [@Prodigy to Spacy Guide](https://support.prodi.gy/t/prodigy-to-spacy-guide/857): > > Hi, I admit that I’m entirely new to this world of NLP, NER, spaCy and Prodigy. Earlier it was very overwhelming while I wanted to learn all this stuff. But ever since I discovered Prodigy, I immediately bought the license as I it feels super welcoming to newcomers like me. I have a use-case, where I have to assign labels to either a group or sometime singular english words into something machine readable. For example if the raw text is “My business phone number is 123-234-2323” I want the mod… > [@ruiyeNLP](#): > > 2 Will the **train-at-once** **influence** the final result? for bad or for good? Ideally, better. See this post: > [@Training Multiple entities at the Same time?](https://support.prodi.gy/t/training-multiple-entities-at-the-same-time/867/2): > > Sure! During annotation, you can always speficy one or more labels via the --label argument – for example, --label PERSON,ORG. We usually recommend focusing on a smaller label set per session and running smaller experiments during development – but once you’re ready to train your final model for production, you should ideally merge your data and include all the labels you’ve annotated, so the model can learn them all at the same time. > [@ruiyeNLP](#): > > 3 How to check the **evaluation** for each NER? See this post: > [@Prodigy NER model evaluation and custom evaluation scripts](https://support.prodi.gy/t/prodigy-ner-model-evaluation-and-custom-evaluation-scripts/653/2): > > It depends on whether you’re evaluating based on binary annotations, or based on the fully-specified manual annotations. If you’re evaluating the binary annotations, the accuracy score is based on how many of the accepted entities you got right, and how many predicted entities are inconsistent with the annotations (either because they cross a correct entity, or because they match a rejected entity). There will also be some predicted entities that can’t be evaluated. If you’re evaluating the ma… **Very important** : make sure to create a dedicated hold-out (evaluation) dataset early on if you're experimenting. It's easy to use the `--eval-split` but that will mean your evaluation dataset will change in each run. Without a very large dataset, your model's evaluation may change wildly because of different evaluation sets. This will confuse your results. If you do this, you can specify your evaluation dataset with the `eval:` prefix in `prodigy train` like: ```python prodigy train --ner train_data,eval:eval_data ... ``` Also, this post explains more: > [@ner.train number of examples](https://support.prodi.gy/t/ner-train-number-of-examples/641/4): > > When looking at the examples, it's important to distinguish between the input hash and the task hash. The input hash is based on only the input data, e.g. the text. So you can easily have lots of annotations on the same input, but with different spans. Prodigy won't replace examples in the dataset – your dataset should always be an exact record of each individual annotation decision. But when you run ner.batch-train, Prodigy will merge all spans on the same input and use the annotation decision… > [@ruiyeNLP](#): > > 4 Do you recommend doing so or training each NER **separately**? See this post: > [@Most efficient way to train/label having separate manual and binary data](https://support.prodi.gy/t/most-efficient-way-to-train-label-having-separate-manual-and-binary-data/4493/2): > > Hi! These are all good questions and it's always good to consider these questions explicitly for each project slightly_smiling_face The annotations don't necessarily have to be on the same sentences, although it's usually good to have at least some overlap. Otherwise, you can more easily end up with imbalanced data, and you'll also never have examples of texts with multiple different entities, which could mean that there's less useful information for the model to learn from, and more unknown… But you may want to check out the NER workflow (fyi we're planning to update this very soon with improved names!): [https://prodi.gy/36f76cffd9cb4ef653a21ee78659d366/prodigy\_flowchart\_ner.pdf](https://prodi.gy/36f76cffd9cb4ef653a21ee78659d366/prodigy_flowchart_ner.pdf) > [@ruiyeNLP](#): > > If separately, which command to use with the existing annotated dataset (i don't want to annotate one more time ☹ )? Separately isn't needed. However, here's a little background if you want to exclude some examples from annotation. Be sure to use the `--exclude` argument where you can pass examples to exclude examples from a stream that you don't want to annotate. Typically, Prodigy will [default in its configuration](https://prodi.gy/docs/install#config) exclude by [`task_hash`](https://prodi.gy/docs/api-loaders#hashing), which is automatically done. This creates a unique code (hash) that identifies every record by its input text + annotation task (e.g., `ner.manual`). You can change this to exclude by `input_hash` by changing `exclude_by` on your prodigy.json (config file). There are [50+ Prodigy Support issues](https://support.prodi.gy/search?q=exclude) that tack the problem of `exclude`. See them for examples of workflows and other questions. Last, an incredibly [powerful design philosophy of Prodigy](https://explosion.ai/blog/prodigy-annotation-tool-active-learning) is that: no one knows what is the **right way** to build your model. Instead, Prodigy is designed to allow you to **rapidly experiment** and **iterate** for your unique problem. In the [Named Entity Documentation page](https://prodi.gy/docs/named-entity-recognition), there's a great section on [How to Choose the right recipe and workflow](https://prodi.gy/docs/named-entity-recognition#workflow). --- _[View the full topic](https://support.prodi.gy/t/questions-on-multi-ners-annotation-training-at-once-in-a-sentence/5960)._