# Using spancat for ner confidence scores **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, spancat\ **Created:** [August 1, 2022, 11:26am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838 "2022-08-01T11:26:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![efratmentel](https://avatars.discourse-cdn.com/v4/letter/e/c6cbf5/32.png) [@efratmentel](https://support.prodi.gy/u/efratmentel)\ **Post date:** [August 1, 2022, 11:26am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/1 "2022-08-01T11:26:28Z")
Hello! I am working on training a NER, and am interested in getting confidence scores for the predicted entities. From what I understand, this is only possible by training a spancat component instead of a NER component. Is this true? Can I use my training data I annotated using ner recipes in prodigy in order to train a spancat? Do I need to change it in any way? Also, I read that the default spancat scorer uses the LinearLogistic layer. Since I do not have overlapping entities, how do I change this to softmax? Thank you! Efrat ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [August 1, 2022, 3:22pm UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/2 "2022-08-01T15:22:31Z")
hi @efratmentel! Thanks for your questions and welcome to the Prodigy community! > [@efratmentel](#): > > I am working on training a NER, and am interested in getting confidence scores for the predicted entities. > From what I understand, this is only possible by training a spancat component instead of a NER component. This is correct. See this spaCy discussions post: > **[How to get confidence score of custom trained NER model using Spacy? ·...](https://github.com/explosion/spaCy/discussions/9189#discussioncomment-1311602)** > > Hi, I have trained NER model and want to get a confidence score for each entity prediction. How that can achieved? Please provide guidelines to achieve it. Thanks You can use `NER` components as well. Take a look at [this documentation on how to create a custom recipe that use predicted scores to modify annotation order](https://prodi.gy/docs/named-entity-recognition#active-learning-custom). > [@efratmentel](#): > > Can I use my training data I annotated using ner recipes in prodigy in order to train a spancat? Do I need to change it in any way? Actually, you should be fine. Check out this helpful past post that describes. > [@transition from ner to spancat](https://support.prodi.gy/t/transition-from-ner-to-spancat/4910/2): > > Hi! Training a span categorizer from data annotated with ner.manual should just work out of the box, because the underlying format is the same slightly_smiling_face So instead of passing your dataset in as --ner when you train or export the Prodigy annotations, you can use --spancat instead. Whether or not it improves the results really depends on the data and type of annotated spans. In case you haven't seen it, here's an overview of the advantages of NER vs. span categorization, depending o… > [@efratmentel](#): > > I read that the default spancat scorer uses the LinearLogistic layer. Since I do not have overlapping entities, how do I change this to softmax? Yep - that makes sense. For questions like this, the SpaCy discussion forum can be helpful. For example, I found a [similar post that suggested the same](https://github.com/explosion/spaCy/discussions/11144). As referenced, you could use the [`softmax`](https://thinc.ai/docs/api-layers#softmax) from `thinc` in a custom config. If you have questions, I would suggest replying to that post as the spaCy developer team can help you (this forum is more for Prodigy-specific questions). Hope this helps and let me know if you have any further questions! ---
**Author:** ![efratmentel](https://avatars.discourse-cdn.com/v4/letter/e/c6cbf5/32.png) [@efratmentel](https://support.prodi.gy/u/efratmentel)\ **Post date:** [August 2, 2022, 8:38am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/3 "2022-08-02T08:38:09Z")
Thank you!! ---
**Author:** ![efratmentel](https://avatars.discourse-cdn.com/v4/letter/e/c6cbf5/32.png) [@efratmentel](https://support.prodi.gy/u/efratmentel)\ **Post date:** [August 4, 2022, 9:30am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/4 "2022-08-04T09:30:19Z")
Hello again, About the "confidence scores" - I am trying to get the predicted scores of the predicted entities so that I will be able to use a custom threshold on the positive predictions of my model (with my trained ner component). Following your link [https://prodi.gy/docs/named-entity-recognition#active-learning-custom](https://prodi.gy/docs/named-entity-recognition#active-learning-custom), I still can't seem to understand how to do this. This is the code I am using: ```python import spacy import copy from prodigy.components.loaders import JSONL nlp = spacy.load('path-to-my-model') def predict(stream): for eg in stream: predictions = nlp(eg["text"]) for score, start, end, label in predictions: example = copy.deepcopy(eg) example["spans"] = [{"start": start, "end": end, "label": label}] print(f'example:{example}, score:{score}') predict(JSONL('path-to-jsonl-with-test-sentences')) ``` Which returns the following error: ```python Traceback (most recent call last): File "path-to-venv-folder/venv/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3398, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in for score, start, end, label in doc: TypeError: cannot unpack non-iterable spacy.tokens.token.Token object ``` What am I doing wrong? Any help would be greatly appreciated. Thank you! ---
**Author:** ![ljvmiranda921](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ljvmiranda921/32/3197_2.png) [@ljvmiranda921](https://support.prodi.gy/u/ljvmiranda921)\ **Post date:** [August 4, 2022, 9:56am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/5 "2022-08-04T09:56:23Z")
Hi @efratmentel , Whenever you iterate on a [`Doc` object](https://spacy.io/api/doc) (i.e., the output of `nlp(eg["text"])`), you're iterating on set of [`Token` objects](https://spacy.io/api/token). What you may want to do is something like: ```python spans_key = "sc" # usually the case unless you set something different for eg in stream: prediction = nlp(eg["text"]) for span in predictions.spans[spans_key]: # do something ``` From the `span` variable, you can access all [Span attributes](https://spacy.io/api/span). However, If you want to access the scores, you might want to do something like: ```python spans_key = "sc" # usually the case unless you set something different scores = prediction.spans[spans_key].attrs["scores"] ``` Without the need for iterating through each Span. ---
**Author:** ![efratmentel](https://avatars.discourse-cdn.com/v4/letter/e/c6cbf5/32.png) [@efratmentel](https://support.prodi.gy/u/efratmentel)\ **Post date:** [August 4, 2022, 12:13pm UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/6 "2022-08-04T12:13:23Z")
Thank you, but unfortunately this still isn't working for me. - I can't seem to find "scores" in the span attributes. - predictions.spans returns an empty dictionary. The recognised entities can be found in prediction.ents, but not the scores. What am I missing here? After loading my trained model (custom ner) I am trying to use it on new sentences, and get a list of the entities it recognises in each sentence, with their predicted scores - the softmax activation output of each entity. Is this possible? ---
**Author:** ![ljvmiranda921](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ljvmiranda921/32/3197_2.png) [@ljvmiranda921](https://support.prodi.gy/u/ljvmiranda921)\ **Post date:** [August 10, 2022, 12:58am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/7 "2022-08-10T00:58:34Z")
Hi @efratmentel , > - predictions.spans returns an empty dictionary. The recognised entities can be found in prediction.ents, but not the scores. This partially explains why there are no spans detected in your `Doc`. To clarify, as of now there's no straightforward way to obtain confidence scores in `ner`. You can only do so via `spancat`. This means you have to train your model using `spancat`, not `ner`. ---
**Author:** ![efratmentel](https://avatars.discourse-cdn.com/v4/letter/e/c6cbf5/32.png) [@efratmentel](https://support.prodi.gy/u/efratmentel)\ **Post date:** [August 10, 2022, 7:02am UTC](https://support.prodi.gy/t/using-spancat-for-ner-confidence-scores/5838/8 "2022-08-10T07:02:02Z")
Okay, thank you very much for your reply!