Using spancat for ner confidence scores

[ryanwesslen]

hi @efratmentel!

Thanks for your questions and welcome to the Prodigy community!

@efratmentel

I am working on training a NER, and am interested in getting confidence scores for the predicted entities.
From what I understand, this is only possible by training a spancat component instead of a NER component.

This is correct. See this spaCy discussions post:

How to get confidence score of custom trained NER model using Spacy? ·...

You can use NER components as well. Take a look at this documentation on how to create a custom recipe that use predicted scores to modify annotation order.

@efratmentel

Can I use my training data I annotated using ner recipes in prodigy in order to train a spancat? Do I need to change it in any way?

Actually, you should be fine. Check out this helpful past post that describes.

transition from ner to spancat

Hi! Training a span categorizer from data annotated with ner.manual should just work out of the box, because the underlying format is the same So instead of passing your dataset in as --ner when you train or export the Prodigy annotations, you can use --spancat instead. Whether or not it improves the results really depends on the data and type of annotated spans. In case you haven't seen it, here's an overview of the advantages of NER vs. span categorization, depending o…

@efratmentel

I read that the default spancat scorer uses the LinearLogistic layer. Since I do not have overlapping entities, how do I change this to softmax?

Yep - that makes sense. For questions like this, the SpaCy discussion forum can be helpful. For example, I found a similar post that suggested the same. As referenced, you could use the softmax from thinc in a custom config. If you have questions, I would suggest replying to that post as the spaCy developer team can help you (this forum is more for Prodigy-specific questions).

Hope this helps and let me know if you have any further questions!