Prediction or probability score for prediction results using ner model developed by ner.teach

[ks1996]

Hey @ines @honnibal.

First of all, thanks for the amazing tool.
I have created a ner model using ner.teach recipe. The next step I do is take this model and apply it over incoming records to predict each record with a label based on the entities present in it along with its score.

The highlighted words are the ones i have tagged as entities. while training the model in prodigy.

I have 2 questions.

1. How to I return a single label for each record and not a label every time a entity appears
2. How do I associate each predicted label with a score or probability which would be another column named score.?

Since ner.teach associates a score while it is being annotated. Is there a way to give a probability for each prediction?

I have just started using spacy. Any help would be much appreciated.

[ines (Ines Montani)]

Hi and thanks!

@ks1996

• How to I return a single label for each record and not a label every time a entity appears
• How do I associate each predicted label with a score or probability which would be another column named score.?

If these are your requirements, are you sure that framing it as a named entity recognition task actually makes sense? It sounds like text classification would be a much better fit? This will give you one or more labels and their scores per label. And if you don't care that much about the exact boundaries, making your model predict them can be counterproductive, because it makes the whole problem harder to learn, for no reason.

[ks1996]

The screenshot contains sample tags that I custom wrote as I cannot share the data. I will be classifying tags which makes no sense to english dictionary like LMT for limit, ANLZ for analyser.
So NER would be better to find entities and we would like to pick on those entities alone too.

That is why I am trying to get a score or a probability of some sort.

Is that possible in the case of NER.

[ines (Ines Montani)]

It's possible, but it's a bit more involved in the current version of spaCy. See these threads for details and background:

Accessing probabilities in NER

I assume that when spaCy runs NER on a document it generates a list of tokens (or token combinations, e.g. ‘United Kingdom’) from that document with a probability score / confidence level that the token is an entity of a given type. I imagine that if the probability score is above a certain threshold then it is marked as an entity of that type and flagged for the user to see, otherwise it remains submerged in a sea of unflagged variables that spaCy helpfully keeps from the user to avoid confusio…

Displaying a confidence score next to a user-defined entity

Hi, I came accross one of your posts for generating scores for the entities using beam. I had a couple of questions regarding this topic: It gives me a “sum of the scores of the parses containing it” based score for named entities defined by out-of-the-box spacy model such as “LOC”, “PER” etc. I build a prodigy model to identity labels such as “LOCATION” which shows a field name in some location. My question is when I run the beam code, what exactly do 103,345,1109 mean? Is it a sum-of-score…

[ks1996]

Thank you. I will look into it