Level of confidence in NER

, ,
1

Is there a way for me to get the probability scores or some other levels of confidence that the model has when it displays a entity it found. We are building custom NER models and would like to display the confidence information in the UI by some color codes once we have the value.

2

The confidence score of the entities should be displayed already, in the lower right. Any information attached in the meta key of the example is displayed there, in the footer of the task card. The score is included by default.

3

To address this part of your question: Prodigy also allows custom colors for labels – see the the bottom of this section for details. The only thing is that the custom colors are matched by the exact label string – so you'd have to create separate labels like PERSON_HIGH, PERSON_LOW etc. For example:

"PERSON_HIGH": "green"

You could then wrap your stream in a function that gets the score and assigns the respective label based on the confidence. For example, something like this:

def add_custom_labels_to_stream(stream):
    for eg in stream:
        for span in spans:  # iterate over the entity spans
            score = span.get('score', 0)  # get the score from the entity span
            if score > 0.5:
                span['label'] = '{}_HIGH'.format(span['label'])
            # etc.
        yield eg
4

Thanks a lot @honnibal & @ines …appreciate the responses. I will try it out.

1 Like