show false negative/false positives in NER

Hey :slight_smile:

I am trying to find out which of the entities annotated for NER are either skipped (false negatives) or which pieces of text the model is incorrectly picking up as entities (false positives). Is there an easy way to do this via the Prodigy/Spacy API?

I hacked my way through the code a bit but couldn't find anything. The closest I could get from the train recipe was the scores object, but that only contained the scoring. It would be really nice to store the predictions. Then we could compute other metrics / plots (conf. matrix, etc.)

Hi! There's no built-in function for this at the moment, but it should be pretty straightforward to implement. You probably want to do this as a separate step, though, after you've trained the model – and you probably also want to use a separate dedicated evaluation set instead of just doing a random split so you can compare the results more reliably (if you're not doing that already).

To get the false positives/negatives, you can then process your evaluation data with your trained model and compare the "spans" against the predicted doc.ents:

data_tuples = ((eg["text"], eg) for eg in your_evaluation_data)
nlp = spacy.load("./your_trained_model")
for doc, eg in nlp.pipe(data_tuples, as_tuples=True):
    correct_ents = [(e["start"], e["end"], e["label"]) for e in eg["spans"])
    predicted_ents = [(e.start_char, e.end_char. e.label_) for e in doc.ents]
    for ent in predicted_ents:
        if ent not in correct_ents:
            print("False positive:", ent)
    for ent in correct_ents:
        if ent not in predicted_ents:
            print("False negative:", ent)
1 Like