Hi many thanks for your quick answer. It was my mistake, now I get exact match with your spacy internal numbers, both precision and recall and f1. I changed my code to
def confusion_matrix(your_evaluation_data=None, ner_model = None):
#
tp,fp,fn,tn = 0,0,0,0
#
data_tuples = [(eg.text, eg) for eg in your_evaluation_data]
# see https://spacy.io/api/language#pipe
for doc, example in ner_model.pipe(data_tuples, as_tuples=True):
# correct_ents
ents_x2y = example.get_aligned_spans_x2y(example.reference.ents)
correct_ents = [(e.start_char, e.end_char, e.label_) for e in ents_x2y]
# predicted_ents
ents_x2y = example.get_aligned_spans_x2y(doc.ents)
predicted_ents = [(e.start_char, e.end_char, e.label_) for e in ents_x2y]
#
for ent in predicted_ents:
if ent not in correct_ents:
print("False positive:", ent)
for ent in correct_ents:
if ent not in predicted_ents:
print("False negative:", ent)
# false positives
fp += len([ent for ent in predicted_ents if ent not in correct_ents])
# true positives
tp += len([ent for ent in predicted_ents if ent in correct_ents])
# false negatives
fn += len([ent for ent in correct_ents if ent not in predicted_ents])
# true negatives
tn += len([ent for ent in correct_ents if ent in predicted_ents])
return tp,fp,fn,tn
However, it would still be nice to get tp,fp,fn,tn out via spacy, maybe one day you could add that feature. Precision and recall are good to have, but it is even better to get raw tp,fp,fn,tn number for detailed debugging. Thanks and well done on doing spacy, amazing great package!!!!