transformers based ner evaluation result is 90 but while predicting using nlp pipeline it's showing no output

Hi,
I trained my NER data spacy with roberta-base transformers.
evaluation result is 90 F1 score. But while predicting using normal spacy nlp pipeline no entity output is showing.
My training data is in "word label" format. where "word" get from whitespace tokenizer and label in BLIOU format.
Can anyone point out the problem here.
Thanks in advance.
regards

Hi! This is a bit difficult to debug from afar because there can be many different causes an explanations. Some things to check would be:

  • How is your evaluation done, and is it representative? If your evaluation set is really small, or doesn't contain many entities, you can end up with a score that doesn't actually tell you very much about how useful your model actually is. For instance, if your evaluation data doesn't contain any entities, your model may report an accuracy of 100%, because it has just learned to never predict entities.
  • How do you perform the checks with our nlp object? Are you just testing on some random examples you can think of? This isn't always the best and most representative way to check if things are working as expected. Even if your model is 90% accurate, it'll still make mistakes.
1 Like