NER: CLI command for Validation set

, ,
1

Hi,
Is there any CLI command to test validation set against newly created prodigy model or spacy?. I would like see P, R, F1 and accuracy scores like how we get after training. So far i have created my own script to test and compare.

Thanks

2

Hi! I think the easiest solution would be to just use spacy evaluate, which was exactly designed for that purpose. You can run prodigy data-to-spacy to convert your evaluation set to spaCy's JSON format if it's a Prodigy dataset (or spacy convert if it's some other format). I'm guessing your validation set is less likely to change so converting it once shouldn't be too much of a hassle and you'll still be able to iterate quickly on your model and training data.

1 Like
3

Thank you so much. I just ran that command on evaluation set. I am getting less score.

My NER model has overall 92% accuracy where evaluate is giving 84%. How do i make the difference less?