Is there any way I can check the accuracy of my prodigy based ner model on unseen sentences i will provide?
Hi! An NER model you train with Prodigy out-of-the-box is a regular spaCy model, so you can use spacy evaluate: https://spacy.io/api/cli#evaluate
Prodigy's train recipe also lets you provide the name of a dataset containing evaluation data via the --eval-id option, so you're actually evaluating against the same data (instead of just holding back a small random portion of the training set).