mismatch between accuracy on holdout set and batch-train accuracy

[ines (Ines Montani)]

@mhigginslp This recipe contributed by @farlee2121 (or a modified version of it) might be what you're looking for:

Evaluating Precision and Recall of NER

In case others like me come looking for a basic scoring recipe, here is what I cooked up. It doesn’t consider threshold, but it evaluates model accuracy without re-training and can output either PRF or the standard prodigy score scheme. import spacy import spacy.scorer from prodigy.components.db import connect from prodigy.core import recipe, recipe_args from prodigy.models.ner import EntityRecognizer, merge_spans from prodigy.util import log from prodigy.components.preprocess import split_sen…