train textcat baseline accuracy

Where can I find a description of the "baseline accuracy" when using "prodigy train textcat"? Is it just the naive approach of classifying everything as the success class or something else?

Hi! The score reported as the baseline accuracy in the regular train recipe is the result of evaluating the base model on the evaluation set. If you're using a blank model, this is the accuracy with randomly initialized weights. Or, expressed in code, the equivalent of this:

nlp = spacy.blank("en")
textcat = nlp.create_pipe("textcat")
textcat.add_label("LABEL_A")  # etc.
nlp.add_pipe(textcat)
nlp.begin_training()
scores = nlp.evaluate(eval_data)

I think in the previous textcat.batch-train, Prodigy was actually calculating a majority class baseline, which is probably a more useful metric here and something we should add back (at least as an option).

1 Like