evaluate text classification model using spacy evaluate?

, ,
1

Hello,

I am trying to use spacy's evaluate with my model made from prodigy along with my evaluation dataset (also exported from prodigy). I already converted the evaluation dataset from .jsonl to .json using

python -m spacy convert file.jsonl . --converter jsonl --lang en

and that seemed to work to get it in a format that spacy's evaluate recipe can iterate over but now im getting an error: Cannot evaluate textcat model on data with different labels
Labels in model: {'LABEL_NAME'}
Labels in evaluation data: set()

when looking at the json file i got from using the convert function i dont see my label in the structure? Am i doing something wrong?

Thank you!

2

Hi, that spacy converter only converts NER annotation, not textcat annotation. Try the new prodigy data-to-spacy command instead, which has support for textcat. (And check that the --textcat_exclusive option is set correctly.)

2 Likes
3

Thanks Adriane! Worked like a charm :clap: