Scorer for Text Classification

I try to evaluate my multilabel textcat with Scorer, but the scorer.scores don't return anything.

def evaluate(nlp, test_data):
    scorer = Scorer()
    for text, label in test_data:
        # text: "my example text"
        # label: {"cats:{"cat1":0.0, "cat2":1.0, "cat3":0.0}}
        doc_gold_text = nlp.make_doc(text)
        gold = GoldParse(doc_gold_text, cats=label["cats"])
        pred_value = nlp(text)
        scorer.score(pred_value, gold)

    return score.scores
    # return: {}
    # actually, also other fields return zero

The model works well to predict and I use this same function to eval NER in other model with success. Do I something wrong in this case? Score don't work to textcat?

I don't know what's wrong with the code above, but I tried a different approach and it's worked! Using nlp.evaluate():

def evaluate(nlp, test_data):
    eval_input = [(nlp.make_doc(text), GoldParse(nlp.make_doc(text), cats=label["cats"])) for text, label in test_data]
    scorer = nlp.evaluate(eval_input)
    return scorer.scores
1 Like