Bootstrapping on NER Accuracy test

,
1

We need to perform bootstrapping to test a NER model accuracy. Does there is a function to do it with a .spacy file?

Thank you,
Victor

2

Hi @vtorres , I'm curious as to what you mean by "bootstrapping" in this context.

  • If you wish to use a particular NER model to label data, you can pass the base_model parameter in the ner.manual recipe. You can also use ner.teach if you wish to do active learning.
  • If it's about model evaluation, you can use spacy evaluate. You only need to pass the model you've trained, and the .spacy file of your test data.
3

Thanks, yes, its about model evaluation. What Im looking is how to do boostraping on the evaluation data. In other words, use test data.spacy and perfome boostraping.

I though spacy had a function like evaluate_boostraping. I know pytorch have a boostrap function but I can apply it to the .spacy format.

Thank you,
Victor

4

Any idea?

5

Hi @vtorres ,

Hmm, I don't think there's native support to the functionality you've mentioned. The best scenario here would be for you to write that bootstrapping function, save that sampled batch as a .spacy file, and evaluate manually for each using spacy evaluate.

6

Yes, that what I ended up doing. Thanks!

1 Like