Using binary accept/reject from NER teach in Spacy

[ines (Ines Montani)]

@justin.epstein

Hi everyone! We are using the ner.teach recipe to generate the binary accept/reject data from our texts. Is there any way to train this using spacy? We currently train our models in databricks using spacy, so if there is a way to import that data and leverage our existing pipelines that would be great!

You can use the data-to-spacy command to convert the annotations to spaCy's training format, and set --ner-missing to treat all unannotated tokens as missing values (so you can represent partial annotations). However, for the binary annotations collected with the active learning recipes, it does mean you're losing some information from the rejected examples. I've explained the difference of the two update mechanisms in more detail here:

Prodigy 1.90 train recipe --ner-missing argument

In a typical training scenario, you're updating a model with examples and the correct answer – e.g. with a text and the entities in it. In some cases you may also have partial annotations: you know some entities but not all. Prodigy's active learning recipes like ner.teach also let you collect binary yes/no decisions. The data you create here is different again: for some spans, you know that they are entities, because you accepted them. For the ones you rejected, you know that they're not of t…

Two Questions on Teach recipes

If you're using ner.batch-train or train with --binary then no – the training process here was specifically designed for incomplete and binary yes/no answers. Before training, Prodigy will merge all annotations on the same texts, and all unannotated tokens will be considered missing values. (Btw, if you're interested in how the updating process works for binary annotations, my slides here show an example.) Depending on the annotations and specific texts and entities, it can always happen that …

spaCy does let you represent "negative" labels with an exclamation mark, though – for instance, !B-PERSON for a token that's not the beginning of a PERSON entity. So you could experiment with that to incporporate some of the rejected answers, if your results end up looking worse.