Custom recipe for Annotating Overlapping Spans

[ines (Ines Montani)]

@bart_factual_design

From your answer I understand that the Spacy model can be trained using overlapping entities. Is that correct?

No, the model predicts one label per token, so it can't be trained on overlapping entities. This is true for most named entity recognition model implementations, since classic "named entities" can't overlap by definition.

@bart_factual_design

And another question: does the Spacy model compare individual token (or span) accuracy and pick the highest value? If so, is it possible to output all entity predictions per token?

This is a bit more complex – see here for details and an explanation:

Accessing probabilities in NER

Hi, The answer to this question is a little bit involved. The short answer is this: # Number of alternate analyses to consider. More is slower, and not necessarily better -- you need to experiment on your problem. beam_width = 16 # This clips solutions at each step. We multiply the score of the top-ranked action by this value, and use the result as a threshold. This prevents the parser from exploring options that look very unlikely, saving a bit of efficiency. Accuracy may also improve, becau…