Two-Step Span Annotation

Hi - is it possible to have a two-step span annotation in which the annotator first labels a span according to a top-level category and then for each labeled span, it is marked by another category? The alternative is to enumerate all combinations of both questions which is a bit cumbersome for the annotators. I've posted an example from a recent paper in which the error type is first chosen (Entity, Event, Noun Phrase, Others) and then each span is further categorized as Extrinsic or Intrinsic.


Hi @griff4692 , there are two ways you can go about this:

  1. If the second-level category (Extrinsic vs. Intrinsic) can be labeled in isolation and not in context, you can perhaps group all the instances of a given entity type together, and ask the second category as a multiple choice question. This means, you label the first category as you do in standard NER, then compile the entities together, and label those compiled entities in a multiple-choice fashion. You can use both custom recipes / interfaces to achieve this.
  2. Another approach may be to use Span Categorization. It allows overlaps in entities so you can label them in "two" categories. The only downside is that you have to label both at the same time. This can be useful if the second-level categories (Extrinsic vs. Intrinsic) needed to be labeled in context to the whole text.

Thanks for the reply and this is really helpful. Those are two great options but I think we were hoping for something even easier for annotators. 1) might be difficult because they will have been thinking about a particular span and asking them to annotate it again after the fact might be a cognitive burden and for 2) it might also be a burden to ask them to select the same span twice (we are asking clinicians to label text so efficiency and ease of use is paramount). We were hoping for a UX where a clinician labels a span of text and then some sort of pop-up comes up that directly asks the clinician "Is this intrinsic or extrinsic?" A similar two-step process is shown in BRAT and would be great to support in Prodigy! Thanks