New to Prodigy: Annotation Structure Advice (Big Section of Text vs Separating Sentences)

[ryanwesslen]

Hi @grahama!

Thanks for your question and welcome to the Prodigy community

Can you provide some context on the problem you're trying to solve? For example:

• what format (e.g., raw text, json/xml, word docs, pdf) do you receive the data in?
• is it consistently in the same format like titles/sections or can it vary?
• are there any known organization conventions you know about the docs (like legal docs that follow certain sectioning guidelines)
• who is annotating the data? only you or do you have other annotators? are annotators experts (i.e., they don't need a lot of annotation guidance) or non-experts that will need lots of guidance?

@grahama

Or, should I break the section text below into lines and annotate them separately?

Overall, in general, Prodigy is best suited for breaking down he documents into the smallest unit of analysis that's possible. We have some advice in our docs (here and here) on this:

For NER annotation, there’s often no benefit in annotating long documents at once, especially if you’re planning on training a model on the data. Annotating with a model in the loop is also much faster if the texts aren’t too long, which is why recipes like ner.teach and ner.correct split sentences by default. NER model implementations also typically use a narrow contextual window of a few tokens on either side. If a human annotator can’t make a decision based on the local context, the model will struggle to learn from the data.

If your documents are longer than a few hundred words each, we recommend applying the annotations to smaller sections of the document. Often paragraphs work well. Breaking large documents up into chunks lets the annotator focus on smaller pieces of text at a time, which helps them move through the data more consistently. It also gives you finer-grained labels : you get to see which paragraphs were marked as indicating a label, which makes it much easier to review the decisions later.

As you may have seen too, there are lots of other posts relating to longer text:

Working with longer texts

Hi Paul, The CNN encodes 4 words of context on either side of each token. So for a token more than 4 words from an edge, the rest of the context doesn't really matter. This does allow one convenience though: it makes it relatively easy to support longer documents, because they can be processed largely in parallel during the token-vector encoding. So on the one hand, you'll be able to pass forward documents of a few thousand words into spaCy and it will be able to process it. But it's not takin…

Here's a related post from spaCy GitHub on textcat vs spancat for documents:

Textcat on large documents - best practice for performance · explosion/spaCy...

@grahama

Personally, I'd prefer to annotate the entire section...but do not want to go down on any bad paths given my lack of experience with Prodigy

If you're the only one annotating, then you can be a little more generous and tend to label longer documents.

However, if you have multiple non-expert annotators, I'd err on the simplest decisions possible. I say this because we sometimes see users who decide it's not worth the effort to break down their docs and provide everything to the annotators leading to poor UX annotation experience. This leads to poor annotations and possibly wasted time as they scale up annotators to do thousands of annotations only to realize that those annotations don't aid model training much.