# New to Prodigy: Annotation Structure Advice (Big Section of Text vs Separating Sentences) **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, spancat\ **Created:** [November 17, 2023, 5:39pm UTC](https://support.prodi.gy/t/new-to-prodigy-annotation-structure-advice-big-section-of-text-vs-separating-sentences/6903 "2023-11-17T17:39:17Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [November 20, 2023, 1:50pm UTC](https://support.prodi.gy/t/new-to-prodigy-annotation-structure-advice-big-section-of-text-vs-separating-sentences/6903/2 "2023-11-20T13:50:04Z")
Hi @grahama! Thanks for your question and welcome to the Prodigy community 👋 Can you provide some context on the problem you're trying to solve? For example: - what format (e.g., raw text, json/xml, word docs, pdf) do you receive the data in? - is it consistently in the same format like titles/sections or can it vary? - are there any known organization conventions you know about the docs (like legal docs that follow certain sectioning guidelines) - who is annotating the data? only you or do you have other annotators? are annotators experts (i.e., they don't need a lot of annotation guidance) or non-experts that will need lots of guidance? > [@grahama](#): > > Or, should I break the section text below into lines and annotate them separately? Overall, in general, Prodigy is best suited for breaking down he documents into the smallest unit of analysis that's possible. We have some advice in our docs ([here](https://prodi.gy/docs/named-entity-recognition#long-text) and [here](https://prodi.gy/docs/text-classification#whole-document)) on this: > For NER annotation, there’s often no benefit in annotating long documents at once, especially if you’re planning on training a model on the data. Annotating with a model in the loop is also much faster if the texts aren’t too long, which is why recipes like [` ner.teach`](https://prodi.gy/docs/recipes#ner-teach) and [` ner.correct`](https://prodi.gy/docs/recipes#ner-correct) split sentences by default. NER model implementations also typically use a **narrow contextual window** of a few tokens on either side. If a human annotator can’t make a decision based on the local context, the model will struggle to learn from the data. > If your documents are longer than a few hundred words each, we recommend applying the annotations to smaller sections of the document. Often paragraphs work well. Breaking large documents up into chunks lets the annotator focus on smaller pieces of text at a time, which helps them move through the data more consistently. It also gives you **finer-grained labels** : you get to see which paragraphs were marked as indicating a label, which makes it much easier to review the decisions later. As you may have seen too, there are lots of other posts relating to longer text: > [@Working with longer texts](https://support.prodi.gy/t/working-with-longer-texts/3378/3): > > Hi Paul, The CNN encodes 4 words of context on either side of each token. So for a token more than 4 words from an edge, the rest of the context doesn't really matter. This does allow one convenience though: it makes it relatively easy to support longer documents, because they can be processed largely in parallel during the token-vector encoding. So on the one hand, you'll be able to pass forward documents of a few thousand words into spaCy and it will be able to process it. But it's not takin… Here's a related post from spaCy GitHub on textcat vs spancat for documents: > **[Textcat on large documents - best practice for performance · explosion/spaCy...](https://github.com/explosion/spaCy/discussions/10174)** > > Hi, I have a textcat task on large documents where speed/performance is important. As the Explosion team point out in the docs, the textcat can struggle on large documents so it might be a better i... > [@grahama](#): > > Personally, I'd prefer to annotate the entire section...but do not want to go down on any bad paths given my lack of experience with Prodigy If you're the only one annotating, then you can be a little more generous and tend to label longer documents. However, if you have multiple non-expert annotators, I'd err on the simplest decisions possible. I say this because we sometimes see users who decide it's not worth the effort to break down their docs and provide everything to the annotators leading to poor UX annotation experience. This leads to poor annotations and possibly wasted time as they scale up annotators to do thousands of annotations only to realize that those annotations don't aid model training much. --- _[View the full topic](https://support.prodi.gy/t/new-to-prodigy-annotation-structure-advice-big-section-of-text-vs-separating-sentences/6903)._