# NER on long texts

**URL:** https://support.prodi.gy/t/ner-on-long-texts/5467
**Category:** Uncategorized
**Tags:** usage, ner
**Created:** [March 23, 2022, 4:01pm UTC](https://support.prodi.gy/t/ner-on-long-texts/5467 "2022-03-23T16:01:36Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![yllwpr](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/yllwpr/32/2967_2.png) [@yllwpr](https://support.prodi.gy/u/yllwpr)
#### Post date: [March 23, 2022, 4:01pm UTC](https://support.prodi.gy/t/ner-on-long-texts/5467/1 "2022-03-23T16:01:37Z")

</div>

I would like to train a NER on scientific texts. The goal is to label different physical properties. The texts are very long and contain the corresponding properties only in certain places. Is it advisable to split the texts? Here would be only in very few parts then a corresponding entity to find. How many words/sentences are optimal for prodigy?

---

<div class="post-metadata">

### Author: ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)
#### Post date: [March 24, 2022, 8:23am UTC](https://support.prodi.gy/t/ner-on-long-texts/5467/2 "2022-03-24T08:23:07Z")

</div>

Hi @yllwpr,  
It is definitely advisable to split texts for NER task annotation. In fact `ner.correct` and `ner.teach` recipes split texts into sentences by default. There are three main reasons for this: 1) NER models usually learn and infer based on a fairly narrow window of tokens 2) Prodigy workflows with a model in the loop are more efficient if the tasks are smaller 3) smaller text chunks make for a less taxing annotation task.  
That said, it is possible to modify the UI to accommodate longer chunks of text as explained [here](https://prodi.gy/docs/named-entity-recognition#long-text).

Given that you expect your target entities to be sparse, the recommended workflow would probably be to:

1. annotate a small gold-standard corpus either fully manually or with the help of patterns using `ner.manual` [recipe](https://prodi.gy/docs/recipes#ner-manual) with `--patterns` option
2. train an initial model based on your initial gold-standard corpus using `train` [recipe](https://prodi.gy/docs/recipes#train)
3. use the model trained in step 2 it in the `ner.teach` [recipe](https://prodi.gy/docs/recipes#ner-teach) to suggest the most relevant examples to annotate in the rest of your original corpus.
4. Train the final model on the full training set.
