# What happens if your annotation has overlapping entity spans? **URL:** \ **Category:** Uncategorized\ **Tags:** usage, spacy\ **Created:** [March 2, 2018, 5:16pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363 "2018-03-02T17:16:39Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/wpm/32/103_2.png) [@wpm](https://support.prodi.gy/u/wpm)\ **Post date:** [March 2, 2018, 5:16pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/1 "2018-03-02T17:16:39Z")
When I fix the catastrophic forgetting problem by adding in entities detected by the baseline model do I have to be careful not to have the new entity spans and the old entity spans overlap? For example, say I am trying to build an NER model that finds sports teams. I have the following sentence. ```nohighlight The Florida Gators won their away game in California last night. ``` Out of the box, spaCy will annotate “Florida” and “California” as GPEs. What I ultimately want is to keep “California” as a GPE, but label “Florida Gators” as a SPORTS\_TEAM. In my training data I’ll label “Florida Gators” as SPORTS\_TEAM, but then in order to combat catastrophic forgetting I’ll run the sentence through the baseline NER, have it tell me that “California” is a GPE, and add that span to my training data. The baseline NER will also tell me that “Florida” is a GPE, and I don’t want to have that information overwrite my SPORTS\_TEAM annotation. Is there some convention that spAcy/Prodigy uses to keep this straight, or do I just have to be careful not to overlap spans when I’m augmenting my training data? ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [March 2, 2018, 7:55pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/2 "2018-03-02T19:55:41Z")
The entity recognizer is constrained to predict only non-overlapping, non-nested spans. The training data should obey the same constraint. If you like, you could have two sentences with the different annotations in your data. I’m not sure whether this would hurt or help your performance, though. If you want spaCy to learn to recover both annotations, you could have two `EntityRecognizer` instances in the pipeline. You would need to move the entity annotations into an extension attribute, because you don’t want the second entity recogniser to overwrite the entities set by the first one. Something like this should work: ```python from spacy.tokens import Doc Doc.set_extension('my_ents', default=None) def move_ents_to_attr(doc): if doc._.my_ents is None: doc._.my_ents = [] doc._.my_ents.extend(doc.ents) doc.ents = [] return doc nlp = spacy.load('en') nlp.entity.postprocess.append(move_ents_to_attr) ``` ---
**Author:** ![sergei3000](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@sergei3000](https://support.prodi.gy/u/sergei3000)\ **Post date:** [December 24, 2019, 12:04pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/3 "2019-12-24T12:04:39Z")
What if I have more than two NER models? Do I need to create an extension attribute for each of them? Or should I code it differently to handle unknown number of models in my NER pipeline? ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [December 25, 2019, 11:50am UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/4 "2019-12-25T11:50:14Z")
You would need to have an extension attribute to hold the spans, yes. Internally spaCy encodes the entity annotations using IOB-style data, so there's no way to represent overlapping entities on the built-in token data. ---
**Author:** ![mkallen](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mkallen](https://support.prodi.gy/u/mkallen)\ **Post date:** [March 23, 2020, 8:21pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/6 "2020-03-23T20:21:29Z")
Tried to implement the code above but got the following error `AttributeError: 'spacy.pipeline.pipes.EntityRecognizer' object has no attribute 'postprocess'` I cant seem to find anything on `postprocess` in the spacy docs ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [March 29, 2020, 9:00am UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/7 "2020-03-29T09:00:14Z")
@mkallen My mistake, sorry. Just add the function to the pipeline with `nlp.add_pipe(move_ents_to_attr, last=True)` ---
**Author:** ![mkallen](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@mkallen](https://support.prodi.gy/u/mkallen)\ **Post date:** [March 30, 2020, 3:09pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/8 "2020-03-30T15:09:18Z")
Thank you 😊 ---
**Author:** ![adsk2050](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/adsk2050/32/4290_2.png) [@adsk2050](https://support.prodi.gy/u/adsk2050)\ **Post date:** [January 10, 2024, 1:11pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/9 "2024-01-10T13:11:17Z")
I have read that [Spans can have overlapping entities](https://spacy.io/usage/rule-based-matching#spanruler-usage): ```python Unlike in doc.ents, overlapping matches are allowed in doc.spans, so no filtering is required, but optional filtering and sorting can be applied to the spans before they’re saved. ``` Can I use this to create training data for my spacy model? If yes, how? Because DocBin doesnt accept overlapping spans. ---
**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\ **Post date:** [January 12, 2024, 12:45pm UTC](https://support.prodi.gy/t/what-happens-if-your-annotation-has-overlapping-entity-spans/363/10 "2024-01-12T12:45:09Z")
Hi @adsk2050 and welcome to the forum! [This note](https://prodi.gy/docs/span-categorization#training) in Prodigy span categorization docs contains just the info you need, I think. You should consider training spaCy [SpanCategorizer](https://spacy.io/api/spancategorizer) Did `data-to-spacy` with `--spancat` dataset did not work for you?