# Train spancat bug

**URL:** <https://support.prodi.gy/t/train-spancat-bug/4781>\
**Category:** Uncategorized\
**Tags:** spacy, training, spancat\
**Created:** [October 8, 2021, 9:26am UTC](https://support.prodi.gy/t/train-spancat-bug/4781 "2021-10-08T09:26:49Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lize](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@Lize](https://support.prodi.gy/u/Lize)\
**Post date:** [October 8, 2021, 9:26am UTC](https://support.prodi.gy/t/train-spancat-bug/4781/1 "2021-10-08T09:26:49Z")

</div>

Hello 🙂 !

I'm doing a spancat test.  
I have annotated a first batch of examples and now I would like to have a first idea of the scores. So I tried to launch a training.  
My command line seems to be correct since everything runs as it should.However the process stops by itself without giving any error code.

The command line I use is the following :

> prodigy train ./testmodelespan/ --spancat SpanClauseClassification -es 0.3 -m fr\_core\_news\_sm --lang "fr" --label-stats

Here is a screenshot of the error, it just says that the "process stopped"

 ![Capture (1)](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/2/2f2c82c6ca4a41bf805bd46af8cb259dd815f958.png)

Thanks for your help !

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 8, 2021, 9:40am UTC](https://support.prodi.gy/t/train-spancat-bug/4781/2 "2021-10-08T09:40:06Z")

</div>

Hi! How much memory do you have and could you run profiling to check whether you're running out of memory? If your process just dies, one of the likely explanations is that you're running out of memory.

---

<div class="post-metadata">

**Author:** ![Lize](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@Lize](https://support.prodi.gy/u/Lize)\
**Post date:** [October 8, 2021, 2:33pm UTC](https://support.prodi.gy/t/train-spancat-bug/4781/3 "2021-10-08T14:33:46Z")

</div>

Hi !

we were configured on 8go, we went to 32go and the problem is still the same.  
What could be the other causes?

Thanks 🙂 !

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 10, 2021, 5:30pm UTC](https://support.prodi.gy/t/train-spancat-bug/4781/4 "2021-10-10T17:30:18Z")

</div>

Is there anything else you can see in the logs that looks relevant? It really sounds like this is more of a system-level problem rather than a problem with the training logic itself. Also, could you try exporting your data with `data-to-spacy` and train with spaCy direclty? If the problem still remains, could you open an issue on the [spaCy tracker](https://github.com/explosion/spaCy/issues)?

---

<div class="post-metadata">

**Author:** ![Lize](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@Lize](https://support.prodi.gy/u/Lize)\
**Post date:** [October 11, 2021, 10:04am UTC](https://support.prodi.gy/t/train-spancat-bug/4781/5 "2021-10-11T10:04:14Z")

</div>

Hi Inès,

Is there a log file where we can see the type of errors ?

I tried to export my data with data-to-spacy but the problem is still the same : "processus stoped".

So, I was wondering if i could juste export my annotations with db-out and train a new model with spacy directly ?  
I use db-out to export my data in .json.

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 11, 2021, 10:53am UTC](https://support.prodi.gy/t/train-spancat-bug/4781/6 "2021-10-11T10:53:19Z")

</div>

Ah, I was referring to your server logs here! For example, if you're using a cloud provider, you can typically view the logs and see everything that's happening.

> [@Lize](#):
>
> I tried to export my data with data-to-spacy but the problem is still the same : "processus stoped".

This does indicate that there's either a problem at the spaCy level, or that you're running out of memory preprocessing your data. 8gb would probably be too little so maybe double-check that you definitely have 32gb available? Also, maybe try re-installing spaCy as well, just in case you ended up with a broken installation somehow.

---

<div class="post-metadata">

**Author:** ![Lize](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@Lize](https://support.prodi.gy/u/Lize)\
**Post date:** [October 11, 2021, 12:14pm UTC](https://support.prodi.gy/t/train-spancat-bug/4781/7 "2021-10-11T12:14:57Z")

</div>

Is it possible that the spans that I annotated are too long and kill the process ?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 12, 2021, 9:40am UTC](https://support.prodi.gy/t/train-spancat-bug/4781/8 "2021-10-12T09:40:46Z")

</div>

> [@Lize](#):
>
> Is it possible that the spans that I annotated are too long and kill the process ?

It's definitely possible that some of the logic used to preprocess the spans and calculate the best-matching suggester range is inefficient. So IF you're running out of memory, that'd explain why. So it'd be important to check whether you're in fact running out of memory.

Also, could you run `data-to-spacy` again with the `PRODIGY_LOGGING=basic` environment variable to show more logs? This should give you an idea where it happens.
