# any solution for this issue even after i've changed batch size its not working

**URL:** <https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717>\
**Category:** Uncategorized\
**Tags:** usage, spacy, training, spancat\
**Created:** [June 20, 2022, 1:17pm UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717 "2022-06-20T13:17:22Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Harsh\_chavhan](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/harsh_chavhan/32/3244_2.png) [@Harsh\_chavhan](https://support.prodi.gy/u/Harsh_chavhan)\
**Post date:** [June 20, 2022, 1:17pm UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/1 "2022-06-20T13:17:22Z")

</div>

Components: spancat  
Merging training and evaluation data for 1 components

- [spancat] Training: 132 | Evaluation: 33 (20% split)  
Training: 132 | Evaluation: 33  
Labels: spancat (5)

ℹ Pipeline: ['tok2vec', 'spancat']  
ℹ Initial learn rate: 0.001  
E # LOSS TOK2VEC LOSS SPANCAT SPANS\_SC\_F SPANS\_SC\_P SPANS\_SC\_R SCORE

* * *

⚠ Aborting and saving the final best model. Encountered exception:  
OutOfMemoryError('Out of memory allocating 1,876,703,232 bytes (allocated so  
far: 6,663,892,992 bytes).')  
Traceback (most recent call last):  
File "/usr/lib/python3.8/runpy.py", line 194, in \_run\_module\_as\_main  
return \_run\_code(code, main\_globals, None,  
File "/usr/lib/python3.8/runpy.py", line 87, in \_run\_code  
exec(code, run\_globals)  
File "/home/harsh/.local/lib/python3.8/site-packages/prodigy/ **main**.py", line 61, in   
controller = recipe(_args, use\_plac=True)  
File "cython\_src/prodigy/core.pyx", line 329, in prodigy.core.recipe.recipe\_decorator.recipe\_proxy  
File "/home/harsh/.local/lib/python3.8/site-packages/plac\_core.py", line 367, in call  
cmd, result = parser.consume(arglist)  
File "/home/harsh/.local/lib/python3.8/site-packages/plac\_core.py", line 232, in consume  
return cmd, self.func(_(args + varargs + extraopts), \*\*kwargs)  
File "/home/harsh/.local/lib/python3.8/site-packages/prodigy/recipes/train.py", line 278, in train  
return \_train(  
File "/home/harsh/.local/lib/python3.8/site-packages/prodigy/recipes/train.py", line 198, in \_train  
spacy\_train(nlp, output\_path, use\_gpu=gpu\_id, stdout=stdout)  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/training/loop.py", line 122, in train  
raise e  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/training/loop.py", line 105, in train  
for batch, info, is\_best\_checkpoint in training\_step\_iterator:  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/training/loop.py", line 203, in train\_while\_improving  
nlp.update(  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/language.py", line 1164, in update  
proc.update(examples, sgd=None, losses=losses, \*\*component\_cfg[name]) # type: ignore  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/pipeline/spancat.py", line 346, in update  
backprop\_scores(d\_scores) # type: ignore  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/layers/chain.py", line 60, in backprop  
dX = callback(dY)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/layers/chain.py", line 60, in backprop  
dX = callback(dY)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/layers/concatenate.py", line 67, in backprop  
gradient = bwd(dY)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/layers/reduce\_mean.py", line 26, in backprop  
return Ragged(model.ops.backprop\_reduce\_mean(dY, lengths), lengths)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/backends/cupy\_ops.py", line 235, in backprop\_reduce\_mean  
return \_custom\_kernels.backprop\_reduce\_mean(d\_means, lengths)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/backends/\_custom\_kernels.py", line 318, in backprop\_reduce\_mean  
out = cupy.zeros((T, O), dtype="f")  
File "/home/harsh/.local/lib/python3.8/site-packages/cupy/\_creation/basic.py", line 211, in zeros  
a = cupy.ndarray(shape, dtype, order=order)  
File "cupy/\_core/core.pyx", line 171, in cupy.\_core.core.ndarray. **init**  
File "cupy/cuda/memory.pyx", line 698, in cupy.cuda.memory.alloc  
File "cupy/cuda/memory.pyx", line 1375, in cupy.cuda.memory.MemoryPool.malloc  
File "cupy/cuda/memory.pyx", line 1396, in cupy.cuda.memory.MemoryPool.malloc  
File "cupy/cuda/memory.pyx", line 1076, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc  
File "cupy/cuda/memory.pyx", line 1097, in cupy.cuda.memory.SingleDeviceMemoryPool.\_malloc  
File "cupy/cuda/memory.pyx", line 1335, in cupy.cuda.memory.SingleDeviceMemoryPool.\_try\_malloc  
cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 1,876,703,232 bytes (allocated so far: 6,663,892,992 bytes).

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [June 20, 2022, 2:12pm UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/2 "2022-06-20T14:12:19Z")

</div>

Hi there.

Could you share the commands that you ran? Without understanding what you tried to run, it's hard to drill deeper and understand what went wrong.

From looking at the traceback it seems like you're hitting an out of memory issue. Are you trying to run on a GPU? Could you share your operating system, Python, spaCy and Prodigy versions?

---

<div class="post-metadata">

**Author:** ![Harsh\_chavhan](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/harsh_chavhan/32/3244_2.png) [@Harsh\_chavhan](https://support.prodi.gy/u/Harsh_chavhan)\
**Post date:** [June 20, 2022, 2:14pm UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/3 "2022-06-20T14:14:52Z")

</div>

> [@koaning](#):
>
> ould you share the commands that you ran? Without understanding what you tried to run, it's hard to drill deeper and understand what went wrong.
> 
> From looking at the traceback it seems like you're hitting an out of memory issue. Are you trying to run on a GPU? Could you share your operating system, Python, spaCy and Prodigy versions?

python3 -m prodigy train "./model\_text\_span\_20\_jun/" --spancat text-span-softskill --gpu-id 0 --verbose

Python = 3.8.10  
spacy = spaCy v3.3.1

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [June 20, 2022, 2:21pm UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/4 "2022-06-20T14:21:46Z")

</div>

How large is your dataset? Do you have very long examples? It's suggested [on GitHub](https://github.com/explosion/spaCy/discussions/7720) that you may get it to run by making the batch size smaller and/or set the `max_batch_items` smaller.

You can [override](https://spacy.io/usage/training#config-overrides) the config in the Prodigy train command as you would in the spaCy train command. You might try running it again with `--training.batch_size 8` at the end. Does that help?

Also, do you see the same error if you don't connect a GPU?

---

<div class="post-metadata">

**Author:** ![Harsh\_chavhan](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/harsh_chavhan/32/3244_2.png) [@Harsh\_chavhan](https://support.prodi.gy/u/Harsh_chavhan)\
**Post date:** [June 21, 2022, 6:48am UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/5 "2022-06-21T06:48:39Z")

</div>

no its showing me this error after changing batch size

=========================== Initializing pipeline ===========================  
✘ Config validation error  
training -\> batch\_size extra fields not permitted

{'dev\_corpus': 'corpora.dev', 'train\_corpus': 'corpora.train', 'seed': 0, 'gpu\_allocator': None, 'dropout': 0.1, 'accumulate\_gradient': 1, 'patience': 1600, 'max\_epochs': 0, 'max\_steps': 20000, 'eval\_frequency': 200, 'frozen\_components': , 'annotating\_components': , 'before\_to\_disk': {'@misc': 'prodigy.todisk\_cleanup.v1'}, 'logger': {'@loggers': 'prodigy.ConsoleLogger.v1'}, 'batch\_size': 8, 'batcher': {'@batchers': 'spacy.batch\_by\_words.v1', 'discard\_oversize': False, 'tolerance': 0.2, 'get\_length': None, 'size': {'@schedules': 'compounding.v1', 'start': 100, 'stop': 1000, 'compound': 1.001, 't': 0.0}}, 'optimizer': {'@optimizers': 'Adam.v1', 'beta1': 0.9, 'beta2': 0.999, 'L2\_is\_weight\_decay': True, 'L2': 0.01, 'grad\_clip': 1.0, 'use\_averages': False, 'eps': 1e-08, 'learn\_rate': 0.001}, 'score\_weights': {'spans\_sc\_f': 1.0, 'spans\_sc\_p': 0.0, 'spans\_sc\_r': 0.0}}

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [June 21, 2022, 7:01am UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/6 "2022-06-21T07:01:51Z")

</div>

Ah! My bad. Could you try:

```python
--training.batcher.size.start 8

```

I just checked the [default configuration](https://spacy.io/usage/training) and these are the available settings for the batch size:

```python
[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

```

---

<div class="post-metadata">

**Author:** ![Harsh\_chavhan](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/harsh_chavhan/32/3244_2.png) [@Harsh\_chavhan](https://support.prodi.gy/u/Harsh_chavhan)\
**Post date:** [June 21, 2022, 8:42am UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/7 "2022-06-21T08:42:57Z")

</div>

Thanks @koaning Vincent for helping me but, it's still not working on GPU because my span size is 96 and again i'm attaching my error for your reference

python3 -m prodigy train "./span\_20\_jun\_2022\_v1/" --spancat text-span-softskill --gpu-id 0 --training.batcher.size.start 8  
ℹ Using GPU: 0  
/home/harsh/.local/lib/python3.8/site-packages/torch/cuda/ **init**.py:145: UserWarning:  
NVIDIA GeForce RTX 3070 Ti with CUDA capability sm\_86 is not compatible with the current PyTorch installation.  
The current PyTorch install supports CUDA capabilities sm\_37 sm\_50 sm\_60 sm\_70.  
If you want to use the NVIDIA GeForce RTX 3070 Ti GPU with PyTorch, please check the instructions at [Start Locally | PyTorch](https://pytorch.org/get-started/locally/)

warnings.warn(incompatible\_device\_warn.format(device\_name, capability, " ".join(arch\_list), device\_name))

========================= Generating Prodigy config =========================  
ℹ Auto-generating config with spaCy  
Using 'spacy.ngram\_range\_suggester.v1' for 'spancat' with sizes 1 to 96 (inferred from data)  
✔ Generated training config

=========================== Initializing pipeline ===========================  
[2022-06-21 13:56:02,879] [INFO] Set up nlp object from config  
Components: spancat  
Merging training and evaluation data for 1 components

- [spancat] Training: 148 | Evaluation: 36 (20% split)  
Training: 148 | Evaluation: 36  
Labels: spancat (5)  
[2022-06-21 13:56:03,231] [INFO] Pipeline: ['tok2vec', 'spancat']  
[2022-06-21 13:56:03,233] [INFO] Created vocabulary  
[2022-06-21 13:56:03,233] [INFO] Finished initializing nlp object  
[2022-06-21 13:56:05,770] [INFO] Initialized pipeline components: ['tok2vec', 'spancat']  
✔ Initialized pipeline

============================= Training pipeline =============================  
Components: spancat  
Merging training and evaluation data for 1 components

- [spancat] Training: 148 | Evaluation: 36 (20% split)  
Training: 148 | Evaluation: 36  
Labels: spancat (5)  
ℹ Pipeline: ['tok2vec', 'spancat']  
ℹ Initial learn rate: 0.001  
E # LOSS TOK2VEC LOSS SPANCAT SPANS\_SC\_F SPANS\_SC\_P SPANS\_SC\_R SCORE

* * *

⚠ Aborting and saving the final best model. Encountered exception:  
OutOfMemoryError('Out of memory allocating 2,098,403,328 bytes (allocated so  
far: 186,237,952 bytes).')  
Traceback (most recent call last):  
File "/usr/lib/python3.8/runpy.py", line 194, in \_run\_module\_as\_main  
return \_run\_code(code, main\_globals, None,  
File "/usr/lib/python3.8/runpy.py", line 87, in \_run\_code  
exec(code, run\_globals)  
File "/home/harsh/.local/lib/python3.8/site-packages/prodigy/ **main**.py", line 61, in   
controller = recipe(_args, use\_plac=True)  
File "cython\_src/prodigy/core.pyx", line 329, in prodigy.core.recipe.recipe\_decorator.recipe\_proxy  
File "/home/harsh/.local/lib/python3.8/site-packages/plac\_core.py", line 367, in call  
cmd, result = parser.consume(arglist)  
File "/home/harsh/.local/lib/python3.8/site-packages/plac\_core.py", line 232, in consume  
return cmd, self.func(_(args + varargs + extraopts), \*\*kwargs)  
File "/home/harsh/.local/lib/python3.8/site-packages/prodigy/recipes/train.py", line 278, in train  
return \_train(  
File "/home/harsh/.local/lib/python3.8/site-packages/prodigy/recipes/train.py", line 198, in \_train  
spacy\_train(nlp, output\_path, use\_gpu=gpu\_id, stdout=stdout)  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/training/loop.py", line 122, in train  
raise e  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/training/loop.py", line 105, in train  
for batch, info, is\_best\_checkpoint in training\_step\_iterator:  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/training/loop.py", line 203, in train\_while\_improving  
nlp.update(  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/language.py", line 1164, in update  
proc.update(examples, sgd=None, losses=losses, \*\*component\_cfg[name]) # type: ignore  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/pipeline/spancat.py", line 344, in update  
scores, backprop\_scores = self.model.begin\_update((docs, spans))  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/model.py", line 309, in begin\_update  
return self.\_func(self, X, is\_train=True)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward  
Y, inc\_layer\_grad = layer(X, is\_train=is\_train)  
File "/home/harsh/.local/lib/python3.8/site-packages/thinc/model.py", line 291, in **call**  
return self.\_func(self, X, is\_train=is\_train)  
File "/home/harsh/.local/lib/python3.8/site-packages/spacy/ml/extract\_spans.py", line 32, in forward  
Y = Ragged(X.dataXd[indices], spans.dataXd[:, 1] - spans.dataXd[:, 0]) # type: ignore[arg-type, index]  
File "cupy/\_core/core.pyx", line 1437, in cupy.\_core.core.ndarray. **getitem**  
File "cupy/\_core/\_routines\_indexing.pyx", line 43, in cupy.\_core.\_routines\_indexing.\_ndarray\_getitem  
File "cupy/\_core/core.pyx", line 782, in cupy.\_core.core.ndarray.take  
File "cupy/\_core/\_routines\_indexing.pyx", line 144, in cupy.\_core.\_routines\_indexing.\_ndarray\_take  
File "cupy/\_core/\_routines\_indexing.pyx", line 834, in cupy.\_core.\_routines\_indexing.\_take  
File "cupy/\_core/core.pyx", line 171, in cupy.\_core.core.ndarray. **init**  
File "cupy/cuda/memory.pyx", line 698, in cupy.cuda.memory.alloc  
File "cupy/cuda/memory.pyx", line 1375, in cupy.cuda.memory.MemoryPool.malloc  
File "cupy/cuda/memory.pyx", line 1396, in cupy.cuda.memory.MemoryPool.malloc  
File "cupy/cuda/memory.pyx", line 1076, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc  
File "cupy/cuda/memory.pyx", line 1097, in cupy.cuda.memory.SingleDeviceMemoryPool.\_malloc  
File "cupy/cuda/memory.pyx", line 1335, in cupy.cuda.memory.SingleDeviceMemoryPool.\_try\_malloc  
cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 2,098,403,328 bytes (allocated so far: 186,237,952 bytes). @ines

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [June 22, 2022, 8:41am UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/8 "2022-06-22T08:41:53Z")

</div>

Please put code/tracebacks in code blocks (by wrapping them in ```). That way, it's easier to look through them and they will also become scrollable, saving screen real estate.

It's becoming more and more clear that this is perhaps less of a Prodigy issue and more of a spaCy/Pytorch issue. One question, a span of size 96 feels very large. Are you using a custom config? Your previous commands suggested that you weren't and the standard config has these span sizes:

```python
[components.spancat.suggester]
@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3]

```

Is there a reason why your span size really needs to be that big?

---

<div class="post-metadata">

**Author:** ![Harsh\_chavhan](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/harsh_chavhan/32/3244_2.png) [@Harsh\_chavhan](https://support.prodi.gy/u/Harsh_chavhan)\
**Post date:** [June 22, 2022, 9:03am UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/9 "2022-06-22T09:03:07Z")

</div>

> [@koaning](#):
>
> It's becoming more and more clear that this is perhaps less of a Prodigy issue and more of a spaCy/Pytorch issue. One question, a span of size 96 feels very large. Are you using a custom config? Your previous commands suggested that you weren't and the s

okay is there any way to limit those large span size while annotating ???

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [June 23, 2022, 9:54am UTC](https://support.prodi.gy/t/any-solution-for-this-issue-even-after-ive-changed-batch-size-its-not-working/5717/10 "2022-06-23T09:54:30Z")

</div>

I'm perhaps a bit confused. My impression is that you've been annotating the data such that there are spans that have 96 tokens in them. If you want to prevent this, you should not create large spans when you annotate.

Do you have an example of such a large span? Are you using a custom config?