# Command "ner.batch-train" returns MemoryError **URL:** \ **Category:** Uncategorized\ **Tags:** ner, solved\ **Created:** [August 20, 2019, 8:47am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919 "2019-08-20T08:47:17Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![rohan](https://avatars.discourse-cdn.com/v4/letter/r/f19dbf/32.png) [@rohan](https://support.prodi.gy/u/rohan)\ **Post date:** [August 20, 2019, 8:47am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919/1 "2019-08-20T08:47:17Z")
When I run: `prodigy ner.batch-train my_dataset en_core_web_sm --output ./ --n-iter 25 --eval-split 0.2 --dropout 0.2` It returns: > ``` > Traceback (most recent call last): > File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main > " __main__", mod_spec) > File "/usr/lib/python3.5/runpy.py", line 85, in _run_code > exec(code, run_globals) > File "/home/rohan/.local/lib/python3.5/site-packages/prodigy/ __main__.py", line 380, in > controller = recipe(*args, use_plac=True) > File "cython_src/prodigy/core.pyx", line 212, in prodigy.core.recipe.recipe_decorator.recipe_proxy > File "/usr/local/lib/python3.5/dist-packages/plac_core.py", line 328, in call > cmd, result = parser.consume(arglist) > File "/usr/local/lib/python3.5/dist-packages/plac_core.py", line 207, in consume > return cmd, self.func(*(args + varargs + extraopts), **kwargs) > File "/home/rohan/.local/lib/python3.5/site-packages/prodigy/recipes/ner.py", line 602, in batch_train > examples = list(split_sentences(model.orig_nlp, examples)) > File "cython_src/prodigy/components/preprocess.pyx", line 39, in split_sentences > File "/home/rohan/.local/lib/python3.5/site-packages/spacy/language.py", line 708, in pipe > for doc, context in izip(docs, contexts): > File "/home/rohan/.local/lib/python3.5/site-packages/spacy/language.py", line 736, in pipe > for doc in docs: > File "nn_parser.pyx", line 221, in pipe > File "/home/rohan/.local/lib/python3.5/site-packages/spacy/util.py", line 460, in minibatch > batch = list(itertools.islice(items, int(batch_size))) > File "nn_parser.pyx", line 226, in pipe > File "nn_parser.pyx", line 244, in spacy.syntax.nn_parser.Parser.predict > File "nn_parser.pyx", line 257, in spacy.syntax.nn_parser.Parser.greedy_parse > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/model.py", line 169, in __call__ > return self.predict(x) > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/model.py", line 133, in predict > y, _ = self.begin_update(X, drop=None) > File "_parser_model.pyx", line 214, in spacy.syntax._parser_model.ParserModel.begin_update > File "_parser_model.pyx", line 262, in spacy.syntax._parser_model.ParserStepModel. __init__ > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update > X, inc_layer_grad = layer.begin_update(X, drop=drop) > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/api.py", line 295, in begin_update > X, bp_layer = layer.begin_update(layer.ops.flatten(seqs_in, pad=pad), drop=drop) > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update > X, inc_layer_grad = layer.begin_update(X, drop=drop) > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/resnet.py", line 25, in begin_update > y, bp_y = self._layers[0].begin_update(X, drop=drop) > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update > X, inc_layer_grad = layer.begin_update(X, drop=drop) > File "/home/rohan/.local/lib/python3.5/site-packages/thinc/neural/_classes/convolution.py", line 33, in begin_update > X__bo = self.ops.seq2col(X__bi, self.nW) > File "ops.pyx", line 557, in thinc.neural.ops.NumpyOps.seq2col > File "ops.pyx", line 401, in thinc.neural.ops.NumpyOps.allocate > MemoryError > > ``` I was reading [This Topic](https://support.prodi.gy/t/memory-error/326) and [This Another Topic](https://support.prodi.gy/t/thinc-neural-ops-ops-allocate-memoryerror/716) and this doesn't help. I checked numpy, and everything seems fine. Still not sure, why am I not able to train it. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [August 20, 2019, 9:18am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919/2 "2019-08-20T09:18:29Z")
How much memory do you have? ---
**Author:** ![rohan](https://avatars.discourse-cdn.com/v4/letter/r/f19dbf/32.png) [@rohan](https://support.prodi.gy/u/rohan)\ **Post date:** [August 20, 2019, 10:15am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919/3 "2019-08-20T10:15:03Z")
By 'memory', I'm supposing you wanna know the RAM size. I'm using Google Cloud Virtual Machine which contains 8GB of it. I know it's quite less, but I have used the same recipe earlier and it worked like a charm. For now, I just want to batch train for a single label. Not sure, if something's wrong with numpy or my setup after the update. It can be the memory, of course. But what alternate solution we can have here, since I'm unable to upgrade the memory? (BTW, thanks for a prompt reply.) ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [August 20, 2019, 11:19am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919/4 "2019-08-20T11:19:26Z")
Yeah, exactly, I meant the RAM. 8gb should be fine, although 16 are usually better. Did you change anything since training successfully and getting the memory error? Also, since you also opened that other thread: [Command "db-in" returns "MySQL server has gone away"](https://support.prodi.gy/t/command-db-in-returns-mysql-server-has-gone-away/1884/7) Did you try to import those ~10mb examples into the same dataset? If only one of them succeeded, this could easily explain the memory error. ---
**Author:** ![rohan](https://avatars.discourse-cdn.com/v4/letter/r/f19dbf/32.png) [@rohan](https://support.prodi.gy/u/rohan)\ **Post date:** [August 22, 2019, 5:45am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919/5 "2019-08-22T05:45:42Z")
Yeah! That is the same dataset I'm trying to train. None of them works. It seems like its all happening because of the size of the documents. They're basically the pdf(s), each of them contains 50+ pages. I'm planning to annotate them page-wise. But, I'm afraid, I might loose the important details in the documents due to separation. Will give it a shot. Thanks! ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [August 22, 2019, 9:55am UTC](https://support.prodi.gy/t/command-ner-batch-train-returns-memoryerror/1919/6 "2019-08-22T09:55:15Z")
> [@rohan](#): > > They're basically the pdf(s), each of them contains 50+ pages. I'm planning to annotate them page-wise. But, I'm afraid, I might loose the important details in the documents due to separation. Will give it a shot. Thanks! Yes, that explains a lot. And you can always separate them by paragraphs and add an `"id"` or something similar to the data with a reference to the original document. This way, you always know what belongs together. The thing is, if your goal is to train an NER model or text classifier with spaCy, there's really nothing you gain from annotating and training on these _huge_ documents. The model is sensitive to the very _local_ context – so if _you_ can't make an annotation decision based on the local context, the model is very unlikely to learn anything meaningful anyways.