I got the following error, hope this helps to understand the problem… i am going to reduce the beam-width e batch-size
python3 -m prodigy ner.batch-train citizenship /home/damiano/blank_model --output /home/damiano/after_citizenship --dropout 0.2 --n-iter 100 --batch-size 8 --beam-width 8
Loaded model /home/damiano/blank_model
Using 20% of accept/reject examples (3172) for evaluation
Using 100% of remaining examples (23010) for training
Dropout: 0.2 Batch size: 8 Iterations: 100
BEFORE 0.039
Correct 90
Incorrect 2241
Entities 25268
Unknown 25178
# LOSS RIGHT WRONG ENTS SKIP ACCURACY
01 1172.850 2146 185 15527 0 0.921
02 470.228 2226 105 11765 0 0.955
03 499.363 2296 35 8709 0 0.985
04 520.938 2250 81 10971 0 0.965
05 561.103 2244 87 6994 0 0.963
06 563.869 2300 31 6790 0 0.987
07 537.077 2299 32 9406 0 0.986
08 338.692 2306 25 7027 0 0.989
09 594.263 2299 32 7235 0 0.986
10 398.136 2286 45 8009 0 0.981
11 631.404 2299 32 8527 0 0.986
12 674.743 2309 22 16163 0 0.991
13 454.832 2296 35 23171 0 0.985
14 514.246 2295 36 11051 0 0.985
15 596.353 2302 29 10663 0 0.988
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.5/dist-packages/prodigy/__main__.py", line 254, in <module>
controller = recipe(*args, use_plac=True)
File "cython_src/prodigy/core.pyx", line 152, in prodigy.core.recipe.recipe_decorator.recipe_proxy
File "/home/damiano/.local/lib/python3.5/site-packages/plac_core.py", line 328, in call
cmd, result = parser.consume(arglist)
File "/home/damiano/.local/lib/python3.5/site-packages/plac_core.py", line 207, in consume
return cmd, self.func(*(args + varargs + extraopts), **kwargs)
File "/usr/local/lib/python3.5/dist-packages/prodigy/recipes/ner.py", line 428, in batch_train
model_to_bytes = model.to_bytes()
File "cython_src/prodigy/models/ner.pyx", line 401, in prodigy.models.ner.EntityRecognizer.to_bytes
File "cython_src/prodigy/models/ner.pyx", line 402, in prodigy.models.ner.EntityRecognizer.to_bytes
File "/usr/local/lib/python3.5/dist-packages/spacy/language.py", line 675, in to_bytes
return util.to_bytes(serializers, exclude)
File "/usr/local/lib/python3.5/dist-packages/spacy/util.py", line 485, in to_bytes
serialized[key] = getter()
File "/usr/local/lib/python3.5/dist-packages/spacy/language.py", line 665, in <lambda>
('vocab', lambda: self.vocab.to_bytes()),
File "vocab.pyx", line 402, in spacy.vocab.Vocab.to_bytes
File "/usr/local/lib/python3.5/dist-packages/spacy/util.py", line 485, in to_bytes
serialized[key] = getter()
File "vocab.pyx", line 395, in spacy.vocab.Vocab.to_bytes.deserialize_vectors
File "vectors.pyx", line 409, in spacy.vectors.Vectors.to_bytes
File "/usr/local/lib/python3.5/dist-packages/spacy/util.py", line 485, in to_bytes
serialized[key] = getter()
File "vectors.pyx", line 404, in spacy.vectors.Vectors.to_bytes.serialize_weights
File "/home/damiano/.local/lib/python3.5/site-packages/msgpack_numpy.py", line 171, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._packer.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._packer.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._packer.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._packer.Packer._pack
MemoryError