# data-to-spacy command error

**URL:** <https://support.prodi.gy/t/data-to-spacy-command-error/2399>\
**Category:** Uncategorized\
**Tags:** spacy\
**Created:** [January 7, 2020, 2:06pm UTC](https://support.prodi.gy/t/data-to-spacy-command-error/2399 "2020-01-07T14:06:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jyothi](https://avatars.discourse-cdn.com/v4/letter/j/5f9b8f/32.png) [@Jyothi](https://support.prodi.gy/u/Jyothi)\
**Post date:** [January 7, 2020, 2:06pm UTC](https://support.prodi.gy/t/data-to-spacy-command-error/2399/1 "2020-01-07T14:06:14Z")

</div>

Hi,  
I have recently upgraded my python to 1.9.3 and using spacy 2.2.3. I used to use convert command to get spacy format to train.

**python -m spacy convert -l en -t json -c jsonl prodigy-data.jsonl /spacy\_dir**

After installed new version of prodigy, used data-to-spacy command to get spacy format.

**python -m prodigy data-to-spacy .\spacy\_dataset\train-data.json .\spacy\_dataset\eval-data.json --lang en --ner data\_shuffled\_cleaned**

data\_shuffled\_cleaned is the dataset which I was successful with prodigy train CLI with 90% overall accuracy. The same dataset I am trying to use for spacy train CLI.

So i ran spacy train command but getting below error. May I know why?

**python -m spacy train en model train-data.json eval-data.json --pipeline ner -v en\_vectors\_web\_lg --verbose**

Traceback (most recent call last):

File "\*\*\*\*\*\*\*\*\*\*\*\*\*/runpy.py", line 193, in \_run\_module\_as\_main

" **main**", mod\_spec)

File "\*\*\*\*\*\*\*\*\*\*\*\*/python3/lib/python3.6/runpy.py", line 85, in \_run\_code

exec(code, run\_globals)

File "\*\*\*\*\*\*\*/python3/lib/python3.6/site-packages/spacy/ **main**.py", line 33, in

[plac.call](https://protect-de.mimecast.com/s/PuSYCMZkWkT5pA0VcJrT25?domain=plac.call)(commands[command], sys.argv[1:])

File "\*\*\*\*\*\*\*\*\*\*\*/python3/lib/python3.6/site-packages/plac\_core.py", line 367, in call

cmd, result = parser.consume(arglist)

File \*\*\*\*\*\*\*/python3/lib/python3.6/site-packages/plac\_core.py", line 232, in consume

return cmd, self.func(\*(args + varargs + extraopts), \*\*kwargs)

File "\*\*\*\*\*\*\*\*\*\*\*\*/python3/lib/python3.6/site-packages/spacy/cli/train.py", line 230, in train

corpus = GoldCorpus(train\_path, dev\_path, limit=n\_examples)

File "gold.pyx", line 224, in spacy.gold.GoldCorpus. **init**

File "gold.pyx", line 235, in spacy.gold.GoldCorpus.write\_msgpack

File "gold.pyx", line 280, in read\_tuples

File "gold.pyx", line 545, in read\_json\_file

File "gold.pyx", line 592, in \_json\_iterate

OverflowError: value too large to convert to int

---

<div class="post-metadata">

**Author:** ![adriane](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@adriane](https://support.prodi.gy/u/adriane)\
**Post date:** [January 8, 2020, 4:44pm UTC](https://support.prodi.gy/t/data-to-spacy-command-error/2399/2 "2020-01-08T16:44:20Z")

</div>

Sorry about that! That's a known bug when the number of documents is too large for an `int`. It will be fixed in the next version of spacy.

The easiest solution with spacy v2.2.3 is to split your training file into multiple `.json` files. If you provide a directory instead of a file as the train or dev path, `spacy train` will find all the `.json` files within that directory (also in all the subdirectories).