I have a few tagged corpus combined where half of the annotations do not have “token_start” and “token_end” in the spans. i injest the same jsonl file into dataset. batch-train works in unix. but it breaks with the following error in Mac.
Loaded model vector_model_boem
Using 20% of accept/reject examples (361) for evaluation
Traceback (most recent call last):
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/prodigy/main.py”, line 259, in
controller = recipe(args, use_plac=True)
File “cython_src/prodigy/core.pyx”, line 253, in prodigy.core.recipe.recipe_decorator.recipe_proxy
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/plac_core.py”, line 328, in call
cmd, result = parser.consume(arglist)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/plac_core.py”, line 207, in consume
return cmd, self.func((args + varargs + extraopts), **kwargs)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/prodigy/recipes/ner.py”, line 426, in batch_train
examples = list(split_sentences(model.orig_nlp, examples))
File “cython_src/prodigy/components/preprocess.pyx”, line 52, in split_sentences
KeyError: ‘token_start’
Is it possible that you have a different (older) version installed on macOS? Try running prodigy stats and compare the version, and updgrade your macOS if necessary 
Thank you. I am running it on High Sierra. I will update it 
Sorry, I just realised my answer was phrased badly – I mean “upgrade your Prodigy installation on macOS if necessary”!
My prodigy version is : Darwin-17.7.0-x86_64-i386-64bit
But i am running it in High Sierra. probably its time to upgrade my os too.
Your macOS version should be fine – but I think you might be running an older version of Prodigy and not the latest, v1.6.1 on your Mac. That’s why you’re seeing this error on Mac and not on your other machine, which is likely running a different Prodigy version.
Yes, its 1.6.0. Thank you for explaining. I will update it.
Thanks @ines the latest version works.