# Prodigy annotations from older from to newer version **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, spacy, solved\ **Created:** [January 14, 2020, 6:40am UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420 "2020-01-14T06:40:46Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Zainpann](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zainpann/32/1150_2.png) [@Zainpann](https://support.prodi.gy/u/Zainpann)\ **Post date:** [January 14, 2020, 6:40am UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420/1 "2020-01-14T06:40:46Z")
Recently I resubscribed Prodigy,I used to have prodigy 1.7.1. I export the annotations from my older prodigy version and insert(db-in) that jsonl file into prodigy newer version(1.9.5). The problem is I am unable to batch train the NER in this newer version. and geeting the below error: Prodigy now comes with a new general-purpose `train` command that supports all components and can be used with binary accept/reject annotations by setting the --binary flag. It also features an improved training loop and more detailed per-entity-type results. Give it a try! ℹ Loaded model en\_core\_web\_sm Using 50% of examples (12948) for evaluation Using 100% of remaining examples (15666) for training Dropout: 0.2 Batch size: 16 Iterations: 10 BEFORE 0.009 Correct 2071 Incorrect 217049 Entities 33920 Unknown 31848 # LOSS RIGHT WRONG ENTS SKIP ACCURACY Traceback (most recent call last): File "C:\Users\BNV\AppData\Local\Programs\Python\Python36\Lib\runpy.py", line 193, in \_run\_module\_as\_main " **main**", mod\_spec) File "C:\Users\BNV\AppData\Local\Programs\Python\Python36\Lib\runpy.py", line 85, in _run\_code exec(code, run\_globals) File "C:\Users\BNV\Envs\spacy221\lib\site-packages\prodigy\_main_.py", line 60, in controller = recipe(_args, use\_plac=True) File "cython\_src\prodigy\core.pyx", line 213, in prodigy.core.recipe.recipe\_decorator.recipe\_proxy File "C:\Users\BNV\Envs\spacy221\lib\site-packages\plac\_core.py", line 328, in call cmd, result = parser.consume(arglist) File "C:\Users\BNV\Envs\spacy221\lib\site-packages\plac\_core.py", line 207, in consume return cmd, self.func(_(args + varargs + extraopts), \*\*kwargs) File "C:\Users\BNV\Envs\spacy221\lib\site-packages\prodigy\deprecated\train.py", line 143, in ner\_batch\_train examples, batch\_size=batch\_size, drop=dropout, beam\_width=beam\_width File "cython\_src\prodigy\models\ner.pyx", line 351, in prodigy.models.ner.EntityRecognizer.batch\_train File "cython\_src\prodigy\models\ner.pyx", line 443, in prodigy.models.ner.EntityRecognizer.\_update File "cython\_src\prodigy\models\ner.pyx", line 436, in prodigy.models.ner.EntityRecognizer.\_update File "cython\_src\prodigy\models\ner.pyx", line 437, in prodigy.models.ner.EntityRecognizer.\_update File "C:\Users\BNV\Envs\spacy221\lib\site-packages\spacy\language.py", line 515, in update proc.update(docs, golds, sgd=get\_grads, losses=losses, \*\*kwargs) File "nn\_parser.pyx", line 445, in spacy.syntax.nn\_parser.Parser.update File "nn\_parser.pyx", line 550, in spacy.syntax.nn\_parser.Parser.\_init\_gold\_batch File "transition\_system.pyx", line 95, in spacy.syntax.transition\_system.TransitionSystem.get\_oracle\_sequence File "transition\_system.pyx", line 156, in spacy.syntax.transition\_system.TransitionSystem.set\_costs ValueError: [E024] Could not find an optimal move to supervise the parser. Usually, this means that the model can't be updated in a way that's valid and satisfies the correct annotations specified in the GoldParse. For example, are all labels added to the model? If you're training a named entity recognizer, also make sure that none of your annotated entity spans have leading or trailing whitespace. You can also use the experimental `debug-data` command to validate your JSON-formatted training data. For details, run: python -m spacy debug-data --help ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 14, 2020, 11:28am UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420/2 "2020-01-14T11:28:28Z")
Hi! The main difference between spaCy v2.0 and v2.1 is the handling of whitespace in entities. This used to be an issue and could cause significantly worse results, because the entity recognizer was allowed to predict newlines and whitespace tokens as entities. So if your data contains spans starting with / consisting of only whitespace, that's the one thing you need to fix and filter out. See these threads for details: > [@ner.batch-train after ner.maual results error (Value error : \[E024\])](https://support.prodi.gy/t/ner-batch-train-after-ner-maual-results-error-value-error-e024/1677/2): > > Hi! Did you use the same Prodigy version for annotation and training, or did you collect the annotations in a previous version? It’s likely that this is related to the update to spaCy v2.1, which is stricter about gold standard data and constraints for the parser and named entity recognizer. See my reply from [this thread](https://support.prodi.gy/t/recipe-ner-batch-train-results-in-valueerror-e030/1555/6): So you might want to double-check the data and see if you have any “illegal” spans in there. It’s usually pretty rare and removing them should be no problem, because in most… > [@Recipe ner.batch-train results in ValueError: \[E030\]](https://support.prodi.gy/t/recipe-ner-batch-train-results-in-valueerror-e030/1555/6): > > The --unsegmented flag only means that Prodigy won’t apply the sentence segmenter to split your texts into sentences. If your examples are already pre-segmented, this is fine – but if your data contains lots of really long texts, you probably want to split them, because otherwise training may be slow and the long texts may throw off the model. So it should be fine in your case. Ahh, I meant examples that consist of only one token. So basically, where "text" has only one word. Do you find any … ---
**Author:** ![Zainpann](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zainpann/32/1150_2.png) [@Zainpann](https://support.prodi.gy/u/Zainpann)\ **Post date:** [January 15, 2020, 1:06pm UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420/3 "2020-01-15T13:06:15Z")
I think the problem is different: I first just db-in "n1.jsonl" in a new dataset, no error with the training. Then I db-in "n2.jsonl" in another new dataset, no error with training. then I db-in n2.jsonl in the first dataset, then there is same ValueError: [E024].So Whenever I am doing "db-in" those 2 jsonl file in a single dataset, I am facing the error. Note:Those two jsonl were produced using prodigy 1.7, but I am facing issues while retraining it on prodigy 1.9.5. Please help me with this issue ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 15, 2020, 1:55pm UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420/4 "2020-01-15T13:55:45Z")
How are you evaluating? If you're not using a dedicated evaluation set (e.g. via `--eval-id`), Prodigy will hold back a random sample from the dataset. So if there's one problematic example in one of your datasets, it may end up in the evaluation set and the model won't be updated with it (and you won't get an error). But when you combine the examples, the held-back evaluation data will be different and the problematic example might end up in the training set. You can test this by using the same dataset (or any other dataset, really) for evaluation and training on the full data. This should always give you an error. ---
**Author:** ![Zainpann](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zainpann/32/1150_2.png) [@Zainpann](https://support.prodi.gy/u/Zainpann)\ **Post date:** [January 16, 2020, 5:04am UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420/5 "2020-01-16T05:04:52Z")
okay if that's an issue then I am having too many(almost 1000 out of 41000) examples like the one below in my json file, is there any script which can handle this, i tried the one you mentioned in above reply but it didnt work. Example: {"text":" ","start":83,"end":84,"id":16} ---
**Author:** ![Zainpann](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/zainpann/32/1150_2.png) [@Zainpann](https://support.prodi.gy/u/Zainpann)\ **Post date:** [January 16, 2020, 7:51am UTC](https://support.prodi.gy/t/prodigy-annotations-from-older-from-to-newer-version/2420/6 "2020-01-16T07:51:46Z")
I solved this issue by removing the script which removes the promblematic entities, by running this script one by one on every jsonl file resolves the issue.