# Reproducing prodigy ner.batch-train in spacy: cross-validation results and outputted model **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner\ **Created:** [September 13, 2018, 7:54pm UTC](https://support.prodi.gy/t/reproducing-prodigy-ner-batch-train-in-spacy-cross-validation-results-and-outputted-model/815 "2018-09-13T19:54:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![timchange](https://avatars.discourse-cdn.com/v4/letter/t/ed655f/32.png) [@timchange](https://support.prodi.gy/u/timchange)\ **Post date:** [September 13, 2018, 7:54pm UTC](https://support.prodi.gy/t/reproducing-prodigy-ner-batch-train-in-spacy-cross-validation-results-and-outputted-model/815/1 "2018-09-13T19:54:35Z")
For a few reasons (such as to facilitating hyper-parameter optimization) I’d like to move from training NER models in Prodigy to Spacy. I’m wondering how I can reproduce the prodigy ner.batch-train results directly in spacy (such as things like cross-validated accuracy). I have a few questions related to this issue. 1. Model evaluation in Prodigy: Based on prodigy documentation, it isn’t clear to me exactly what is being used for model training and evaluation in `ner.batch-train`. As an example, for one model, the following is printed to console during ner.batch-train, but the numbers don’t exactly add up precisely to anything in the database that I can figure out: This output is for a databse with 150 examples labeled via `manual` and 1350 examples labeled via `teach` (715 of which were accepted). E.g., 50% of the accept/reject examples does not equal 383. The numbers are closer (but don’t seem exact), if I assume that items in the db with the same \_input\_hash are being merged. If possible, can you clarify exactly what’s being used for both the training and evaluation here? - Are data from both ‘manual’ and ‘teach’ being used during training? During evaluation? - Are both positive and negative (accept and reject) examples being used during training? During evaluation? - Are rows with matching \_input\_hash values combined? For instances with multiple accepted entity spans, is the model treated as correct only if it identifies all labeled entities? 1. Training models from Prodigy db’s, in Spacy: I’ve seen how to export prodigy data to a format for spacy here: [Mixing in gold data to avoid catastrophic forgetting](https://support.prodi.gy/t/mixing-in-gold-data-to-avoid-catastrophic-forgetting/126/3) and it seems relatively easy to modify that code so that I can transform examples highlighted via the `manual` method. Given the reformatted data, will updating an NER model (either starting from a blank model or a pretrained model) with the method given here: [https://spacy.io/usage/training#example-train-ner](https://spacy.io/usage/training#example-train-ner) or here: [https://spacy.io/usage/training#training-simple-style](https://spacy.io/usage/training#training-simple-style) approximately replicate what prodigy is doing during ner.batch-train? ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [September 14, 2018, 10:28pm UTC](https://support.prodi.gy/t/reproducing-prodigy-ner-batch-train-in-spacy-cross-validation-results-and-outputted-model/815/2 "2018-09-14T22:28:12Z")
It's definitely a good idea to move the data out form Prodigy and experiment with general-purpose NLP tools at some point. Obviously we typically use spaCy, but other tools can be useful as well, especially for text classification. So, the data export is definitely something we care about. However, there are a couple of subtleties to keep in mind. > [@timchange](#): > > - Are data from both ‘manual’ and ‘teach’ being used during training? During evaluation? Yes, although mixing data from the two modes is a bit awkward currently. The problem is that the `batch-train` command currently takes a flag, `--no-missing`, that tells it about the nature of the data: is the data fully annotated (like the data from manual), or does the data have entities which may be correct, but are not annotated (like the data from `ner.teach`)? This flag is set over the whole dataset, so you can't get optimal results from mixing the two types of example currently. > [@timchange](#): > > - Are both positive and negative (accept and reject) examples being used during training? During evaluation? Yes, in both training and evaluation, for both types of data (manual and teach). > [@timchange](#): > > - Are rows with matching \_input\_hash values combined? For instances with multiple accepted entity spans, is the model treated as correct only if it identifies all labeled entities? We do merge annotations from the same example together, using the input hash. Each entity is marked as correct or incorrect during evaluation --- evaluation isn't over the single text. > [@timchange](#): > > Given the reformatted data, will updating an NER model (either starting from a blank model or a pretrained model) with the method given here: > [Training Pipelines & Models · spaCy Usage Documentation](https://spacy.io/usage/training#example-train-ner) > or here: > [Training Pipelines & Models · spaCy Usage Documentation](https://spacy.io/usage/training#training-simple-style) > approximately replicate what prodigy is doing during ner.batch-train? For data from `manual` yes, but for data from `ner.teach` --- not quite! Prodigy implements a custom algorithm for training from the binary annotations produced by `ner.teach`. spaCy assumes that the data has no missing labels, which isn't true for the binary annotations. The best solution is to use the `ner.print-best` command, which will run the parser over your `ner.teach` dataset, and use your annotations to find the best parse. During training, this best parse would be marked as the gold-standard (even though it might be incorrect). If you output these best parses and use them to train spaCy, you'll be doing much the same thing as the Prodigy training algorithm. An even better approach is to pipe the output of `ner.print-best` into the `ner.manual` recipe. This lets you correct any errors in the data, effectively up-converting the `ner.teach` data into normal gold-standard annotations, which are much easier to use in other tools. ---
**Author:** ![timchange](https://avatars.discourse-cdn.com/v4/letter/t/ed655f/32.png) [@timchange](https://support.prodi.gy/u/timchange)\ **Post date:** [October 3, 2018, 10:08pm UTC](https://support.prodi.gy/t/reproducing-prodigy-ner-batch-train-in-spacy-cross-validation-results-and-outputted-model/815/3 "2018-10-03T22:08:51Z")
First of all–thanks for the detailed response! I tried using the ner.print-best method, as you suggested, to produce a dataset that could then be passed to spacy for training. But the outputs produced by the print-best method seemed very problematic. Perhaps I’m doing it wrong though? This is what I have done: 1. Created a labeled dataset in prodigy with about 1500 examples for a single new entity (200 using ner.manual and 1300 using ner.teach). 2. I fit two models: (a) was trained starting from the ‘en\_core\_web\_sm’ model, and (b) was trained starting from a blank\_english model I generated via spacy. In both cases I used this command to train the model, changing the initial spacy model as appropriate: `prodigy ner.batch-train new_entity en_core_web_sm --output new_entity_model --dropout 0.5 --n-iter 25 -U` (3) I then use print-best as follows: `prodigy ner.print-best pulse models/pulse_1 > new_entity_print_best.jsonl` The outputs from the print-best seem very poor (despite prodigy saying it is getting upwards of 90% accuracy during it’s internal cross-validation procedure, independent of which spacy\_model I start with). - For the model trained starting from a blank\_english model, the model way over-predicts the new entity (often assigning the label 3 or more times within my single-sentence excerpts, that have at most 1 instance of the label). - For the model trained starting from en\_core\_web\_sm, the new entity seems to be applied for judiciously, but the default ent-labels in en\_core\_web\_sm get applied all over the place, despite it not being relevant to our texts (e.g., “WORK\_OF\_ART” is applied on average 5x per excerpt) . I’m wondering if you know what might be going on somehow? And is it possible that the cross-validation is somehow (especially when training from a blank model) ignoring False positives that the model is producing. Maybe this happens because it isn’t assuming the our model has `gold-standard` labels, so it can’t assume that the new\_entity doesn’t apply to the texts except when an explicit “reject” response has been given to that highlighted section? ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [October 5, 2018, 10:34am UTC](https://support.prodi.gy/t/reproducing-prodigy-ner-batch-train-in-spacy-cross-validation-results-and-outputted-model/815/4 "2018-10-05T10:34:34Z")
> [@timchange](#): > > Maybe this happens because it isn’t assuming the our model has `gold-standard` labels, so it can’t assume that the new\_entity doesn’t apply to the texts except when an explicit “reject” response has been given to that highlighted section? I'd say that's very likely to be the problem. Try adding the `--no-missing` argument to the batch training. The model directory also has jsonl files with the training and evaluation split data, so you can pipe exactly that data through the model to see why it thinks it's getting 90% accuracy.