# How to convert prodigy dataset to .spacy object? **URL:** \ **Category:** Uncategorized\ **Tags:** usage, spacy, solved\ **Created:** [November 25, 2020, 3:43am UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662 "2020-11-25T03:43:04Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![yw2903](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/yw2903/32/1614_2.png) [@yw2903](https://support.prodi.gy/u/yw2903)\ **Post date:** [November 25, 2020, 3:43am UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/1 "2020-11-25T03:43:04Z")
I'm trying to use the new spacy-nightly training pipeline. How can I convert a prodigy dataset to .spacy object? Thanks, ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [November 25, 2020, 3:55am UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/2 "2020-11-25T03:55:27Z")
You can use Prodigy's `data-to-spacy` to conver one or more datasets to a JSON training file and then convert that to `.spacy` using `spacy convert`. Also see here: - [https://nightly.spacy.io/api/cli#convert](https://nightly.spacy.io/api/cli#convert) - [https://nightly.spacy.io/usage/projects#prodigy](https://nightly.spacy.io/usage/projects#prodigy) Under the hood, the `.spacy` format is just a serialized `DocBin` object, so for full flexibility, you can also create `Doc` objects from your existing annotations and add them to a `DocBin`. See here for details: [https://nightly.spacy.io/api/docbin](https://nightly.spacy.io/api/docbin) ---
**Author:** ![do12siwu](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/do12siwu/32/3090_2.png) [@do12siwu](https://support.prodi.gy/u/do12siwu)\ **Post date:** [November 24, 2022, 6:02pm UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/3 "2022-11-24T18:02:07Z")
Hello guys, I have a problem to convert the dataset to .spacy. I labelled the data with this code: python -m prodigy rel.manual koText en\_core\_web\_sm koText.json --label INVESTS,RECEIVES,ACQUIRES --span-label ORG,MONEY,PERSON,REGEX --add-ents –wrap. And now I wanted to convert everything to spacy with this code: python -m prodigy data-to-spacy ./hp --ner allnoTitles --eval-split 0 (The new merged dataset is called allnoTitles) Now I get this error: ============================== Generating data ============================== Components: ner Merging training and evaluation data for 1 components - [ner] Training: 1715 | Evaluation: 0 (0% split) ✘ Invalid data for component 'ner' spans -\> 16 -\> start field required spans -\> 16 -\> end field required Why does it not work? Kind regards. ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [November 25, 2022, 9:19pm UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/4 "2022-11-25T21:19:36Z")
Hi @do12siwu! It's likely something is wrong with the data. Check out these related posts: > [@rel.manual to train ner and dependency](https://support.prodi.gy/t/rel-manual-to-train-ner-and-dependency/3091/2): > > Hi! That's strange – this would indicate that somewhere in your data, it ended up with a span that doesn't specify a start and end thinking Are you able to find this example in your data and if so, can you share what it looks like? > [@rel.manual training: Invalid data for component 'ner' & ValueError: Could not find gold transition](https://support.prodi.gy/t/rel-manual-training-invalid-data-for-component-ner-valueerror-could-not-find-gold-transition/5768): > > I created a dataset with the joint entity and relation annotations (rel.manual). I attempted to train a model with the dataset I created via rel.manual recipe and received the error shown below. I assume the span labels should be creating valid data for the 'ner' component, so unsure what could be causing this issue. Would appreciate any support, thanks! =========================== Initializing pipeline =========================== [2022-07-01 20:28:54,391] [INFO] Set up nlp object from config … They seemed to find cleaning up annotations helped and then found a related issue dealing with their config file. Could you try to export the annotations `db-out` to `.jsonl` to inspect and find the span? Perhaps try to remove it. Let us know if this helps! ---
**Author:** ![do12siwu](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/do12siwu/32/3090_2.png) [@do12siwu](https://support.prodi.gy/u/do12siwu)\ **Post date:** [November 26, 2022, 12:14pm UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/5 "2022-11-26T12:14:22Z")
Thanks a lot. I just used this code from Ines. > [@No start and end of span using data-to-spacy after rel.manual](https://support.prodi.gy/t/no-start-and-end-of-span-using-data-to-spacy-after-rel-manual/3376/2): > > Hi! Do you know which version of Prodigy you created the annotations with? The same version you're currently using? It seems like for some reason, you ended up with an invalid span here. The easiest way to find and exclude it would be to just go over your data, check the spans and if they include a start/end and only keep the valid spans in a new dataset: from prodigy.components.db import connect db = connect() examples = db.get\_dataset("myDataset2") filtered\_examples = [] for eg in examples:… It worked! ---
**Author:** ![do12siwu](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/do12siwu/32/3090_2.png) [@do12siwu](https://support.prodi.gy/u/do12siwu)\ **Post date:** [January 13, 2023, 10:37am UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/6 "2023-01-13T10:37:08Z")
Hi guys, I converted the dataset to spacy with this code: python -m prodigy data-to-spacy ./hp --ner allnoTitles\_filtered --eval-split 0 And before that, I annotated the dataset with this code: python -m prodigy rel.manual koText en\_core\_web\_sm koText.json --label INVESTS,RECEIVES,ACQUIRES --span-label ORG,MONEY,PERSON,REGEX --add-ents –wrap. The problem is that the .spacy file contain the span-label ORG,MONEY,PERSON,REGEX, but not the relations INVESTS,RECEIVES,ACQUIRES. I already checked the documentation. What am I doing wrong? Kind regards ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [January 13, 2023, 6:04pm UTC](https://support.prodi.gy/t/how-to-convert-prodigy-dataset-to-spacy-object/3662/7 "2023-01-13T18:04:09Z")
hi @do12siwu! > [@do12siwu](#): > > The problem is that the .spacy file contain the span-label ORG,MONEY,PERSON,REGEX, but not the relations INVESTS,RECEIVES,ACQUIRES. This post explains that `relations` annotations aren't part of `data-to-spacy`: > [@prodigy data-to-spacy for relation extraction](https://support.prodi.gy/t/prodigy-data-to-spacy-for-relation-extraction/5620/2): > > At the time of writing, spaCy doesn't natively support relation extraction models. The example that we list on our docs [here](https://spacy.io/usage/layers-architectures#component-rel) is meant to be a tutorial on how to set up a custom component, not a guide on a feature in spaCy. The crux of the issue is that the Doc object in spaCy currently has no support for relationships. That is also why, in turn, the .spacy object does not support them. The config file that you see can be changed via the --config flag ([docs](https://prodi.gy/docs/recipes#data-to-spacy)). If this flag is not set, which is y…