# Convert spacy binary data to jsonl

**URL:** <https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113>\
**Category:** Uncategorized\
**Tags:** ner, spacy, solved, nightly\
**Created:** [April 8, 2021, 8:48am UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113 "2021-04-08T08:48:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tristan19954](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/tristan19954/32/2079_2.png) [@tristan19954](https://support.prodi.gy/u/tristan19954)\
**Post date:** [April 8, 2021, 8:48am UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113/1 "2021-04-08T08:48:04Z")

</div>

Hello,

I just recently got prodigy, and before that I was using another free program to annotate data for my spacy model. From this program, I could download my annotated data in iob format which I then converted to spacy binary format with the spacy convert command.

I would like to use this data that I already annotated in prodigy, partially to try to run the "train-curve" command to see how much improvement I could still get by adding more data.  
I can't seem to find an easy way to convert a ".spacy", a spacy json file or ".iob" file to jsonl. I found the depreciated "ner.iob-to-gold", so I guess there might be a new recipe for that purpose but I can't find it. I also found the "data-to-spacy" that can do the reverse.

Is there a way to do this?

I'm using the latest version of spacy (3.0.5 I believe) and prodigy nightly.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [April 12, 2021, 12:43am UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113/2 "2021-04-12T00:43:32Z")

</div>

Hi! Under the hood, the binary `.spacy` file is a serialized `DocBin`, which you can always load back as spaCy `Doc` objects: [https://spacy.io/api/docbin](https://spacy.io/api/docbin) This gives you access to the annotations, e.g. the `doc.ents`.

```python
import spacy
from spacy.tokens import DocBin

nlp = spacy.blank("en")
doc_bin = DocBin().from_disk("./data.spacy") # your file here
examples = [] # examples in Prodigy's format
for doc in doc_bin.get_docs(nlp.vocab):
    spans = [{"start": ent.start_char, "end": ent.end_char, "label": ent.label_} for ent in doc.ents]
    examples.append({"text": doc.text, "spans": spans})

```

In your case, this is probably the most straightforward solution because you already have the `.spacy` files and you need them for training anyways.

For completeness (and others who come across this thread later): of course, you'd generally don't need the `.spacy` conversion to use IOB data with Prodigy. Prodigy's format expects entities and other spans to be defined as character offsets. So if you have token-based annotations like IOB or BILUO, you can convert them to offsets – spaCy provides handy utility functions for that: [https://prodi.gy/docs/named-entity-recognition#tip-biluo-offsets](https://prodi.gy/docs/named-entity-recognition#tip-biluo-offsets)

---

<div class="post-metadata">

**Author:** ![tristan19954](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/tristan19954/32/2079_2.png) [@tristan19954](https://support.prodi.gy/u/tristan19954)\
**Post date:** [April 12, 2021, 9:12am UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113/3 "2021-04-12T09:12:07Z")

</div>

Thank you Ines, this worked like a charm!

As a little note to anyone who plans to reuse this script, just replace ent.label with ent.label\_ to get the char instead of the label ID

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [April 13, 2021, 1:28am UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113/4 "2021-04-13T01:28:07Z")

</div>

> [@tristan19954](#):
>
> As a little note to anyone who plans to reuse this script, just replace ent.label with ent.label\_ to get the char instead of the label ID

Ah sorry, that was a typo 😅 Just edited my previous post!

---

<div class="post-metadata">

**Author:** ![milos-cuculovic](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/milos-cuculovic/32/2937_2.png) [@milos-cuculovic](https://support.prodi.gy/u/milos-cuculovic)\
**Post date:** [April 27, 2022, 2:36pm UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113/5 "2022-04-27T14:36:02Z")

</div>

Thanks for the hint @ines . I have a slightly different problem but very much related to your reply.

I have .spacy train and valid files and am trying to reuse my code that was used to work with JSON files (SpaCy v2). So what I was tying:

```python
    nlp = spacy.load("my-model")

    doc_bin = DocBin().from_disk(path_test_data)
    examples = []
    for doc in doc_bin.get_docs(nlp.vocab):
        spans = [{"start": ent.start_char, "end": ent.end_char, "label": ent.label_} for ent in doc.ents]
        examples.append(Example.from_dict(nlp.make_doc(doc.text), spans))

```

However, this part is failing due to:

```python
TypeError: Argument 'example_dict' has incorrect type (expected dict, got list)

```

Initially, my code was working as:

```python
examples = []
    for text, annotations in TEST_DATA:
        examples.append(Example.from_dict(nlp.make_doc(text), annotations))

```

With the TEST\_DATA being a JSON

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![ljvmiranda921](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ljvmiranda921/32/3197_2.png) [@ljvmiranda921](https://support.prodi.gy/u/ljvmiranda921)\
**Post date:** [April 28, 2022, 6:07am UTC](https://support.prodi.gy/t/convert-spacy-binary-data-to-jsonl/4113/6 "2022-04-28T06:07:55Z")

</div>

Hi @milos-cuculovic !

I think you've already posted in the spaCy discussions forum. For posterity, here's the [link](https://github.com/explosion/spaCy/discussions/10717) with the answer.