# jsonl loading question **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, solved\ **Created:** [February 28, 2019, 7:04am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252 "2019-02-28T07:04:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ByronJames](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/byronjames/32/676_2.png) [@ByronJames](https://support.prodi.gy/u/ByronJames)\ **Post date:** [February 28, 2019, 7:04am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/1 "2019-02-28T07:04:33Z")
Hi, New-ish to Prodigy, not very new to Python, not new at all to working with data and code. That aside, I’m having a spot of trouble with getting Prodigy to load my JSON file: [dsample.jsonl](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/1X/b6d9dd67157810307a1e52694b249d9c8cc1b31e.jsonl) (19.7 KB) using: `prodigy ner.teach mc_apad_listening en_core_web_lg /Users/User/Desktop/myfreshcorpus/input/dsample.jsonl` error: 15:38:29 - Task queue depth is 1 15:38:29 - Task queue depth is 2 15:38:29 - Exception when serving /get\_questions Traceback (most recent call last): File “cython\_src/prodigy/components/loaders.pyx”, line 145, in prodigy.components.loaders.JSONL ValueError: Expected object or value During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/anaconda2/lib/python3.7/site-packages/waitress/channel.py”, line 336, in service task.service() File “/anaconda2/lib/python3.7/site-packages/waitress/task.py”, line 175, in service self.execute() File “/anaconda2/lib/python3.7/site-packages/waitress/task.py”, line 452, in execute app\_iter = self.channel.server.application(env, start\_response) File “hug/api.py”, line 423, in hug.api.ModuleSingleton. **call**.api\_auto\_instantiate File “/anaconda2/lib/python3.7/site-packages/falcon/api.py”, line 244, in **call** responder(req, resp, \*\*params) File “hug/interface.py”, line 793, in hug.interface.HTTP. **call** File “hug/interface.py”, line 766, in hug.interface.HTTP. **call** File “hug/interface.py”, line 703, in hug.interface.HTTP.call\_function File “hug/interface.py”, line 100, in hug.interface.Interfaces. **call** File “/anaconda2/lib/python3.7/site-packages/prodigy/app.py”, line 173, in get\_questions tasks = controller.get\_questions() File “cython\_src/prodigy/core.pyx”, line 129, in prodigy.core.Controller.get\_questions File “cython\_src/prodigy/components/feeds.pyx”, line 56, in prodigy.components.feeds.SharedFeed.get\_questions File “cython\_src/prodigy/components/feeds.pyx”, line 61, in prodigy.components.feeds.SharedFeed.get\_next\_batch File “cython\_src/prodigy/components/feeds.pyx”, line 131, in prodigy.components.feeds.SessionFeed.get\_session\_stream File “/anaconda2/lib/python3.7/site-packages/toolz/itertoolz.py”, line 368, in first return next(iter(seq)) File “cython\_src/prodigy/components/sorters.pyx”, line 151, in **iter** File “cython\_src/prodigy/components/sorters.pyx”, line 61, in genexpr File “cython\_src/prodigy/models/ner.pyx”, line 292, in **call** File “cython\_src/prodigy/models/ner.pyx”, line 259, in get\_tasks File “cytoolz/itertoolz.pyx”, line 1047, in cytoolz.itertoolz.partition\_all. **next** File “cython\_src/prodigy/models/ner.pyx”, line 209, in predict\_spans File “cytoolz/itertoolz.pyx”, line 1047, in cytoolz.itertoolz.partition\_all. **next** File “cython\_src/prodigy/components/preprocess.pyx”, line 35, in split\_sentences File “/anaconda2/lib/python3.7/site-packages/spacy/language.py”, line 548, in pipe for doc, context in izip(docs, contexts): File “/anaconda2/lib/python3.7/site-packages/spacy/language.py”, line 572, in pipe for doc in docs: File “nn\_parser.pyx”, line 367, in pipe File “cytoolz/itertoolz.pyx”, line 1047, in cytoolz.itertoolz.partition\_all. **next** File “nn\_parser.pyx”, line 367, in pipe File “cytoolz/itertoolz.pyx”, line 1047, in cytoolz.itertoolz.partition\_all. **next** File “pipeline.pyx”, line 431, in pipe File “cytoolz/itertoolz.pyx”, line 1047, in cytoolz.itertoolz.partition\_all. **next** File “/anaconda2/lib/python3.7/site-packages/spacy/language.py”, line 746, in \_pipe for doc in docs: File “/anaconda2/lib/python3.7/site-packages/spacy/language.py”, line 551, in docs = (self.make\_doc(text) for text in texts) File “/anaconda2/lib/python3.7/site-packages/spacy/language.py”, line 544, in texts = (tc[0] for tc in text\_context1) File “cython\_src/prodigy/components/preprocess.pyx”, line 34, in genexpr File “cython\_src/prodigy/components/filters.pyx”, line 35, in filter\_duplicates File “cython\_src/prodigy/components/filters.pyx”, line 16, in filter\_empty File “cython\_src/prodigy/components/loaders.pyx”, line 22, in \_rehash\_stream File “cython\_src/prodigy/components/loaders.pyx”, line 152, in JSONL ValueError: Failed to load task (invalid JSON). [ … [ Nothing loads - which has me wondering what I’ve missed. ADDENDUM: Tried adding labels (Spacy NER) same result - error loading Prodigy on localhost. The JSONL file is 23MB. Is this a possible issue? Thoughts? Slings? Arrows? Outrageous fortunes? Best, B ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [February 28, 2019, 8:41am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/2 "2019-02-28T08:41:28Z")
Hi! If I read this correctly, the problem might be that your file is actually JSON and not JSONL ([newline-delimited JSON](http://jsonlines.org/))? If you load in JSONL, the data is expected to have one JSON object on each line. So internally, all the loader really does is read in the file line-by-line and call `json.loads()` on each line. This fails for your file, because the first line is `[`, the second line is `{` and so on. So if you just rename your file to `.json`, it should work as expected, because the file will then be parsed as a whole. For large files, this is going to be slow-ish, so you might still prefer to convert it to JSONL. You can write your own little script (it’s really as straightforward as writing `json.dumps(obj) + '\n'` to each line) or use Prodigy’s built-in helper: ```python prodigy.util.write_jsonl('/path/to/file.jsonl', list_of_dicts) ``` Btw, another small thing I noticed when looking at your sample data: Prodigy expects the text to be available as the key `"text"` – otherwise, it won’t know which entry to load from your data. So you probably want to rename one of `"Title"`, `"Snippet"` etc. to `"text"`. You can also add an entry `"meta"`, a dictionary of key/value pairs that will be shown in the bottom right corner of the annotation card. For example, `{"meta": {"link": "http://www.valleyscw.com/..."}}`. ---
**Author:** ![ByronJames](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/byronjames/32/676_2.png) [@ByronJames](https://support.prodi.gy/u/ByronJames)\ **Post date:** [February 28, 2019, 9:32am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/3 "2019-02-28T09:32:46Z")
Hi! I’ll sit down and make the changes and see what happens! Thanks! ---
**Author:** ![ByronJames](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/byronjames/32/676_2.png) [@ByronJames](https://support.prodi.gy/u/ByronJames)\ **Post date:** [February 28, 2019, 9:53am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/4 "2019-02-28T09:53:37Z")
Well look at that! Worked like a charm! This, this is great! B ---
**Author:** ![ronaldcturner](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@ronaldcturner](https://support.prodi.gy/u/ronaldcturner)\ **Post date:** [January 6, 2021, 10:26am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/5 "2021-01-06T10:26:05Z")
Pardon my resurrecting this "dead horse," but I am unable to use **ner.manual** with my JSONL patterns file (value of the **--patterns** argument). Prodigy's error message: ✘ Failed to load task (invalid JSON on line 1) This error pretty much always means that there's something wrong with this line of JSON and Python can't load it. Even if you think it's correct, something must confuse it. Try calling json.loads(line) on each line or use a JSON linter. The Prodigy error message is consistent with your helpful comments above. The JSONL file is my own creation, each line a valid JSON object. So with a Python script (attached) I have attempted to replicate what I _think_ Prodigy does internally with my JSONL patterns file: consume a JSONL, then transform it to to JSON. ![JSON_test](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/0/0555b5966e9bf57ec089ff023ad456b61bf86743.png) . Everything in this "round-trip" bench test appears to work. So I'm clearly still missing something and will appreciate your kind advice. Installations: Python (3.8.6), spaCy (2.3.4) and Prodigy (1.10.5) ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 7, 2021, 2:41am UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/6 "2021-01-07T02:41:32Z")
> [@ronaldcturner](#): > > ✘ Failed to load task (invalid JSON on line 1) Hi! Based on the error message, it seems like this is occuring when Prodigy is trying to load your _input data_, not the patterns file. So it looks like your patterns are fine (and your code snippet is indeed what Prodigy is doing under the hood 🙂) but something in the first line of your input file doesn't parse correctly as JSON. Can you double-check that and see if it passes your manual `json.loads` test? ---
**Author:** ![ronaldcturner](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@ronaldcturner](https://support.prodi.gy/u/ronaldcturner)\ **Post date:** [January 7, 2021, 10:00pm UTC](https://support.prodi.gy/t/jsonl-loading-question/1252/7 "2021-01-07T22:00:33Z")
Thank you Ines once again for kindly supporting me, even with basic (dumb!) errors. The INPUT (source) file should be also a .jsonl (for sensible chunking). but I violated basic JSONL. Both of the enclosing braces ("{" and "}") were on _separate lines_, clearly NOT conforming JSONL.