error while training a new entity type ner.teach

[ameyn21]

I tried the commands to train a new entity type in Prodigy on the trial VM provided.

All commands worked fine except the below one,

prodigy ner.teach drug_ner en_core_web_md train --loader Reddit --label DRUG --patterns drugs_patterns1.jsonl

The error thrown is,

Using 1 labels: DRUG
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/prodigy/pgy-env/lib/python3.6/site-packages/prodigy/main.py", line 380, in
controller = recipe(args, use_plac=True)
File "cython_src/prodigy/core.pyx", line 212, in prodigy.core.recipe.recipe_decorator.recipe_proxy
File "/home/prodigy/pgy-env/lib/python3.6/site-packages/plac_core.py", line 328, in call
cmd, result = parser.consume(arglist)
File "/home/prodigy/pgy-env/lib/python3.6/site-packages/plac_core.py", line 207, in consume
return cmd, self.func((args + varargs + extraopts), **kwargs)
File "/home/prodigy/pgy-env/lib/python3.6/site-packages/prodigy/recipes/ner.py", line 117, in teach
source, api=api, loader=loader, rehash=True, dedup=True, input_key="text"
File "cython_src/prodigy/components/loaders.pyx", line 65, in prodigy.components.loaders.get_stream
File "cython_src/prodigy/components/loaders.pyx", line 252, in prodigy.components.loaders.Reddit.init
OSError: Can't find file path: train

Could anyone please guide ?

[ines (Ines Montani)]

Hi! The argument after the model name is the text source you want to load in and annotate. In the tutorial video, we've used a directory called train containing the Reddit data. From the error, it looks like you're trying to load data from a path train, but that path doesn't exist.

I've explained this in more detail here:

@ines

Sorry if this was confusing in the video – I’m copying over my reply from this thread, which asked the same question:

The fourth argument of the command after the dataset and the model is the data source , i.e. the texts you’re loading in. So the train in the command above is the path to the data we’re loading in – for training, we’ve created a directory /train containing the data files. Iin the video, you’ll see that it’s underlined, because it points to a directory.) Here’s a more explicit version of the command:

ner.teach drugs_ner en_core_web_lg /path/to/reddit/data --loader reddit --label DRUG --patterns drug_patterns.jsonl

The data loaded in is a portion of the Reddit Comments Corpus, which you can download for free. The built-in Reddit loader in Prodigy is available via --loader reddit and can take either a single .bz2 archive (the format the corpus is shipped in), or a directory containing multiple archives, which are then loaded in order.

If you’re following the video example, note that we’ve pre-processed the Reddit data and divided it into a training set, an evaluation set and a test set (to make sure we can actually evaluate the model properly). We’ve also extracted only comments from /r/opiates . However, if you want to try out a similar approach with a broader category (like, slang terms or technology companies or whatever else you come up with), you can also easily stream in texts from all subreddits.

Hope this helps!