# Using Loaders **URL:** \ **Category:** Uncategorized\ **Tags:** usage, solved\ **Created:** [February 25, 2018, 3:02pm UTC](https://support.prodi.gy/t/using-loaders/340 "2018-02-25T15:02:45Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![marchdown](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/marchdown/32/206_2.png) [@marchdown](https://support.prodi.gy/u/marchdown)\ **Post date:** [February 25, 2018, 3:02pm UTC](https://support.prodi.gy/t/using-loaders/340/1 "2018-02-25T15:02:45Z")
Hello! I’m trying to follow the video tutorial ([https://prodi.gy/docs/video-new-entity-type](https://prodi.gy/docs/video-new-entity-type)), and at around 13:10 mark there Matthew runs a command `prodigy ner.teach drugs_ner en_core_web_lg train --loader reddit --label DRUG --patterns drug_patterns.jsonl`. It fails with my setup. It seems to be choking on `test` in the command. I can’t find the docs that clarify what’s it for. Maybe the issue is that I don’t have the reddit corpus installed locally, again I couldn’t find the relevant documentation: where should I put it to make it available to Prodigy? How do I use `--loader` at all? Thanks. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [February 25, 2018, 3:15pm UTC](https://support.prodi.gy/t/using-loaders/340/2 "2018-02-25T15:15:56Z")
Sorry if this was a little confusing in the video! The fourth argument of the command after the dataset and the model is the **data source** , i.e. the texts you’re loading in. So the `train` in the command above is the path to the data we’re loading in – for training, we’ve created a directory `/train` containing the data files. Iin the video, you’ll see that it’s underlined, because it points to a directory.) Here’s a more explicit version of the command: ```bash ner.teach drugs_ner en_core_web_lg /path/to/reddit/data --loader reddit --label DRUG --patterns drug_patterns.jsonl ``` The data loaded in is a portion of the [Reddit Comments Corpus](https://archive.org/details/2015_reddit_comments_corpus), which you can download for free. The built-in Reddit loader in Prodigy is available via `--loader reddit` and can take either a single `.bz2` archive (the format the corpus is shipped in), or a directory containing multiple archives, which are then loaded in order. If you’re following the video example, note that we’ve pre-processed the Reddit data and divided it into a training set, an evaluation set and a test set (to make sure we can actually evaluate the model properly). We’ve also extracted only comments from `/r/opiates`. However, if you want to try out a similar approach with a broader category (like, slang terms or technology companies or whatever else you come up with), you can also easily stream in texts from all subreddits. Hope this helps! ---
**Author:** ![marchdown](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/marchdown/32/206_2.png) [@marchdown](https://support.prodi.gy/u/marchdown)\ **Post date:** [February 25, 2018, 3:20pm UTC](https://support.prodi.gy/t/using-loaders/340/3 "2018-02-25T15:20:59Z")
Thank you, it does help a lot! How can I see a list of all the available loaders? I have my data as a list text files in a flat directory, I’ve reformatted it as JSONL for the first example in the video. Neither of those works without specifying a loader. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [February 25, 2018, 3:34pm UTC](https://support.prodi.gy/t/using-loaders/340/4 "2018-02-25T15:34:47Z")
You can find an overview of the data and live API loaders [on the website](https://prodi.gy/docs/#files) or in your `PRODIGY_README.html`, available for download with Prodigy. The README also includes more detailed API docs for each individual loader, including how to call it from Python in your own recipe scripts. > [@marchdown](#): > > I have my data as a list text files in a flat directory, I’ve reformatted it as JSONL for the first example in the video. Neither of those works without specifying a loader. Except for the Reddit and Images loader, Prodigy currently expect the input data to be single files – so maybe that was the problem? Alternatively, you can also check the "Input formats" section of the README, which lists the expected formats of the individual file types supported by Prodigy. If you don't specify a `--loader` on the command line, the appropriate loader will be selected based on the file extension. So if your file is called `data.jsonl`, Prodigy should use the JSONL loader. (Btw, there's currently an [open feature request](https://support.prodi.gy/t/feature-request-directories-of-text-files-as-a-source-format/186) for allowing data loading from directories – if this would be useful to you as well, we could make this happen for a future release.) ---
**Author:** ![vatsala](https://avatars.discourse-cdn.com/v4/letter/v/35a633/32.png) [@vatsala](https://support.prodi.gy/u/vatsala)\ **Post date:** [November 12, 2018, 2:26pm UTC](https://support.prodi.gy/t/using-loaders/340/5 "2018-11-12T14:26:47Z")
Sorry to say but the ‘Prodigy tutorial’ IS HIGHLY CONFUSING - I am spending most of my time FIXING stuff and making out how to do stuff rather than actually using the tool itself. I am even thinking of writing my own annotation program to just do some basic annotation for me. This is all I wanted out of Prodigy but it turns out I am still navigating between this and that video tutorial to get things done. Your tutorials are supposed to make the lives of people easier, and not the other way round! ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [November 12, 2018, 2:40pm UTC](https://support.prodi.gy/t/using-loaders/340/6 "2018-11-12T14:40:31Z")
Hi @vatsala, I’m sorry you’re having a bad experience, but without understanding what you’re struggling with it’s pretty hard to know how to help! What are you trying to do, and which tutorials are you following? You’re on a free research license, right? For some research topics, writing your own tooling really is the best solution, especially since you can then release it alongside the data. Best, Matt ---
**Author:** ![vatsala](https://avatars.discourse-cdn.com/v4/letter/v/35a633/32.png) [@vatsala](https://support.prodi.gy/u/vatsala)\ **Post date:** [November 12, 2018, 3:01pm UTC](https://support.prodi.gy/t/using-loaders/340/7 "2018-11-12T15:01:53Z")
Hi Matt, Thanks for your reply. I figured out how to use the tool for annotation finally, but so far the tool annotates only one word at a time. I need to annotate plant species names (which are tricky, and some having a latin form - most of them consist of 2 words). Although I have created a database of plant terms (with the labels consisting of 2 words), I fail to understand why the tool picks up only one word instead of picking two consecutive words (termed as a plant label). Best, Vatsala ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [November 12, 2018, 3:19pm UTC](https://support.prodi.gy/t/using-loaders/340/8 "2018-11-12T15:19:48Z")
It sounds like you might not have defined your patterns file correctly, or alternatively, if you're using terms.teach, you might need to change how the vectors are trained. These threads might be useful: > [@Train a new NER entity with multi-word tokens](https://support.prodi.gy/t/train-a-new-ner-entity-with-multi-word-tokens/227): > > I’m not sure if I’m dealing with a bug or if I’m doing something wrong. But also if it’s the latter case you might be interested in why I’m doing this, so I’m going to write a quite verbose message here that will let you follow my thought process. My goal is to train a new NER entity with the name DISASTER which will recognize for example floods, storms and volcano eruptions. Yesterday I followed your video in which you train a DRUG entity and got the results I wanted in the end. But now I’m st… > [@ner.teach does not suggest multiple tokens](https://support.prodi.gy/t/ner-teach-does-not-suggest-multiple-tokens/884): > > Hi, this question is very closely related to [How to score incompletely highlighted entities?](https://support.prodi.gy/t/how-to-score-incompletely-highlighted-entities/625). I am trying to learn a new entity. The dataset looks like so: {"text": " The notes will mature on August 15, 2018, and will be paid in U.S. dollars against presentation and surrender thereof at the corporate trust office of the Trustee. However, we may redeem the notes at our option prior to that date. See \"—Optional Redemption.\" The notes will not be entitled to the benefit of, and are not su… > [@Two word entities](https://support.prodi.gy/t/two-word-entities/912): > > Hi, Does prodigy catch/train two word entities (new label) like some skincare brands ‘Estee Lauder’ , ‘Bobbi brown’ , ‘SK-II’ ? Right now its catching ‘estee’ , ‘lauder’ , ‘SK’ , ‘II’ seperately. Finally, if you just want to label text in an uncomplicated way, remember that you can always use the `ner.manual` mode, which doesn't do anything fancy -- it just lets you highlight text, without having a model suggest things. ---
**Author:** ![vatsala](https://avatars.discourse-cdn.com/v4/letter/v/35a633/32.png) [@vatsala](https://support.prodi.gy/u/vatsala)\ **Post date:** [November 12, 2018, 3:31pm UTC](https://support.prodi.gy/t/using-loaders/340/9 "2018-11-12T15:31:10Z")
thanks for the pointer, i will go through them. Vatsala