Feature request: directories/archives of text files as a source format

[wpm (W.P. McNeill)]

In addition to jsonl, csv etc., have Prodigy accept a directory containing text files as input, one example per file. Equivalently, support a .tgz filetype.

[kyoungrok_jang (Riddler)]

Hi, can you please add support for gzip files? I usually store dataset in that format but have to decompress them when to use with Prodigy.

[honnibal (Matthew Honnibal)]

Sure, do you mean gzipped jsonl, for the loading?

[kyoungrok_jang (Riddler)]

Yes. and text also.

[ines (Ines Montani)]

Merged these two threads, since they're related! Happy to add this feature in a future release – in the meantime, I've outlined a simple custom loader solution in this thread:

Does Prodigy allow loading all files from a filepath

Out-of-the-box, Prodigy currently supports loading in data from single files of various types – for text, that’s .jsonl, .json, .txt and .csv. You can specify the loader via the --loader argument on the command line. If no loader is set, Prodigy will use the file extension to pick the respective loader. prodigy ner.teach your_dataset en_core_web_sm /path/to/data.txt So if you have multiple .txt files and want to use them all, the easiest way would be to combine them into one file. Alternative…

So instead of decompressing the file, you could also write a simple script that does this for you, loads the individual files and outputs the examples.