Feature request: directories/archives of text files as a source format

[ines (Ines Montani)]

Merged these two threads, since they're related! Happy to add this feature in a future release – in the meantime, I've outlined a simple custom loader solution in this thread:

Does Prodigy allow loading all files from a filepath

Out-of-the-box, Prodigy currently supports loading in data from single files of various types – for text, that’s .jsonl, .json, .txt and .csv. You can specify the loader via the --loader argument on the command line. If no loader is set, Prodigy will use the file extension to pick the respective loader. prodigy ner.teach your_dataset en_core_web_sm /path/to/data.txt So if you have multiple .txt files and want to use them all, the easiest way would be to combine them into one file. Alternative…

So instead of decompressing the file, you could also write a simple script that does this for you, loads the individual files and outputs the examples.