Welcome to the forum @MarieCo
Apologies for a slightly delayed response but for some reason I've missed your post until now!
The reason why the apply
method from the example doesn't work is that it is meant to be used with the Stream
data structure, while your custom loader is returning a generator object.
I admit it might be a bit confusing because Prodigy built-in loaders used to return the generator
until version 1.12. Then we made a complete refactor and introduced the Stream
data structure for better tractability of the stream of tasks (some more context). We've updated most of the examples in the docs to use the updated Stream
API, without stating that explicitly in the custom loaders section (note taken!)
You can still use "the old" generator way (and not use the apply
method) but I'll show you how to edit your code to make it compatible with the new Stream
data structure.
The easiest and the smallest edit would be to convert your generator into Stream
using Stream
's .from_iterable
method:
from prodigy.components.stream import Stream
stream_as_generator = custom_csv_loader(source) #load stream with custom loader
stream = Stream.from_iterable(stream_as_generator) # convert it into Stream
That should make the rest of your code run as expected - let me know if it doesn't