Custom loader and Stream not compatible?

Welcome to the forum @MarieCo :wave:

Apologies for a slightly delayed response but for some reason I've missed your post until now!

The reason why the apply method from the example doesn't work is that it is meant to be used with the Stream data structure, while your custom loader is returning a generator object.
I admit it might be a bit confusing because Prodigy built-in loaders used to return the generator until version 1.12. Then we made a complete refactor and introduced the Stream data structure for better tractability of the stream of tasks (some more context). We've updated most of the examples in the docs to use the updated Stream API, without stating that explicitly in the custom loaders section (note taken!)

You can still use "the old" generator way (and not use the apply method) but I'll show you how to edit your code to make it compatible with the new Stream data structure.
The easiest and the smallest edit would be to convert your generator into Stream using Stream's .from_iterable method:

from prodigy.components.stream import Stream
stream_as_generator = custom_csv_loader(source) #load stream with custom loader
stream = Stream.from_iterable(stream_as_generator) # convert it into Stream

That should make the rest of your code run as expected - let me know if it doesn't :slight_smile: