Display stream length

[ines (Ines Montani)]

Yes, this actually just came up in a different thread. Here's some background on when the stream length is shown vs. when the stream is displayed as infinite:

@ines

By default, Prodigy streams are generators and if a file can be read in line-by-line, Prodigy will do so and start yielding out tasks immediately. Generators have no length, because they don’t know how many items there are in total. And if Prodigy doesn’t know how many items there are in total, it can’t display the progress.

You can try making the following edit in your recipe:

@ines

the idea is that the recipe needs to return a stream that exposes a __len__ (i.e. a length, which is the case for regular lists, but not generators). So try converting the stream as late as possible in the recipe:

stream = add_tokens(nlp, stream)
stream = list(stream)

You can also supply your own custom progress function via the recipe options. This is useful if you want to calculate the remaining examples dynamically (e.g. by checking your database).