Wrong percentage when using TXT loader and "auto_count_stream": True in custom recipe

1

Good day,

I created a custom recipe that uses "view_id": "spans" and the TXT loader. I have also set "auto_count_stream": True so I can see the percentage while going through the set. But the percentage is always not correct. It even does not reach 100% when there are no tasks anymore.

Regards,
Joe

2

Hi Joe,

what version of Prodigy are you using? The auto_count_stream has been deprecated as of v1.12. Did you not see a warning on this version?

More details on the new progress bar system can be found here:

3

We are using 1.11.14 currently.

4

Then this might help explain the behavior. If you switch to v1.12 you should see an improvement, although we have dropped support for auto_count_stream. If you stumble apon another edge case though; we're all ears!

5

Hello. Are there any examples/recipes on how to use the new progress bar system?

6

The new progress system should kick in automatically. If you'd like to see it in action, the announcement video does a quick demo:

Alternatively, the docs on custom progress bars has a section titled "what is a source" that gives a bit more information on the implementation.