prodigy review --auto-accept exhausting stream before all annotations saved to gold dataset

Hello!

I have collected a number of annotations that I'd like to review. The stats for the annotations dataset (let's call it ds) are as follows:

New Unique Total Unique
Aug 2021 278 278 278 278
Sep 2021 899 880 1177 1158
Oct 2021 327 322 1504 1480

I want to review the dataset into ds_gold to have a set of unique gold standard annotations, and anticipate that I'd have to review at most 1504-1480=24 annotations to get there.

I think the review --auto-accept flag is right for my use case, and used prodigy review ds_gold ds --auto-accept.

I'm running into a situation where the stream is exhausted and doesn't "refresh" until I restart the server. If I restart the server I get 23 annotations auto-accepted into the dataset and then a "No Tasks" message. Sometimes I get a duplicate annotation to review before restarting the server but usually not.

I think it could be because of the sparsity of annotations with conflicts. I am going to dive into the review recipe to check, but thought I'd log this issue here in case there's something obvious I'm missing. I'll update if I solve it.


Update
Here's the output of logging=basic, as I think that's pointed me to the fact that most of the action happens before I access the app. I need to look into what the controller and feed are supposed to be doing.

Running command: prodigy review ds_gold ds --auto-accept
20:05:12: INIT: Setting all logging levels to 20
20:05:12: RECIPE: Calling recipe 'review'
20:05:12: RECIPE: Starting recipe review
20:05:12: CONFIG: Using config from global prodigy.json
20:05:12: CONFIG: Using config from working dir
20:05:12: DB: Initializing database PostgreSQL
20:05:12: DB: Connecting to database PostgreSQL
20:05:12: DB: Loading dataset 'ds' (1504 examples)
20:05:12: RECIPE: Merged 1498 examples from 1 datasets
20:05:12: CONFIG: Using config from global prodigy.json
20:05:12: CONFIG: Using config from working dir
20:05:12: VALIDATE: Validating components returned by recipe
20:05:12: CONTROLLER: Initialising from recipe
20:05:12: VALIDATE: Creating validator for view ID 'review'
20:05:12: VALIDATE: Validating Prodigy and recipe config
20:05:12: DB: Creating dataset '2021-10-10_20-05-12'
20:05:12: FEED: Initializing from controller
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:12: DB: Getting dataset 'ds_gold'
20:05:12: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: DB: Getting dataset 'ds_gold'
20:05:13: DB: Added 1 examples to 1 datasets
20:05:13: CORS: initialized with wildcard "*" CORS origins

:sparkles: Starting the web server at http://localhost:8080 ...
Open the app in your browser and start annotating!

INFO: Started server process [63004]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://localhost:8080 (Press CTRL+C to quit)
INFO: ::1:58039 - "GET / HTTP/1.1" 401 Unauthorized
INFO: ::1:58040 - "GET / HTTP/1.1" 200 OK
INFO: ::1:58040 - "GET /bundle.js HTTP/1.1" 200 OK
20:05:46: GET: /project
INFO: ::1:58040 - "GET /project HTTP/1.1" 200 OK
20:05:47: POST: /get_session_questions
20:05:47: CONTROLLER: Getting batch of questions for session: None
20:05:47: FEED: Finding next batch of questions in stream
20:05:47: FEED: re-adding open tasks to stream
20:05:47: FEED: Stream is empty
20:05:47: FEED: adding tasks from other sessions to None queue.
20:05:47: FEED: batch of questions requested for session None: 0
20:05:47: RESPONSE: /get_session_questions (0 examples)
INFO: ::1:58040 - "POST /get_session_questions HTTP/1.1" 200 OK

Hello,
I tried out the new release and am still seeing this issue-- any tips on troubleshooting you might be able to offer?
Thanks as always!
Adam