I was asked to create an interface where a user could paste in unlabelled examples (for NER tagging) in a text box, submit, and immediately annotate the example.
I wanted to use Prodigy for this, so I thought of using an on-disk queue that the text-box callback appends to, and a custom Prodigy loader pops tasks from. This means that when the queue is empty, the loader has to either yield some dummy value or simply wait to yield a value until there is an example in the queue. I obviously have to set the batch_size to 1, otherwise Prodigy waits for multiple examples to be submitted before displaying any to be labelled.
I have stumbled across some issues here. At first, the annotation UI just shows "Loading..." when it's waiting for a task to be available. I then submit a task, and the loader immediately picks it up from the queue. After the task has been annotated, if there is another task in the queue Prodigy loads that, but if it's empty it's just supposed to stay on "Loading..." until a task comes in. But sometimes (and I haven't been able to figure out what the circumstances are, it seems quite random) it shows up with "No tasks available". As soon as there is a task available this message disappears and the next task shows up for annotation, but the problem is that previously annotated tasks seem to get lost when the "No tasks available" screen shows up.
Another issue I've seen is that if the "Loading..." screen is showing and I want to stop annotating and save the progress, pressing the Save button doesn't do anything until a new example comes in. Meaning data isn't saved at all unless I submit a new unlabelled example.
I know this is quite niche and not really the intended usage, but if you have any idea what's going on and how to fix it that would be great. It's such a good UI for annotation, so would love to be able to use it to work on a dynamic task queue.
I can try to post some minimal code examples if needed.