Custom end-screen (“no tasks available”) fires prematurely in multi-recipe workflow

, , ,
1

Hello, first time posting

I'm currently running a fairly complex multi-user annotation setup on multiple Prodigy instances with custom recipes and have participants report about an issue that occasionally occurs.

To give a brief overview of my setup:

Participants will be sent a link that shows them a page created using Flask. There they can register and login to start the annotation process. The login is required to provide the participant with their unique session ID used in the url: /?session=<session_id>

Once logged in, they will start with the first task. Once that task is finished and all annotations have been completed for this task, usually Prodigy would show a "no tasks available" screen. This screen has been completely overwritten with custom CSS, HTML, and Javascript. Instead, the page now shows the instructions for the next task, and a button, that when pressing sends them over to the next prodigy task on a different port, with their same session_id.
This repeats one more time to send them to the third task. This is a span categorization annotation task. For this one, they will be presented with 40 short text (loaded from JSONL stream) that they have to annotate. Once they have completed the 40 texts, they will again be presented with a custom "no tasks available" page that simply shows them their progress with a progress bar, a button to redirect them to the next port that also has 40 short texts, and another button to end the annotation task that redirects them to a end page created using flask.

Now to the issue that some participants have run into:
Since I have upgraded to Prodigy version 1.18, some participants have reported that when reaching the third task, they would immediately be presented with the custom "no tasks available" screen instead of the first short text out of the 40 to annotate.
I could only recreate this problem a handful of times, for which a simple reload of the page was enough to show the first text of the batch and solve the issue. However, this problem can ruin the annotation study in general as participants then have the option to immediately jump to the next annotation task.

Summary of the problem:

I’m running into an issue where my custom “no tasks available” / end-of-study screen sometimes appears immediately at the start of Task 3 rather than only after participants complete all annotation items. I’d love any advice on how to reliably suppress the end screen until there are no text available to annotate in the task.

Environment:

  • Prodigy version: 1.18
  • OS / Python: Ubuntu 20.04, Python 3.8
  • Browsers tested: Chrome latest, Firefox latest
  • Workflox: Multiple chained Prodigy recipes, each on its own port, sharing the same session_id

What I've tried:

  1. 200 ms setTimeout in the prodigyend handler - still brittle under certain conditions.
  2. window.taskPresented + MutationObserver to detect real task nodes before allowing the end screen - reports say it still misfires occasionally.
  3. Tweaking instant_submit and other Prodigy config flags - no change.

Relevant JS snippet (inside config["javascript"] of the custom prodigy recipe:

            "javascript": """
                function acceptTask() {
                    window.prodigy.answer('accept');
                }

                function ignoreTask() {
                    window.prodigy.answer('ignore');
                }

                function endTask() {
                    window.location.href = ''; //redirecting to next task on different port
                }

                // Prevent the browser back button as this has created issues with the custom no-tasks-available page in the past
                window.history.pushState(null, "", window.location.href);
                window.addEventListener("popstate", function () {
                    window.history.pushState(null, "", window.location.href);
                });

                // Code to Track Task Presentation 
                // Set a global flag indicating whether a task has been rendered.
                window.taskPresented = false;

                // Use of a MutationObserver to watch for insertion of task elements. -> this doesn't seem to work
                const observer = new MutationObserver(function(mutationsList, observerInstance) {
                    mutationsList.forEach(function(mutation) {
                        if (mutation.addedNodes.length > 0) {
                            mutation.addedNodes.forEach(function(node) {
                                // Only check element nodes.
                                if (node.nodeType === 1) {
                                    // If an element with class 'prodigy-block' or 'prodigy-spans' is added (or found within the node)
                                    if (node.matches('.prodigy-block, .prodigy-spans') || node.querySelector('.prodigy-block, .prodigy-spans')) {
                                        window.taskPresented = true;
                                        // Task detected; no need to observe further.
                                        observerInstance.disconnect();
                                    }
                                }
                            });
                        }
                    });
                });

                // Start observing changes in the main content container.
                const contentContainer = document.querySelector('.prodigy-content');
                if (contentContainer) {
                    observer.observe(contentContainer, { childList: true, subtree: true });
                }

                // Custom end-of-study page
                document.addEventListener("prodigyend", function() {
                    setTimeout(function () {
                        const taskContainer = document.querySelector('.prodigy-content');
                        const taskVisible = !!document.querySelector('.prodigy-block, .prodigy-spans');

                        if (!taskVisible) {
                    
                            const sessionId = window.prodigy.config.session;
                            const containerHtml = `
                                // Content for custom no-tasks-available page
                            `;
                            
                            // Replace the body with the custom page
                            document.body.innerHTML = containerHtml;
                            document.body.style = "font-family: Arial, sans-serif; background-color: #f9f9f9; margin: 0; padding: 0; overflow-y: auto;";

                            // Append custom CSS styles for the new page
                            const style = document.createElement("style");
                            style.innerHTML = `
                            `;

                            // Define the two button actions:
                            window.continueTask = function() {
                                window.location.href = `webpage/?session=${sessionId}`; //redirecting to the next task on different port
                            };

                            window.endStudy = function() {
                                window.location.href = 'webpage/end_page'; //show flask end_page
                            };
                        }
                    }, 200); //wait briefly to ensure no tasks are actually present
                });
            """,
        },
    }

I apologize for the lengthy code snippet, yet I believe it to be necessary to understand the context of the setup.

Questions

  1. Is there a built-in Prodigy hook or event that fires only once the first annotation is actually rendered and accepted?
  2. Am I missing a more reliable client-side approach than MutationObservers + timeouts?
  3. Any other best practice for chaining multiple Prodigy ports/recipes with a single session, where you only want the final “no tasks” screen after all annotations are done?
  4. Is there anything that has changed between Prodigy version 1.15 and 1.18 that generates this problem?

Thanks in advance for any pointers!

2

Welcome to the forum @Hijacqued !

Thank you for sharing all the details and apologies for the late reply!

Is there anything that has changed between Prodigy version 1.15 and 1.18 that generates this problem?

In 1.18 we have modfied the UI update cycle a bit to address the issue of the custom js code not being available in some conditions. I can't be 100% sure, but I suspect that this might have made a race condition more likely between the get_questions request and the empty queue detection which is what triggers the prodigyend event.
This interpretation would be in line with the fact that that the problem occurs only occasionally and can't be consistently reproduced. We'll definitely look into it, but in the meantime let's try to work around your problem.

Am I missing a more reliable client-side approach than MutationObservers + timeouts?

I think we can simplify it by relying on Prodigy events and just making sure the prodigyend can only be triggered once at least one task has been loaded. This of course assumes that all of your servers are expected to have non empty source.
We could inspect the DOM like you tries, but we can also listen to prodigyanswer event and access the task structure from there. It could be enough just to detect prodigyanswer event, but you could also store some indicator in the meta dictionary of the task (you'd need to modify your input JSONL file) just make it more tractable. What I mean is:

let firstTaskAnswered = false;

document.addEventListener('prodigyend', () => {
    console.log("Prodigy end event")
    if (!firstTaskAnswered) {
        console.warn("Blocked premature prodigyend event");
        return;
    }
    showYourCustomEndScreen(); // whatever you normally do
});

document.addEventListener('prodigyanswer', event => {

    const { answer, task } = event.detail;
    if (task && task.meta && task.meta.first) {
        console.log('First task was answered');
        firstTaskAnswered = true;
    }
});

This should result in "Loading..." screen until the first task is loaded.

I hope that also answers your first question:

Is there a built-in Prodigy hook or event that fires only once the first annotation is actually rendered and accepted?

There's not such event currently, but you could work around as described above.

Any other best practice for chaining multiple Prodigy ports/recipes with a single session, where you only want the final “no tasks” screen after all annotations are done?

I think your strategy of customizing prodigyend in certain conditions is pretty good - it's just unfortunate that that this race condition kicks in. Hopefully the suggested workaround is enough to unblock you.
In general, we recommend having a separate server per annotation tasks, but if sharing the state is not a concern you might take a look at this alternative approach here.
I do think, though that using separate servers should be more reliable.