Custom end-screen (“no tasks available”) fires prematurely in multi-recipe workflow

Hello, first time posting

I'm currently running a fairly complex multi-user annotation setup on multiple Prodigy instances with custom recipes and have participants report about an issue that occasionally occurs.

To give a brief overview of my setup:

Participants will be sent a link that shows them a page created using Flask. There they can register and login to start the annotation process. The login is required to provide the participant with their unique session ID used in the url: /?session=<session_id>

Once logged in, they will start with the first task. Once that task is finished and all annotations have been completed for this task, usually Prodigy would show a "no tasks available" screen. This screen has been completely overwritten with custom CSS, HTML, and Javascript. Instead, the page now shows the instructions for the next task, and a button, that when pressing sends them over to the next prodigy task on a different port, with their same session_id.
This repeats one more time to send them to the third task. This is a span categorization annotation task. For this one, they will be presented with 40 short text (loaded from JSONL stream) that they have to annotate. Once they have completed the 40 texts, they will again be presented with a custom "no tasks available" page that simply shows them their progress with a progress bar, a button to redirect them to the next port that also has 40 short texts, and another button to end the annotation task that redirects them to a end page created using flask.

Now to the issue that some participants have run into:
Since I have upgraded to Prodigy version 1.18, some participants have reported that when reaching the third task, they would immediately be presented with the custom "no tasks available" screen instead of the first short text out of the 40 to annotate.
I could only recreate this problem a handful of times, for which a simple reload of the page was enough to show the first text of the batch and solve the issue. However, this problem can ruin the annotation study in general as participants then have the option to immediately jump to the next annotation task.

Summary of the problem:

I’m running into an issue where my custom “no tasks available” / end-of-study screen sometimes appears immediately at the start of Task 3 rather than only after participants complete all annotation items. I’d love any advice on how to reliably suppress the end screen until there are no text available to annotate in the task.

Environment:

  • Prodigy version: 1.18
  • OS / Python: Ubuntu 20.04, Python 3.8
  • Browsers tested: Chrome latest, Firefox latest
  • Workflox: Multiple chained Prodigy recipes, each on its own port, sharing the same session_id

What I've tried:

  1. 200 ms setTimeout in the prodigyend handler - still brittle under certain conditions.
  2. window.taskPresented + MutationObserver to detect real task nodes before allowing the end screen - reports say it still misfires occasionally.
  3. Tweaking instant_submit and other Prodigy config flags - no change.

Relevant JS snippet (inside config["javascript"] of the custom prodigy recipe:

            "javascript": """
                function acceptTask() {
                    window.prodigy.answer('accept');
                }

                function ignoreTask() {
                    window.prodigy.answer('ignore');
                }

                function endTask() {
                    window.location.href = ''; //redirecting to next task on different port
                }

                // Prevent the browser back button as this has created issues with the custom no-tasks-available page in the past
                window.history.pushState(null, "", window.location.href);
                window.addEventListener("popstate", function () {
                    window.history.pushState(null, "", window.location.href);
                });

                // Code to Track Task Presentation 
                // Set a global flag indicating whether a task has been rendered.
                window.taskPresented = false;

                // Use of a MutationObserver to watch for insertion of task elements. -> this doesn't seem to work
                const observer = new MutationObserver(function(mutationsList, observerInstance) {
                    mutationsList.forEach(function(mutation) {
                        if (mutation.addedNodes.length > 0) {
                            mutation.addedNodes.forEach(function(node) {
                                // Only check element nodes.
                                if (node.nodeType === 1) {
                                    // If an element with class 'prodigy-block' or 'prodigy-spans' is added (or found within the node)
                                    if (node.matches('.prodigy-block, .prodigy-spans') || node.querySelector('.prodigy-block, .prodigy-spans')) {
                                        window.taskPresented = true;
                                        // Task detected; no need to observe further.
                                        observerInstance.disconnect();
                                    }
                                }
                            });
                        }
                    });
                });

                // Start observing changes in the main content container.
                const contentContainer = document.querySelector('.prodigy-content');
                if (contentContainer) {
                    observer.observe(contentContainer, { childList: true, subtree: true });
                }

                // Custom end-of-study page
                document.addEventListener("prodigyend", function() {
                    setTimeout(function () {
                        const taskContainer = document.querySelector('.prodigy-content');
                        const taskVisible = !!document.querySelector('.prodigy-block, .prodigy-spans');

                        if (!taskVisible) {
                    
                            const sessionId = window.prodigy.config.session;
                            const containerHtml = `
                                // Content for custom no-tasks-available page
                            `;
                            
                            // Replace the body with the custom page
                            document.body.innerHTML = containerHtml;
                            document.body.style = "font-family: Arial, sans-serif; background-color: #f9f9f9; margin: 0; padding: 0; overflow-y: auto;";

                            // Append custom CSS styles for the new page
                            const style = document.createElement("style");
                            style.innerHTML = `
                            `;

                            // Define the two button actions:
                            window.continueTask = function() {
                                window.location.href = `webpage/?session=${sessionId}`; //redirecting to the next task on different port
                            };

                            window.endStudy = function() {
                                window.location.href = 'webpage/end_page'; //show flask end_page
                            };
                        }
                    }, 200); //wait briefly to ensure no tasks are actually present
                });
            """,
        },
    }

I apologize for the lengthy code snippet, yet I believe it to be necessary to understand the context of the setup.

Questions

  1. Is there a built-in Prodigy hook or event that fires only once the first annotation is actually rendered and accepted?
  2. Am I missing a more reliable client-side approach than MutationObservers + timeouts?
  3. Any other best practice for chaining multiple Prodigy ports/recipes with a single session, where you only want the final “no tasks” screen after all annotations are done?
  4. Is there anything that has changed between Prodigy version 1.15 and 1.18 that generates this problem?

Thanks in advance for any pointers!

Welcome to the forum @Hijacqued !

Thank you for sharing all the details and apologies for the late reply!

Is there anything that has changed between Prodigy version 1.15 and 1.18 that generates this problem?

In 1.18 we have modfied the UI update cycle a bit to address the issue of the custom js code not being available in some conditions. I can't be 100% sure, but I suspect that this might have made a race condition more likely between the get_questions request and the empty queue detection which is what triggers the prodigyend event.
This interpretation would be in line with the fact that that the problem occurs only occasionally and can't be consistently reproduced. We'll definitely look into it, but in the meantime let's try to work around your problem.

Am I missing a more reliable client-side approach than MutationObservers + timeouts?

I think we can simplify it by relying on Prodigy events and just making sure the prodigyend can only be triggered once at least one task has been loaded. This of course assumes that all of your servers are expected to have non empty source.
We could inspect the DOM like you tried, but we can also listen to prodigyanswer event and access the task structure from there. It could be enough just to detect prodigyanswer event, but you could also store some indicator in the meta dictionary of the task (you'd need to modify your input JSONL file) just make it more tractable. What I mean is:

let firstTaskAnswered = false;

document.addEventListener('prodigyend', () => {
    console.log("Prodigy end event")
    if (!firstTaskAnswered) {
        console.warn("Blocked premature prodigyend event");
        return;
    }
    showYourCustomEndScreen(); // whatever you normally do
});

document.addEventListener('prodigyanswer', event => {

    const { answer, task } = event.detail;
    if (task && task.meta && task.meta.first) {
        console.log('First task was answered');
        firstTaskAnswered = true;
    }
});

This should result in "Loading..." screen until the first task is loaded.

I hope that also answers your first question:

Is there a built-in Prodigy hook or event that fires only once the first annotation is actually rendered and accepted?

There's not such event currently, but you could work around as described above.

Any other best practice for chaining multiple Prodigy ports/recipes with a single session, where you only want the final “no tasks” screen after all annotations are done?

I think your strategy of customizing prodigyend in certain conditions is pretty good - it's just unfortunate that that this race condition kicks in. Hopefully the suggested workaround is enough to unblock you.
In general, we recommend having a separate server per annotation tasks, but if sharing the state is not a concern you might take a look at this alternative approach here.
I do think, though that using separate servers should be more reliable.

1 Like

Thank you very much, @magdaaniol, for your reply (and for the warm welcome)!

I’ve implemented the code snippet you provided, and so far it appears to be working well - it correctly loads the short texts for the annotation study before displaying my custom "no tasks available" page. (However, I’ll need to wait for more participants to test it thoroughly, as the issue it addresses is quite situational.)

That said, I’ve encountered a different issue now:

When a participant reaches the custom "no tasks available" screen and idles for a while, then reloads the page or navigates back and forth in the browser, Prodigy’s default "no tasks available" screen replaces my custom one. After that, the default screen appears permanently, even after subsequent reloads or navigation.

Steps to reproduce:

  1. Complete all 40 annotations in Task 3-1.
  2. Custom "no tasks available" screen appears as expected.
  3. Wait a few minutes.
  4. Reload the page or use the browser’s back/forward buttons.
  5. The default Prodigy "no tasks available" screen appears and persists, overriding the custom one.

I'm happy to share the full recipe code if helpful.


Question

Is there any supported way—such as an event hook, configuration flag, or a recommended pattern—to persistently override Prodigy’s "no tasks available" screen, so that my custom version is shown on every load (including after reloads, navigation, or idling)?

Thanks again for your help!

Hi @Hijacqued!

The reason why you see the built-in end screen after refresh is that firstTaskAnswered variable is not persistent between browser sessions. In other words, if you hit refresh the custom javascript is being reloaded so the condition won't apply. After several returns from prodigyend, the app will fall back to the built-in screen. So, to prevent that, you're right that you might as well overwrite the end screen persistently.

You can do that by injecting custom CSS. For example:

[class*="Annotator-message"] {
    display: flex !important;
    justify-content: center;
    align-items: center;
    min-height: 100vh;
    font-family: Arial, sans-serif;
    font-size: 1.8em;
    color: #222;
    text-align: center;
}

[class*="Annotator-message"] * {
    display: none !important;
}

[class*="Annotator-message"]::before {
    content: "🎉 Custom end screen — you're done!";
}

Also, that made me realize that you might want to improve the condition:

 if (task && task.meta && task.meta.first) {
        console.log('First task was answered');
        firstTaskAnswered = true;
    }

to apply whenever prodigyanswer was detected. If the user refreshes their session and restart form the middle of the file, they won't have the "first" in their meta anymore, but we still want this condition to be true.

If this doesn't work please share the entire current custom js just to make sure I'm not missing anything. Thank you!

Dear @magdaaniol,

Thank you very much for your reply.

To give you the full context of my workflow, I believe it’s best to share the entire custom JavaScript file. Please let me know if you also need the complete custom recipe or any other specific details.

"javascript": """
    // --- BLOCK premature prodigyend until at least one annotation is submitted ---
    let firstTaskAnswered = false;

    // Whenever the user submits an answer, flip the flag
    document.addEventListener('prodigyanswer', event => {
        firstTaskAnswered = true;
    });            
    
    function acceptTask() {
        window.prodigy.answer('accept');
    }

    function ignoreTask() {
        window.prodigy.answer('ignore');
    }

    function endTask() {
        window.location.href = 'placeholder.com/end_page';
    }

    // Prevent the browser back button
    window.history.pushState(null, "", window.location.href);
    window.addEventListener("popstate", function () {
        window.history.pushState(null, "", window.location.href);
    });

    // Hide Prodigy's default buttons
    document.addEventListener('prodigymount', function() {
        const buttons = document.querySelector('.prodigy-buttons');
        if (buttons) {
            buttons.style.display = 'none';
        }
    });

    // Only show end screen after an answer has been given
    document.addEventListener('prodigyend', () => {
        setTimeout(() => {
            if (!firstTaskAnswered) {
                console.warn(\"Blocked premature prodigyend event\");
                return;
            }                

            // Custom end-of-study page
            const sessionId = window.prodigy.config.session;
            const containerHtml = `
                <div class="container">
                    <h1>Thank you for participating!</h1>
                    <!-- Custom HTML/CSS progress bar -->
                    <div class="w3-light-grey progress-bar-custom">
                        <div class="w3-container w3-blue w3-center" style="width:33%">33%</div>
                    </div>
                    <br>
                    <div class="button-container">
                        <button class="btn continue-btn" onclick="continueTask()">Continue</button>
                        <button class="btn end-btn" onclick="endStudy()">Conclude Task</button>
                    </div>
                </div>
            `;
            
            // Replace the body with our custom page
            document.body.innerHTML = containerHtml;
            document.body.style = "font-family: Arial, sans-serif; background-color: #f9f9f9; margin: 0; padding: 0; overflow-y: auto;";

            // Append custom CSS styles for the new page
            const style = document.createElement("style");
            style.innerHTML = `
                html, body {
                    height: 100%;
                    margin: 0;
                    padding: 0;
                    overflow-y: auto;
                    font-family: Arial, sans-serif;
                    background-color: #f9f9f9;
                    font-size: 1em;
                }
                .container {
                    max-width: 800px;
                    margin: 50px auto;
                    text-align: center;
                    padding: 20px;
                    background-color: white;
                    box-shadow: 0 2px 10px rgba(0,0,0,0.1);
                }
                h1 {
                    font-size: 2rem;
                    color: #333;
                }
                .highlighted-text {
                    background-color: #ffe184;
                    color: black;
                    padding: 2px 4px;
                    border-radius: 4px;
                }
                /* Custom progress bar styles */
                .w3-light-grey {
                    background-color: #f1f1f1;
                    border-radius: 4px;
                    margin: 20px auto;
                    width: 80%;
                }
                .w3-container {
                    padding: 5px;
                }
                .w3-blue {
                    background-color: #4594f1 !important;
                    border-radius: 4px;
                }
                .w3-center {
                    text-align: center;
                }
                .progress-bar-custom {
                    width: 80%;
                    margin: 0 auto;
                }                        
                .button-container {
                    margin: 30px 0;
                }
                .btn {
                    display: inline-block;
                    padding: 12px 25px;
                    margin: 10px;
                    color: white;
                    background-color: #4fd364;
                    text-decoration: none;
                    border-radius: 5px;
                    font-size: 1.2rem;
                    cursor: pointer;
                    border: none;
                }
                .btn:hover {
                    background-color: #3cb654;
                }
                .end-btn {
                    background-color: #d9534f;
                }
                .end-btn:hover {
                    background-color: #c9302c;
                }
            `;
            document.head.appendChild(style);

            // Define the two button actions:
            window.continueTask = function() {
                window.location.href = `placeholder.com/?session=${sessionId}`;
            };

            window.endStudy = function() {
                window.location.href = 'placeholder.com/end_page';
            };
        }, 200); //wait briefly to ensure no tasks are actually present
    });
""",

I look forward to working together on finding a solution. Thank you very much in advance.