Hello, first time posting
I'm currently running a fairly complex multi-user annotation setup on multiple Prodigy instances with custom recipes and have participants report about an issue that occasionally occurs.
To give a brief overview of my setup:
Participants will be sent a link that shows them a page created using Flask. There they can register and login to start the annotation process. The login is required to provide the participant with their unique session ID used in the url: /?session=<session_id>
Once logged in, they will start with the first task. Once that task is finished and all annotations have been completed for this task, usually Prodigy would show a "no tasks available" screen. This screen has been completely overwritten with custom CSS, HTML, and Javascript. Instead, the page now shows the instructions for the next task, and a button, that when pressing sends them over to the next prodigy task on a different port, with their same session_id.
This repeats one more time to send them to the third task. This is a span categorization annotation task. For this one, they will be presented with 40 short text (loaded from JSONL stream) that they have to annotate. Once they have completed the 40 texts, they will again be presented with a custom "no tasks available" page that simply shows them their progress with a progress bar, a button to redirect them to the next port that also has 40 short texts, and another button to end the annotation task that redirects them to a end page created using flask.
Now to the issue that some participants have run into:
Since I have upgraded to Prodigy version 1.18, some participants have reported that when reaching the third task, they would immediately be presented with the custom "no tasks available" screen instead of the first short text out of the 40 to annotate.
I could only recreate this problem a handful of times, for which a simple reload of the page was enough to show the first text of the batch and solve the issue. However, this problem can ruin the annotation study in general as participants then have the option to immediately jump to the next annotation task.
Summary of the problem:
I’m running into an issue where my custom “no tasks available” / end-of-study screen sometimes appears immediately at the start of Task 3 rather than only after participants complete all annotation items. I’d love any advice on how to reliably suppress the end screen until there are no text available to annotate in the task.
Environment:
- Prodigy version: 1.18
- OS / Python: Ubuntu 20.04, Python 3.8
- Browsers tested: Chrome latest, Firefox latest
- Workflox: Multiple chained Prodigy recipes, each on its own port, sharing the same session_id
What I've tried:
- 200 ms
setTimeoutin theprodigyendhandler - still brittle under certain conditions. window.taskPresented+MutationObserverto detect real task nodes before allowing the end screen - reports say it still misfires occasionally.- Tweaking instant_submit and other Prodigy config flags - no change.
Relevant JS snippet (inside config["javascript"] of the custom prodigy recipe:
"javascript": """
function acceptTask() {
window.prodigy.answer('accept');
}
function ignoreTask() {
window.prodigy.answer('ignore');
}
function endTask() {
window.location.href = ''; //redirecting to next task on different port
}
// Prevent the browser back button as this has created issues with the custom no-tasks-available page in the past
window.history.pushState(null, "", window.location.href);
window.addEventListener("popstate", function () {
window.history.pushState(null, "", window.location.href);
});
// Code to Track Task Presentation
// Set a global flag indicating whether a task has been rendered.
window.taskPresented = false;
// Use of a MutationObserver to watch for insertion of task elements. -> this doesn't seem to work
const observer = new MutationObserver(function(mutationsList, observerInstance) {
mutationsList.forEach(function(mutation) {
if (mutation.addedNodes.length > 0) {
mutation.addedNodes.forEach(function(node) {
// Only check element nodes.
if (node.nodeType === 1) {
// If an element with class 'prodigy-block' or 'prodigy-spans' is added (or found within the node)
if (node.matches('.prodigy-block, .prodigy-spans') || node.querySelector('.prodigy-block, .prodigy-spans')) {
window.taskPresented = true;
// Task detected; no need to observe further.
observerInstance.disconnect();
}
}
});
}
});
});
// Start observing changes in the main content container.
const contentContainer = document.querySelector('.prodigy-content');
if (contentContainer) {
observer.observe(contentContainer, { childList: true, subtree: true });
}
// Custom end-of-study page
document.addEventListener("prodigyend", function() {
setTimeout(function () {
const taskContainer = document.querySelector('.prodigy-content');
const taskVisible = !!document.querySelector('.prodigy-block, .prodigy-spans');
if (!taskVisible) {
const sessionId = window.prodigy.config.session;
const containerHtml = `
// Content for custom no-tasks-available page
`;
// Replace the body with the custom page
document.body.innerHTML = containerHtml;
document.body.style = "font-family: Arial, sans-serif; background-color: #f9f9f9; margin: 0; padding: 0; overflow-y: auto;";
// Append custom CSS styles for the new page
const style = document.createElement("style");
style.innerHTML = `
`;
// Define the two button actions:
window.continueTask = function() {
window.location.href = `webpage/?session=${sessionId}`; //redirecting to the next task on different port
};
window.endStudy = function() {
window.location.href = 'webpage/end_page'; //show flask end_page
};
}
}, 200); //wait briefly to ensure no tasks are actually present
});
""",
},
}
I apologize for the lengthy code snippet, yet I believe it to be necessary to understand the context of the setup.
Questions
- Is there a built-in Prodigy hook or event that fires only once the first annotation is actually rendered and accepted?
- Am I missing a more reliable client-side approach than MutationObservers + timeouts?
- Any other best practice for chaining multiple Prodigy ports/recipes with a single session, where you only want the final “no tasks” screen after all annotations are done?
- Is there anything that has changed between Prodigy version 1.15 and 1.18 that generates this problem?
Thanks in advance for any pointers!