Hi! In Prodigy v1.10, you could implement something like that using the
validate_answer callback. It's not 100% what that function was originally designed to do, but it should work The annotator will then see an alert when 100 accepted answers are submitted and won't be able to submit any more.
To count the existing accepted answers, you could use the
update callback, which gives you access to the batches of annotations that come back to the server. Here's a simple example:
total_accepted = 0
total_accepted += len([eg for eg in answers if eg["answer"] == "accept"])
if eg["answer"] == "accept" and total_accepted >= 100:
raise ValueError("Enough accepted answers, you can stop :)")
One thing to keep in mind is that depending on the batch size, there may be a small delay until the annotator sees the alert, because the batches of answers first have to be sent back to the server until Prodigy can know that 100 annotations are there. You could minimise that by using a lower batch size or setting
"instant_submit": True to immediately submit each answer as it's made in the app.