We are annotating texts with multiple non exclusive labels.
- Is it possible to enforce a human annotator to select at least one label?
- Alternatively, in the case where no label was selected we want to treat the task as ignored. Is it possible?
Maybe not exactly "force", but you could alert the user if an answer was submitted and no options were selected. For instance, assuming you're using the choice interface, you could add JavaScript that looks like this:
prodigy.addEventListener('prodigyanswer', event => {
const selected = event.detail.task.accept || []
if (!selected.length) {
alert('Task with no selected options submitted.')
}
})
Yes, that'd also be possible. After annotation, you could export the data, check for examples that don't have selected options and change the "answer" to "ignore". Datasets in Prodigy are append-only (to make sure you can't so easily destroy work), so if you want to keep working with the data in Prodigy, you'd have to import it to a new dataset.
from prodigy.components.db import connect
db = connect()
examples = db.get_dataset("your_dataset")
for eg in examples:
if not eg.get("accept", []):
eg["answer"] = "ignore"
db.add_dataset("new_dataset")
db.add_examples(examples, datasets=["new_dataset"])
Thank you, how can I involve this piece of code into the "textcat.manual" built-in recipe?
How accepted annotation task without selected labels can be represented in history by "X" sign (like in ignored annotation case)?
You can add it to the "javascript" setting in your prodigy.json. You probably want to make it if (!selected.length && window.prodigy.viewId == 'choice') to only show the alert in the choice interface and not everywhere.
The other code snippet I posted is a postprocessing step, so that wouldn't have to go in any recipe.
The icon here reflects the annotation decision: "accept", "reject" or "ignore". So if a task is accepted, it will get the accept icon, and it's not easily possible to show a different icon here. Otherwise, the behaviour would be very inconsistent and confusing.