Hi, I am not a Prodigy user yet, my annotation use cases doesn't seem to be supported out of the box, so just looking to find out whether it can be done with a workaround or may be a custom interface?
Our ML systems outputs a list of strings for a given input string. The order of the output string matters.
What we would like to be able to provide in annotations:
Rerank (ideally drag and drop but having input fields instead of checkboxes in the classification interface where the annotators would add/edit rank would be sufficient.
Ability to add a missing string (ideally validated against a finite set)
Ability to remove a string that should not be in the output
What you're describing should definitely be achievable with a custom interface. It will require some amount of front-end coding on your part, though. Good news is that it Prodigy UI was designed to be extendable. It exposes a number of app events and CSS classes for easier modification of the existing elements and you can specify custom HTML, CS and JS including new events to define custom interfaces.
You can even load remote JavaScript and CSS files in case you'd like to use an external front-end library in the project.
Your interface could consist of two view_id stacked in a block:
text_input interface were users could add a string. The validation of the input can be implemented on the recipe level via validate_answer callback
As for removing a string, you could perhaps define buttons as part of 1) html interface that would be linked to a custom back-end callback to remove the string from the final annotation via custom events.. Or, assuming the strings are numbered, add another text_input field to enter the number of the string to discard.
I really recommend browsing through our docs on custom interfaces to get and idea of what's possible and the complexity involved.
