Hi and sorry for only getting to this now, I somehow missed the thread!
One option would be to show all your texts with a simple
<input type="number"> to specify the rank (with a min/max value based on the number of options). Each input can have an event listener that fires on
change, i.e. when the user changes the number, and adds the information to the task JSON with
window.prodigy.update. If you wanted to restrict it so that one number can only be present once, this would be a bit more involved to implement on the client – but you could also just use the
validate_answer callback for that, check the added ranks in Python and raise an error if a value is present more than once.
If you want the UI to be more in a drag-and-drop style, you could adapt something like this: https://codepen.io/crouchingtigerhiddenadam/pen/qKXgap The minimum code needed is actually very straightforward and you could almost copy-paste it directly into your
window.prodigy.update after each drag event (
dragEnd) that checks the order of elements and updates the JSON accordingly. Each list element could have an
id that corresponds to the document title ID in your data, so you could just loop over the list elements in order to extract the rank. So if the first element has document title ID
123, you know that title
123 was ranked
0, depending on how you index).
This is probably something you could do with the
choice UI, the query as the main content and options for the document titles. You could then select one (or more) most relevant queries. Framing this as a choice problem might be very helpful here, because in cases like this, you realistically only care about the top 1 or 2 selections – you don't necessarily want annotators to waste time on deciding whether a title should rank 9th or 10th, and you'd likely end up with a lot of disagreements here anyways. If you already have a model you could also let it pre-select the most relevant option(s), so you can calculate how often the annotator agrees with the model, or how often different annotators agree with each other.