We want to create an IR dataset for which we want to manually label the relevance of some search results shown for a set of queries and also classify the queries themselves.
Please see the image below for a graphical example of the intended annotation task for one query example (design details are not important as this one was created using simple online editors to clarify our requirements). In this example:
- Fruit: is the query to be annotated
- Food, Automobiles, Animals, Nature: are the possible classes for query some of which should be selected
- Numbers: for each search result (image with its title) exactly one number should be selected indicating a relevance score for this result to the query.
We checked the online demos for a similar use case but we could not find any, although some are close. Can we use Prodigy for such an annotation task? If so could you please point us to the relevant documentation entries that we should check or propose how can this be achieved?
Thank you for your support!