Hi @ryanwesslen and thanks for the welcome and the answer, I think this will save me from some further headaches, haha!
That's an interesting use case. I'm curious, could you describe a bit more?
What's the logic of which questions each example will get? Is it fixed like based on rules of the input text or metadata? Also are the questions predefined or can they change over time?
We're basically using an LLM to perform several classical and less-classical NLP tasks simultanously, and need to annotate some of the data post-hoc to get estimates on the model performance on the tasks. So, the questions are the same for each text item.
We considered the option of splitting the annotation tasks apart but since the texts are quite long, we felt like it's better for the annotators (and therefore for the annotation quality) if they the only have to read the text once and answer the questions all at once. Having just a single recipe/workflow to deal with also simplifies the related data engineering work.
In any case, custom html gets the job done, but extracting the annotator answers is a bit more work (as they end up getting stored within an html string).
Thanks for the answer!