We are doing clinical evaluation of LLM outputs and have multiple dimensions of SME inputs (such as clinical relevance, clinical harmfulness, etc) each of which needs to be represented by a multiple choice question. We want our SMEs to be able to do the evaluation for each output on one page rather than splitting into separate tasks. From this question, we can infer that custom HTML is required to permit multiple multiple choice questions, but there is no solution to that question, so I don't know where to start on implementing this custom HTML. How can we implement a setup with multiple fixed multiple choice questions?
Welcome to the forum @freepskov 
Apologies for a slightly delayed response!
Would the answer provided here solve your problem?
Thank you for your reply!
It looks like what I need, but I thought from the documentation though that the blocks interface wasn't designed to support multiple multiple choice blocks?
Also, I don't have the get_stream function on my Linux server, probably because my software version is old (1.11.14). On my Mac laptop, I have prodigy 1.12.7 and the get_stream function exists. Is there a way to update the version on the server? Does my company need to renew the license to update the Linux version further?
Hi @freepskov,
It looks like what I need, but I thought from the documentation though that the blocks interface wasn't designed to support multiple multiple choice blocks?
That's right, the solution proposed is a bit of workaround. We are currently working on a new annotation interface pages that would support better multiple choice block, but in the meantime adding multiple choice blocks with different values for options is the recommended solution.
Does my company need to renew the license to update the Linux version further?
1.11.14 came out in May 2023 so your company license should still allow to download 1.12.7. It depends when exactly it was purchased. You can try upgrading on Linux with the following command:
python -m pip install --upgrade prodigy --extra-index-url https://XXXX-XXXX-XXXX-XXXX@download.prodi.gy
where XXX would be your company license. Let me know if that doesn't work and we can try to make the snippet compatible with 1.11.14 although it is, of course, recommended to update to the latest version your license permits.
I managed to upgrade to 1.12.7. I think it was an issue of the dependency versions that were installed on my base environment. I created a prodigy environment, isolated it, and then install prodigy, as follows (using Anaconda)
conda create -n prodigy python=3.11
conda install -c conda-forge conda-ecosystem-user-package-isolation
python -m pip install prodigy --extra-index-url https://XXXX-XXXX-XXXX-XXXX@download.prodi.gy
Looks good! Thanks for the update.