Hi @justindujardin, thanks for your answers.
So, I guess my slightly off-topic question regarding whether that log message is okay is answered now 
Thanks.
Regarding the order issue, I guess we were/are basically doing the same thing and when I tested this locally just by myself, e.g., using different tabs and different session ids, for say 10 or 20 tasks, I found the task shown the correct order without duplicates in each session, independently of what I had already labeled in the other sessions. That's why I was surprised to see that this is not the case, while conducting our main study with three different annotators for a while. I'm not sure what the key difference of both setups (local pre-test vs. main study on our server) is, but just to name a few that come to my mind:
- strongly different times of annotation, e.g., two coders started their annotations within a few hours, whereas the other coder started a day later
- longer annotation times, i.e., each coder spent a couple of hours coding, also incl. intermediate breaks; then hours or a day later they continued their annotation
- one coder reported that they were having the same tab open all the time (also between longer breaks), whereas another one closed the tab when they finished their work for the day
Besides such differences, there should be no technical difference, since in both cases I was using the same docker image to run prodigy, i.e., only the computer running the container changed.
If you have any idea of what could be potential root causes of this issue, I'll gladly try to provide more information.
Just FYI and maybe others who are reading this thread because they have similar issues: Right now, we have changed the setup slightly: we set up the main study using three prodigy processes, each with their own PRODIGY_HOME, PRODIGY_PORT and also own dataset. Moreover, we have applied the stream function as suggested by Ines (Struggling to create a multiple choice image classification). These two changes seem to work better now, but I couldn't have checked it in detail yet, since not all coders resumed their annotations.