Refresh browser fix with force_stream_order

@snd507 I confirmed that force_stream_order=True and feed_overlap=False cannot be combined when using named sessions. I've come up with a fix that lets you combine the flags (with a warning), and it should be available in the next release. The reason that combining the flags will produce a warning is that prodigy doesn't know which questions have been asked but not answered, and so it can still show overlapping examples if multiple users are annotating at the same time.

To understand why this happens, consider the following pseudo-configuration:

batch_size = 2
dataset = ["one", "two", "three", "four", "five", "six"]
sessions = ["user1", "user2"]
force_stream_order = True
feed_overlap = False

Because force stream order repeats the same questions until they're answered, when "user1" and "user2" open their browsers to annotate at the same time, they're sent the same initial batch of questions ["one", "two"]. When "user1" then answers the first question, there is no client/server communication to let "user2" know that question has been answered, and that they shouldn't answer it. If "user2" were to refresh though, they would see a new batch ["two", "three"] because "user1" answered the first question.

While the configuration may still produce some overlap, it does minimize it. The overlapping entries can later be resolved to single answers if needed by using the "review" recipe or a custom script to remove duplicates.

1 Like