I'm not 100% sure why this would matter (maybe something related to existing hashes?), but could you try again with a new and blank dataset? I was able to make it behave weirdly with one of my messy test datasets, but if I try with a fresh dataset and textcat.manual, I can't reproduce the problem.
As mentioned on the other thread, the only scenario where a glitch could still happen is if the annotator holds down the answer key – but that's also typically not a common use case.
Btw, just to clarify, you shouldn't be using a forced stream order with active learning recipes like textcat.teach, ner.teach or a recipe using a stream that continuously responds to outside state. That's currently not supported.
Hi @ines The dataset I am using in Example 1 is a new and blank dataset without any existing hashes. However, it does have some atypical key:value pairs. Here is a sample record:
{"html":"<p>here is the sentence</p>", "text":"here is the same sentence again", "meta": {"docId":"Doc123456", "pageNumber":"3", "sentenceLocation":{"b":"2896","r":"2260","t":"304","l":"445"}, "documentSentenceNumber":"97"} }
When force_stream_order:false the annotator can come back hours later, re-open the browser and pick up where she left off. Is there a way to use force_stream_order:true with my particular jsonl dataset formatting and still pick up where she left off?
As for Example 2, I observed similar behavior that the glitch only happens when I go through the tasks quicker than any annotator would likely annotate. So, it's not really a problem.
The structure of the task shouldn't matter – unlesss any of these properties are computed dynamically and change. (Like, if your "text contained the current timestamp for instance, that could lead to problems because the same example would receive a different hash every time.)
Yes, that should be the main benefit of setting "force_stream_order": true. Assuming you've saved the progress, you should be able to close the tab etc. and always re-start with the last unannotated example, in the same order.
Hi @ines I tried, as you suggested, with a new and blank dataset and it seems to have fixed the problem, but it doesn't seem to be caused by the existing hashes. Here are the experiments I conducted:
(1) I used db-out and then db-in using the --re-hash parameter to see if recreating the hashes would fix the hashes. That did not solve the problem.
(2) Then, I compared the new hashes from the new and blank dataset to the hashes from the old 700+ annotated example dataset. Both the task and input hashes are identical. In fact, the entire record (i.e., every key:value pair is identical)
So, there doesn't seem to be any evidence to support the hypothesis that the existing hashes are causing the problem. Can you see any flaw in my experiment or logic that might guide me to figure out the root cause and ultimately how to get force_stream_order to work as expected with an existing dataset?
Another hypothesis I came up with is that force_stream_order works for small datasets, but once you get to several hundred annotations it causes a race condition that causes it to restart the tasks. Do you see this as a possibility?
I can confirm that I am still experiencing duplicate tasks in Prodigy 1.9.10 as well when setting "force_stream_order":true, though it seems to be more rare than it was in 1.9.9. The duplicates didn't appear in my test set of 100 samples, but when I scaled up the data set to a few thousand samples they appear again.
That's an interesting idea, although I'm not entirely sure how the dataset size would be related Maybe the exlude mechanism, but that also doesn't seem very logical. The hashes explanation didn't make much sense to me to be honest, but it was the first idea I could some up with that at least seemed somewhat related. Anyway, we'd have to investigate and debug this in more detail. Thanks for your debugging help so far, I'm confident that we'll figure out the mystery here
For now, maybe just use a new dataset and keep your datasets small? There shouldn't be much of a downside here – you can still use the --exclude option to exclude examples, or just annotate separately and then merge the sets before training.
Thanks for testing! How are you scaling up the set, are you just annotating really fast or holding down the hotkey? Because in that case, it can still happen in the current implementation and we weren't able to just quickly cover that by the hotfix – but it's also luckily not a very realistic scenario.
Thanks for releasing 1.9.10. In my case it also helped reduce but not eliminate the problem. In my workflow I do end up having bursts of pressing Accept naturally because I can really annotate some examples that fast.
If the speed is the issue, would introducing a speed bump somewhere help?
Would increasing the batch_size to say 100 help?
It's not easy to test these things and without knowing the underlying issue I don't have a guess whether the have a theoretical chance of working.
So interestingly when I was testing on the small sample set I just spammed one of the answers randomly, (either accept or reject) and did not encounter issues. When I scaled up the data set I was doing the annotations "for real" so the pace should have been much slower. My annotation pattern is similar to @geniki in that there could be short bursts of quick annotation if a sample can be annotated at a glance, though I don't know if this accounts for the duplicates. Duplication is fairly rare, right now I have a 2% duplication rate, but not so uncommon that it isn't noticeable when performing the annotation tasks. Thanks for working on getting to the bottom of this! Let me know if there's any scenarios you'd like tested.
Hi @ines In regards to the Example 1 problem, I think I have ruled out my hypothesis about the small dataset cause. Even if I create a new dataset and then add a couple of the annotations from the previous dataset, it still breaks it. It definitely has something to do with the annotations from the old dataset. If I start off with a new dataset and perform some annotations, it works correctly, but as soon as I add some of the annotations from the old dataset to the new dataset using db.add_examples(old_dataset[:2], datasets=("new_dataset",)), it breaks it again.
However, not all of the old annotations cause problems. A couple times I added old annotations it did not break it. So, I have no idea what the problem is, but I hope this narrows it down for you.
FYI, I ported over the posts from the other thread to keep the discussion in one place (otherwise it's harder to keep track of the reports, updates etc.)
Here's @justindujardin's latest update again so it doesn't get buried:
Next, we'll investigate the issue with existing datasets and why some examples in the pre-existing dataset cause examples to be re-sent, while others don't
Thanks for your continued efforts on this bug! I wanted to let you know I'm encountering duplicates even on a fresh dataset, so it doesn't seem to be isolated to just pre-existing datasets. I'm setting force_stream_order=True and feed_overlap=True. I do have annotators working at the same time though if that's related.
The fix is under review, and I don't know when the next release will be available. Also, I'd like to not get in the habit of committing to dates on the forums.
I expect it will be ready in about as much time as it takes for other bug fixes. We'll be sure to update this thread when it's released.
@snd507 From another thread, can I ask why you thought that? Is it the usage of feed_overlap=False that gives you that expectation?
One behavior change we are looking at for the combination of force_stream_order=True and feed_overlap=False is to have the server filter out the duplicates, only accepting the first annotation for an input. This would lead to there being only one of each example in the database, but would mean that only the first annotation is accepted, even if the second one was better. Would this behavior meet your expectations?
Yes, ideally we wouldn't want replayed tasks but that won't be an issue if only the first annotation is saved, and subsequent answers ignored if they're exist already.
@snd507@cgreco@geniki@dshefman@Kairine Thanks for all of your reports! Before the next release, I'd like to get some verification that the latest fixes resolve your problems. Since we don't have clear reproduction cases for each of your problems, I'd like to ask you to test out a preview build of v1.10. If you're willing to help out, send me a private message or let me know here and I'll send you a link. Thanks!
