# textcat.manual Duplicate Samples

**URL:** <https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962>\
**Category:** Uncategorized\
**Tags:** usage, textcat, done, streams\
**Created:** [June 1, 2020, 3:21pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962 "2020-06-01T15:21:54Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![cgreco](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@cgreco](https://support.prodi.gy/u/cgreco)\
**Post date:** [June 1, 2020, 3:21pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/1 "2020-06-01T15:21:54Z")

</div>

Hello I am currently manually annotating a collection of text using the textcat.manual recipe. I'm having two annotators assign labels to the samples in the dataset in order to calculate an inner-annotator agreement utilizing the `/?session=id` feature. I've noticed that the annotators are receiving the same sample multiple times to annotate even if that annotator has annotated that sample before, once more prodigy seems to continue to present examples as if the dataset is of infinite size. For example. if the dataset has 100 samples, the annotation session will continue after the 100th sample, and the progress bar on prodigy displays an infinity symbol.

This:  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/e/e1f6a87a253915a3421451658dcc469aff4239ad.png)  
Versus This (which can been seen in the demo on the prodigy website):  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/0/0d42b6c8c4394e83271d7dd8c6015623e1b423f6.png)

I'm using Prodigy 1.9.9

This is my Prodigy.json file:

```json
{
    "port":8081,
    "force_stream_order": true,
    "feed_overlap": true,
    "exclude_by": "input"
}

```

I've confirmed that the duplicate annotated samples have identical values for both `_input_hash` and `_task_hash` so I'm not sure why they are showing up mulitple times in the same session.

Any ideas?

Thanks!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 4, 2020, 8:59am UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/2 "2020-06-04T08:59:12Z")

</div>

Hi! Thanks for the report – we're currently trying to debug a potential problem with `force_stream_order` over here on this thread. If we can track it down, it's likely that it's related:

> [@Refresh browser fix with force\_stream\_order](https://support.prodi.gy/t/refresh-browser-fix-with-force-stream-order/2949):
>
> Hi, I've tried the using the "force\_stream\_order" to fix issue of losing data when users refresh their browser. As expected, the same tasks they left off is resent. Problem is somehow prodigy loops back to resend duplicate tasks as well. So instead of say 60 datasets from your source, it'd allow 90 annotations. This is a multi-user prodigy instance btw.

---

<div class="post-metadata">

**Author:** ![cgreco](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@cgreco](https://support.prodi.gy/u/cgreco)\
**Post date:** [June 4, 2020, 3:23pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/3 "2020-06-04T15:23:36Z")

</div>

Hello! Yes after running a few tests it looks like the issue is the `force_stream_order` parameter, setting it to `false` prevents the duplicates from appearing in the stream.

---

<div class="post-metadata">

**Author:** ![cgreco](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@cgreco](https://support.prodi.gy/u/cgreco)\
**Post date:** [June 4, 2020, 4:04pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/4 "2020-06-04T16:04:12Z")

</div>

Ok, I ran into a second issue that may or may not be related. If you drop a dataset then reload it, the samples that were previously annotated in the dataset are still treated as labeled by the annotator and are not sent again.

**Example:** (With a 100 sample dataset)  
`prodigy textcat.manual my_dataset data/my_dataset.jsonl --label='MyLable'`  
_Annotator proceeds to annotate 50 examples_  
An issue is discovered that requires re-annotation of the data  
`prodigy drop my_dataset` - This should clear out the old data  
`prodigy textcat.manual my_dataset data/my_dataset.jsonl --label='MyLable'`  
The total reads as 0 in the Prodigy UI however the annotator is only able to annotate 50 samples, not the full 100

---

<div class="post-metadata">

**Author:** ![justindujardin](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/justindujardin/32/169_2.png) [@justindujardin](https://support.prodi.gy/u/justindujardin)\
**Post date:** [June 4, 2020, 8:47pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/5 "2020-06-04T20:47:36Z")

</div>

Hi @cgreco,

I posted information about a fix for the next release to address duplicates while using `force_stream_order`: [Refresh browser fix with force\_stream\_order - #11 by justindujardin](https://support.prodi.gy/t/refresh-browser-fix-with-force-stream-order/2949/11)

> [@cgreco](#):
>
> Ok, I ran into a second issue that may or may not be related. If you drop a dataset then reload it, the samples that were previously annotated in the dataset are still treated as labeled by the annotator and are not sent again.
> 
> **Example:** (With a 100 sample dataset)  
> `prodigy textcat.manual my_dataset data/my_dataset.jsonl --label='MyLable'`  
> _Annotator proceeds to annotate 50 examples_  
> An issue is discovered that requires re-annotation of the data  
> `prodigy drop my_dataset` - This should clear out the old data  
> `prodigy textcat.manual my_dataset data/my_dataset.jsonl --label='MyLable'`  
> The total reads as 0 in the Prodigy UI however the annotator is only able to annotate 50 samples, not the full 100

I think this is a separate issue related to the `drop` command. Can you confirm that this happens when you are using named sessions, but not if you exclude the `?session=something` argument? 🤔

---

<div class="post-metadata">

**Author:** ![geniki](https://avatars.discourse-cdn.com/v4/letter/g/ecb155/32.png) [@geniki](https://support.prodi.gy/u/geniki)\
**Post date:** [June 5, 2020, 7:39am UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/6 "2020-06-05T07:39:32Z")

</div>

@cgreco this sounds the same as what I reported here [Dropping dataset from code doesn't properly delete examples](https://support.prodi.gy/t/dropping-dataset-from-code-doesnt-properly-delete-examples/437/12)

@justindujardin is this something you plan to address in the next release?

---

<div class="post-metadata">

**Author:** ![cgreco](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@cgreco](https://support.prodi.gy/u/cgreco)\
**Post date:** [June 5, 2020, 2:29pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/7 "2020-06-05T14:29:05Z")

</div>

Hello @justindujardin! Thanks for your working debugging the issue for the next release! I look forward to when it comes out.

With regards to the second issue with the `drop` command. This does indeed appear to be the same issue that @geniki has reported. I can confirm that this issue occurs when using named sessions, and goes away when I remove the `?session=something` argument.

Thanks for your work!

---

<div class="post-metadata">

**Author:** ![justindujardin](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/justindujardin/32/169_2.png) [@justindujardin](https://support.prodi.gy/u/justindujardin)\
**Post date:** [June 5, 2020, 2:45pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/8 "2020-06-05T14:45:19Z")

</div>

> [@cgreco](#):
>
> With regards to the second issue with the `drop` command. This does indeed appear to be the same issue that @geniki has reported. I can confirm that this issue occurs when using named sessions, and goes away when I remove the `?session=something` argument.

@cgreco thanks for the confirmation, I think I have a workaround for you. The trouble is that when you use named sessions, it creates the source dataset entries, and another "session" dataset of the form "[dataset\_name]-[session\_id]". So if you used a dataset "foo" and a session "bar" you would end up with examples in the datasets "foo" and "foo-bar". When you drop "foo" it leaves "foo-bar" behind. So in your case, you can work around the problem by dropping the named session datasets in addition to the regular one:

```bash
prodigy drop my-dataset
prodigy drop my-dataset-justin

```

---

<div class="post-metadata">

**Author:** ![cgreco](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@cgreco](https://support.prodi.gy/u/cgreco)\
**Post date:** [June 5, 2020, 2:53pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/9 "2020-06-05T14:53:46Z")

</div>

Gotcha, thanks for the workaround. Is this the intended behavior or something that may be changed in the future? If this is the intended behavior it may be useful to make a note somewhere so that people know they have to clear out the session datasets as well.

Thanks!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [June 5, 2020, 6:57pm UTC](https://support.prodi.gy/t/textcat-manual-duplicate-samples/2962/10 "2020-06-05T18:57:36Z")

</div>

Just released v1.9.10, which should fix the underlying problem with `force_stream_order` (explained in detail by @justindujardin [in this post](https://support.prodi.gy/t/refresh-browser-fix-with-force-stream-order/2949/11)). The only case where a glitch may still be possible with the current implementation is if you hold down a hotkey and rapid fire – but that should also be a pretty unusual scenario.

> [@cgreco](#):
>
> If this is the intended behavior it may be useful to make a note somewhere so that people know they have to clear out the session datasets as well.

Good point, I'll add a note to the docs for now! We definitely want to have a clearer link between regular datasets and session datasets in the future so Prodigy can automatically remove "orphaned" session datasets – but this may require some changes to the database model.