# Keeping Duplicates in Stream

**URL:** <https://support.prodi.gy/t/keeping-duplicates-in-stream/7195>\
**Category:** Uncategorized\
**Tags:** textcat, solved\
**Created:** [April 23, 2024, 4:16pm UTC](https://support.prodi.gy/t/keeping-duplicates-in-stream/7195 "2024-04-23T16:16:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vera-bernhard](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vera-bernhard/32/4370_2.png) [@vera-bernhard](https://support.prodi.gy/u/vera-bernhard)\
**Post date:** [April 23, 2024, 4:16pm UTC](https://support.prodi.gy/t/keeping-duplicates-in-stream/7195/1 "2024-04-23T16:16:46Z")

</div>

Hi there,

I have a dataset.jsonl looking something like this:  
{'text': 'xyz', annotation\_task': 'a'}  
{'text': 'xyz', 'annotation\_task': 'b'}  
{'text': 'xyz', 'annotation\_task': 'c'}  
{'text': 'stu', 'annotation\_task': 'a'}  
{'text': 'stu', 'annotation\_task': 'b'}  
{'text': 'stu', 'annotation\_task': 'c'}  
...  
According to the 'annotation\_task' a different set of labels are added to the options. This set is necessary, as the annotation tasks need to be broken down but the same sample should be annotated in a row. I've created a custom recipe and use choice\_style multiple. I'm working with prodigy==1.15.2.

I'm using the following to read in the data:

```python
from prodigy.components.stream import get_stream
from prodigy import set_hashes

 stream = get_stream(dataset, dedup=False, rehash=True)
 stream = (set_hashes(eg, input_keys=("text"), task_keys=("annotation")) for eg in stream)

```

I specify dedup as False and assign different task\_keys, however, the input data is always deduplicated and a sample only shows up with annotation\_task 'a'. I've tried all sorts of rehashing but it seems as long as the 'text' key is the same in the input stream, deduplication happens. What can I do to prevent deduplication?

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [April 24, 2024, 8:39pm UTC](https://support.prodi.gy/t/keeping-duplicates-in-stream/7195/2 "2024-04-24T20:39:28Z")

</div>

Hi @vera-bernhard ,

Could you try adding `overwrite` argument set to `True` in your call to `set_hashes`, that's

```python
 stream = (set_hashes(eg, task_keys=("annotation",), overwrite=True) for eg in stream)

```

Also, please note that I added a comma to `task_keys` value as it needs to be a tuple (I just realized this detail is missing from the docs)

You'll need that because you're setting hashes already in the call to `get_stream` so to make your custom hashing effective, you'd need to overwrite these default ones.

Let's see of fixing the call to `set_hashes` solves your problem.

In any case your examples:

> {'text': 'xyz', annotation\_task': 'a'}  
> {'text': 'xyz', 'annotation\_task': 'b'}  
> {'text': 'xyz', 'annotation\_task': 'c'}  
> {'text': 'stu', 'annotation\_task': 'a'}  
> {'text': 'stu', 'annotation\_task': 'b'}  
> {'text': 'stu', 'annotation\_task': 'c'}

would get each different \_task\_hash by default, because if no default keys are present the entire dictionary is used to generate the \_task\_hash and in this toy example each line is a different combination of keys and values.  
Also, one thing to have in mind is that `_input_hash` as always prefixed to the string from which the `_task_hash` is computed.

---

<div class="post-metadata">

**Author:** ![vera-bernhard](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vera-bernhard/32/4370_2.png) [@vera-bernhard](https://support.prodi.gy/u/vera-bernhard)\
**Post date:** [April 25, 2024, 8:36am UTC](https://support.prodi.gy/t/keeping-duplicates-in-stream/7195/3 "2024-04-25T08:36:54Z")

</div>

Thanks for your quick response. The issue remained after having set task \_keys and overwrite. However, I realised that a PatternMatcher further down the line, was overwriting the correctly set task\_keys. Now everything is working correctly.

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [April 25, 2024, 9:25am UTC](https://support.prodi.gy/t/keeping-duplicates-in-stream/7195/4 "2024-04-25T09:25:39Z")

</div>

Awesome! Thanks for reporting back 🙂 Also, not sure if you saw it: prodigy has basic and verbose logging which is usually helpful for debugging. You can turn it on via `PRODIGY_LOGGING=basic` or `PRODIGY_LOGGING=verbose` prepended to your Prodigy cmd