# Parameter "dedup" in get\_stream function

**URL:** <https://support.prodi.gy/t/parameter-dedup-in-get-stream-function/539>\
**Category:** Uncategorized\
**Tags:** usage, solved\
**Created:** [May 11, 2018, 9:00pm UTC](https://support.prodi.gy/t/parameter-dedup-in-get-stream-function/539 "2018-05-11T21:00:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![soumyagk](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/soumyagk/32/317_2.png) [@soumyagk](https://support.prodi.gy/u/soumyagk)\
**Post date:** [May 11, 2018, 9:00pm UTC](https://support.prodi.gy/t/parameter-dedup-in-get-stream-function/539/1 "2018-05-11T21:00:26Z")

</div>

I created a simple prodigy recipe that streams from a JSONL file to the prodigy interface and stores along with answer, to a dataset.  
My JSONL file contains tasks in the prodigy `ner` format, i.e., each line is a dictionary with keys `'text'` and `'spans'`. It has repeated sentences but with different spans specified in a given task.  
When I read the file through the `get_stream` function like below:

```python
return {
        'dataset': dataset,
        'stream': get_stream(source, rehash=True, dedup=True),
        'view_id': 'ner',
        'update': None,
    }

```

it appears to be filtering based on input hash rather than task hash. Is that the intended bahavior?  
I could use `filter_duplicates` function too, but I would avoid it if it can be done with the `dedup` parameter instead.  
I am using prodigy version 1.4.2

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [May 11, 2018, 9:45pm UTC](https://support.prodi.gy/t/parameter-dedup-in-get-stream-function/539/2 "2018-05-11T21:45:59Z")

</div>

> [@soumyagk](#):
>
> it appears to be filtering based on input hash rather than task hash. Is that the intended bahavior?

Yes, the `dedup` parameter is explicitly for filtering out duplicate incoming examples – i.e. if your data contains the exact same text twice. `get_stream` mostly really cares about the input and input hashes, because in most recipes, it's run _before_ the model or some other process, which then sets the spans, labels etc.

So in your case, using `filter_duplicates` seems like a good solution! 👍

---

<div class="post-metadata">

**Author:** ![nuno](https://avatars.discourse-cdn.com/v4/letter/n/65b543/32.png) [@nuno](https://support.prodi.gy/u/nuno)\
**Post date:** [November 4, 2019, 5:56pm UTC](https://support.prodi.gy/t/parameter-dedup-in-get-stream-function/539/3 "2019-11-04T17:56:26Z")

</div>

Hi Ines,

What would be the correct way of using the `dedup` argument if you don't want duplicates to be filtered out? I tried `dedup=False` but it still filtered all my examples from the stream and showed "No tasks available".

This is my code:

```
stream = get_stream(source, api=api, loader=loader, rehash=True,
                    dedup=False, input_key='text')

return {
    'dataset': dataset, # save annotations in this dataset
    'view_id': 'choice', # use the choice interface
    'stream': stream,
    "choice_auto_accept": True,
    'config': {'choice_style': 'multiple', 'show_stats': True},

```

Thank you!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [November 4, 2019, 6:50pm UTC](https://support.prodi.gy/t/parameter-dedup-in-get-stream-function/539/4 "2019-11-04T18:50:53Z")

</div>

Hi! Setting `dedup=True` will filter out identical examples in the stream – e.g. `{"text": "hello"}` and `{"text": "hello"}`. Do you have anything in your dataset already? By default, Prodigy will also filter out questions that have already been answered – so that's the effect you might be seeing here. If you don't want this behaviour, you can set `"auto_exclude_current": False` in the `"config"`. This means that the current dataset won't be automatically excluded.