# Exclude not functioning / duplicate tasks

**URL:** <https://support.prodi.gy/t/exclude-not-functioning-duplicate-tasks/3137>\
**Category:** Uncategorized\
**Tags:** done, streams\
**Created:** [July 2, 2020, 1:28pm UTC](https://support.prodi.gy/t/exclude-not-functioning-duplicate-tasks/3137 "2020-07-02T13:28:45Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [July 2, 2020, 2:36pm UTC](https://support.prodi.gy/t/exclude-not-functioning-duplicate-tasks/3137/2 "2020-07-02T14:36:54Z")

</div>

Hey, thanks a lot for the super detailed report! 🙏

> [@pmbaumgartner](#):
>
> I'm a little stumped here. Is this the intended behavior of `--exclude` and I'm missing part of the wider annotation process? Are the patterns overriding examples that would be excluded? Is it an issue with the hashing logic?

Before looking into this in more detail, I can definitely confirm what the intended behaviour is: `ner.teach` excludes by task hash, so two examples with the same task hash are considered duplicates and you should never be asked about them twice. The task hash is based on the text and span/label so you may be asked about different suggestions on the same text, but never about the same text + span + label combination. If an incoming example has the same task hash as an example in one of the excluded sets (via `--exclude` or the current dataset) and it's presented to you, that's a bug.

Other recipes, mostly the manual ones like `ner.manual`, exclude by input (via the `"exclude_by": "input"` config setting) because the assumption here is that you want to create one gold-standard annotation for each text and don't want to see the same text again, even if it comes in later with different pre-highlighted suggestions.

[This thread](https://support.prodi.gy/t/textcat-not-excluding-dataset/3101/8) made me a little suspicious about the `--exclude` option with a separate dataset. Although nothing really changed around this, so I'm not entirely sure where the problem would be 🤔 But it's probably the first thing we should double-check.

> [@pmbaumgartner](#):
>
> Running `prodigy db-merge test_1,test_2 test_3` doesn't merge any of the examples, and then `prodigy db-out test_3 test_3_out` tells me that there's 8 (expecting 4) examples in the merged dataset.

`db-merge` currently only appends and doesn't do any hashing/filtering/combining (that's currently only done during training and when you run `data-to-spacy`). So if you have 4 examples + 4 duplicates, you'll end up with one set of 8 examples.

---

_[View the full topic](https://support.prodi.gy/t/exclude-not-functioning-duplicate-tasks/3137)._