# duplicate example **URL:** \ **Category:** Uncategorized\ **Tags:** usage\ **Created:** [September 8, 2022, 10:04am UTC](https://support.prodi.gy/t/duplicate-example/5922 "2022-09-08T10:04:01Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![aisha\_harbi](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@aisha\_harbi](https://support.prodi.gy/u/aisha_harbi)\ **Post date:** [September 8, 2022, 10:04am UTC](https://support.prodi.gy/t/duplicate-example/5922/1 "2022-09-08T10:04:01Z")
Hi, the prodigy app is showing duplicate example multiple times, how do I stop this? Noting that I am using PRODIGY\_ALLOWED\_SESSION but only one user/session is currently is active and "feed\_overlap": false in prodigy.json. Thanks so much. ---
**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\ **Post date:** [September 8, 2022, 12:37pm UTC](https://support.prodi.gy/t/duplicate-example/5922/2 "2022-09-08T12:37:09Z")
Hi Aisha, there can be a few reasons why you might see duplicates. In Prodigy the task and the examples are hashed separately, and the combination of the two determines uniqueness. So it might be related to you how your tasks are defined in your examples. It might also be that you're running a custom recipe without running [filter\_duplicates](https://prodi.gy/docs/api-components#filter_duplicates) or [get\_stream without dedup](https://prodi.gy/docs/api-components#get_stream). Could you share the Prodigy command that you're trying to run? Perhaps with a small example dataset so that I might reproduce locally? ---
**Author:** ![aisha\_harbi](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@aisha\_harbi](https://support.prodi.gy/u/aisha_harbi)\ **Post date:** [September 8, 2022, 1:21pm UTC](https://support.prodi.gy/t/duplicate-example/5922/3 "2022-09-08T13:21:09Z")
Thanks for replying! I tired using `stream = prodigy.get_stream(source, dedup = True)` and I got the following error: ```python for eg in stream: File "/home/ubuntu/relation_extraction/custom_relACE05.py", line 121, in add_tokens for eg in stream: File "cython_src/prodigy/components/loaders.pyx", line 29, in _add_attrs File "cython_src/prodigy/components/filters.pyx", line 48, in filter_duplicates KeyError: '_input_hash' ``` ---
**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\ **Post date:** [September 8, 2022, 1:24pm UTC](https://support.prodi.gy/t/duplicate-example/5922/4 "2022-09-08T13:24:04Z")
You can use [prodigy.set\_hashes](https://prodi.gy/docs/api-components#set_hashes) to specify the hashes that will be used for deduplication. Did you apply that beforehand? Got an example of an item in your source? ---
**Author:** ![aisha\_harbi](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@aisha\_harbi](https://support.prodi.gy/u/aisha_harbi)\ **Post date:** [September 8, 2022, 1:44pm UTC](https://support.prodi.gy/t/duplicate-example/5922/5 "2022-09-08T13:44:52Z")
**It consists of 3 keys: tokens, text, and relations\_x which consist of arg1 and arg2 and the relation's type (in addition to the arg text, type and span) what the custom recipe does is add entity spans and labels (NER labels from arg1 and arg2 types) and then adds the relations within relations\_x** the output fed to the UI similar to this: ```python { "text": "My mother’s name is Sasha Smith. She likes dogs and pedigree cats.", "tokens": [ {"text": "My", "start": 0, "end": 2, "id": 0, "label": O}, {"text": "mother", "start": 3, "end": 9, "id": 1, "label": LOC}, .... ], {"text": "pedigree", "start": 52, "end": 60, "id": 12, "ws": true}, "relations": [ { "head": 0, "child": 1, "label": "POSS", "head_span": {"start": 0, "end": 2, "token_start": 0, "token_end": 0, "label": null}, "child_span": {"start": 3, "end": 9, "token_start": 1, "token_end": 1, "label": null} }, { "head": 1, "child": 8, "label": "COREF", "head_span": {"start": 3, "end": 9, "token_start": 1, "token_end": 1, "label": null}, "child_span": {"start": 33, "end": 36, "token_start": 8, "token_end": 8, "label": null} }, { "head": 9, "child": 13, "label": "OBJECT", "head_span": {"start": 37, "end": 42, "token_start": 9, "token_end": 9, "label": null}, "child_span": {"start": 52, "end": 65, "token_start": 12, "token_end": 13, "label": "NP"} } ] } ``` Here's an example of the source input: it's in Arabic. `{"tokens": ["\u0623\u0639\u0644\u0646", "\u0648\u0632\u064a\u0631", "\u062f\u0627\u062e\u0644\u064a\u0629", "\u0643\u0648\u062f\u064a\u0641\u0648\u0627\u0631", "\u0627\u0633\u062a\u0639\u0627\u062f.....], "text": "\u0623\u0639\u0644\u0646 \u0648\u0632\u064a\u0631 \u062f\u0627\u062e\u0644\u064a\u0629 \u0643\u0648\u062f\u064a\u0641\u0648\u0627\u0631 \u0627\u0633\u062a\u0639\u0627\u062f\u0629 .....", "relations\_x": [ {"label": "PART-WHOLE:Geographical", "arg1-span": [11, 12], "arg1-text": "\u0623\u0631\u062c\u0627\u0621", "arg1-type": "LOC:Region-General", "arg2-span": [12, 13], "arg2-text": "\u0627\u0644\u0628\u0644\u0627\u062f", "arg2-type": "GPE:Nation"}, {"label": "ORG-AFF:Employment", "arg1-span": [1, 2], "arg1-text": "\u0648\u0632\u064a\u0631", "arg1-type": "PER:Individual", "arg2-span": [2, 3], "arg2-text": "\u062f\u0627\u062e\u0644\u064a\u0629", "arg2-type": "ORG:Government"}, ...... }]}` ---
**Author:** ![aisha\_harbi](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@aisha\_harbi](https://support.prodi.gy/u/aisha_harbi)\ **Post date:** [September 8, 2022, 2:12pm UTC](https://support.prodi.gy/t/duplicate-example/5922/6 "2022-09-08T14:12:42Z")
@koaning never mind I think my dataset might already contain duplicate examples 😕 Thanks! ---
**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\ **Post date:** [September 8, 2022, 2:13pm UTC](https://support.prodi.gy/t/duplicate-example/5922/7 "2022-09-08T14:13:26Z")
Ah! Yeah, that's also a possibility 🙂 !