# Duplicates in ner.manual
**URL:** https://support.prodi.gy/t/duplicates-in-ner-manual/2278
**Category:** Uncategorized
**Tags:** enhancement, usage, done, ner
**Created:** [December 3, 2019, 1:45pm UTC](https://support.prodi.gy/t/duplicates-in-ner-manual/2278 "2019-12-03T13:45:17Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@blume](https://support.prodi.gy/u/blume)
#### Post date: [December 3, 2019, 1:45pm UTC](https://support.prodi.gy/t/duplicates-in-ner-manual/2278/1 "2019-12-03T13:45:17Z")
Hello,
we are using ner.manual. However, we have a lot of duplicates.
Roughly, the first 1000+x annotations were made twice, as they appeared twice in prodigy and at first, no one noticed. What attracts attention, is that the first „set“ (first half of the duplicates) has got the \_view\_id ‚NaN‘ (None/NULL), the second half has got „ner\_manual“.
However, we later annotated into the same dataset using ner.make-gold. This only produced duplicates by starting out from the beginning of the input file.
After having dropped the duplicates and re-inserted the data into an empty dataset, ner.manual starts anew with the input csv-file.
How can we handle duplicates and prevent them? It does cost a lot of time.
(The annotations have been done since April and we constantly upgraded prodigy, currently we run version 1.8.3. )
Thanks!
---
### Author:  [@ines](https://support.prodi.gy/u/ines)
#### Post date: [December 4, 2019, 8:40pm UTC](https://support.prodi.gy/t/duplicates-in-ner-manual/2278/2 "2019-12-04T20:40:49Z")
> [@blume](#):
>
> What attracts attention, is that the first „set“ (first half of the duplicates) has got the \_view\_id ‚NaN‘ (None/NULL), the second half has got „ner\_manual“.
Thanks for the report! I think this might explain what's going on: the `_view_id` property was added more recently and it seems like it's taken into account when generating the hashes to decide whether two examples are the same or not. I think it probably makes sense to exclude the `_view_id` from the hashing, since I can't think of many use cases where you would want to treat the same task displayed in a different interface as different questions.
In the meantime, you can export the data, make sure all of the examples have `"_view_id": "ner_manual"` set, call `prodigy.set_hashes(example, overwrite=True)` on each example to make sure the hashes are updated, and then re-import it. (For the next release, we'll also add a `--rehash` flag to `db-in` that takes care of this automatically.)
> [@blume](#):
>
> However, we later annotated into the same dataset using ner.make-gold. This only produced duplicates by starting out from the beginning of the input file.
This is currently expected, because Prodigy will filter based on the task hashes, i.e. the question. So if you annotate an example once with `ner.manual` (no pre-highlights) and then with `ner.make-gold` (pre-highlighted spans), both examples will have the same input hash but different task hashes, so they're treated as different questions on the same text.
For the next release, want to add a `"filter_by"` config setting that lets recipes specify whether to filter by task hashes (e.g. in binary recipes where you want to answer many accept/reject questions about the same text) or by input hashes (where you only want to annotate an example once).
In the meantime, you could add your own filter like this:
```python
def filter_stream(stream):
seen = set()
for eg in stream:
# Get the hash idenfitying the original input, e.g. the text
input_hash = eg["_input_hash"]
if input_hash not in seen:
yield eg
seen.add(input_hash)
stream = filter_stream(stream)
```
---
### Author:  [@blume](https://support.prodi.gy/u/blume)
#### Post date: [December 10, 2019, 8:27am UTC](https://support.prodi.gy/t/duplicates-in-ner-manual/2278/3 "2019-12-10T08:27:36Z")
Okay. Thank you, Ines for the explanation!
---
### Author:  [@ines](https://support.prodi.gy/u/ines)
#### Post date: [December 18, 2019, 11:31pm UTC](https://support.prodi.gy/t/duplicates-in-ner-manual/2278/4 "2019-12-18T23:31:10Z")
@blume Now shipped in v1.9 😃 `ner.manual` now sets `"exclude_by": "input"` by default, meaning that Prodigy will use only the raw text to decide whether two questions are identical. So if you've annotated a text before (with suggestions or without), you shouldn't be seeing it again (even if the suggestions are different).