# Adding data to a Prodigy dataset using db-in - is there a way to filter out/remove duplicate annotations? **URL:** \ **Category:** Uncategorized\ **Tags:** usage, solved\ **Created:** [December 29, 2022, 1:15pm UTC](https://support.prodi.gy/t/adding-data-to-a-prodigy-dataset-using-db-in-is-there-a-way-to-filter-out-remove-duplicate-annotations/6207 "2022-12-29T13:15:37Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![jspinella](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/jspinella/32/2898_2.png) [@jspinella](https://support.prodi.gy/u/jspinella)\ **Post date:** [December 29, 2022, 1:15pm UTC](https://support.prodi.gy/t/adding-data-to-a-prodigy-dataset-using-db-in-is-there-a-way-to-filter-out-remove-duplicate-annotations/6207/1 "2022-12-29T13:15:37Z")
I have a process whereby data from an external database is converted to a Prodigy-supported JSONL file, which then feeds it into Prodigy via `db-in`. This process runs daily, and each time it runs, the same annotations are added over and over again to this dataset, in addition to records added to this external database since the last run. Does Prodigy have the ability to skip importing what it identifies as duplicate annotations? I am building the JSONL file programmatically, so I can create the JSONL file however I need to in order for Prodigy to recognize duplicate annotations. For example, would I need to generate hashes for the input and task when building the JSONL file, and does Prodigy know to skip re-adding the same annotations with `db-in` by looking at those values? The Prodigy documentation on deduplication doesn't seem to mention deduplication per sé, rather it focuses on the ability to manually exclude annotations from recipes by specifying hashes. I recognize this is an atypical use-case! ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [December 29, 2022, 5:45pm UTC](https://support.prodi.gy/t/adding-data-to-a-prodigy-dataset-using-db-in-is-there-a-way-to-filter-out-remove-duplicate-annotations/6207/2 "2022-12-29T17:45:31Z")
hi @jspinella! Thanks for your question. > [@jspinella](#): > > Does Prodigy have the ability to skip importing what it identifies as duplicate annotations? Yes! Have you seen the [`get_stream`](https://prodi.gy/docs/api-components#get_stream) function? It's a generalization loader and can deduplicate by setting its `dedup` argument to `True` (it is `False` by default). In this case, it'll then dedup by whatever you specify as `exclude_by` in your configuration. By default, it is by the `"task"` hash but can be changed to the `"input"` hash. Typically, most recipes will load the file as a stream using `get_stream` then pass that stream through the recipe. However, in your case where you want to use `db-in` with deduplication is that you could modify the existing `db-in` recipe by replacing the loader with `get_stream` with how it is currently loaded (i.e., `get_loader`). To do this, you can find the Python script for `db-in` by first finding the location of your Prodigy installation. You can find this by running `python -m prodigy stats` and find the `Location:` path. Open that folder and then look for the `/recipes/commands.py` script, where you'll see the recipe for `db-in`. I'd then recommend using the `get_stream` function to load your file. > [@jspinella](#): > > For example, would I need to generate hashes for the input and task when building the JSONL file, and does Prodigy know to skip re-adding the same annotations with `db-in` by looking at those values? If you can open up the `db-in` and view the recipe, you'll be able to see where part of that script it has `set_hashes` to generate them on your own. One nice feature with `get_stream` is that you can rehash again if you wanted by setting that argument to `True`. > [@jspinella](#): > > The Prodigy documentation on deduplication doesn't seem to mention deduplication per sé, rather it focuses on the ability to manually exclude annotations from recipes by specifying hashes. Have you seen the [`filter_duplicates` function](https://prodi.gy/docs/api-components#filter_duplicates) too? As that part of the docs mention, it's how the deduplication is done in `get_stream`. ```python from prodigy.components.filters import filter_duplicates from prodigy import set_hashes stream = [{"text": "foo", "label": "bar"}, {"text": "foo", "label": "bar"}, {"text": "foo"}] stream = [set_hashes(eg) for eg in stream] stream = filter_duplicates(stream, by_input=False, by_task=True) # [{'text': 'foo', 'label': 'bar', '_input_hash': ..., '_task_hash': ...}, {'text': 'foo', '_input_hash': ..., '_task_hash': ...}] stream = filter_duplicates(stream, by_input=True, by_task=True) # [{'text': 'foo', 'label': 'bar', '_input_hash': ..., '_task_hash': ...}] ``` Essentially, an alternative approach could be just to run `filter_duplicates` in your `db-in` after the hashes are set. I generally prefer using `get_stream` as it is more general but this is an alternative. Hopefully, this information should give you enough to dedup as you deem fit. Let me know if this works and if you have any further questions. ---
**Author:** ![jspinella](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/jspinella/32/2898_2.png) [@jspinella](https://support.prodi.gy/u/jspinella)\ **Post date:** [January 4, 2023, 9:31pm UTC](https://support.prodi.gy/t/adding-data-to-a-prodigy-dataset-using-db-in-is-there-a-way-to-filter-out-remove-duplicate-annotations/6207/3 "2023-01-04T21:31:41Z")
This is perfect, thank you Ryan!