# Filter already annotated text

**URL:** https://support.prodi.gy/t/filter-already-annotated-text/5124
**Category:** Uncategorized
**Tags:** solved, usage, streams
**Created:** [December 23, 2021, 12:15pm UTC](https://support.prodi.gy/t/filter-already-annotated-text/5124 "2021-12-23T12:15:32Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![joaomsimoes](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/joaomsimoes/32/2562_2.png) [@joaomsimoes](https://support.prodi.gy/u/joaomsimoes)
#### Post date: [December 23, 2021, 12:15pm UTC](https://support.prodi.gy/t/filter-already-annotated-text/5124/1 "2021-12-23T12:15:32Z")

</div>

Hallo,

I built a custom recipe to do text classification according to a query. The dataset is a big CSV file. When I start a new session I basically start from the beginning. I tried to use the filter\_inputs to filter the inputs saved from the last session but I still have the same problem.

```python
@prodigy.recipe(
    "semanticsearch",
    dataset=("The dataset to use", "positional", None, str),
    source=("The source data as a CSV file", "positional", None, str)
)
def semanticsearch(
        dataset: str,
        source: str):

    db = connect()
    input_hashes = db.get_input_hashes(dataset)

    stream = CSV(source)
    stream = filter_inputs(stream, input_hashes)

    blocks = [
        {"view_id": "html",
         "html_template": "<div style='background-color:SlateBlue;'><h1 style='color:White;'>{{label}}</h1></div>"},
        {"view_id": "html", "html_template": "<div>{{text}}</div>"}
    ]

    return {
        "view_id": "blocks", # Annotation interface to use
        "dataset": dataset, # Name of dataset to save annotations
        "stream": stream, # Incoming stream of examples
        "config": {"blocks": blocks}
    }

```

Any help?

LG

---

<div class="post-metadata">

### Author: ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)
#### Post date: [December 24, 2021, 1:09pm UTC](https://support.prodi.gy/t/filter-already-annotated-text/5124/2 "2021-12-24T13:09:03Z")

</div>

Hi! Is it possible that the hashes somehow change between the runs? I don't immediately see anything in your code that indicates that (like a timestamp in the main content or something like that). But it's definitely something to check.

The way you're using `filter_inputs` here won't work if your stream doesn't yet include hashes. So you either want to call `prodigy.set_hashes` on all your incoming examples before filtering, or write out the logic explicitly like this:

```python
def filter_stream(stream, input_hashes):
    for eg in stream:
        eg = prodigy.set_hashes(eg)
        if eg["_input_hash"] not in input_hashes:
            yield eg

stream = filter_stream(stream, input_hashes)

```

This also makes it easy to print the hashes if you need to, to double-check if they correctly reflect the "uniqueness" of the content.

---

<div class="post-metadata">

### Author: ![joaomsimoes](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/joaomsimoes/32/2562_2.png) [@joaomsimoes](https://support.prodi.gy/u/joaomsimoes)
#### Post date: [December 27, 2021, 7:02am UTC](https://support.prodi.gy/t/filter-already-annotated-text/5124/3 "2021-12-27T07:02:05Z")

</div>

Hi Ines!

It worked with no problems!

Thanks for the help 👍