# Repeating examples when using \`--exclude\` with \`ner.correct\`

**URL:** <https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699>\
**Category:** Uncategorized\
**Tags:** usage, ner\
**Created:** [December 3, 2020, 8:40pm UTC](https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699 "2020-12-03T20:40:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rdonaldson](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@rdonaldson](https://support.prodi.gy/u/rdonaldson)\
**Post date:** [December 3, 2020, 8:40pm UTC](https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699/1 "2020-12-03T20:40:09Z")

</div>

Hi All,

When I use the `--exclude` flag in `ner.correct` I see the same 26 examples over and over. When I shut down the server and restart, I get a new set of 26 examples and can continue annotating, but restarting the annotation process every 26 examples doesn't seem like a workflow that is in the spirit of prodi.gy. Am I doing something wrong?

In the spirit of Ines' "ingredients" NER model, I'm running a command like this (names changed to protect the innocent):

```python
(prodigy-1.10.5) [~/work]$ prodigy ner.correct dataset_2 ./tmp_model prepped-data.jsonl --label FRUIT,VEG,MEAT,DAIRY,GRAIN --exclude dataset_1

```

The same command without the `--exclude` argument serves all the examples in `prepped-data.jsonl` (but of course doesn't exclude the examples in `dataset_1`).

Here's my `prodigy.json` config (most are default settings):

```python
{
  "theme": "basic",
  "custom_theme": {},
  "buttons": ["accept", "reject", "ignore", "undo"],
  "batch_size": 10,
  "history_size": 10,
  "port": 8080,
  "host": "0.0.0.0",
  "cors": true,
  "db": "sqlite",
  "db_settings": {},
  "api_keys": {},
  "validate": true,
  "auto_exclude_current": true,
  "instant_submit": false,
  "feed_overlap": false,
  "ui_lang": "en",
  "project_info": ["dataset", "session", "lang", "recipe_name", "view_id", "label"],
  "show_stats": false,
  "hide_meta": false,
  "show_flag": false,
  "instructions": false,
  "swipe": false,
  "split_sents_threshold": false,
  "html_template": false,
  "global_css": null,
  "javascript": null,
  "writing_dir": "ltr",
  "show_whitespace": false,
  "exclude_by": "input"
}

```

I'm running `Prodigy 1.10.5` on Linux, `Python 3.6.9`.

I'm working around this by piping my data in using bash `tail` thusly:

```python
(prodigy-1.10.5) [~/work]$ tail -n +100 prepped-data.jsonl | prodigy ner.correct dataset_2 ./tmp_model - -l FRUIT,VEG,MEAT,DAIRY,GRAIN

```

I increment the `-n` argument for each batch of data to correct, but this still breaks the flow a little. Is there a way to use `--exclude` and see a full stream of data using prodi.gy alone?

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [December 4, 2020, 1:55am UTC](https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699/2 "2020-12-04T01:55:00Z")

</div>

Hi! That definitely doesn't sound like the intended behaviour. Could you double-check whether the incoming examples end up with the same hashes as the examples already in the dataset that's excluded? (Quick debugging tip: if you make Prodigy execute any JavaScript, like `"javascript": "console.log('js!')"`, you can log `window.prodigy.content` in your developer console to see the JSON of whatever is currently on the screen).

If the hashes are the same, it's more likely related to the stream orchestration 🤔 In that case, can you check if setting `"force_stream_order": false` solves the problem?

---

<div class="post-metadata">

**Author:** ![rdonaldson](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@rdonaldson](https://support.prodi.gy/u/rdonaldson)\
**Post date:** [December 4, 2020, 8:25pm UTC](https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699/3 "2020-12-04T20:25:40Z")

</div>

Thanks for the tips! (And for being so quick!)

I tried out your debugging tip and was able to confirm that the hashes of the examples that repeat over and over indeed are unchanged when I see them on subsequent cycles through the set of repeating examples.

So...setting `"force_stream_order": false` does indeed solve the problem!

This doesn't exactly feel like "case closed" since `"force_stream_order": false` is the default, and even if it were not, the docs suggest that setting `"force_stream_order": true` should still register annotated examples as "finished". However, this does solve _my_ problem, so thank-you! I hope it solves somebody else's problem too!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [December 8, 2020, 5:27am UTC](https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699/4 "2020-12-08T05:27:38Z")

</div>

Yes, this is definitely not the "solution", but it helps narrow in on the underlying problem. (The recipe in this case sets `"force_stream_order": True`, so it sounds like you've hit some edge case here that makes Prodigy believe that a batch hasn't been annotated yet, even though it's present in the dataset.)

---

<div class="post-metadata">

**Author:** ![jdagdelen](https://avatars.discourse-cdn.com/v4/letter/j/db5fbb/32.png) [@jdagdelen](https://support.prodi.gy/u/jdagdelen)\
**Post date:** [May 22, 2021, 1:41am UTC](https://support.prodi.gy/t/repeating-examples-when-using-exclude-with-ner-correct/3699/5 "2021-05-22T01:41:38Z")

</div>

We were able to recreate this bug exactly (up to the 26 and repeat) on our setup as well.