# Avoid restarting from zero...

**URL:** <https://support.prodi.gy/t/avoid-restarting-from-zero/457>\
**Category:** Uncategorized\
**Tags:** enhancement, usage, solved\
**Created:** [March 30, 2018, 8:02am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457 "2018-03-30T08:02:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 30, 2018, 8:02am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/1 "2018-03-30T08:02:35Z")

</div>

Hi,  
i have noticed that ner.match start the annotation from the beginning at every annotation session. Why? I see the same samples i already have annotated.

When i reboot my pc and start the session again with:

```bash
python3 -m prodigy ner.match email it_core_news_sm cv.jsonl --patterns patterns.jsonl

```

prodigy start from the first sample on my cv.jsonl data source.

I have around 80k sentences, i would like to continue with new sentences, is that possible?

Thanks

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 30, 2018, 9:24am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/2 "2018-03-30T09:24:40Z")

</div>

By default, Prodigy makes a little assumptions about your stream as possible. But you can tell it to exclude annotations from one or more datasets via the `--exclude` option, e.g. `--exclude email`. You can exclude one or more dataset names (comma-separated list), including the current dataset. This is also very useful for evaluation data, because you want to make sure that no training examples end up in your evaluation set, and vice versa.

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 30, 2018, 11:46am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/3 "2018-03-30T11:46:29Z")

</div>

@ines thank you for your quick reply but i still have the same problem.

I have created a custom recipe that is 95% the same as ner.match. Basically this recipe truncates the text near the entity i should accept/reject. It is quite useful because as i already told you i am dealing with long texts. The code is:

```
def my_generator(stream, window):
	for _, eg in stream:                                 
		start = eg['spans'][0]['start'] - window if eg['spans'][0]['start']-window>=0 else 0
		end = eg['spans'][0]['end'] + window if eg['spans'][0]['end']+window<=len(eg['text']) else len(eg['text'])        
		eg['text'] = eg['text'][start:end]
		eg['spans'][0]['start'] = eg['spans'][0]['start'] - start
		eg['spans'][0]['end'] = eg['spans'][0]['end'] - start        
		yield eg

@recipe('custom.ner.match',
        dataset=recipe_args['dataset'],
        spacy_model=recipe_args['spacy_model'],
        patterns=recipe_args['patterns'],
        source=recipe_args['source'],
        window=plac.Annotation("window", type=int),
        api=recipe_args['api'],
        loader=recipe_args['loader'],        
        exclude=recipe_args['exclude']) 
def custom_ner_match(dataset, spacy_model, patterns, source, window=100,
                     api=None, loader=None, exclude=None):                         
    """
    Suggest phrases that match a given patterns file, and mark whether they
    are examples of the entity you're interested in. The patterns file can
    include exact strings, regular expressions, or token patterns for use with
    spaCy's `Matcher` class.
    """
    log("RECIPE: Starting recipe ner.match", locals())
    DB = connect()
    # Create the model, using a pre-trained spaCy model.
    model = PatternMatcher(spacy.load(spacy_model)).from_disk(patterns)
    log("RECIPE: Created PatternMatcher using model {}".format(spacy_model))
    if dataset is not None and dataset in DB:
        existing = DB.get_dataset(dataset)
        log("RECIPE: Updating PatternMatcher with {} examples from dataset {}"
            .format(len(existing), dataset))
        model.update(existing)
    stream = get_stream(source, api=api, loader=loader, rehash=True,
                        dedup=True, input_key='text')

    return {
        'view_id': 'ner',
        'dataset': dataset,
        'stream': my_generator(model(stream), window),
        'exclude': exclude
    }

```

Running it via

```bash
python3 -m prodigy custom.ner.match email it_core_news_sm cv.jsonl --patterns patterns.jsonl -F /home/damiano/recipe.py --exclude email

```

I still see the first sentence that i have annotated (i double-checked it with db-out and i already see the annotation).

However ner.match gives me the same problem. Any workaround?  
Thanks

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 30, 2018, 11:51am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/4 "2018-03-30T11:51:35Z")

</div>

Just to confirm: You’re using the latest version, `v1.4.1`, right?

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 30, 2018, 12:06pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/5 "2018-03-30T12:06:10Z")

</div>

Yes @ines

```
damiano@damiano:~$ python3 -m prodigy stats

  ✨ Prodigy stats

  Version 1.4.1              
  Total Sessions 77                 
  Prodigy Home /home/damiano/.prodigy 
  Location /home/damiano/.local/lib/python3.5/site-packages/prodigy 
  Platform Linux-4.13.0-37-generic-x86_64-with-Ubuntu-16.04-xenial 
  Total Datasets 3                  
  Database Name SQLite             
  Python Version 3.5.2              
  Database Id sqlite
```

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 31, 2018, 8:52am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/6 "2018-03-31T08:52:48Z")

</div>

@ines I have noticed that the problem only occurs when i open the browser the first time.  
Let me explain… I run this command:

`python -m prodigy ner.match test it_core_news_sm cv.jsonl --patterns patterns.jsonl --exclude test`

then i see the normal prodigy output:

```
  ? Starting the web server at http://localhost:8080 ...
  Open the app in your browser and start annotating!

```

When i open the browser i am ABLE to annotate the **same** (already annotated) sentences BUT if i call [http://localhost:8080](http://localhost:8080) a second time i see: `No tasks available.`

so, basically a `ctrl- r` fix the problem for the moment 😃

Hope this helps.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 31, 2018, 9:49am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/7 "2018-03-31T09:49:12Z")

</div>

Thanks for updating! And that’s interesting – this would indicate that some examples are not filtered out correctly in the first batch. Basically, what happens when you reload the web app is that it makes a new request to `/get_questions`, which asks Prodigy for a new batch of tasks.

Another thing you could try is rehashing the tasks manually in your `my_generator` function:

```python
from prodigy import set_hashes

# your other code here
yield set_hashes(eg)

```

This will ensure that each examples receives unique hashes that let Prodigy determine whether a question has been asked before. If this produces different hashes than the ones Prodigy created before, you might still see the same examples one more time – but once the hashes are in the dataset, the `--exclude` should work as expected.

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 31, 2018, 10:21am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/8 "2018-03-31T10:21:28Z")

</div>

@ines i am looking at ner.py and i see:

```
return {
    'view_id': 'ner',
    'dataset': dataset,
    'stream': (eg for _, eg in model(stream)),
    'exclude': exclude
}

```

in **ner.match** , so do you also need to put `set_hashes` there?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 31, 2018, 10:30am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/9 "2018-03-31T10:30:05Z")

</div>

No, the hashing should actually be taken care of automatically. Adding the manual to your custom recipe was just an idea to try in your custom `my_generator` function, to see if it helps us debug the problem.

But I can’t think of a reason why the filtering wouldn’t work, since the hashes created for the same match should be identical. (You could also check this by printing each example in your stream, and comparing the `_task_hash` with the one you already have in your dataset.)

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 31, 2018, 10:34am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/10 "2018-03-31T10:34:09Z")

</div>

As you have seen my recipe truncates the text but if the hash is created after the return…yes, it looks strange. Is the hash a simple md5 of the text?

Ok, i can do a test to see if the **\_task\_hash** change

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 31, 2018, 10:39am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/11 "2018-03-31T10:39:21Z")

</div>

By default, Prodigy’s hashes are 32-bit signed integers and we use [`mmh3`](https://pypi.python.org/pypi/mmh3). They can take various task properties into account – like the text or image for the input hash, and the input plus the spans, options etc. for the task hash. The `set_hashes` docs in the `PRODIGY_README.html` have some more info on this.

That said, this is just the default solution – if you ever feel like you want to implement your own, custom hashing solution, that’s also possible. Prodigy will respect already pre-set hashes on the data. and doesn’t really care about how they’re generated. (For example, depending on your data, you might want to set the same hashes, regardless of the capitalisation.)

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 31, 2018, 6:05pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/12 "2018-03-31T18:05:37Z")

</div>

@ines the **\_task\_hash** is the same:

```
damiano@damiano:~$ python3 -m prodigy custom.ner.match test it_core_news_sm cv.jsonl --patterns patterns.jsonl -F /home/damiano/recipe.py

  ✨ Starting the web server at http://localhost:8080 ...
  Open the app in your browser and start annotating!

{'text': 'testo di prova', '_input_hash': 397300493, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, '_task_hash': 1271078817, 'spans': [{'score': 0.7142857313156128, 'text': 'prova', 'pattern': 0, 'end': 14, 'label': 'TEST', 'start': 9, 'priority': 0.7142857313156128}]}
######################################
{'text': 'testo di er tret ert prova ert ert ert er', '_input_hash': -466006437, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, '_task_hash': 1234389887, 'spans': [{'score': 0.7142857313156128, 'text': 'prova', 'pattern': 0, 'end': 26, 'label': 'TEST', 'start': 21, 'priority': 0.7142857313156128}]}
######################################
{'text': 'testo di prova prova sdwef wefwef wef wefwe323', '_input_hash': -1827486154, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, '_task_hash': 1488445579, 'spans': [{'score': 0.7142857313156128, 'text': 'prova', 'pattern': 0, 'end': 14, 'label': 'TEST', 'start': 9, 'priority': 0.7142857313156128}]}
######################################
{'text': 'testo di prova prova sdwef wefwef wef wefwe323', '_input_hash': -1827486154, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, '_task_hash': 1572291564, 'spans': [{'score': 0.7142857313156128, 'text': 'prova', 'pattern': 0, 'end': 20, 'label': 'TEST', 'start': 15, 'priority': 0.7142857313156128}]}
######################################
{'text': 'testo di ddd d ddd d dd dprova ew ewew weprova prova wewef f 4 43', '_input_hash': 395734250, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, '_task_hash': -6107930, 'spans': [{'score': 0.7142857313156128, 'text': 'prova', 'pattern': 0, 'end': 52, 'label': 'TEST', 'start': 47, 'priority': 0.7142857313156128}]}
######################################

```

i have annotated those task, then after reloading it with:

```
damiano@damiano:~$ python3 -m prodigy custom.ner.match test it_core_news_sm cv.jsonl --patterns patterns.jsonl -F /home/damiano/recipe.py --exclude test

  ✨ Starting the web server at http://localhost:8080 ...
  Open the app in your browser and start annotating!

{'_input_hash': 397300493, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, 'spans': [{'priority': 0.7142857313156128, 'end': 14, 'text': 'prova', 'score': 0.7142857313156128, 'start': 9, 'pattern': 0, 'label': 'TEST'}], '_task_hash': 1271078817, 'text': 'testo di prova'}
######################################
{'_input_hash': -466006437, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, 'spans': [{'priority': 0.7142857313156128, 'end': 26, 'text': 'prova', 'score': 0.7142857313156128, 'start': 21, 'pattern': 0, 'label': 'TEST'}], '_task_hash': 1234389887, 'text': 'testo di er tret ert prova ert ert ert er'}
######################################
{'_input_hash': -1827486154, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, 'spans': [{'priority': 0.7142857313156128, 'end': 14, 'text': 'prova', 'score': 0.7142857313156128, 'start': 9, 'pattern': 0, 'label': 'TEST'}], '_task_hash': 1488445579, 'text': 'testo di prova prova sdwef wefwef wef wefwe323'}
######################################
{'_input_hash': -1827486154, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, 'spans': [{'priority': 0.7142857313156128, 'end': 20, 'text': 'prova', 'score': 0.7142857313156128, 'start': 15, 'pattern': 0, 'label': 'TEST'}], '_task_hash': 1572291564, 'text': 'testo di prova prova sdwef wefwef wef wefwe323'}
######################################
{'_input_hash': 395734250, 'meta': {'score': 0.7142857313156128, 'pattern': 0, 'source': 'Tuoagente'}, 'spans': [{'priority': 0.7142857313156128, 'end': 52, 'text': 'prova', 'score': 0.7142857313156128, 'start': 47, 'pattern': 0, 'label': 'TEST'}], '_task_hash': -6107930, 'text': 'testo di ddd d ddd d dd dprova ew ewew weprova prova wewef f 4 43'}
######################################

```

As you can see the hash is the same, so i wonder how a re-hash could help here.  
After a `ctrl-r` i see **No task available**.

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [March 31, 2018, 7:08pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/13 "2018-03-31T19:08:02Z")

</div>

Should i follow this approach: [Continue to annotate same data in new session](https://support.prodi.gy/t/continue-to-annotate-same-data-in-new-session/118/8) ?

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [April 1, 2018, 8:28am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/14 "2018-04-01T08:28:23Z")

</div>

unfortunately, **set\_hashes** did not fix the problem i still see already annotated tasks.

---

<div class="post-metadata">

**Author:** ![damiano](https://avatars.discourse-cdn.com/v4/letter/d/ebca7d/32.png) [@damiano](https://support.prodi.gy/u/damiano)\
**Post date:** [April 1, 2018, 10:10am UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/15 "2018-04-01T10:10:46Z")

</div>

**SOLVED**  
@ines sorry for stressing you out so much! 🙂  
I have changed my generator in this way:

```
def my_generator(stream, window):
	for _, eg in stream:                                 
		start = eg['spans'][0]['start'] - window if eg['spans'][0]['start']-window>=0 else 0
		end = eg['spans'][0]['end'] + window if eg['spans'][0]['end']+window<=len(eg['text']) else len(eg['text'])        
		eg['text'] = eg['text'][start:end]
		eg['spans'][0]['start'] = eg['spans'][0]['start'] - start
		eg['spans'][0]['end'] = eg['spans'][0]['end'] - start        	
		eg['_task_hash'] = mmh3.hash(eg['text'])		
		yield eg 

```

forcing a new `_tash_hash` based on `text`  
Hope this help someone.

Happy Easter 🙂

---

<div class="post-metadata">

**Author:** ![cheerfulstoic](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/cheerfulstoic/32/266_2.png) [@cheerfulstoic](https://support.prodi.gy/u/cheerfulstoic)\
**Post date:** [April 17, 2018, 7:31pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/16 "2018-04-17T19:31:26Z")

</div>

I’m trying to use the `--exclude` flag, but I think it isn’t working how I would expect. From researching, it seems to be based off of the `_task_hash` which is generated from the text and other properties. What properties is it based off of? For example, I’m seeing for a specific example that the priority / score of the span are different whereas everything else (the full text and the span text/start/end/pattern/label) is the same. Is the `_task_hash` based off of priority and/or score? If so, how do all of the things get combined to be hashed by `mmh3`? If they are included, if I’m understanding right, perhaps I could make my own generator which simply generates my own `_task_hash` which is based on everything but those properties (??)

Thanks!

---

<div class="post-metadata">

**Author:** ![cheerfulstoic](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/cheerfulstoic/32/266_2.png) [@cheerfulstoic](https://support.prodi.gy/u/cheerfulstoic)\
**Post date:** [April 17, 2018, 8:29pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/17 "2018-04-17T20:29:17Z")

</div>

(I’m using version 1.4.1, by the way)

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [April 18, 2018, 2:11pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/18 "2018-04-18T14:11:34Z")

</div>

Yes, you can definitely customise how the hashes are generated – check out the docs for the `set_hashes` helper function. The `input_keys` and `task_keys` let you define the string names of the keys you want to include in the respective hashes. The input keys default to `('text', 'image', 'html', 'input')` and the task keys to `('spans', 'label', 'options')`. A custom generator could then look like this:

```python
stream = (set_hashes(eg, input_keys=('text', 'custom_text')) for eg in stream)

```

---

<div class="post-metadata">

**Author:** ![cheerfulstoic](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/cheerfulstoic/32/266_2.png) [@cheerfulstoic](https://support.prodi.gy/u/cheerfulstoic)\
**Post date:** [May 9, 2018, 8:34pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/19 "2018-05-09T20:34:14Z")

</div>

Thanks so much! Sorry, I neglected to look at the documentation on this before. I still think I have a question about the Task hash, though. By default it’s based on `('spans', 'label', 'options')`, but the issue that I’m trying to figure out is that while I like that it’s based on the `spans` (so that it can differentiate the different things it might try to highlight in the same fragment), the `spans` have a `score` which seems to change and thus change the `task_hash` unnecessarily (at least for what I’m trying to do). Here is an example to hopefully make it clearer:

```json
{  
   "text":"This matter was originally scheduled for May 29, 2015, but it was properly noticed for this session in order for this matter to be heard.",
   "_input_hash":-603642722,
   "_task_hash":391498376,
   "spans":[  
      {  
         "start":45,
         "end":47,
         "text":"29",
         "rank":4,
         "label":"WEAPON",
         "score":0.036862158,
         "source":"en_core_web_sm",
         "input_hash":-603642722
      }
   ],
   "meta":{  
      "score":0.036862158
   },
   "answer":"reject"
}
{  
   "text":"This matter was originally scheduled for May 29, 2015, but it was properly noticed for this session in order for this matter to be heard.",
   "_input_hash":-603642722,
   "_task_hash":557712048,
   "spans":[  
      {  
         "start":45,
         "end":47,
         "text":"29",
         "rank":3,
         "label":"WEAPON",
         "score":0.1024216291,
         "source":"en_core_web_sm",
         "input_hash":-603642722
      }
   ],
   "meta":{  
      "score":0.1024216291
   },
   "answer":"reject"
}   

```

The `input_hash` in the same for both, but the `task_hash` is different because the `rank` and `score` were different (I assume, at least), and I’m not sure why. Ideally I’d like to not see this same sentence and span again

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [May 10, 2018, 5:30pm UTC](https://support.prodi.gy/t/avoid-restarting-from-zero/457/20 "2018-05-10T17:30:19Z")

</div>

@cheerfulstoic Thanks a lot for the update – this is interesting and a point we hadn’t really considered before. This might also explain some of the inconsistencies in the past.

The `score` is the model’s prediction, which varies depending on where you’re at in the annotation process. If the model in the loop was already updated with various annotations, it will score an entity differently than the base model that hasn’t learned anything new yet. That’s why you end up with different scores – and different hashes.

We need to think of a good solution for this. Of course, the hashing shouldn’t just ignore random attributes, because that’d make it unreliable… on the other hand, the score shouldn’t influence the hash, because that makes it a lot less useful 🤔

In the meantime, if you’re okay with filtering on the `input_hash`, you could just add a wrapper around your stream that gets all input hashes from the datbase (`db.get_input_hashes`) and then filters examples with the same hash from the stream. You can also use the built-in filter `prodigy.components.filters.filter_inputs` that does exactly that – see the docs for the API details.