function set_hashes

[ryanwesslen]

Hi @Vincent!

set_hashes calls get_hash (from prodigy.util import get_hash), which will set input_hash based on any values for keys "text", "image", "html", "input" using murmurhash.

task_hash is similar but will prefix the task with the input_hash and a new murmurhash hash for values with keys "spans", "label", "options", "arcs".

Also, it's important to be aware of a small quirk if you have any keys named in the ignore list. See this post for more details:

How does "set_hashes" work ?

Hi Jim. You have stumbled apon a keyname that's in the default ignore list and I can totally imagine the confusion. Let's go over it step-by-step. If the key is just text, nothing unexpected is happening. from prodigy import set_hashes paths = ["a", "b", "c"] [set_hashes({"text": p}) for p in paths] # [{'text': 'a', '_input_hash': -1808989213, '_task_hash': -1053809049}, # {'text': 'b', '_input_hash': 748838038, '_task_hash': -150095526}, # {'text': 'c', '_input_hash': -600324218, '_task_…

Hopefully this helps!