Which number of training labels should I trust

[nvasil (Nikolaos Vasiloglou)]

I see this pattern

• [ner] Training: 1738 | Evaluation: 326 (20% split)
  Training: 352 | Evaluation: 88

Why is this discrepancy? Which one should I trust?
When I do prodigy tats the dataset has 1738 annotations and they are all accepted

[ryanwesslen]

hi @nvasil!

Have you seen this related post?

Difference number examples dataset and batch-train

Hi! I wrote about this in some more detail here: I'm not sure what's in your data and how many unique examples you have – you could check that looking at how many unique input hashes there are: from prodigy.components.db import connect db = connect() input_hashes = db.get_input_hashes(["energy_patterns"]) print(len(set(input_hashes))) You could also set PRODIGY_LOGGING=basic to see if anything else is being skipped.

I suspect you either have duplicates or you have merged entity spans of annotations on the same data. In the second case, if you’ve accepted/rejected several entities on the same text, those will be combined into one example.

Be sure to use logging PRODIGY_LOGGING=basic that should show the dedup step explicitly.

@nvasil

Which one should I trust?

The final one is what you should go with (if you're comfortable with how Prodigy's is defaulting its behavior by deduping/merging entities, etc.).

Let me know if this helps!