Difference number examples dataset and batch-train

[ines (Ines Montani)]

Hi! I wrote about this in some more detail here:

@ines

There are several steps here that change the total number of individual records:

1. Merging entity spans of annotations on the same data. So if you’ve accepted/rejected several entities on the same text, those will be combined into one example.
2. Splitting sentences. If you don’t set --unsegmented , long texts will be split into sentences.
3. Filtering out ignored examples with "answer": "ignore" .
4. Filtering out duplicates or otherwise invalid annotations.

I'm not sure what's in your data and how many unique examples you have – you could check that looking at how many unique input hashes there are:

from prodigy.components.db import connect
db = connect()
input_hashes = db.get_input_hashes(["energy_patterns"])
print(len(set(input_hashes)))

You could also set PRODIGY_LOGGING=basic to see if anything else is being skipped.