filter_inputs still causes duplicated image

[ines (Ines Montani)]

@zhiyan114 Thanks for checking, it's definitely good to know that it's actually duplicate input hashes and not different hashes generated for the same image etc.

How quickly are you annotating? I wonder if you might be hitting the race condition described here:

duplicate images when annotating

Thanks for clarifying about the jsonl file, and sharing your DB timings, it was helpful! I was able to reproduce seeing duplicates by holding down the shortcut key to accept images as quickly as possible until all 100 were done. When I did that, prodigy would show me 101 or 102 images, but only store the expected 100 in the database. I think the trouble is that when you answer questions at the same time as the app asks for more, it can miss reporting one or two items that you've …