Yes, I think what's going on here is that you're adding your new, reviewed annotations to the same dataset. So your dataset now contains the old annotations, as well as the new ones. By default, Prodigy is designed to always keep a record of each individual annotation decision so you can always reproduce it – that’s also why it doesn’t just silently overwrite existing records in your dataset.
You might find this thread interesting, which discusses a very similar workflow. I've also posted a more advanced recipe to automate the reviewing process and take random samples.