How to produce a full dataset without conflicts and duplicates.

I am reviewing one dataset and its only save the reviews notes. Is there any way to combine two datasets without duplicate text? :upside_down_face:

Hi Alvaro,

could you elaborate on which commands you used and what you expected? Do you have a reproducible example that we can run on our own machine? Are you referring to the review recipe for NER?

sorry for the delay , i dont know why but its seems that I was not notified. I explain better.

He have a team and we are annote together, some times prodigy gives us the same text to two of us. Then for that cases we use review recipt , but when you are reviews the dataset that prodigy creates only save the revied notes, can we save all notes together in the new dataset ?

Thanks for the response, Álvaro. :slight_smile:

some times prodigy gives us the same text to two of us

This can happen, but it also depends on how you've set Prodigy up. Are you running Prodigy with feed_overlap set to true? This section of the docs shares more details on what to expect:

reviews the dataset that prodigy creates only save the revied notes, can we save all notes together in the new dataset

It might help to have a tangible example to help me understand what you mean. Could you share a screenshot of what you're trying to do and also explain what you expect to happen?