As the dataset grow it’s hard to maintain consistency across the entire set.
Especially when the first annotations was months ago, and now I have more in-depth knowledge on the data.
Can active learning help with this? For example by comparing to a threshold, pick up annotations that "surprise" the model the most, and ask the annotator "does this annotation still looks OK to you?"
If this could be done it could help cleanup inconsistent and wrong annotations. I'm currently doing that manually.