Yeah, that makes total sense and is pretty common – however, I'm not sue I'd call it "split annotation from modelling", because that's not something we'd recommend doing. The annotation scheme is crucial for the modelling decisions, so that should be tightly coupled with the development process. Who does the annotation in the end is of course a different question.
I'm not sure where validate_answer could fit in here. But even if you use a stream generator that checks for the most recent answer, and "instant_submit": True to make sure an answer is submitted immediately when the user makes a decision, it's difficult to prevent a race condition or have the user wait until the next question is computed on the server.
If you want full flexibility, I think a better approach would be to just implement your own form using an HTML template with checkboxes and custom JavaScript that updates the task object (and possible shows/hides further options). I'm going through this in more detail in my comment here: