Ah, the custom JS/CSS/HTML features were specifically designed to allow custom UI components like buttons, checkboxes etc. If you want a button that does something to the current data, you can just add a <button> and call into window.prodigy.update to update the current annotation task. There are a few threads on the forum on implementing custom dropdowns, checkboxes etc. (like this one).
One thing that doesn't really make sense in the context of Prodigy is to have the annotator choose or filter the data in the UI. If your goal is to train a model, you typically want control over what is annotated and what will be used as training data – that can't really be decoupled from the development process, and you can't really expect the annotator to make a meaningful decision for you.
The active learning capabilities exclusively happen on the server, though, and don't really have anything to do with the UI components. The idea is to use the model in the loop to decide what to present for annotation – and for that, you ideally want a focused annotation session that collects enough annotations to allow the model to be updated in the loop and make meaningful suggestions. That wouldn't really work if you only collect very sporadic feedback.
Also, it's probably quite rare that the output the model you're training produces is exactly what you'll be showcasing to the business users – often, you'll do something with the predictions, and what the end user sees is the result of that. So the feedback you may be collecting in your application may not be feedback on the direct model output.
So if you want to implement a tighter feedback loop and allow your users to make corrections to the model's predictions, one solution would be to set up a dedicated annotation session for it and stream in selected examples that need attention from domain experts etc. Those could be predictions with very uncertain scores (e.g. close to 0.5 where the answer could go either way), or predictions that caused mistakes somewhere in your downstream application. This also gives you a more measurable evaluation, lets you calculate agreement (do end users even agree on the problems?) and actually create data you can update the model with to improve it.