Okay, first update on my experiment: the rough idea works 
There are still some open questions around how the underlying data should be structured and saved but it's looking promising. What happens when you click a bounding box can be customised in code via the click_bounding_box event hook, so it can be applied to all kinds of use cases.