If we want to have a separate data set for a separate annotation, do we need to spin up a new prodigy container, or is there a way to organize different annotation tasks within one instance?
Welcome to the forum @rbannon 
It would be best to have a separate Prodigy server per annotation task, yes.
You should be able to define the view id (UI) on the task level (in your custom recipe you would have to add the right view_id to the task dictionary) and also interact with the database programmatically to make sure the tasks get saved in the right datasets but it's much cleaner and less error prone and, importantly, much more efficient (thinking here of the Database transactions) to have a dedicated Prodigy instance per annotation task.