I'm trying to make sense of the documentation on how to have multiple annotators work on a single (fairly large dataset). It's not clear to me, for instance, what happened to 'Prodigy teams'. Could you please assist? And is there a way to do things like measuring inter-coder reliability or automated task assignment? Thanks much
The upcoming Prodigy Teams product is still under active development – if you've filled in the form, we'll let you know once more beta spots are available.
You can definitely use Prodigy with multiple named annotators – see here for details: https://prodi.gy/docs/api-web-app#multi-user-sessions The session ID will be saved with the annotations, as well as unique hashes that let you indentify annotations on the same input. You can use this to calculate agreement between sessions (how you define agreement can obviously be use-case specific so it depends on the type of annotations you collect).
1 Like
I have not been able to access the form for quite some time now... But thanks for the link
I guess it did work again afterwards. I've signed up - hoping to be able to try it out soon with our team