Split dataset for multiple annotators

[livyreal]

Hi! I have 3k items to classify (using textcat.manual) to be annotated by 5 annotators. Ideally, each item would be annotated by two different annotators. Is there any way to automatically split the data in this scenario?
Thanks!

[ines (Ines Montani)]

Hi! There's no built-in mechanism for this exact scenario at the moment, but if you know who should annotate what in advance, you can write a custom loader that sends out different examples, depending on who is annotating. I shared an example and some ideas here:

Give a specific order of presentation depending on the annotator

Hi and thanks, that's nice to hear The easiest way to implement something like this would be to start a separate instance for each annotator and use a custom stream for each of them, with a different order of examples (randomly, or specific based on the annotator). Since the only difference between the instances is the stream, you might not even need a custom recipe and can just use a custom loader. See here for an example: https://prodi.gy/docs/api-loaders#loaders-custom If you want …