I ran the ner.eval-ab recipe and wanted to explore the rejected examples from my model B (I assume this is the model you pass second).
I'm confused by different values in the column accept (which should give me the model I accepted) and mapping.B which I thought should be accept if I accepted the model B and reject if I rejected it. But I have lots of examples when I accepted the model B but mapping.B for these examples is reject (screenshot below).
I couldn't find a description of the columns anywhere so not sure how to interpret these examples.
Hi! Sorry if this is confusing – I think what happened here is that we changed the interface to use a multiple choice UI instead of the less intuitive "compare" interface that expected you to hit accept for the green option and reject for the red option. But we didn't change the underlying JSON format to prevent backwards incompatibility.
The mapping is supposed to indicate which model (A and B) ended up being option 1 vs. option 2, which is random by default. Instead of accept and reject, think of it as 0 and 1. If mapping.A is 1, options[1] (the second multiple choice) option is model A (the first model). In the example you posted, the data indicates that you always selected the output of the second model.
I'll put this on my list of enhancements btw, this is definitely something I want to fix and simplify for he next bigger release where we can introduce some breaking changes!
