# Is A/B evaluation workflow gone?

**URL:** <https://support.prodi.gy/t/is-a-b-evaluation-workflow-gone/1078>\
**Category:** Uncategorized\
**Tags:** docs, usage\
**Created:** [December 17, 2018, 12:37pm UTC](https://support.prodi.gy/t/is-a-b-evaluation-workflow-gone/1078 "2018-12-17T12:37:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![henrye](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/henrye/32/569_2.png) [@henrye](https://support.prodi.gy/u/henrye)\
**Post date:** [December 17, 2018, 12:37pm UTC](https://support.prodi.gy/t/is-a-b-evaluation-workflow-gone/1078/1 "2018-12-17T12:37:22Z")

</div>

The link to the A/B evaluation workflow [https://prodi.gy/docs/workflow-evaluation](https://prodi.gy/docs/workflow-evaluation) now appears to redirect to [https://prodi.gy/features/ab-evaluation](https://prodi.gy/features/ab-evaluation) . Is this feature still supported?

I’m hoping to use the A/B feature for human evaluation. That is, to do pairwise evaluation of model outputs and rank models using the trueskill algorithm. This would presumably work well in conjunction with the active learning / sorting feature.

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [December 17, 2018, 1:30pm UTC](https://support.prodi.gy/t/is-a-b-evaluation-workflow-gone/1078/2 "2018-12-17T13:30:02Z")

</div>

Yes, absolutely – in fact, we think of A/B evaluation as a super important and often very underrated feature. (Since the original workflow page was actually more of a feature description anways, we ended up merging it with the feature page.)

Your idea sounds cool, so if you end up implementing this, definitely let us know how you go!

Prodigy ships with built-in workflows like [`ner.eval-ab`](https://prodi.gy/docs/recipes#ner-eval-ab), but you can also put together your own recipes using the `compare` interface ([see here](https://prodi.gy/demo?view_id=compare) for an example). It takes data in the following format:

```json
{
    "id": 1,
    "input": { "text": "NLP" },
    "accept": { "text": "Natural Language Processing" },
    "reject": { "text": "Neuro-Linguistic Programming" },
    "mapping": { "A": "accept", "B": "reject" }
}

```

`"input"` is the original input (optional) and `"accept"` / `"reject"` are the outputs displayed in green and red, respectively. Instead of `"text"`, they could also specify `"html"` or an `"image"`. The `"mapping"` lets you resolve the two randomised outputs back to the A and B models.

In your [custom recipe](https://prodi.gy/docs/workflow-custom-recipes) function, you can create this data however you like and return it as the `"stream"` setting. In order to use Prodigy’s built in sorter components like `prefer_uncertain`, your ranking function needs to yield out `(score, example)` tuples. You can then wrap it in the sorter function:

```python
from prodigy.components.sorters import prefer_uncertain

stream = get_your_ranked_stream()
stream = prefer_uncertain(stream)

```

You can also find more info and API docs in your `PRODIGY_README.html`, available for download with Prodigy.

---

<div class="post-metadata">

**Author:** ![henrye](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/henrye/32/569_2.png) [@henrye](https://support.prodi.gy/u/henrye)\
**Post date:** [December 17, 2018, 2:09pm UTC](https://support.prodi.gy/t/is-a-b-evaluation-workflow-gone/1078/3 "2018-12-17T14:09:04Z")

</div>

Perfect! Thanks for the response. Will let you know how it gets on. Thanks as well for the academic licence!