"Depth-first" workflows?

,
1

In our work, we often have to go through workflows where the tasks in subsequent steps depend on the results of the previous step. The different steps often require different interfaces. Think of workflows along the lines of "first preselect something, then do a deeper analysis of the selection, then do quality control on the results". For example, we might first mark some text in images (to be independent of OCR errors), then extract and correct the texts, then do NER on those texts.

Prodigy basically assumes "breadth-first" traversal of workflow tasks. That is fine for many cases. However, whenever tasks are complex, it is much more efficient to avoid context switching and keep working on the same item, i.e. do a "depth-first" traversal of the "workflow graph".

Firstly, are "depth-first" scenarios planned in Prodigy Scale or Prodigy Teams? They would be an amazing feature for our work.

Secondly, do you have any advice on how to create a system to allow to work through tasks in "depth-first" fashion? We think it should be possible to do that with switching between tabs and refreshing pages. get_examples is called whenever there are no more tasks left in the current batch, correct? Then, if we work with a batch size of 1 and wire in some additional behavior upon events, we should be able to build something like this -- or are we missing something?

2

Hi! I definitely see the point and there are use cases where this type of workflow makes sense and you should be able to implement something like it in Prodigy.

First, you can overwrite any config setting (including view_id used to render the content) via the "config" provided by an individual task. So in theory, you can set up a stream with different examples using different interfaces, different content, labels, and so on.

Yes, whenever the queue is running low in the background, the app will make a request to the server to ask for a new batch of questions. One thing to be aware of is that a certain number of examples will be kept on the client to allow the annotator to hit undo (without having to reconcile the conflicting answers on the server). You can set "instant_submit": true in your config to skip that part and send an answer back to the server immediately.

If you're generating the next example on the fly based on the previous answer, you do have to keep the stream busy so you're not running out of examples if the server is taking a bit longer to produce the next question. Similarly, the experience might be a lot less smooth because the annotator might have to wait between questions. One way to mitigate this would be to "go deeper" in small batches – for example, ask 3 questions at the top level first, and then generate the lower-level questions for this batch in the background, and so on. You might have to experiment a bit to find the right batch size but I think you can end up with a workflow that's both snappy and also gives you the very tight feedback loop with the server.

3

Hi Ines! Thanks a lot for your reply :slight_smile:

First, you can overwrite any config setting (including view_id used to render the content) via the "config" provided by an individual task. So in theory, you can set up a stream with different examples using different interfaces, different content, labels, and so on.

That's really cool! How does that interact with custom interfaces? Are we limited to one custom interface or can we create several? I don't immediately see how one would specify several custom interfaces.

If you're generating the next example on the fly based on the previous answer, you do have to keep the stream busy so you're not running out of examples if the server is taking a bit longer to produce the next question.

What happens if the stream is "lagging behind", for lack of a better term? Is there a timeout? If so, how long is it? Or is this purely about using the annotator's time efficiently?

Similarly, the experience might be a lot less smooth because the annotator might have to wait between questions.

You're very right about that! I think to mitigate that, it would make a lot of sense to start processing partial annotations as they are created (whenever that is possible). I'm a bit fuzzy on the events Prodigy uses, but we should be able to hook into them for that, correct? Is there an overview of the processes and events somewhere, or could you give us one? That would be fantastic.

4

In theory, you can have each example specify its own "html" or "html_template", or even use example-specific "blocks", each with their own content and config settings. Something like this should be a totally valid stream:

{"text": "...", "config": {"view_id": "blocks", "blocks": [...]}}
{"image": "...", "config": {"view_id": "image"}}
{"image": "...", "config": {"view_id": "image_manual", "labels": [...]}}
{"custom": "...", "config": {"view_id": "html", "html_template": "...", "javascript": "..."}}

It just makes the individual examples a bit more verbose because you'd have to specify the overrides on each example. If no config overrides are provided, Prodigy will use the defaults returned by the recipe.

If the stream is blocking, you'll see the "Loading..." message. (You've probably come across this before during regular annotation but if not, just add an infinite recursion or something to your stream generator and you'll see it.) So even if this just pops up for a second between the examples, it can potentially be quite disruptive for the annotator.

If your stream is "over" and returns an empty batch, Prodigy will show "No tasks available". So you typically want to avoid that and make sure that your stream doesn't run out while it waits for new answers to come in (e.g. by having a while loop that keeps keeps checking whether your update callback received answers/hashes you previously didn't know about).

Ah, if that's possible for what you're doing, that's definitely a clever idea! You can find the custom events fired by the app here (a bit further down – I should add a link anchor to that section :sweat_smile:): Custom Interfaces · Prodigy · An annotation tool for AI, Machine Learning & NLP prodigyupdate is fired whenever the current example is updated and prodigyanswer is called when an answer is submitted. In both cases, event.detail gives you access to the exmple content and ID.

Then you just need a way to let the running recipe know about those updates. I haven't tried this yet but you should actually be able to just reuse the /give_answers endpoint for that, which invokes the update callback of your recipe. For example, something like this with an added "partial": true that lets you distinguish partial answers from final answers. Examples will always include the _task_hash, which lets you identify them.

Untested, but something like this should work (hope I got the headers right but I think so!):

document.addEventListener('prodigyupdate', event => {
    const { task } = event.detail
    const body = JSON.stringify({ ...task, partial: true, answer: 'accept' })
    const headers = {Accept: 'application/json', 'Content-Type': 'application/json', }
    fetch('/give_answers', { method: 'POST', body, headers, credentials: 'same-origin' })
        .then(json => console.log('Sent partial answers', json))
})
1 Like
5

Thanks a lot for all that info, I'll play around with that :smiley:

I'm still a tad confused about custom recipes. How do the definitions in tasks and in the return object of a custom recipe interact? Does what a custom recipe returns override what is found in tasks, is it the other way around, or yet different?