Session-specific progress

,
1

I’m hoping to understand a bit more about customizing the progress reporting within the web app.

In my scenario, I’m using custom task routing to ensure that examples are assigned to multiple annotators from a specific sub-pool of annotators (based on example metadata). This means that each annotator (i.e., session) can be assigned wildly different number of annotations (due to the imbalance of the input data). So, I was hoping I could customize the progress reporting so that it would better reflect each annotator’s actual progress.

It looks like the controller’s get_progress function reports the specific progress for a given session_id, but it doesn’t look like the customizable progresscallback has access to this session_id, so I’m at a loss on how session-specific progress could be reported. Is there something I’m missing about how progress reporting within the web app works?

2

I believe I may have figured it out by looking at the source code. It looks like the signature for the progress function has been updated so that it takes four inputs rather than two with the second input being the session in question.

3

Hi @laurejt,

You're totally right! I was about to say that the progress callback actually supports an extended signature that gives you access to the current session. This was added as part of an internal refactor to make progress tracking work correctly in multi-session setups, but we hadn't documented it yet — the original 2-argument signature still works and covers the common case. Since you're using custom task routing, you're exactly the kind of user who needs it.

You have already seen it, but for completeness in case other users come across this post: if your progress function accepts four arguments instead of two, Prodigy will pass the current Session object and the list of answers that were just received:

  def progress(ctrl, session, answers, update_return_value):
      target = get_target_for_session(session.id)
      return session.total_annotated / target

The session object gives:

  • session.id — the current annotator's session ID
  • session.total_annotated — total annotations for this session
  • session.session_annotated — annotations in the current server run

Prodigy auto-detects which signature you're using — no extra configuration needed. Under the hood, the old 2-argument signature is wrapped to match the new one (see _translate_old_progress_protocol in the source), so the session is always there; the old signature just doesn't expose it.

We've now added this to the docs as well — see the extended progress signature.