AFK, and multi-curator "total_example_target" questions

, , ,
1

Hi community, to whom ever read this I hope everything is fine in their lives, nevertheless, I was wondering if it would be possible to be able to implement an AFK add-on to a custom recipe. I would first need to find how to add a counter, or possibly request the timestamp from the API? But it would be sick if there has been no updates of annotations for any of the users for more than 5 minutes for example, save and quit + save their time without including those 5 AFK minutes. My question would be the type of approach, do I have to add the counter to the custom recipe? Or separately add the counter to the status for each user.

Another question I had; is the "total_example_target" configuration number targeted for all users or for each?

Thanks :slight_smile:

2

That's an interesting idea! Off the top of my head, one way you could implement this is by adding a custom endpoint that the app can post to, and use custom JavaScript to add an event listener to listen to prodigyupdate. This is called whenever the user makes an update in the app, e.g. to add an entity, select an option etc. So on each update, you could send a request to your endpoint with the timestamp and the session ID if needed (from the URL query parameters). If your endpoint hasn't received a request for a given session ID in, say, 5 minutes, your endpoint will know it's inactive.

However, there's not currently a way to save and quit the session remotely – that's something the annotator will have to do. If the server is still running and you want to keep it running, there's also no direct equivalent of "quitting" a session because sessions can connect and re-connect at any time without having to be terminated. The only thing you can do is stop the server.

That's the total number of annotations you want to collect to constitute a progress of 100%.

1 Like
3

Oh wow thank you for the fast response. Yeah that's one of the approaches I was thinking of, once the endpoint knows it inactive, afk time recorded would be removed from the users performance stats. When adding a custom Javascript function like this to textcat.manual function, how does it work, do I add it to the controller / return of the function?

I guess I can just provide a message for when they have reached their number of annotations using Javascript as well. Thanks Ines

4

You could either add it to the "config" returned by the recipe, or to your prodigy.json, e.g.:

{
    "javascript": "console.log('hello world!')"
}

Since the functionality isn't necessarily recipe-specific, I think adding it to your prodigy.json could be a fine solution.

1 Like
5

All right perfect, thank you very much!

6

Hi @ines, hope I everything is doing well. I was wondering if I can set an UI alert when the number of annotations of each given dataset_id reaches a set amount. I have been looking at the custom interfaces, and I'm a bit confused on how I can apply a Custom JavaScript to my custom recipe.

Here is my custom recipe. I will have this also linked to the AFK feature to save time stamp once this alert also pops up.

def condition(stats):
    for user in stats:
        if user[1] == n_examples:
            # custom javascript alert : "You have reached set number of annotations, please exit"


def add_label_options_to_stream(stream, labels,annotations):
    options = [{"id": label, "text": label} for label in labels]
    for task in stream:
        task["options"] = options
        yield task

def add_labels_to_stream(stream, labels):
    for task in stream:
        task["label"] = label[0]
        yield task

@prodigy.recipe(
    "title_classification",
    dataset=("The dataset to use", "positional", None, str),
    source=("The source data as a JSONL file", "positional", None, str),
    label=("One or more comma-separated labels", "option", "l", split_string),
    n_examples = ("Number of examples to randomly review, -1 for all", "option", "n", int),
    exclusive=("Treat classes as mutually exclusive", "flag", "E", bool),
    exclude=("Names of datasets to exclude", "option", "e", split_string),
)

def title_classification(
    dataset: str,
    source: str,
    label: Optional[List[str]] = None,
    n_examples : int,
    exclusive: bool = False,
    exclude: Optional[List[str]] = None,
):  

    db = connect()

    stats = []
    
    # I will have to add a while loop to update stats for condition function

    for dataset_id in db.sessions:
        annotations = db.get_dataset(dataset_id)
        stats.append([str(dataset_id), len(annotations)])

    stream = JSONL(source)
    print(stream)

    #Add labels to each task in stream
    has_options = len(label) > 1
    if has_options:
        stream = add_label_options_to_stream(stream, label)
    else:
        stream = add_labels_to_stream(stream, label)```
7

Ah would a viable solution is to create a controller and input a config javascript alert, if the condition function is met?

8

Answered your question on the other thread: