The task returns to the beginning with a custom recipe (it now happens after v1.11.0).

, ,
1

Hello!
I'm using a custom recipe that combines "view_id": "classification" with "view_id": "diff". In addition, I'm overwriting prodigy.json to combine view_id: "html".
It worked fine until now.
However, when I save the annotations in the middle and restart the session on the command line, I now see the first task in the source data. Until now, it was displayed from the continuation of the annotated task, but the behavior has changed.
I think it happened after updating to v1.11.0.
Below is my custom recipe and prodigy.json. Am I overlooking something?
My custom recipe is simple based on this documentation.

#My custom recipe
import prodigy
from prodigy.components.loaders import JSONL
from prodigy.util import split_string
from typing import List, Optional

@prodigy.recipe(
    "anno_diff",
    dataset=("The dataset to use", "positional", None, str),
    source=("The source data as a JSONL file", "positional", None, str),
    label=("One or more comma-separated labels", "option", "l", split_string)
    )
def grammar_error(
    dataset,
    source,
    label: Optional[List[str]]
):
    stream = JSONL(source)
    blocks = [{"view_id": "classification", "text":None},{"view_id": "diff", "text":None}] #Overwritten with prodigy.json
    stream = add_options(stream,label)
    return {
        "stream": stream,
        "dataset": dataset,
        "view_id": "blocks",
        "config": {
            "blocks": blocks
        }
    }

def add_options(stream, label):
    for task in stream:
        task["label"] = label
        yield task

prodigy.json
{
    "port": 8011,
    "global_css": ".prodigy-content{text-align: left}",
    "blocks": [
        {
            "view_id": "classification",
            "text": ""
        },
        {
            "view_id": "diff",
            "text": ""
        },
        {
            "view_id": "html",
            "html_template": "<small>[original]<br>{{original}}<br>[revised]<br>{{revised}}</small>"
        },
        {
            "view_id": "text_input",
            "field_id": "input_anottator_name",
            "field_rows": 1,
            "field_placeholder":"input comment..."
        }
    ]
}

I used prodigy stats and prodigy db-out. I confirmed that the annotation was recorded halfway. However, the annotator is surprised on the screen because the task returns to the beginning.

2

Hi! Are you able to share one example from your input JSONL data? Also, just to confirm, nothing in your data or its format changed between now and when you previously ran it, right?

3

Thank you for your reply. The input data and the result of db-out are shown below. (Individual information is processed)
There is one change from the previous one. I added a "meta" key to display meta information. However, I think this issue happened after the v1.11.0 update and before I added the "meta" key.

#input data
{
	"original": "This is sample text 1.",
	"revised": "This is sample text 2.",
	"domain": "something",
	"label": "some label",
	"id": 329,
	"text": "This is sample text 1.\n\nThis is sample text 2.",
	"added": "This is sample text 1.",
	"removed": "This is sample text 2.",
	"meta": {
		"domain": "something",
		"id": 329
	}
}

#db-out data
{
	"original": "This is sample text 1.",
	"revised": "This is sample text 2.",
	"domain": "something",
	"label": [
		"some label"
	],
	"id": 329,
	"text": "This is sample text 1.\n\nThis is sample text 2.",
	"added": "This is sample text 1.",
	"removed": "This is sample text 2.",
	"meta": {
		"domain": "something",
		"id": 329
	},
	"_input_hash": 217917166,
	"_task_hash": -622372976,
	"_session_id": "test_cleaning_name-name",
	"_view_id": "blocks",
	"answer": "accept",
	"_timestamp": 1629883528,
	"_annotator_id": "test_cleaning_name-name"
}
4

Does adding "exclude_by": "input" to the "config" returned by your recipe help? I still don't really know why the task hashes would be different, and adding the "meta" shouldn't have an impact here. But if you're excluding by input hash, the hashes should match and Prodigy should skip the already annotated examples.

5

I modified prodigy.json as below and restarted the session. However, no change was seen. (Am I following your advice?)

{
    "port": 8011,
    "global_css": ".prodigy-content{text-align: left}",
    "exclude_by": "input",
    "blocks": [
        {
            "view_id": "classification",
            "text": ""
        },
        {
            "view_id": "diff",
            "text": ""
        },
        {
            "view_id": "html",
            "html_template": "<small>[original]<br>{{original}}<br>[revised]<br>{{revised}}</small>"
        },
        {
            "view_id": "text_input",
            "field_id": "input_annotator_name",
            "field_rows": 1,
            "field_placeholder":"input comment..."
        }
    ]
}
6

Yes, although I'd suggest adding it to the "config" returned by the recipe in your Python file. The prodigy.json overrides the default settings defined in the recipe, so you can easily end up with potentially confusing results here. I'd also suggest moving the "blocks" out ouf your prodigy.json and into the recipe – otherwise, all recipes will receive this setting, which is usually not what you want (the blocks to use are typically pretty specific to the recipe).

7

Thanks for your advice.
I wrote and customized various unique settings in prodigy.json, but I realized it wasn't good. I made prodigy.json simple and modified and My custom recipe.py has been modified as follows.
(I added "exclude_by": "input" to the config in My custom recipe.py.)

#prodigy.json
{
    "global_css": ".prodigy-content{text-align: left}"
}

#My custom recipe

import prodigy
from prodigy.components.loaders import JSONL
from prodigy.util import split_string
from typing import List, Optional

@prodigy.recipe(
    "anno_diff",
    dataset=("The dataset to use", "positional", None, str),
    source=("The source data as a JSONL file", "positional", None, str),
    label=("One or more comma-separated labels", "option", "l", split_string)
    )
def grammar_error(
    dataset,
    source,
    label: Optional[List[str]]
):
    stream = JSONL(source)
    blocks = [
        {
            "view_id": "classification",
            "text": None
        },
        {
            "view_id": "diff",
            "text": None
        },
        {
            "view_id": "html",
            "html_template": "<small>[original]<br>{{original}}<br>[revised]<br>{{revised}}</small>"
        },
        {
            "view_id": "text_input",
            "field_id": "input_anottator_name",
            "field_rows": 1,
            "field_placeholder":"input comment..."
        }
    ]
    stream = add_options(stream,label)
    return {
        "stream": stream,
        "dataset": dataset,
        "view_id": "blocks",
        "config": {
            "exclude_by": "input",
            "blocks": blocks
        }
    }

def add_options(stream, label):
    for task in stream:
        task["label"] = label
        yield task

However, it does not solve the original problem. Where is the problem?

8

After updating to the latest version v1.11.3 and rebooting, the issue was resolved. According to changelog, this related issue seemed to be fixed. Thank you for the update!