--label-field and --choice-field error with v.1.14.0

I sent along two custom recipes in case they are helpful. Thanks!

1 Like

Hey Cheyanne, we just did a release of v1.14.1 and v1.14.2. That last release will get announced next week but it should be available for download right now and it should also fix the issue that you have.

If it doesn't, do let me know!

1 Like

I installed v.1.14.4 and I am still getting the error above, along with this:

AttributeError: 'str' object has no attribute 'pipe_labels'

I had another look at your code and spotted the issue related to the AttributeError. You are re-using the spans_manual recipe inside of Prodigy. That's a good practice (great even!) but doing so will prevent the CLI helpers from kicking in. In particular on line 46 if your custom recipe file:

result = prodigy_spans_manual(dataset, spacy_model, source, label=label)

That spacy_model really needs to be a spaCy model, and you're currently passing a string. This does work on my end:

nlp = spacy.load(spacy_model)
result = prodigy_spans_manual(dataset, nlp, source, label=label)

Could you share a traceback of the aforementioned issue? If that persists I'd love to have another look.

1 Like

That worked! I'm going to go through all my custom recipes and will edit/test.

(prodigy) cheyannebaird@Cheyannes-MBP:~/posh/annotation-service$ PRODIGY_ALLOWED_SESSIONS=cheyanne prodigy stt-spans stt_spans_install_test blank:en /Users/cheyannebaird/posh/stt_error_validation/final_annotated_output/final/joel_updated_output/diff_with_raw/stt_error_validation_joel.jsonl -F /Users/cheyannebaird/posh/annotation-service/src/annotator/recipes/stt_spans.py --label STT_ERROR
Using 1 label(s): STT_ERROR
⚠ Config setting 'exclude_by' defined in recipe is overwritten by a
different value set in the global or local prodigy.json. This may lead to
unexpected results and potentially changes to the core behavior of the recipe.
If that's surprising, you should probably remove the setting 'exclude_by' from
your prodigy.json.

✨  Starting the web server at http://localhost:8080 ...
Open the app in your browser and start annotating!
1 Like

Ok, so I made updates to 3 custom recipes that referenced spacy_model -- I updated the code, and I am no longer getting errors on any of these recipes.

The other recipes I have do not reference spacy_model, and I'm still getting this error:

the following arguments are required: --label-field, --choice-field

Is there something else I need to add to handle these label and choice field errors? I can send a few examples if that is helpful. These args were never required in previous versions.

Just to double check, what version are you running with this error? It's not 1.14.0?

I'm running v1.14.4

I will test out v1.14.5 since it was just released, but blocked on updating for now.

That's interesting ... I'm not seeing that error while I am on 1.14.4.

Since this is going to be about recipe specific code I'll respond in the email.

Offering a bit more info about what is working, and what is not working with v1.14.4. I was able to get various span recipes (3 out of 17, meaning I am not able to run 14 of my custom recipes with v1.14.4) working with the fix Vincent mentioned above. I have not tried v1.14.5 yet.

Here's an example of a custom recipe that I am able to run with the latest update:

stt_spans.py -- in this recipe, I edited this code, and it now runs:

CLI:

PRODIGY_ALLOWED_SESSIONS=cheyanne prodigy stt-spans stt_spans_install_test blank:en /Users/cheyannebaird/posh/stt_error_validation/final_annotated_output/final/joel_updated_output/diff_with_raw/stt_error_validation_joel.jsonl -F /Users/cheyannebaird/posh/annotation-service/src/annotator/recipes/stt_spans.py --label STT_ERROR

Recipe:

Edits made:

nlp = spacy.load(spacy_model)
result = prodigy_spans_manual(dataset, nlp, source, label=label)
import json
import prodigy
from typing import Dict, Generator, Iterable, List, Optional, Union

import spacy
from prodigy.components.loaders import get_stream
from prodigy.components.loaders import JSONL
from prodigy.components.preprocess import add_tokens
from prodigy.core import recipe
from prodigy.recipes.spans import manual as prodigy_spans_manual
from prodigy.types import RecipeSettingsType
from prodigy.util import get_labels

@prodigy.recipe(
    "stt-spans",
    dataset=("dataset to save annotations to", "positional", None, str),
    spacy_model=(
        "spaCy pipeline for tokenization (e.g. blank:en)", "positional", None,
        str
    ),
    source=("file path with data to annotate", "positional", None, str),
    loader=(
        "loader (guessed from file extension if not set)", "option", "lo",
        str
    ),
    label=("comma-separated label(s) to annotate", "option", "l", get_labels)
)

def manual(
    dataset: str,
    spacy_model: str,
    source: Union[str, Iterable[dict]],
    loader: Optional[str] = None,
    label: Optional[List[str]] = None
) -> RecipeSettingsType:
    """
    Overrides the built-in Prodigy recipe to be able to present the data to the
    annotators with our custom view_id and stream configuration.
    """
    nlp = spacy.load(spacy_model)
    stream = add_tokens(
        nlp,
        get_stream(
            source, loader=loader, rehash=True, dedup=True, input_key="text"
        ),
        use_chars=False
    )

    nlp = spacy.load(spacy_model)
    result = prodigy_spans_manual(dataset, nlp, source, label=label)
    #result = prodigy_spans_manual(dataset, spacy_model, source, label=label)
    result.update(
        stream=list(stream),
        view_id="blocks",
        config={
            "lang": nlp.lang,
            "blocks": [
                {"view_id": "spans_manual"},
                {"view_id": "text_input", "field_label": "1st STT Error", "field_id": "1st_stt_error"},
                {"view_id": "text_input", "field_label": "2nd STT Error", "field_id": "2nd_stt_error"},
                {"view_id": "text_input", "field_label": "3rd STT Error", "field_id": "3rd_stt_error"},
                {"view_id": "html", "html_template": "<div style=\"padding: 0 10px; border: 1px solid #ddd; border-radius: 4px; text-align: left;\">" +
                    "<label style=\"font-size: 12px; font-weight: bold; opacity: 0.75; margin-bottom: 10px;\">deployment</label>" +
                    "<div style=\"max-height: 300px; overflow-y: scroll; margin-bottom: 10px; margin: 0; padding: 0;\">{{deployment}}</div>" +
                "</div>"
                },
                {"view_id": "html", "html_template": "<div style=\"padding: 0 10px; border: 1px solid #ddd; border-radius: 4px; text-align: left;\">" +
                    "<a href=\"https://app.poshdevelopment.com/chatlogs?chatId={{chat_id}}\" target=\"_blank\" style=\"margin-bottom: 10px;\">View Full Chat</a>" +
                "</div>"
                },
                {"view_id": "text_input", "field_label": "notes"}
            ],
            "labels": label,
            "exclude_by": "input",
            "ner_manual_highlight_chars": False,
            "auto_count_stream": True
        }
    )

    return result

This is a custom recipe that I am not able to get working:
eup_corpus_validation_with_options.py:

CLI:

PRODIGY_ALLOWED_SESSIONS=cheyanne prodigy eup-corpus-validation-with-options eup_val_options_install_test /Users/cheyannebaird/posh/eup_corpus/eup_task_april2023/final_final/eup_low_scoring_data_v1.jsonl -F /Users/cheyannebaird/posh/annotation-service/src/annotator/recipes/eup_corpus_validation_with_options.py

Error:

the following arguments are required: --label-field, --choice-field

Recipe:

import json
from typing import Dict, Generator

import prodigy
from prodigy.components.loaders import JSONL

from annotator.recipes.utils import DIFFERENT_EUP

UNLIMITED_ROWS = [
    {"view_id": "html", "html_template":
        "<div style=\"padding: 0 10px; border: 1px solid #ddd; border-radius: 4px; text-align: center;\">" +
        "<div style=\"max-height: 300px; font-size: 30px; overflow-y: scroll; margin-bottom: 10px;\">{{message}}" +
      "</div>"
    },
    {"view_id": "choice"},
    {"view_id": "html", "html_template": "<div style=\"padding: 0 10px; border: 1px solid #ddd; border-radius: 4px; text-align: left;\">" +
        "<label style=\"font-size: 12px; font-weight: bold; opacity: 0.75; margin-bottom: 10px;\"></label>" +
        "<a href=\"https://coda.io/d/Research-Notes_dbuGQfaDy5r/EUP-Tree_suVEG#_lu8Uk\" target=\"_blank\" style=\"margin-bottom: 10px;\">EUP Tree</a>" +
      "</div>"
    },
    {
      "view_id": "html",
      "html_template": 
        "<style>" +
          ".checkbox-container {" +
            "float: left;" +
            "margin-right: 25px;" +
          "}" +
        "</style>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='ambig' name='ambig' value='Ambig' data-id='{{chat_id}}' onchange='updateAmbig()'>" +
          "<label for='ambig' style='margin-left: 5px;'>Ambig</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='contextual' name='contextual' value='Contextual' data-id='{{chat_id}}' onchange='updateContextual()'>" +
          "<label for='contextual' style='margin-left: 5px;'>Contextual</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='multi' name='multi' value='Multi-EUP' data-id='{{chat_id}}' onchange='updateMulti()'>" +
          "<label for='multi' style='margin-left: 5px;'>Multi-EUP</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='partial' name='partial' value='Partial' data-id='{{chat_id}}' onchange='updatePartial()'>" +
          "<label for='partial' style='margin-left: 5px;'>Partial</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='side_conversation' name='side_conversation' value='Side Conversation' data-id='{{chat_id}}' onchange='updateSideConversation()'>" +
          "<label for='side_conversation' style='margin-left: 5px;'>Side Conversation</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='stt_error' name='stt_error' value='STT Error' data-id='{{chat_id}}' onchange='updateSttError()'>" +
          "<label for='stt_error' style='margin-left: 5px;'>STT Error</label>" +
        "</div>"
    },
    {"view_id": "text_input", "field_label": "Proposed EUP(s)", "field_id": "proposedEUP"}, 
    {"view_id": "text_input", "field_label": "notes"}
]

def add_options(
        stream, label_field="eup", choices=DIFFERENT_EUP
) -> Generator[Dict, None, None]:
    """
    Convert each line in the ``stream`` to a ``task`` with a text and an
    options field

    :param stream: the input stream
    :param label_field: key; defaults to "label"
    :param choices: the different choices
    :yield: a task Dict with text and options
    """

    for line in stream:
        if label_field in line:
            options = json.loads(line[label_field])
        else:
            options = []
        for word in (options + choices):
            if word not in options:
                options.append(word)
        task = {
            "eup": line["eup"],
            "message": line["message"],
            "ambig": False,
            "contextual": False,
            "multi": False,
            "partial": False,
            "side_conversation": False,
            "stt_error": False,
            "options": [
                {"id": o, "deployment": o, "prompt": o,
                "text": o} for o in options
            ]
        }
        yield task

@prodigy.recipe(
    "eup-corpus-validation-with-options",
    dataset=("The dataset to save to", "positional", None, str),
    file_path=("Path to texts", "positional", None, str),
    label_field=("Label to use for the accept task", "option", "f", str),
    choice_field=("Choices to add to the input", "option", "c", str)
)
def custom_labels(dataset, file_path, label_field, choice_field):
    """
    Annotate the text with labels from the list from the ``label_field`` in
    the input file. Augmented with choices from ``choice_field``.
    """

    blocks = UNLIMITED_ROWS

    stream = JSONL(file_path)
    stream = add_options(stream)  # add options to each task

    javascript = """
    // Set ambig to false by default
    prodigy.update({ ambig: false });

    function updateAmbig() {
      prodigy.update({ ambig: document.getElementById('ambig').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset ambig to false
      prodigy.update({ ambig: false });

      document.getElementById('ambig').checked = false;
    });

    // Set contextual to false by default
    prodigy.update({ contextual: false });

    function updateContextual() {
      prodigy.update({ contextual: document.getElementById('contextual').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset contextual to false
      prodigy.update({ contextual: false });

      document.getElementById('contextual').checked = false;
    });

    // Set multi to false by default
    prodigy.update({ multi: false });

    function updateMulti() {
      prodigy.update({ multi: document.getElementById('multi').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset multi to false
      prodigy.update({ multi: false });

      document.getElementById('multi').checked = false;
    });

    // Set partial to false by default
    prodigy.update({ partial: false });

    function updatePartial() {
      prodigy.update({ partial: document.getElementById('partial').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset partial to false
      prodigy.update({ partial: false });

      document.getElementById('partial').checked = false;
    });

    // Set side_conversation to false by default
    prodigy.update({ side_conversation: false });

    function updateSideConversation() {
      prodigy.update({ side_conversation: document.getElementById('side_conversation').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset side_conversation to false
      prodigy.update({ side_conversation: false });

      document.getElementById('side_conversation').checked = false;
    });

    // Set stt_error to false by default
    prodigy.update({ stt_error: false });

    function updateSttError() {
      prodigy.update({ stt_error: document.getElementById('stt_error').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset stt_error to false
      prodigy.update({ stt_error: false });

      document.getElementById('stt_error').checked = false;
    });
    """
 
    return {
        "dataset": dataset,
        "view_id": "blocks",
        "stream": list(stream),
        "config": {
          "blocks": blocks,
          "javascript": javascript
        }
    }

With the recipe that is not working with the latest update, I did add dummy labels and choice to the call / command line, and then I was able to launch the recipe... but why is this required? This will require us to make an update to our internal annotation-service if these fields are required, so I'm trying to understand this requirement.

PRODIGY_ALLOWED_SESSIONS=cheyanne prodigy eup-corpus-validation-with-options eup_val_options_install_test /Users/cheyannebaird/posh/eup_corpus/eup_task_april2023/final_final/eup_low_scoring_data_v1.jsonl -F /Users/cheyannebaird/posh/annotation-service/src/annotator/recipes/eup_corpus_validation_with_options.py  --label foo,bar --choice-field foo,bar

I don't have access to annotator.py nor your datafiles. But I have been able to adapt the recipe slightly such that I receive the same error.

import json
from typing import Dict, Generator

import prodigy
from prodigy.components.loaders import JSONL

DIFFERENT_EUP = [{"id": "a", "text": "a"}, {"id": "b", "text": "b"}]

UNLIMITED_ROWS = [
    {"view_id": "html", "html_template":
        "<div style=\"padding: 0 10px; border: 1px solid #ddd; border-radius: 4px; text-align: center;\">" +
        "<div style=\"max-height: 300px; font-size: 30px; overflow-y: scroll; margin-bottom: 10px;\">{{message}}" +
      "</div>"
    },
    {"view_id": "choice"},
    {"view_id": "html", "html_template": "<div style=\"padding: 0 10px; border: 1px solid #ddd; border-radius: 4px; text-align: left;\">" +
        "<label style=\"font-size: 12px; font-weight: bold; opacity: 0.75; margin-bottom: 10px;\"></label>" +
        "<a href=\"https://coda.io/d/Research-Notes_dbuGQfaDy5r/EUP-Tree_suVEG#_lu8Uk\" target=\"_blank\" style=\"margin-bottom: 10px;\">EUP Tree</a>" +
      "</div>"
    },
    {
      "view_id": "html",
      "html_template": 
        "<style>" +
          ".checkbox-container {" +
            "float: left;" +
            "margin-right: 25px;" +
          "}" +
        "</style>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='ambig' name='ambig' value='Ambig' data-id='{{chat_id}}' onchange='updateAmbig()'>" +
          "<label for='ambig' style='margin-left: 5px;'>Ambig</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='contextual' name='contextual' value='Contextual' data-id='{{chat_id}}' onchange='updateContextual()'>" +
          "<label for='contextual' style='margin-left: 5px;'>Contextual</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='multi' name='multi' value='Multi-EUP' data-id='{{chat_id}}' onchange='updateMulti()'>" +
          "<label for='multi' style='margin-left: 5px;'>Multi-EUP</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='partial' name='partial' value='Partial' data-id='{{chat_id}}' onchange='updatePartial()'>" +
          "<label for='partial' style='margin-left: 5px;'>Partial</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='side_conversation' name='side_conversation' value='Side Conversation' data-id='{{chat_id}}' onchange='updateSideConversation()'>" +
          "<label for='side_conversation' style='margin-left: 5px;'>Side Conversation</label>" +
        "</div>" +
        "<div class='checkbox-container'>" +
          "<input type='checkbox' id='stt_error' name='stt_error' value='STT Error' data-id='{{chat_id}}' onchange='updateSttError()'>" +
          "<label for='stt_error' style='margin-left: 5px;'>STT Error</label>" +
        "</div>"
    },
    {"view_id": "text_input", "field_label": "Proposed EUP(s)", "field_id": "proposedEUP"}, 
    {"view_id": "text_input", "field_label": "notes"}
]

def add_options(
        stream, label_field="eup", choices=DIFFERENT_EUP
) -> Generator[Dict, None, None]:
    """
    Convert each line in the ``stream`` to a ``task`` with a text and an
    options field

    :param stream: the input stream
    :param label_field: key; defaults to "label"
    :param choices: the different choices
    :yield: a task Dict with text and options
    """

    for line in stream:
        if label_field in line:
            options = json.loads(line[label_field])
        else:
            options = []
        for word in (options + choices):
            if word not in options:
                options.append(word)
        task = {
            "eup": line["eup"],
            "message": line["message"],
            "ambig": False,
            "contextual": False,
            "multi": False,
            "partial": False,
            "side_conversation": False,
            "stt_error": False,
            "options": [
                {"id": o, "deployment": o, "prompt": o,
                "text": o} for o in options
            ]
        }
        yield task

@prodigy.recipe(
    "eup-corpus-validation-with-options",
    dataset=("The dataset to save to", "positional", None, str),
    file_path=("Path to texts", "positional", None, str),
    label_field=("Label to use for the accept task", "option", "f", str),
    choice_field=("Choices to add to the input", "option", "c", str)
)
def custom_labels(dataset, file_path, label_field, choice_field):
    """
    Annotate the text with labels from the list from the ``label_field`` in
    the input file. Augmented with choices from ``choice_field``.
    """

    blocks = UNLIMITED_ROWS

    stream = JSONL(file_path)
    stream = add_options(stream)  # add options to each task

    javascript = """
    // Set ambig to false by default
    prodigy.update({ ambig: false });

    function updateAmbig() {
      prodigy.update({ ambig: document.getElementById('ambig').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset ambig to false
      prodigy.update({ ambig: false });

      document.getElementById('ambig').checked = false;
    });

    // Set contextual to false by default
    prodigy.update({ contextual: false });

    function updateContextual() {
      prodigy.update({ contextual: document.getElementById('contextual').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset contextual to false
      prodigy.update({ contextual: false });

      document.getElementById('contextual').checked = false;
    });

    // Set multi to false by default
    prodigy.update({ multi: false });

    function updateMulti() {
      prodigy.update({ multi: document.getElementById('multi').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset multi to false
      prodigy.update({ multi: false });

      document.getElementById('multi').checked = false;
    });

    // Set partial to false by default
    prodigy.update({ partial: false });

    function updatePartial() {
      prodigy.update({ partial: document.getElementById('partial').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset partial to false
      prodigy.update({ partial: false });

      document.getElementById('partial').checked = false;
    });

    // Set side_conversation to false by default
    prodigy.update({ side_conversation: false });

    function updateSideConversation() {
      prodigy.update({ side_conversation: document.getElementById('side_conversation').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset side_conversation to false
      prodigy.update({ side_conversation: false });

      document.getElementById('side_conversation').checked = false;
    });

    // Set stt_error to false by default
    prodigy.update({ stt_error: false });

    function updateSttError() {
      prodigy.update({ stt_error: document.getElementById('stt_error').checked });
    }

    document.addEventListener('prodigyanswer', (event) => {
      // Reset stt_error to false
      prodigy.update({ stt_error: false });

      document.getElementById('stt_error').checked = false;
    });
    """
 
    return {
        "dataset": dataset,
        "view_id": "blocks",
        "stream": list(stream),
        "config": {
          "blocks": blocks,
          "javascript": javascript
        }
    }

Running this:

PRODIGY_ALLOWED_SESSIONS=cheyanne prodigy eup-corpus-validation-with-options eup_val_options_install_test examples.jsonl -F eup_corpus_validation_with_options.py

Yields this:

the following arguments are required: --label-field, --choice-field

This seems to be the same error message mentioned here. So I'll dive in.

The error message seems sound to me looking at the recipe declaration.

@prodigy.recipe(
    "eup-corpus-validation-with-options",
    dataset=("The dataset to save to", "positional", None, str),
    file_path=("Path to texts", "positional", None, str),
    label_field=("Label to use for the accept task", "option", "f", str),
    choice_field=("Choices to add to the input", "option", "c", str)
)
def custom_labels(dataset, file_path, label_field, choice_field):
    """
    Annotate the text with labels from the list from the ``label_field`` in
    the input file. Augmented with choices from ``choice_field``.
    """

We can see label_field in the decorator and this isn't referenced in the command line. But we aren't passing --label-field from the command line. So that would explain the error message. But my editor also suggests that label_field and choice_field aren't used in the recipe at all either.

Do tell if I'm glancing over something, but shouldn't those two inputs be removed? When I remove them I get other issues, but I'd like to make sure what the intention is here before moving on.

I removed choice_field and that error is now gone. Yes, that wasn't being used and should have been removed.

label_field: labels are imported from the .jsonl file. If I use the following command with dummy labels when I call the recipe, I can launch the recipe, but this recipe is basically validating our intent model: it's taking the top three results from our model (in the .jsonl dataset) and providing them as choices to the annotator along with two custom options (a different intent, an intent that doesn't exist`) that I'm importing from a utils file.

(prodigy) cheyannebaird@Cheyannes-MBP:~/posh/annotation-service$ PRODIGY_ALLOWED_SESSIONS=cheyanne prodigy eup-corpus-validation-with-options eup_val_options_install_test /Users/cheyannebaird/posh/eup_corpus/eup_task_april2023/final_final/eup_low_scoring_data_v1.jsonl -F /Users/cheyannebaird/posh/annotation-service/src/annotator/recipes/eup_corpus_validation_with_options.py --label foo,bar
⚠ Prodigy automatically assigned an input/task hash because it was
missing. This automatic hashing will be deprecated as of Prodigy v2 because it
can lead to unwanted duplicates in custom recipes if the examples deviate from
the default assumptions. More information can found on the docs:
https://prodi.gy/docs/api-components#set_hashes

✨  Starting the web server at http://localhost:8080 ...
Open the app in your browser and start annotating!

Just to check, does that mean that you're no longer blocked?

I may have to adjust our task definitions to add dummy labels when using the recipes that are throwing this error -- so far only span, review and NER recipes are not throwing this error because I'm explicitly defining a label set. We create a task definition that is tied to a specific dataset, and if the dataset contains the labels or the task involves classification, then I'm not calling the labels directly in the task definition. Since we're now getting this label error in the latest update, I am going to test adding dummy labels to the task definition to see if that's a workaround for us.

Ok! Let me know if you have something reproducible that I can check out locally with a bug in it. I'm eager to fix any bugs, but I will need to be able to run the same code as you to make sure I'm adressing the issue correctly.

Let me know!

I think I figured out the issue, and I was able to get the recipe to run without dummy labels. I had another label_field declared that was unused. Removed it, and now we're good!

1 Like

Happy to hear it!

The errors in this thread were solved, but I have a new one that is preventing us from upgrading to v1.14.6.

We are getting an error msg that states it cannot find any of our custom recipes:

✘ Can't find recipe or command 'ner_task_v3_b_validation'.
Run prodigy --help for more details. If you're using a custom recipe, provide
the path to the Python file using the -F argument.
Available recipes: ab.llm.tournament, ab.openai.prompts, ab.openai.tournament,
abandonment, account-matching, agent-speedbump, audio.manual, audio.transcribe,
compare, coref.manual, custom-intent-calibration, custom-labels,
custom-labels-prompt-multi-intent, data-to-spacy, db-in, db-merge, db-out,
dep.correct, dep.teach, drop, eup-corpus-validation,
eup-corpus-validation-with-options, filter-by-patterns, helpful-banking-moments,
helpful-banking-moments-repeat-intents, image.manual, login-failure, mark,
match, metric.iaa.binary, metric.iaa.doc, metric.iaa.span, ner-spans,
ner.correct, ner.eval-ab, ner.llm.correct, ner.llm.fetch, ner.manual,
ner.model-annotate, ner.openai.correct, ner.openai.fetch, ner.silver-to-gold,
ner.teach, pos.correct, pos.teach, print-dataset, print-stream, progress,
rel.manual, review, sent.correct, sent.teach, spacy-config, spans.correct,
spans.llm.correct, spans.llm.fetch, spans.manual, spans.model-annotate, stats,
stt-error-validation, stt-spans, terms.llm.fetch, terms.openai.fetch,
terms.teach, terms.to-patterns, textcat.correct, textcat.llm.correct,
textcat.llm.fetch, textcat.manual, textcat.model-annotate,

The error only appears when we run the prodigy server programmatically, i.e., using the prodigy.serve() method as explained here (https://prodi.gy/docs/api-components#serve), and that it worked just fine until we upgraded to v1.14.6.

And this error appears for all our custom recipes, not just the one in the error message above.

We tried adding the recipe path to the serve() command (with -F, as I do when I invoke prodigy directly) but that did not solve the error.

Were there any changes in this update that would have caused this?

The only changes in v1.14.6 should be the Pydantic/spaCy version bumps. But I can't imagine that would cause this issue.

Looking at your output though ... I can't help but notice stt-spans and stt-error-validation in the list of known recipes. So it is able to detect some of your custom recipes it seems.

I also just downloaded Prodigy v1.14.6 to try and reproduce your error. I have this custom recipe:

import prodigy

@prodigy.recipe(
    "my-custom-recipe",
    dataset=("Dataset to save answers to", "positional", None, str),
    view_id=("Annotation interface", "option", "v", str)
)
def my_custom_recipe(dataset, view_id="text"):
    # Load your own streams from anywhere you want
    stream = [{"text": f"omg {i}"} for i in range(1000)]

    def update(examples):
        # This function is triggered when Prodigy receives annotations
        print(f"Received {len(examples)} annotations!")

    return {
        "dataset": dataset,
        "view_id": view_id,
        "stream": stream,
        "update": update
    }

I'm able to confirm that this runs fine:

python -m prodigy my-custom-recipe xxx --view-id text -F recipe.py

When I move it into a Python script, like so:

import prodigy 

prodigy.serve("prodigy my-custom-recipe xxx --view-id text -F recipe.py", port=9000)

Then I seem to hit the same issue.

✘ Can't find recipe or command 'my-custom-recipe'.
Run prodigy --help for more details. If you're using a custom recipe, provide
the path to the Python file using the -F argument.
Available recipes: ab.llm.tournament, ab.openai.prompts, ab.openai.tournament,
audio.manual, audio.transcribe, compare, coref.manual, data-to-spacy, db-in,
db-merge, db-out, dep.correct, dep.teach, drop, filter-by-patterns,
image.manual, mark, match, metric.iaa.binary, metric.iaa.doc, metric.iaa.span,
ner.correct, ner.eval-ab, ner.llm.correct, ner.llm.fetch, ner.manual,
ner.model-annotate, ner.openai.correct, ner.openai.fetch, ner.silver-to-gold,
ner.teach, pos.correct, pos.teach, print-dataset, print-stream, progress,
rel.manual, review, sent.correct, sent.teach, spacy-config, spans.correct,
spans.llm.correct, spans.llm.fetch, spans.manual, spans.model-annotate, stats,
terms.llm.fetch, terms.openai.fetch, terms.teach, terms.to-patterns,
textcat.correct, textcat.llm.correct, textcat.llm.fetch, textcat.manual,
textcat.model-annotate, textcat.openai.correct, textcat.openai.fetch,
textcat.teach, train, train-curve

When I revert to version v1.14.1 however, I seem to get the same error.

> python -m pip install prodigy==1.14.1 -f https://<lincense-key>@download.prodi.gy
> python serve.py

Just to check, what version of Prodigy does work for you here? I'm definately eager to dive into this, but it would help to know when this feature might've broken.