could not implement custom task

, ,
1

Hello! Trying to create my own task to customize llm prompt, but the error occures:

catalogue.RegistryError: [E893] Could not find function 'my_textcat.v1' in function registry 'llm_tasks'. If you're using a custom function, make sure the code is available. If the function is provided by a third-party package, e.g. spacy-transformers, make sure the package is installed in your environment.

My config:

[paths]
examples = "fewshot_examples.jsonl"

[nlp]
lang = "en"
pipeline = ["llm"]
batch_size = 128

[components]

[components.llm]
factory = "llm"

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.0}

[components.llm.task]
@llm_tasks = "my_textcat.v1"

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = ${paths.examples}

[components.llm.task.normalizer]
@misc = "spacy.LowercaseNormalizer.v1"

my_task.py:

from spacy_llm.tasks.textcat import TextCatTask
from spacy_llm.registry import registry

@registry.llm_tasks("my_textcat.v1")
def my_textcat():
    return TextCatTask(
        labels=["POS", "NEG"],
        prompt_template="Classify the following guidance as POS or NEG:\n\n{text}",
        parse_responses=parse_response
    )

def parse_response(responses, *, examples):
    results = []
    for response in responses:
        label = response.strip().upper()
        if label in ["POS", "NEG"]:
            results.append({"cats": {label: 1.0}})
        else:
            results.append({"cats": {}})
    return results

Stack trace:

psql sa -c "\copy (select * from (select guidance as text, build_meta(symbol, quarter) as meta from app.guidance) as sub where meta not in (select get_meta(content) from example)) to stdout WITH (FORMAT CSV, HEADER TRUE);" | prodigy textcat.llm.correct annotated-recipes guidance_neg_pos.cfg - --loader csv
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/prodigy/__main__.py", line 50, in <module>
    main()
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/prodigy/__main__.py", line 44, in main
    controller = run_recipe(run_args)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/prodigy/cli.py", line 124, in run_recipe
    components = recipe_command.func(**parsed_args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/prodigy/recipes/llm/textcat.py", line 64, in llm_correct_textcat
    nlp = assemble(config_path, overrides=config_overrides)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy_llm/util.py", line 48, in assemble
    return assemble_from_config(config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy_llm/util.py", line 28, in assemble_from_config
    nlp = load_model_from_config(config, auto_fill=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy/util.py", line 587, in load_model_from_config
    nlp = lang_cls.from_config(
          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy/language.py", line 1889, in from_config
    nlp.add_pipe(
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy/language.py", line 821, in add_pipe
    pipe_component = self.create_pipe(
                     ^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy/language.py", line 709, in create_pipe
    resolved = registry.resolve(cfg, validate=validate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/confection/__init__.py", line 760, in resolve
    resolved, _ = cls._make(
                  ^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/confection/__init__.py", line 809, in _make
    filled, _, resolved = cls._fill(
                          ^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/confection/__init__.py", line 864, in _fill
    filled[key], validation[v_key], final[key] = cls._fill(
                                                 ^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/confection/__init__.py", line 863, in _fill
    promise_schema = cls.make_promise_schema(value, resolve=resolve)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/confection/__init__.py", line 1069, in make_promise_schema
    func = cls.get(reg_name, func_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maksimgurenkov/miniconda3/envs/py312/lib/python3.12/site-packages/spacy/util.py", line 163, in get
    raise RegistryError(
catalogue.RegistryError: [E893] Could not find function 'my_textcat.v1' in function registry 'llm_tasks'. If you're using a custom function, make sure the code is available. If the function is provided by a third-party package, e.g. spacy-transformers, make sure the package is installed in your environment.

Available names: prodigy.Terms.v1, prodigy.TextPrompter.v1, spacy.EntityLinker.v1, spacy.Lemma.v1, spacy.NER.v1, spacy.NER.v2, spacy.NER.v3, spacy.NoOp.v1, spacy.NoOpNoShards.v1, spacy.REL.v1, spacy.Raw.v1, spacy.Sentiment.v1, spacy.SpanCat.v1, spacy.SpanCat.v2, spacy.SpanCat.v3, spacy.Summarization.v1, spacy.TextCat.v1, spacy.TextCat.v2, spacy.TextCat.v3, spacy.Translation.v1
2

Welcome to the forum @mkgurenkov :waving_hand: !

How are you importing your custom code? Normally, you can make the recipe import your custom code by passing the .py file as -F argument to the recipe. So in the case of textcat.llm.correct that would be:

dotenv run -- python -m prodigy textcat.llm.correct test config.cfg ../../source.jsonl -F my_task.py

That said they way you initialize the TextCatTask is not entirely correct. There's a number of arguments missing - please see spaCy source for details.
Notably, the built-in TextCatTask follows sharding interface so you might need to implement non-sharding generate_prompts as well.

Alternatively, if it easier, you could keep using the built-in TextCatTask with llm.textcat.fetch and then posprocess the results to obtain valid textcat annotations, which you could then use to further curate with textcat.correct.