Unable to save annotation: TypeError: can only concatenate list (not "float") to list

,
1

I wrote a custom recipe which let annotator to annotate data into different fields. I was able to start the annotation server and annotate the data. But when I click the save button, it cannot save the annotation. Here is the full log of error message I received:

future: <Task finished name='Task-19' coro=<RequestResponseCycle.run_asgi() done, defined at /opt/homebrew/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py:402> exception=TypeError('can only concatenate list (not "float") to list')>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 409, in run_asgi
    self.logger.error(msg, exc_info=exc)
  File "/opt/homebrew/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 1518, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/opt/homebrew/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/opt/homebrew/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 1643, in handle
    if (not self.disabled) and self.filter(record):
                               ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 830, in filter
    result = f.filter(record)
             ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/prodigy/__init__.py", line 21, in filter
    raise rec.exc_info[1]
  File "/opt/homebrew/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response
    await self.app(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/prodigy/app.py", line 522, in give_answers
    controller.receive_answers(
  File "/opt/homebrew/lib/python3.11/site-packages/prodigy/core.py", line 585, in receive_answers
    progress = self._cb_progress(self, session, mod_answers, update_return)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/prodigy/core.py", line 874, in wrapper
    return func_(ctrl, update_return_value=update_return_value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/prodigy/components/progress.py", line 29, in __call__
    logloss = math.log(loss + 1e-8)
                       ~~~~~^~~~~~
TypeError: can only concatenate list (not "float") to list

I cannot find a solution for this. Can anyone help?

2

Welcome to the forum @benxue! :waving_hand:

Does your custom recipe also define a custom progress estimator? The update_return_value you pass to it should be a float or None and it looks like it's a list instead?
(I have just updated the docs to make it clear).
If that doesn't help, could you perhaps share your recipe for further debugging? Also, which Prodigy version are you running? Thank you.

3

Hi there, @MAGDAANIOL, thanks for the response. I dont think I have defined a custom progress estimator. Here is my recipe if you can help debug. Thank you!

import prodigy
from prodigy.components.stream import get_stream
from prodigy.util import set_hashes
import uuid
from datetime import datetime
import json
import os
FIELDS = [
    "timestamp", "chat_history", "last_user_utterance", "feedback", "language",
    "locale", "lineage", "annotation", "holdout", "qa", "author", "annotator",
    "description", "EventTime", "index"
]
# Global list to store all annotations
all_annotations = []
@prodigy.recipe(
    "feedback-annotation-recipe",
    dataset=("Dataset to save annotations to", "positional", None, str),
    output_file=("Output JSON file path", "option", "o", str),
    view_id=("Annotation interface", "option", "v", str)
)
def feedback_annotation(dataset: str, output_file: str, view_id: str = "blocks"):
    # Create an empty stream that prompts for new annotations
    def generate_stream():
        while True:
            # Yield an empty task to trigger new annotation creation
            yield {"id": str(uuid.uuid4()), "text": "Annotation Form"}

    stream = generate_stream()

    # Define the annotation interface using blocks
    blocks = [
        {
            "view_id": "text_input",
            "field_id": "chat_history",
            "field_label": "Chat History",
            "field_placeholder": '["user: hello", "bot: hi"]',
            "field_rows": 3
        },
        {
            "view_id": "text_input",
            "field_id": "last_user_utterance",
            "field_label": "Last User Utterance"
        },
        {
            "view_id": "text_input",
            "field_id": "feedback",
            "field_label": "Feedback",
            "field_rows": 3
        },
        {
            "view_id": "text_input",
            "field_id": "language",
            "field_label": "Language (e.g., English)"
        },
        {
            "view_id": "text_input",
            "field_id": "locale",
            "field_label": "Locale (e.g., en-US)"
        },
        {
            "view_id": "text_input",
            "field_id": "lineage",
            "field_label": "Lineage"
        },
        {
            "view_id": "text_input",
            "field_id": "annotation",
            "field_label": "Annotation"
        },
        {
            "view_id": "text_input",
            "field_id": "holdout",
            "field_label": "Holdout Set"
        },
        {
            "view_id": "text_input",
            "field_id": "qa",
            "field_label": "QA Notes"
        },
        {
            "view_id": "text_input",
            "field_id": "author",
            "field_label": "Author"
        },
        {
            "view_id": "text_input",
            "field_id": "annotator",
            "field_label": "Annotator"
        },
        {
            "view_id": "text_input",
            "field_id": "description",
            "field_label": "Description",
            "field_rows": 3
        }
    ]
    
    def update(answers):
        for answer in answers:
            # Add auto-generated fields
            now = datetime.utcnow().isoformat()
            answer["timestamp"] = now
            answer["EventTime"] = now
            answer["index"] = str(uuid.uuid4())
            if "text" in answer:
                del answer["text"]
            # Set default values for all fields
            for field in FIELDS:
                if field not in answer:
                    if field == "chat_history":
                        answer[field] = []
                    else:
                        answer[field] = ""
            
            # Add dummy loss value to prevent type error
            answer["_loss"] = float(0.0)

            # Extract only desired fields for saving
            filtered_answer = {field: answer[field] for field in FIELDS}
            all_annotations.append(filtered_answer)

        if output_file and all_annotations:
            directory = os.path.dirname(output_file)
            if directory:
                os.makedirs(directory, exist_ok=True)
            with open(output_file, "w", encoding="utf-8") as f:
                json.dump(all_annotations, f, ensure_ascii=False, indent=2)
        return answers
    
    return {
        "dataset": dataset,
        "view_id": view_id,
        "stream": (set_hashes(eg) for eg in stream),
        "update": update,
        "progress": progress,
        "config": {
            "blocks": blocks,
            "global_css": """
                .prodigy-wrapper { padding: 20px; }
                .prodigy-interface { max-width: 800px; margin: 0 auto; }
            """,
            "title": "Feedback Annotation Pipeline",
        }
    }
4

Hi @benxue!

It shouldn't be too hard to fix! The problem is the return value of your update callback. If the update callback returns a value (it doesn't have to, it's optional) it is used as theupdate_value_parameter of the progress measuring function. For this reason, if it returns a value, it should be a float or int (you can see the linked docs for more details on how these functions are connected in the backend).
In your case, the update callback returns a list of answers:

 return answers

Looking at your implementation of the update, you are using it to postprocess the annotations to save them in the desired format. There's a another callback for this purpose: before_db. This one returns the transformed examples list.
The update callback is meant for updating the model in the loop which is why its return must be int/float. It can also be used for sending any other updates about the state of annotation, in which case it doesn't have to return anything.

I also noticed that you're writing the annotation to the file inside the update callback? Is there any special reason you'd choose to save your annotations manually from the recipe level? There's really no need for that. The annotations will be saved in the database in the dataset specified as dataset argument. No special configuration is required for custom recipes, it will work exactly the same as for the built-in recipes.
Apart from persistence, saving in the DB has a lot of other benefits such as preventing duplicates, ensuring the right format, automatic stats computation and easy use with other recipes. You can easily extract your dataset at any point of annotation with the db-out command.
Let me know if you have any follow up questions or doubts about this!