Receiving "Couldn't save annotations" error

1

I have two annotators working on the same datasets, and they each occasionally get the following error:

ERROR: Couldn't save annotations. Make sure the server is running correctly.

The only patterns I can see are that the errors seems to occur more often when they are annotating at the same time, but we have also seen this error when only one annotator is annotating. This will sometimes result in:

  • loss of annotated data
  • duplicate utterances shown to the annotator (dupes aren't always present in the database, but the annotator notices that they are annotating the same document twice)

I updated instant_submit today from false to true in order to auto-save in the hopes that we would not lose any additional documents the annotators had annotated. We are saving the annotations in a CloudSQL database, which the prodigy web servers connect to via a proxy.

These are our prodigy.json settings:

{
  "theme": "dark",
  "custom_theme": {"smallText": 16, "bgCard": "#d9d9d9", "bgMeta": "#ffe184", "colorText": "#384451"},
  "buttons": ["accept", "reject", "ignore", "undo"],
  "batch_size": 10,
  "history_size": 10,
  "port": 8080,
  "host": "localhost",
  "cors": true,
  "db": "postgresql",
  "db_settings": {
    "postgresql": {
      "host": "127.0.0.1",
      "port": 5432,
      "dbname": "prodigy",
      "user": "annotator"
    }
  },
  "validate": true,
  "auto_exclude_current": true,
  "instant_submit": true,
  "feed_overlap": false,
  "auto_count_stream": false,
  "total_examples_target": 0,
  "ui_lang": "en",
  "global_css": ".prodigy-content mark span { font-size: 10px }",
  "project_info": ["dataset", "session", "lang", "recipe_name", "view_id", "label"],
  "show_stats": false,
  "hide_meta": false,
  "show_flag": true,
  "swipe": false,
  "swipe_gestures": { "left": "accept", "right": "reject" },
  "split_sents_threshold": false,
  "html_template": false,
  "javascript": null,
  "writing_dir": "ltr",
  "show_whitespace": false,
  "exclude_by": "task",
  "ner_manual_label_style": "dropdown",
  "field_id": "user_input",
  "field_label": "User input field",
  "field_placeholder": "Type here...",
  "field_rows": 1,
  "field_autofocus": false
}

While one annotator has some connection issues (slow connection), the other does not, and they are both experiencing this. Restarting our server, logging in and out, and closing and reopening their browser doesn't seem to improve the situation. The annotators annotate quickly, and I'm wondering if this is causing the issue as I saw that issue in another forum post. However, annotating quickly shouldn't result in an error. Annotators are annotating the same dataset in order to assess inter-annotator agreement. Right now, the current set is 5k documents (documents = short sentences), but this also occurred with a much smaller dataset of 500.

Are there settings I should adjust, or is this a larger issue? Are there meaningful logs I can access?

Thank you,
Cheyanne

2

Hi Cheyanne.

You can run Prodigy with the PRODIGY_LOGGING=basic environment variable set. This should give more information as things are running.

Could you share some more information about the task that you're running? Are you using a custom recipe?

3

I have set this variable. Where can I locate the logs? I am running a custom recipe. We have been using this custom recipe for months, and haven't seen this error before.

4

The logs should appear in stdout. Are you running it inline? Something like;

PRODIGY_LOGGING=basic prodigy ...

If it's of interest, more details are explained in our documentation here.

5

Thank you, I added PRODIGY_LOGGING=verbose so I am now seeing information on each doc submitted. If I see another error, I will check the logs and may post here again with more details.

6

This is the error we keep seeing (see below). The annotator annotated several documents, but the logs show that it is stuck on the first document and the annotations are not being saved.

The prodigy.json file above shows my settings. I'm wondering if there is a setting that I need to change to work alongside instant_submit? I added instant_submit in order to auto-save the documents. The dataset is shared between two annotators, and they are annotating the same documents (inter-annotator agreement), but only one annotator is working on the set right now.

Errors:

ERROR: Couldn't save annotations. Make sure the server is running correctly.

textPayload: "e[1;38;5;135m14:29:58e[0m: FEED: Finding next batch of questions in stream"

textPayload: "[1;38;5;135m14:27:05[0m: FEED: re-adding open tasks to stream"
7

Let's check a few things then.

  1. What version of prodigy are you running? A while ago we fixed a bug but this might not be reflected if you're running an older version.
  2. Can you confirm that if you turn off instant_submit that the errors go away?
  3. Can you confirm that your custom recipe doesn't have an error that's raised?
  4. The text payload seems a bit strange. Can you find the corresponding raw text field in your .jsonl file? I'm wondering if there's something strange happening with the text encoding that might be causing this.
8
  1. The latest: 1.11.7
  2. We get the error with instant_submit as true as well as false. I prefer setting it to true so that we can use the auto-save function. I am wondering about the other settings in the prodigy.json file above (auto_count_stream, feed_overlap, auto_exclude_current, and batch_size).
  3. The custom recipe does not produce any errors and there are no server errors.
  4. encoding is utf-8 on the input .jsonl file. I don't see anything odd about the file or the utterance that produced the error. We've seen the error on other datasets and other utterances, and I'm not sure I see any patterns. The log entries are coming from a file on our server rather than a local terminal (aka standard out); so we get the color codes ([1;38;5;135m14:27:05[0m: ) rather than the actually colored text strings.
9

Just going to add my personal experience from annotating to my computer through ngrok: I need to make sure I refresh my browser window when I come back to annotating after a break or else I tend to get this error. This leads to a lot of saving after one or two annotations because I get paranoid. YMMV but thought Iā€™d add!

1 Like
10

One more issue to report: our server was restarted, and when I restarted the tasks for each annotator, I am seeing the document where the saved annotations error showed up last time, but many documents have been annotated since that one was in the queue. I don't see anything odd about this particular line in the .jsonl file, though.

11

I checked your settings again but they don't seem strange. The auto_exclude_current setting is set to true, which is the default value just like auto_count_stream, so that seems fine. A batch_size of 1 certainly isn't unheard of. The feed_overlap setting also seems fine, although this depends more on how you'd like your colleagues to label. It shouldn't cause an error.

I'll ping some colleagues to check if they can spot something, but in the meantime, I am curious about the logs from CloudSQL or the proxy that you use. Can you see anything on the Google Cloud side that might indicate a connection/authentication mishap?

12

Thank you @koaning for looking into this. Here are some more details about our setup:

  • We have a service running in a Docker container on a GCP cluster, which is the "gateway" for annotators: the annotators use it to log in and sign up for an annotation task.
  • An annotation task is identified by a unique name, a recipe, a dataset name and an input file.
  • For each unique task, we spin up a Prodigy web server inside the service container (using the prodigy.serve() command), forward the internal port to a separate service on our cluster, and redirect the annotator to this service for the given annotation task
  • Each Prodigy web server connects to the CloudSQL database via the standard proxy provided by GCP; we use the same set-up on developers' machines that run Prodigy from the terminal: each has a Docker container running the same proxy to connect to our CloudSQL instance (over VPN).
  • We see a lot of entries like this in the CloudSQL logs (on the order of 400 per 24-hour period); they always come in pairs, separated by less than 100 ms:
New connection for prodigy-annotation-db 
Client closed local connection on 127.0.0.1:5432

We assume the "Client" is the prodigy web server, which presumably uses a pool of connections that are periodically refreshed? Our own code does nothing more than provide the proxy's host name and port number to the prodigy servers.

13

I'm thinking out loud here, but I recall from previous work with Postgress that a maximum number of connections can cause trouble. I think the default setting is 100 connections, which I think is also what Google assumes. The Google Cloud Docs also reflect this quota here and here.

If you run prodigy.serve() for each permutation of input file/name/recipe, is it likely that you hit this limit? Does your proxy close down connections to the database after a "task" is considered completed? I'm not 100% sure if this is the issue, but it seems good to check and rule out.

Are there any logs from your proxy service that might shed some light on this?

14

Thanks @koaning. Some additional info:

  • we limit the number of Prodigy servers to 5, and typically not more than two are running concurrently
  • when the annotation task is complete, we shut down the Prodigy server associated with the task manually; presumably that closes all the connections
  • we haven't seen any errors in the CloudSQL proxy log that mention running out of connections or exceeding our quota
15

One of my colleagues suggested you may want to try out our new alpha release. Details are listed here. It feels like it might be worth a try given what you're experiencing. One of the key differences is that we're switching from peewee to SQLAlchemy as our ORM, which might address some of the database issues you're experiencing as a result. Do let me know if this does/doesn't help!

Could you also check if the error persists across browsers? We mainly want to rule out that this is a Chrome/Firefox/Safari mishap.

17

@koaning We are using Chrome. Testing the alpha release will possibly require a lot of reconfiguration on our part, so I think we will wait for the beta release or full release.

18

Understood. If you gain any more insights from any logs in the meantime, do let us know! It feels like there might be a bug here, and we're eager to address it once it's better understood.

19

So it turns out that our annotators may have been using Edge as their browser, but not consistently. So that is a possible issue. I have them exclusively using Chrome, and will check to see if we have any additional issues.

1 Like
20

That's interesting to know! Thanks for reporting back :slight_smile:

21

We are thinking of testing this out, as we recently saw this error again in another task and are unable to diagnose what might be causing it. What is the latest alpha version?