Error saving annotations

,
1

I started having a mysterious error while saving annotations. A red window appears saying "ERROR: couldn't save annotations. Make sure the server is running correctly.

The log shows the following:

INFO:     73.169.118.24:49716 - "POST /get_session_questions HTTP/1.1" 200 OK
2021-12-28 22:48:17 [ ERROR] routing.py:111 : Error getting request body:

Nothing else, even with verbose logging. The server is still up, and works for other sessions. The document doesn't make it to the database.

I am using a simple custom NER + choice recipe, that I've been using for a long time, nothing has changed. And same kind of texts. It started happening in old Prodigy, v 1.10 or so. I installed the latest one, but the error persists. I experimented with several datasets, I'm able to save about 20-30 records, and then this happens again. Nothing special is about the document that cannot be saved, but I tried different sessions on the same annotation process, and it stops working on the same doc. After I remove the offending doc from the input, saving works for another 10-20 records, and then it happens again. The offending document can be saved fine in a separate dataset.

I was wondering if anyone have seen this error. Google found nothing except for this :

but I am not sure what to make out of it, since I am not familiar with prodigy code.

I am using sqlite database, and it is approaching 1G.

I added before_db callback to the recipe, but that code doesn't get executed. validate_answer doesn't get executed, either - for the doc that causes the error. For other sessions, that haven't reached the offending doc yet, those 2 callbacks work.

2

Hi! Thanks for the detailed report, that's very strange and I haven't seen this come up before. It definitely looks like the error happens on the server level, which then terminates the server and means Prodigy can't pull the next batch of questions anymore (because the server stopped running).

Which Python version are you running and did you change Python versions recently? And could you check which versions of fastapi and starlette you're running? If nothing changed about your recipe and Prodigy configuration or installation, maybe something you installed pulled in a newer version of one of the dependencies that somehow causes this problem (maybe in combination with your Python version).

3

Both seem to be appropriate:
fastapi - 0.68.2
starlette - 0.14.2

I did a bit more experimenting. Not sure what JS tools you are using, and bundle.js in not readable, but I will paste a few things, maybe you have some ideas.

The JS code throws an exception in the browser:
Error: TypeError: NetworkError when attempting to fetch resource.
It is thrown from a huge function that contains:

  f = (t.mapDispatchToProps = function (e) {
    return {
      getProject: function (t) {
        return e(f(t))
      },
      updateQueue: function () {
        return e(d())
      },
      answerQuestion: function (t, n, r) {
        return e(g(t, n, r))
      },
      updateQuestion: function (t) {
        return e(function (e) {
          return {
            type: 'UPDATE_TASK',
            updates: e
          }
        }(t))
      },
...

The exact fragment is this (in pretty-printed bundle.js in Firefox it is line #50938):

  x = function (e, t, n) {
    return function (r) {
      throw n && r(n),
      r(_(t, 'error')),
      Error(e) # Here the exception is thrown
    }
  }

Call stack is a bit confusing, but the code above is called from (line 23130):

function (e, t, n) {
  'use strict';
  function r(e) {
    return function (t) {
      var n = t.dispatch,
      r = t.getState;
      return function (t) {
        return function (o) {
          return 'function' == typeof o ? o(n, r, e) : t(o) # Called from here
        }
      }
    }
  }

And then dispatch, a promise callback, and this:

    t(m('give_answers', 'POST', {
        answers: e,
        session_id: r,
        annotator_id: r
      })).then((function (e) {
        return t(200 === e.status ? {
          type: 'RECEIVE_PROGRESS',
          progress: e.progress
        }
         : o())
      })).catch((function (e) {
        return t(o(e)) # Here e = "TypeError: NetworkError when attempting to fetch resource."
      }))
    }

Any ideas what can it be? Or any suggestions on how to moved towards finding a cause of it?

4

Both seem to be appropriate:

fastapi == '0.68.2'
starlette == '0.14.2'

I installed older versions, it didn't help.

I tried to find out more, and here is what's happening (I'm using Firefox, and code fragments are from pretty-printed bundle.js). Same errors happen in chrome and safari, and when connecting to prodigy from a different computer.

bundle.js throws an exception:
TypeError: NetworkError when attempting to fetch resource.
It doesn't get caught, I assume, because eventually it shows up in the log as:
Uncaught (in promise) TypeError: NetworkError when attempting to fetch resource.

It is thrown from this function (line 50934):

  x = function (e, t, n) {
    return function (r) {
      throw n && r(n),
      r(_(t, 'error')),
      Error(e)
    }
  }

It is a part of a larger function that contains prodigy-related stuff:

  f = (t.mapDispatchToProps = function (e) {
    return {
      getProject: function (t) {
        return e(f(t))
      },
      updateQueue: function () {
        return e(d())
      },
      answerQuestion: function (t, n, r) {
        return e(g(t, n, r))
      },
      updateQuestion: function (t) {
        return e(function (e) {
          return {
            type: 'UPDATE_TASK',
            updates: e
          }
        }(t))
      },

The call stack is hard to understand, but a few levels down (line 50874):

      t(m('give_answers', 'POST', {
        answers: e,
        session_id: r,
        annotator_id: r
      })).then((function (e) {
        return t(200 === e.status ? {
          type: 'RECEIVE_PROGRESS',
          progress: e.progress
        }
         : o())
      })).catch((function (e) { # e here is " TypeError: NetworkError when attempting to fetch resource."
        return t(o(e))
      }))

The stack also mentions dispatch and promise callback among single-letter-named functions.

Any ideas what can it be? And how to start moving towards figuring this out?

It seems that documents on which the process gets stuck are on a larger side (12K characters) where normally they are about 4K. The disturbing thing is that the process gets stuck on the same document consistently for a while (after restarts, renaming datasets, using different session ids, etc.), but then occasionally it gets saved successfully. Couldn't detect any pattern.

5

The errors on the client are all kind of expected and side-effects of the server going down: if the server is going down, the requests the app makes to it will fail. And they'll fail with a network error because the endpoint isn't even available anymore. So this is all consistent with the underlying problem, which is that something in FastAPI triggers an exception.

Which Python version are you using?

6

Python 3.7.10
When this error occurs, the server would still up and other sessions would be functional, until they reach the same doc.

7

Could you try running the same command in a different virtual env with Python 3.8? I know that the async serving sometimes has subtle differences in different Python versions so maybe this problem doesn't occur in a newer version.

8

I think we figured out something - it seems to be happening only on port 8080. I will investigate further and share any useful findings. I will try out python 3.8 as well, but not in the next couple of days. Thank you for your help!

1 Like