Error invalidSave with no explanatory output to browser or logs.

Hi, I’ve got prodigy starting behind an authenticating proxy, and everything appears to be working correctly. However, everytime it tries to autosave, or a user tries to manually save by hitting the ‘save’ button, an ‘Error invalidSave’ dialog appears and then disappears with no further explanation.

There’s also no log output – or any output really, besides a cute emoji on startup – from the prodigy process that might help diagnose the error. Follows is the entire session output from starting the process, to when the error appears. The werkzeug log messages are from from the wrapper flask app on port 8080 I have that’s enforcing authentication and proxying requests to the prodigy process listening on localhost:80801.

I have confirmed the sqlite db is being recreated created by prodigy if I delete it prior to start up.

Any suggestions?

ubuntu@ip-172-31-95-135:/var/riverdrop-prodigy/current$ /bin/bash -c "(PRODIGY_HOME=/var/riverdrop-prodigy/prodigy python3 /var/riverdrop-prodigy/current/app.py 2>&1)"
process id of parent is: 30513
INFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
Using 1 labels: FURNITURE_PIP
Added dataset page_categories to database SQLite.

  ✨  Starting the web server at http://localhost:8081 ...
  Open the app in your browser and start annotating!

INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:31] "GET / HTTP/1.1" 401 -
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:31] "GET /bundle.js HTTP/1.1" 401 -
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:31] "GET /fonts/lato-regular.woff2 HTTP/1.1" 401 -
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:31] "GET /fonts/robotocondensed-bold.woff2 HTTP/1.1" 401 -
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:31] "GET /fonts/lato-bold.woff2 HTTP/1.1" 401 -
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:31] "GET /project HTTP/1.1" 401 -
INFO:__main__:Proxying get_questions (<class 'str'>) to http://127.0.0.1:8081/get_questions (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:33] "GET /get_questions HTTP/1.1" 200 -
INFO:__main__:Proxying fonts/robotocondensed-bold.woff2 (<class 'str'>) to http://127.0.0.1:8081/fonts/robotocondensed-bold.woff2 (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:33] "GET /fonts/robotocondensed-bold.woff2 HTTP/1.1" 200 -
INFO:__main__:Proxying fonts/lato-regular.woff2 (<class 'str'>) to http://127.0.0.1:8081/fonts/lato-regular.woff2 (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:33] "GET /fonts/lato-regular.woff2 HTTP/1.1" 200 -
INFO:__main__:Proxying fonts/lato-bold.woff2 (<class 'str'>) to http://127.0.0.1:8081/fonts/lato-bold.woff2 (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:33] "GET /fonts/lato-bold.woff2 HTTP/1.1" 200 -
INFO:__main__:Proxying bundle.js (<class 'str'>) to http://127.0.0.1:8081/bundle.js (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:33] "GET /bundle.js HTTP/1.1" 200 -
INFO:__main__:Proxying  (<class 'str'>) to http://127.0.0.1:8081/ (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET / HTTP/1.1" 200 -
INFO:__main__:Proxying project (<class 'str'>) to http://127.0.0.1:8081/project (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET /project HTTP/1.1" 200 -
INFO:__main__:Proxying bundle.js (<class 'str'>) to http://127.0.0.1:8081/bundle.js (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET /bundle.js HTTP/1.1" 200 -
INFO:__main__:Proxying project (<class 'str'>) to http://127.0.0.1:8081/project (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET /project HTTP/1.1" 200 -
INFO:__main__:Proxying fonts/robotocondensed-bold.woff2 (<class 'str'>) to http://127.0.0.1:8081/fonts/robotocondensed-bold.woff2 (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET /fonts/robotocondensed-bold.woff2 HTTP/1.1" 200 -
INFO:__main__:Proxying fonts/lato-regular.woff2 (<class 'str'>) to http://127.0.0.1:8081/fonts/lato-regular.woff2 (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET /fonts/lato-regular.woff2 HTTP/1.1" 200 -
INFO:__main__:Proxying fonts/lato-bold.woff2 (<class 'str'>) to http://127.0.0.1:8081/fonts/lato-bold.woff2 (<class 'str'>)
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:34] "GET /fonts/lato-bold.woff2 HTTP/1.1" 200 -
INFO:werkzeug:100.38.105.173 - - [17/Apr/2018 16:58:39] "POST /give_answers HTTP/1.1" 405 -

Found the issue immediately after posting. For the record, if you write basic authentication proxy to shim authentication in front of prodigy, you must proxy POST and PUT requests as well.

Follows is the flask auth proxy app that’s now working:

from flask import Flask
from functools import wraps
from flask import request, Response
import os
import requests
import json
import logging
import subprocess
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = Flask(__name__)

deploy_prodigy_home = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'prodigy')
if os.environ.get('PRODIGY_HOME') is not None:
    prodigy_home = os.environ.get('PRODIGY_HOME')
else:
    if os.path.exists(deploy_prodigy_home):
        prodigy_home = deploy_prodigy_home
    else:
        prodigy_home = os.path.join(os.path.dirname(__file__), 'prodigy')
    logger.info("Setting PRODIGY_HOME to {}".format(prodigy_home))
    os.environ['PRODIGY_HOME'] = prodigy_home

logger.info("PRODIGY_HOME is {}".format(os.environ.get('PRODIGY_HOME')))
prodigy_config_file = os.path.join(prodigy_home, "prodigy.json")

if not os.path.exists(prodigy_config_file):
    raise FileExistsError("Prodigy config file '{}' does not exist!".format(prodigy_config_file))

prodigy_config=json.loads(open(prodigy_config_file, "r").read())
prodigy_port = prodigy_config.get('port')
if prodigy_port is None:
    raise ValueError("port setting ('port') not found in prodigy config file".format(prodigy_config_file))


def check_auth(username, password):
    """This function is called to check if a username /
    password combination is valid.
    """
    return username == 'USERNAME' and password == 'PASSWORD'


def authenticate():
    """Sends a 401 response that enables basic auth"""
    return Response('Access Denied', 401, {'WWW-Authenticate': 'Basic realm="Login Required"'})


def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = request.authorization
        if not auth or not check_auth(auth.username, auth.password):
            return authenticate()
        return f(*args, **kwargs)
    return decorated


@app.route('/')
@app.route('/<path:uri>', methods=['GET', 'POST', 'PUT'])
@requires_auth
def proxy(uri:str=""):
    proxy_url = 'http://127.0.0.1:{}/{}'.format(prodigy_port, uri)
    logger.info("Proxying {} ({}) to {} ({})".format(uri, type(uri), proxy_url, type(proxy_url)))
    resp = requests.request(
        method=request.method,
        url=proxy_url,
        headers={key: value for (key, value) in request.headers if key != 'Host'},
        data=request.get_data(),
        cookies=request.cookies,
        allow_redirects=False)
    excluded_headers = ['content-encoding', 'content-length', 'transfer-encoding', 'connection']
    headers = [(name, value) for (name, value) in resp.raw.headers.items()
               if name.lower() not in excluded_headers]
    response = Response(resp.content, resp.status_code, headers)
    return response


def start_prodigy():
    model = os.path.join(os.path.dirname(__file__), 'MODEL_NAME')
    jsonl = os.path.join(os.path.dirname(__file__), 'TRAINING.JSONL')
    subprocess.call(['python3',
                     '-m',
                     'prodigy',
                     'textcat.teach',
                     'page_categories',
                     model,
                     jsonl,
                     '--label',
                     'LABEL_TO_TRAIN'])

  

if __name__ == '__main__':
    p = os.fork()
    if p == 0:
        start_prodigy()
    else:
        print("process id of parent is: %d" % os.getpid())
        app.run(debug=False, host='0.0.0.0', port=8080)

Thanks so much for updating with your solution :+1:

Btw, we’re also thinking about whether we can ship some of those proxy or authentication components with Prodigy – we just need to decide how to set it up to still allow users to mix and match them (because the individual requirements are often very specific).