Custom view templates with scripts

[justindujardin]

@ines

The main problem is that in order to interact with the current annotation task that’s displayed, user scripts would also need to be able to call the pre-compiled app’s internals (more specifically, dispatch an action). This is pretty difficult.

One trick I've used in the past is to place functions on the root object from some appropriate place in my app, e.g. a singleton or app component init:

window.prodigy = {
  addExample: (...) => {
    this.store.dispatch(...);
  }
}

By using global functions, you can hide your app/framework internals from the consumer of the API. The implementation is also relatively straightforward, so it could be an easy way to dip your toes into exposing an API without building up too much technical debt.

@ines

I did experiment with a few possible solutions, including some crazy stuff like evaling raw JSX with Babel. To my surprise, it worked – but it still feels very wrong.

Nice! Sometimes hacks are needed for building compelling tools. There was also a post on that react thread that suggested a less hacky approach using createContextualFragment

@ines

Another question is how opinionated we want the Prodigy app to be, and to what extent the front-end should be allowed to modify the incoming data.

I would be very opinionated about the data that comes into your system and leave the frontend as open for hacking as you can tolerate. This way you can make guarantees about not letting a user screw up the dataset by doing something stupid while also letting them execute basic project-specific scripts/hacks/workarounds/prototypes inside the prodigy frame.

@ines

Allowing it to freely edit annotation tasks is potentially problematic – a small bug or inconsistency can easily ruin an entire dataset without the user even noticing.

Perhaps something like JSON Schema could help? You define schema/s for the types of JSON data your system uses, and you can use them to validate objects before they touch your data layer. There are implementations in most major languages, and modern IDEs can usually pick up on the schemas for inline validation.

Here's what a text annotation schema could look like:

{
  "title": "TextAnnotation",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "text": { "type": "string" },
    "answer": {
      "type": "string",
      "enum": ["accept", "reject", "ignore"]
    },
    "meta": { "type": "object" },
    "_input_hash": { "type": "number" },
    "_task_hash": { "type": "number" },
  },
  "required": ["text", "answer", "_input_hash", "_task_hash"]
}

You could allow for user-specific data by adding something like a "user" field to your schema and lock down the rest of the object. When an object fails to validate, the specific properties that failed are available for generating helpful error messages. If it proved useful you could even apply it to validation during other prodigy things like db-in or prodigy.json.