How to incorporate document metadata?

[ines (Ines Montani)]

@hugoduncan

We then have to separately call Language.make_doc to create a Doc instance, add the required data, and then thread the document through a loop over the pipeline components (is there a better way of handling this in Spacy?).

You could use Language.pipe and set as_tuples=True, to process the documents as (text, context) tuples. For example:

messages = [('This is a message', {'user_id': 123})]
for doc, meta in nlp.pipe(messages, as_tuples=True):
    # do something with the docs and meta, e.g. add it to a custom attribute
    doc._.meta_data = meta

@hugoduncan

I’m wondering how we can use Prodigy with this pipeline, as I see no way to pass in the information required.

Annotation tasks are simple dictionaries that can be structured however you like. Depending on the recipe and annotation interface you want to use, Prodigy expects some keys to be present (for example, "text", "spans" or "label") – but aside from that, you can freely add your own fields and data that should be stored with the task. For example:

{"text": "This is a message", "user_id": 123}

When you annotate the task, the annotations are added and its saved to the database, together with your custom fields.

@hugoduncan

I see the Prodigy meta field, but don’t think that does what we need.

The "meta" field is mostly intended for metadata that should be displayed to the annotator (in the bottom right corner of the annotation card). You could consider adding some of your metadata here as well, especially in the early development / exploration phase. If you come across strange or interesting examples, it's often good to know their context.