Support in syntax highlighting

Hey! :slight_smile:
We would like to use the tool for tagging code, and as part of the tagging process we would like that the text will be highlighted by relevant keywords using syntax highlighting.
Is there any option to use syntax highlighting in the tagged documents (for example, using method / plugin / define it as part of the task)?
If so - how is it possible?

Hi! This is an interesting use case and it should in theory already work if you can pre-tokenize your code with the info about the different syntax pieces. After all, it's still just tokens – only a different kind :sweat_smile:

The ner_manual interface expects the following JSON format with a list of "tokens": https://prodi.gy/docs/api-interfaces#ner_manual You could generate those based on our tokenized code. Each token can also take a "style" attribute, which you can use for the highlighting colour, and a boolean "ws" attribute that indicates whether the token is followed by a space or not. So your data would end up looking like this, which you can probably auto-generate based on the AST produced by the syntax highlighting algorithm:

{
    "text": "import spacy",
    "tokens": [
        {"text": "import", "start": 0, "end": 6, "id": 0, "style": "color: red", "ws": true},
        {"text": "spacy", "start": 7, "end": 12, "id": 1, "style": "color: black", "ws": false},
    ]
}

Make sure to also include tokens for newlines, whitespace and tabs, since those are all part of the code as well. In your theme, you can also change the fontPrimary to a monospace font to make it easier to read the code.

If you try this approach, definitely keep us updated on how you go! :blush: