Does Prodigy support HTML annotation for NER

[ines (Ines Montani)]

Annotating rendered HTML might sound appealing at first, but there's actually not really an easy answer for how the annotations should be resolved back to the underlying raw text and how to ensure that annotations are consistent. After all, what your model will get to see is the raw text.

I discuss some of these considerations in more detail on this thread:

@ines

If you pass in "<strong>hello</strong>" , there’s no clear solution for how this should be handled. How should it be tokenized, and what are you really labelling here? The underlying markup or just the text, and what should the character offsets point to? And how should other markup be handled, e.g. images or complex, nested tags?

Similarly, if you’re planning on training a model later on, that model will also get to see the raw text, including the markup – so if you are working with raw HTML (like, web dumps or something), you usually always want to see the original raw text that the model will be learning from. Otherwise, the model might be seeing data/markup that you didn’t see during annotation, which is always problematic.

One common solution is to write a function that takes raw HTML, strips out the markup, tokenizes the text and stores each token's character offset into the original raw HTML. This way, you can work with raw text without markup, while still being able to resolve the character offsets of your annotations back to the original input.