HTML Source Sentence Boundary Detection Prodigy

[nix411 (Nicolai Bjerre Pedersen)]

@SmoothRecursion

Does anyone have any good strategies on preprocessing html to get it to a usable form in Prodigy?

I've made a parser that segments html into a list of paragraphs (if you see a new line on the HTML page then you get a new paragraph). Is that something you could use? However note what Matthew Honnibal said

@honnibal

Mark-up languages like HTML often provide important non-text information in the formatting. If you extract to plain text, you might lose information about what’s a list, what’s a section, etc. Obviously spaCy expects to work on raw text — but that doesn’t mean the non-text aspects aren’t going to be important in other ways.

I have this exact issue. I actually want some of the non-text aspects as features but I don't know how to achieve that in a spaCy framework. I've laid out my challenge in another thread.