Pattern-based recognition using negation detection

Maybe you could experiment with detecting disorders first, and then use the syntax to resolve the diagnosis type? For example, with a combination of statistical NER and a dictionary, you could probably build a decent system to detect "infarct" and "ventricular abnormalities". You can then write rules that take advantage of the dependency parse and how the disorders and similar entities are connected to the rest of the sentence to determine whether it's negated, uncertain and so on.

It might sound trivial, but depending on your data, you might be able to resolve a large number of "negated diagnosis" relationships by simply looking at the previous token and whether it's "no". You might also be able to determine other grammatical constructs that are very strong indicators of certain types of diagnoses. Even out-of-the-box, you should get a pretty decent accuracy for part-of-speech tags and syntactic dependencies – and if not, you can use Prodigy to tweak the model on your data and improve the results of your rules. You could also combine this with your existing text classifier – maybe it turns out that it's pretty good for some cases, so you can rely on it for the very high confidence predictions and handle the rest by your other system.

See here for more details how to extract dependency relationships with spaCy and this page for how to write token-based match patterns to extract certain information. Also, here are some threads that discuss similar problems and approaches:

1 Like