Expanding NER to include neighbouring tokens

[nix411 (Nicolai Bjerre Pedersen)]

Hi @ines

Thanks a lot for the helpful answers you always seem to deliver!

@ines

I mean, in theory, you could label some data manually that annotates the spans lik this, update the model with it and hope that it adjusts to your new concept of MONEY . The question is whether it’s worth it. If the pre-trained model you’re using was trained with an annotation scheme that considered the currency not part of an entity, all its current weights are based on that policy. It might take a lot of work and data to teach it a very different definition of the entity type MONEY .

Edit: Just checked and it seems like the annotation scheme does include the currency by default. It just doesn’t seem to be recognised correctly in this case. I can double-check to see how MONEY was annotated in the corpus we’re using for English.

Alright. Good to know. I also read this post where you suggest training a new NER model from scratch. I might take that approach as well - at least test and compare. Is there a way to omit some pretrained labels but keep others?

@ines

Another thing to think about: What’s your end goal once you have the money entities? Will you be converting them to some type of structured format like {'amount': 113000000, 'currency': 'SEK'} ? If so, it might make more sense to leave the entities the way they are and add custom attributes like ._.currency to them.

That is exactly what I need to do and I think I will go for the attributes approach indeed.

@ines

Btw, I’ currently writing spaCy docs for v2.1 and want to include a section on combining statistical models with rules. Would you mind if I used your example or something very similar for this?

Sure. Let me know if I can be of any help. I have lots of data. My data is a4 pages like of HTML reports but I have chopped them up by parsing the HTML and then added the parsed content to spacy. Is it possible to give the whole thing to spacy and then do some preprocessing to keep track of the origin of the tokens or should I add that logic as a combination of some custom logic and spacy?

@ines

Also, totally forgot we had a spaCy code example for entity relation extraction for MONEY entities: see here – looks pretty relevant to you?

It doesn't get more relevant than that!