Using a text classifier instead of NER

From the example you posted above, it definitely looks like text classification is a good approach for the task. The text classifier will be able to take all words in the text into account, so if certain keywords commonly occur in summer texts, it's something that your model will be able to learn and generalise from.

I think what Matt is referring to in his comment is more related to using a text classifier to decide on fine-grained categories for spans mentioned in the text. This is easier if you're expecting the text to be about one concept only, which you can then apply the text category to.

For instance, let's say you have a sentence like "The Yeezy Foam RNNR is the perfect shoe for sultry city nights". What you're looking to extract here is the product (Yeezy Foam RNNR), the type (shoe) and the season (summer). You could now go and create training data that labels "Yeezy Foam RNNR" as SUMMER_SHOE – but that'd likely be a very inefficient approach. There's nothing about the product name that makes it inherently a summer shoe, and you might come across it in different contexts that have nothing to do with the season at all. The model is likely gonig to struggle to learn the distinction on top of the span boundaries. So a better approach would be to predict the season over the whole text (just like what you're doing) and then use an entity recognizer or just a simple database or product catalogue lookup to detect the product.

This works well if you're dealing with product reviews etc. and you know that the text is about one span and/or product. Where it gets more difficult is if you have a sentence that talks about multiple products with different attributes (e.g. a summer shoe vs. a winter shoe). Here, you couldn't just predict a label over the whole text and apply that to all products.

1 Like