SpanCat Training Error on Custom Preprocessed Dataset

[ryanwesslen]

hi @daffahilmyf!

Thanks for your question and welcome to the Prodigy community

Thanks as well for providing your examples. This helped a lot!

I noticed that in your updated file it looks like you stemmed and removed stop words. Is this correct?

• Original (worked): "text": "In order to fulfil the requirements of some railways, it should be possible to provide an alternative means of link assurance indication."
• New (didn't work): "text": "in order fulfil requir some railway possibl provid altern mean link assur indic"

Can you explain more on why the pre-processing is needed?

There's no need to do pre-processing like this and we generally recommend against doing it. For example, this is a good post on the background on stop words:

Do stop words affect text classification in Prodigy?

Hey, In general I would advise against stop-word removal, and I'd definitely advise against storing data that's been transformed (especially a lossy transformation like stop-word removal). The history of stop-word removal goes like this. Early work on text classification was heavily influenced by information retrieval (search engines, basically). In early search engines, you started out by making a term-document matrix, a table where you could look up a term and get all the documents mentionin…