what exactly "--long-text" does in textcat.teach?

[jdixosnd]

I was going through the documentation for textcat and found one parameter "--long-text" in textcat.teach. I wanted to know more about its usage but sadly I couldn't find any info. It would be helpful if one share more details around this parameter.

P.S. I went through the recipe of textcat.teach on github as well. (however, the last commit was a year old and also i couldn't find anything related to --long-text)

[ines (Ines Montani)]

The --long-text flag was an early experiment and the option on the recipe has been deprecated for a while and mostly still exists for backwards compatibility. You can read more about it here:

What is the meaning of the long_text flag?

In full hindsight, I think it might have been better to handle this in the pre-processing, instead of having it as an argument in the model. The long_text argument applied sentence splitting, and then asks questions about each sentence. For evaluation, it runs the model over each sentence in the document, and sums the scores. You can achieve the same thing easily enough by modifying the data or the recipe, which I would probably recommend.

What is the meaning of the long_text flag?

We ended up more or less deprecating that mode because it didn't end up being very useful and a bit too "magical" (too much going on under the hood and not very transparent). The flag still exists because we didn't want to break backwards compatibility, but it's not necessarily a workflow we'd recommend. Instead, you can just split the sentences yourself and decide how you want to interpret the per-sentence scores, which makes the whole process more transparent as well