In full hindsight, I think it might have been better to handle this in the pre-processing, instead of having it as an argument in the model.
The long_text argument applied sentence splitting, and then asks questions about each sentence. For evaluation, it runs the model over each sentence in the document, and sums the scores.
You can achieve the same thing easily enough by modifying the data or the recipe, which I would probably recommend.