Hi Pat :-)

Sphinx 0.9.9 introduced the setting max_packet_size which defaults to 8M, limiting the length of a query that can be run in Sphinx ( http://sphinxsearch.com/docs/current/conf-max-packet-size.html ). ThinkingSphinx has a hardcoded value of MAXIMUM_STATEMENT_LENGTH = (2 ** 23) - 5 matching Sphinx' default limit. However, the value is hardcoded in lib/thinking_sphinx.rb:20 and cannot be changed.

Because I have huge amounts of text in a single document (over 8 MB of text), reducing the batch size to 1 would not help, thus I experimented by adding a custom max_packet_size to config/thinking_sphinx.yml and overwriting MAXIMUM_STATEMENT_LENGTH in a monkey patch. Result: Actions like callbacks, ts:rebuild, ts:index, which had crashed due to excessive QL length before, appear to work fine now.

For this reason, I think that MAXIMUM_STATEMENT_LENGTH should be customizable in Thinking Sphinx. Would you agree or are the problems I oversee?

Best,
Kalsan