"Response payload is not completed" in Prefect Log #16948
Description
Bug summary
Hello,
This issue might be related to #16210.
When running long flows, the error below comes up. Exactly 30 minutes passed when the issue is encountered.
Prefect server was upgraded to 3.1.15 version along with the worker to prefecthq/prefect:3.1.15-python3.11-kubernetes and flow pod running as prefecthq/prefect:3.1.15-python3.11.
This error doesn't stop execution of the flow. The flow managed to be completed successfully.
The worker's watch for job 'solid-okapi-j7qvb' has exited early. Check the logs for more information. The job is still running, but the worker will not wait for it to complete.
12:26:10 PM
prefect.flow_runs.worker
Error during task execution: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>
NoneType: None
12:26:10 PM
prefect.flow_runs.worker
Error occurred while streaming logs - Job will continue to run but logs will no longer be streamed to stdout.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/aiohttp/client_proto.py", line 92, in connection_lost
uncompleted = self._parser.feed_eof()
^^^^^^^^^^^^^^^^^^^^^^^
File "aiohttp/_http_parser.pyx", line 508, in aiohttp._http_parser.HttpParser.feed_eof
aiohttp.http_exceptions.TransferEncodingError: 400, message:
Not enough data for satisfy transfer length header.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 881, in _stream_job_logs
async for line in logs.content:
File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 52, in __anext__
rv = await self.read_func()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 352, in readline
return await self.readuntil()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 386, in readuntil
await self._wait("readuntil")
File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 347, in _wait
await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>
Version info
Version: 3.1.15
API version: 0.8.4
Python version: 3.10.12
Git commit: 3ac3d548
Built: Thu, Jan 30, 2025 11:31 AM
OS/Arch: linux/x86_64
Profile: local
Server type: server
Pydantic version: 2.9.2
Integrations:
prefect-kubernetes: 0.5.2
prefect-dbt: 0.6.3
prefect-docker: 0.6.2
prefect-github: 0.3.0
prefect-shell: 0.3.0