Dispatch API returning 500 only for some Payloads #11385
Description
Nomad version
1.1.4
Operating system and Environment details
debian:bullseye-slim docker container on Ubuntu 20.04 base OS
Issue
When dispatching a job using the Dispatch Job api, some Payloads cause a 500 error with the response body rpc error: EOF. The docs indicate that the base64 Payload string in the request body must be <= 16384 but these payloads are all under that limit.
Reproduction steps
- Create a parameterized job, test-job, and attempt to dispatch jobs using the HTTP API
Example of a decoded payload that causes the 500 response:
{
"function": "export_client_bundle",
"call_id": "export_client_bundle-c198f2f992eca4397a1aeb7d5977a495",
"args": {
"options": {
"artifactId": "artifact_2021-10-24.00:38:47-full"
}
},
"nomad_job": "test-job",
"project": "talon",
"data_source": "PROD",
"user": "talon",
"cluster": "talon",
"reuse": false,
"nomadToken": "--redacted--"
}
When encoded, this is the exact payload sent to nomad:
{
"Payload": "eyJmdW5jdGlvbiI6ICJleHBvcnRfY2xpZW50X2J1bmRsZSIsICJjYWxsX2lkIjogImV4cG9ydF9jbGllbnRfYnVuZGxlLWMxOThmMmY5OTJlY2E0Mzk3YTFhZWI3ZDU5NzdhNDk1IiwgImFyZ3MiOiB7Im9wdGlvbnMiOiB7ImFydGlmYWN0SWQiOiAiYXJ0aWZhY3RfMjAyMS0xMC0yNC4wMDozODo0Ny1mdWxsIn19LCAibm9tYWRfam9iIjogInRlc3Qtam9iIiwgInByb2plY3QiOiAidGFsb24iLCAiZGF0YV9zb3VyY2UiOiAiUFJPRCIsICJ1c2VyIjogInRhbG9uIiwgImNsdXN0ZXIiOiAidGFsb24iLCAicmV1c2UiOiBmYWxzZSwgIm5vbWFkVG9rZW4iOiAiLS1yZWRhY3RlZC0tIn0="
}
With a small change to the payload, it will return a 200 and submit the job:
{
"function": "export_client_bundle",
"call_id": "export_client_bundle-ec5f6840a550c773939e47128d66f9ac",
"args": {
"options": {
"artifactId": "111111111111111111111111111111111"
}
},
"nomad_job": "test-job",
"project": "talon",
"data_source": "PROD",
"user": "talon",
"cluster": "talon",
"reuse": false,
"nomadToken": "--redacted--"
}
Raw request body sent to nomad:
{
"Payload": "eyJmdW5jdGlvbiI6ICJleHBvcnRfY2xpZW50X2J1bmRsZSIsICJjYWxsX2lkIjogImV4cG9ydF9jbGllbnRfYnVuZGxlLWVjNWY2ODQwYTU1MGM3NzM5MzllNDcxMjhkNjZmOWFjIiwgImFyZ3MiOiB7Im9wdGlvbnMiOiB7ImFydGlmYWN0SWQiOiAiMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIn19LCAibm9tYWRfam9iIjogInRlc3Qtam9iIiwgInByb2plY3QiOiAidGFsb24iLCAiZGF0YV9zb3VyY2UiOiAiUFJPRCIsICJ1c2VyIjogInRhbG9uIiwgImNsdXN0ZXIiOiAidGFsb24iLCAicmV1c2UiOiBmYWxzZSwgIm5vbWFkVG9rZW4iOiAiLS1yZWRhY3RlZC0tIn0="
}
Expected Result
Receive a 200 response for both payloads and the jobs to be submitted.
Actual Result
Receive a 500 rpc error: EOF error for one of the payloads and no job is submitted.