KillingInvocation seems to not properly work with Killed invocation status #2876
Description
In this test run https://github.com/restatedev/restate/actions/runs/13764395844/job/38488506837?pr=2873#step:11:434, the test KillInvocation is stuck. From the looks of the sys_invocation table, we are not properly killing all recursive calls. It seems as if one child is still running while the direct parent and the startCallTree are completed. Since the final verification call runs on the same VO as the recursive call, the test is stuck.
{"rows":[{"id":"inv_12dw5UBHriBL6zDR4IfF2Ev9TdQHvK6rsZ","target":"KillTestSingleton/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/recursiveCall","target_service_name":"KillTestSingleton","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"recursiveCall","target_service_ty":"virtual_object","invoked_by":"service","invoked_by_service_name":"KillTestSingleton","invoked_by_id":"inv_12dw5UBHriBL2DJ0pgfNmE46yAptW2rrMJ","invoked_by_target":"KillTestSingleton/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/recursiveCall","journal_size":1,"created_at":"2025-03-10T12:36:10.914Z","modified_at":"2025-03-10T12:36:11.211Z","inboxed_at":"2025-03-10T12:36:10.914Z","running_at":"2025-03-10T12:36:11.211Z","retry_count":1,"last_start_at":"2025-03-10T12:36:11.212Z","last_attempt_deployment_id":"dp_10tsC1MOgC0nVFsmaix9uHn","last_attempt_server":"restate-sdk-java/2.0.0-SNAPSHOT_624fd2fa","status":"running"},
{"id":"inv_12dw5UBHriBL4FkVUseKdvH5CqopfY5eTS","target":"KillTestRunner/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/startCallTree","target_service_name":"KillTestRunner","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"startCallTree","target_service_ty":"virtual_object","idempotency_key":"449df2b9-ceb8-4ebd-898b-cf37de34e464","invoked_by":"ingress","created_at":"2025-03-10T12:36:06.998Z","modified_at":"2025-03-10T12:36:11.208Z","running_at":"2025-03-10T12:36:06.999Z","completed_at":"2025-03-10T12:36:11.208Z","status":"completed","completion_result":"failure","completion_failure":"[409 Aborted] killed"},
{"id":"inv_12dw5UBHriBL5asgtMrkbz2gWGD29ThwA8","target":"AwakeableHolder/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/unlock","target_service_name":"AwakeableHolder","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"unlock","target_service_ty":"virtual_object","idempotency_key":"2d5aa487-076c-4925-b092-836cdf5e0ff2","invoked_by":"ingress","created_at":"2025-03-10T12:36:10.837Z","modified_at":"2025-03-10T12:36:10.900Z","running_at":"2025-03-10T12:36:10.837Z","completed_at":"2025-03-10T12:36:10.900Z","status":"completed","completion_result":"success"},
{"id":"inv_12dw5UBHriBL76f2hy65GbO0KVfkDU6Sdj","target":"KillTestSingleton/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/isUnlocked","target_service_name":"KillTestSingleton","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"isUnlocked","target_service_ty":"virtual_object","invoked_by":"ingress","created_at":"2025-03-10T12:36:11.422Z","modified_at":"2025-03-10T12:36:11.422Z","inboxed_at":"2025-03-10T12:36:11.422Z","status":"pending"}]}
Metadata
Metadata
Assignees
Labels
No labels