Bug summary

When a task in a flow fails, we attempt to bypass it by changing its state from "failed" to "completed." However, upon re-triggering the flow, the task that we marked as "completed" executes again instead of being skipped. Conversely, when we intentionally change a completed task to "failed" and retry the flow, it also fails to work as expected. We anticipated that the failed task would be skipped, but it gets executed again in Prefect 3.

In contrast, this functionality worked perfectly in Prefect 2.

I suspect this issue may be related to recent changes in the retry mechanism. It appears that for each retry within the flow, a new task instance is created instead of executing the existing task, as the task ID is different each time. This change in the task creation logic seems to be influenced by the dynamic_key, which affects how tasks are identified and executed in Prefect 3.

Version info (prefect version output)

Version:             3.0.5
API version:         0.8.4
Python version:      3.12.5
Git commit:          1d1b92a7
Built:               Tue, Oct 8, 2024 5:55 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud
Pydantic version:    2.8.2
Integrations:
  prefect-kubernetes: 0.5.0
  prefect-aws:       0.5.0
  prefect-gcp:       0.6.0
  prefect-github:    0.3.0

Additional context

No response