Pydantic 2.0 broke compatibility with Ray (see #37019), so we currently have pydantic<2.0 pinned in our dependencies.

There is a fix for Ray core that will get cherry-picked to Ray 2.6: #37055, but there may be an issue between FastAPI and pydantic that would prevent us from removing the pin.

Someone from the Serve team needs to investigate if Serve/FastAPI will work with pydantic>=2.0 after the above fix is merged.

The first step is likely running the CI tests with pydantic pinned to the most recent version.

The outcome of this is a decision between:

1. Removing the pin for Ray 2.6 because we're confident that pydantic>=2.0 works with both core and serve.
2. Leaving the pin in for Ray 2.6 because fastapi/serve do not yet work with pydantic>=2.0.

(2) is the more conservative option which we should go with by default if we are unsure.