Unable to supply an access token in any other format besides a Secret block the GitRepository object. #11257
Description
First check
- I added a descriptive title to this issue.
- I used the GitHub search to find a similar issue and didn't find it.
- I searched the Prefect documentation for this issue.
- I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
Many users are prevented from using Prefect as a secret storage solution based on their internal secret storage protocols.
If the error message is to be believed, when configuring remote github storage for flow.from_source(), a user is forced into using a Secret block for this.
ValueError: Please save your access token as a Secret block before converting this storage object to a pull step.
GitCredentials shows that you can pass a str, but it looks like it isn't any case to handle strings in to_pull_step.
Reproduction
from child_flows import child_flow_d
from prefect.runner.storage import GitRepository, GitCredentials
import os
GITHUB_ACCESS_TOKEN = os.getenv("GITHUB_ACCESS_TOKEN")
if __name__ == "__main__":
child_flow_d.from_source(
source=GitRepository(
url="https://github.com/taylor-curran/private-test.git",
credentials=GitCredentials(
username="taylor-curran", access_token=GITHUB_ACCESS_TOKEN
),
),
entrypoint="child_flows.py:child_flow_d",
).deploy(
name="my-dep",
work_pool_name="my-k8s-pool",
image="docker.io/taycurran/child-d:demo",
build=False,
)Error
Traceback (most recent call last):
File "/Users/taylorcurran/Documents/november/dot-deploy-examples/raw_string.py", line 16, in <module>
).deploy(
^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
return call()
^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 398, in __call__
return self.result()
^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 284, in result
return self.future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 355, in _run_async
result = await coro
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/flows.py", line 969, in deploy
deployment_ids = await deploy(
^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/deployments/runner.py", line 830, in deploy
await deployment.apply(image=image_ref, work_pool_name=work_pool_name)
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/deployments/runner.py", line 267, in apply
[self.storage.to_pull_step()] if self.storage else []
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/dot-deploy/lib/python3.11/site-packages/prefect/runner/storage.py", line 277, in to_pull_step
raise ValueError(
ValueError: Please save your access token as a Secret block before converting this storage object to a pull step.Versions
Version: v2.14.6
API version: 0.8.4
Python version: 3.11.5
Git commit: b99932a4
Built: Wed, Nov 22, 2023 3:19 PM
OS/Arch: darwin/arm64
Profile: prefect-patterns
Server type: cloud
Additional context
For context, the user would like to do something like the following:
from <file_name> import <flow_name>
from prefect.runner.storage import GitRepository
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential
keyVaultName = '<kv-name>'
KVUri = f"https://{keyVaultName}.vault.azure.net"
credential = DefaultAzureCredential()
client = SecretClient(vault_url=KVUri, credential=credential)
secretName = '<secret-name>'
if __name__ == "__main__":
<flow_name>.from_source(
source = GitRepository(
url = "https://github.com/<org-name>/<repo-name>.git",
branch = "<branch-name>",
credentials = {"access_token": client.get_secret(secretName).value}),
entrypoint = "<path>/<file_name>.py:<flow_name>").deploy(
name = '<deployment-name>',
work_pool_name = '<work-pool-name>')