Skip to content

SDK is broken when installed by git+https #2422

@Electronic-Waste

Description

@Electronic-Waste

What happened?

When I run the following code with kubeflow-katib installed by: pip install git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1

import kubeflow.katib as katib

# Step 1. Create an objective function with push-based metrics collection.
def objective(parameters):
    # Import required packages.
    import time
    import kubeflow.katib as katib
    time.sleep(5)
    # Calculate objective function.
    result = 4 * int(parameters["a"]) - float(parameters["b"]) ** 2
    # Push metrics to Katib DB.
    katib.report_metrics({"result": result})

# Step 2. Create HyperParameter search space.
parameters = {
    "a": katib.search.int(min=10, max=20),
    "b": katib.search.double(min=0.1, max=0.2)
}

# Step 3. Create Katib Experiment with 4 Trials and 2 CPUs per Trial.
# We choose to install the latest changes of Python SDK because `report_metrics` has not been
# supported yet. Thus, the base image must have `git` command to download the package.
katib_client = katib.KatibClient(namespace="kubeflow")
name = "tune-experiment"
katib_client.tune(
    name=name,
    objective=objective,
    parameters=parameters,
    base_image="electronicwaste/push-metrics-collector:v0.0.9", # python:3.11-slim + git
    objective_metric_name="result",
    max_trial_count=4,
    resources_per_trial={"cpu": "2"},
    packages_to_install=["git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1"],
    # packages_to_install=["kubeflow-katib==0.18.0"],
    metrics_collector_config={"kind": "Push"},
)

# Step 4. Wait until Katib Experiment is complete
katib_client.wait_for_experiment_condition(name=name)

# Step 5. Get the best HyperParameters.
print(katib_client.get_optimal_hyperparameters(name))

An error occurred:

Traceback (most recent call last):
  File "/home/ws/katib-example/push.py", line 1, in <module>
    import kubeflow.katib as katib
  File "/home/ws/miniconda3/envs/katib/lib/python3.10/site-packages/kubeflow/katib/__init__.py", line 73, in <module>
    from kubeflow.katib.api.katib_client import KatibClient
  File "/home/ws/miniconda3/envs/katib/lib/python3.10/site-packages/kubeflow/katib/api/katib_client.py", line 27, in <module>
    from kubeflow.katib.types.trainer_resources import TrainerResources
ModuleNotFoundError: No module named 'kubeflow.katib.types'

And I went to dir /home/ws/miniconda3/envs/katib/lib/python3.10/site-packages/kubeflow/katib/, finding that dir types was missing:

(katib)  ws@master1  ~/miniconda3/envs/katib/lib/python3.10/site-packages/kubeflow/katib  ls
api            configuration.py  exceptions.py  katib_api_pb2_grpc.py  models       rest.py
api_client.py  constants         __init__.py    katib_api_pb2.py       __pycache__  utils

What did you expect to happen?

The code should be executed without error when I installed the SDK with:

pip install git+https://github.com/kubeflow/katib.git@a524f33830e02189476efaf6d9045cbd2ce605f0#subdirectory=sdk/python/v1beta1

Environment

Kubernetes version:

$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1

Katib controller version:

$ kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"
docker.io/kubeflowkatib/katib-controller:latest

Katib Python SDK version:

$ pip show kubeflow-katib
Name: kubeflow-katib
Version: 0.17.0
Summary: Katib Python SDK for APIVersion v1beta1
Home-page: https://github.com/kubeflow/katib/tree/master/sdk/python/v1beta1
Author: Kubeflow Authors
Author-email: premnath.vel@gmail.com
License: Apache License Version 2.0
Location: /home/ws/miniconda3/envs/katib/lib/python3.10/site-packages
Requires: certifi, grpcio, kubernetes, protobuf, setuptools, six, urllib3
Required-by:

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions