k3s: write kubeconfig to KUBECONFIG defined file #978

Sumoa · 2024-02-06T10:01:13Z

Currently when starting a vm + k8s cluster the kubeconfig is always written to "~/.kube/config" which is the default path. By defining the env variable "KUBECONFIG" it is possible to override the default path. This does currently not apply to colima.
As it is possible to configure multiple locations with "KUBECONFIG" I opted to always use the first configured one as the location to save the config.

environment/container/kubernetes/kubeconfig.go

abiosoft · 2024-03-27T07:03:15Z

@Sumoa can you kindly rebase? I fixed the broken CI.

Currently when starting a vm + k8s cluster the kubeconfig is always written to "~/.kube/config" which is the default path. By defining the env variable "KUBECONFIG" it is possible to override the default path. This does currently not apply to colima. As it is possible to configure multiple locations with "KUBECONFIG" I opted to always use the first configured one as the location to save the config.

environment/container/kubernetes/kubeconfig.go

salzig · 2025-03-04T13:04:44Z

i would prefer a command to show me the kubeconfig so i can redirect to a file as i like, in addition to a config option to NOT alter my kubeconfig. I need to work with quite many clusters, which are also changing quite often. So i started to use separate readonly files (~/.kube/config.d/) and load them all as KUBECONFIG (export KUBECONFIG=$(ls -1 ~/.kube/config.d/*.yaml | tr "\n" ":")). Altering ~/.kube/config isn't that useful for me.

Respecting the KUBECONFIG would at least allow me to workaround the problem. Thanks @Sumoa.

hedefalk · 2025-04-27T11:49:35Z

I'd love to get this too! I'm always using $KUBECONFIG using direnv's .envrc files. That's the best way (only?) to have different default kube contexts per folder.

I've tried to forward the k3s arg --write-kubeconfig via --k3s-arg but I think that is all in vain as I guess colima does this another way because I need it written out to the host machine and what I forward to --write-kubeconfig would need to be the mount point within the vm, right? Maybe a a combination of that at mounting the KUBECONFIG into the vm would be an option but I'm just feeling too thick right now :)

Currently I need to manually copy over the sections from ~/.kube/config into my contextual KUBECONFIG. I also get this error message in the end of a start because it can't find the config it just thought it wrote out (since it writes it out to a different place than what kubectl sees):

FATA[0025] error starting kubernetes: error at 'updating config': error running [kubectl config use-context colima-kb], output: "error: no context exists with the name: \"colima-kb\"", err: "exit status 1"

hedefalk · 2025-04-27T12:12:23Z

A PR to the PR for the review feedback.

https://github.com/Sumoa/colima/pull/1/files

Please let me know if I can help in any way to get this in, I'd really love to streamline my colima+k3s workflow with this. What @salzig suggests would also be awesome, to be able to pipe the kubeconfig manually from a colima command would be golden, but this PR is good too.

hedefalk · 2025-04-28T18:28:11Z

While looking at the source code I realized I could get away with a script, so I added this bash script to my dotfiles/stow repo that kinda does all I want:

#!/bin/bash

# Check if COLIMA_PROFILE is provided
if [ -z "$1" ]; then
    echo "Usage: $0 <COLIMA_PROFILE>"
    exit 1
fi

COLIMA_PROFILE=$1
COLIMA_KUBE_CONTEXT=colima-$COLIMA_PROFILE
TEMP_COLIMA_KUBECONFIG=$(mktemp)
TEMP_MERGED_KUBECONFIG=$(mktemp)
colima -p $COLIMA_PROFILE ssh -- kubectl config view --raw  | sed "s/default/$COLIMA_KUBE_CONTEXT/g" > $TEMP_COLIMA_KUBECONFIG
EXISTING_KUBECONFIG=$KUBECONFIG
KUBECONFIG=$KUBECONFIG:$TEMP_COLIMA_KUBECONFIG
kubectl config view --raw > $TEMP_MERGED_KUBECONFIG
cp $TEMP_MERGED_KUBECONFIG $EXISTING_KUBECONFIG
KUBECONFIG=$EXISTING_KUBECONFIG
rm $TEMP_COLIMA_KUBECONFIG $TEMP_MERGED_KUBECONFIG
kubectl config set-context $COLIMA_KUBE_CONTEXT

With that I can keep working wih direnv and have a .envrc file in each "contextual" dev folder doing:

export KUBECONFIG=$(pwd)/kubeconfig

Then I can first setup say a local stage cluster with k3sup (that is already honoring $KUBECONFIG):

k3sup install --skip-install --host pi1 --user pi --merge --context k3s

and then add colima k3s to this:

colima start -p kb
export-colima-profile kb

kubectl config get-contexts

CURRENT   NAME        CLUSTER     AUTHINFO    NAMESPACE
*         colima-kb   colima-kb   colima-kb
          k3s         k3s         k3s

environment/container/kubernetes/kubeconfig.go

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [abiosoft/colima](https://github.com/abiosoft/colima) | patch | `v0.8.1` -> `v0.8.2` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>abiosoft/colima (abiosoft/colima)</summary> ### [`v0.8.2`](https://github.com/abiosoft/colima/releases/tag/v0.8.2) [Compare Source](abiosoft/colima@v0.8.1...v0.8.2) #### Highlights This is an incremental release with bug fixes. #### Fixes - Report errors that occur while reading configuration file. Fixes [#1239](abiosoft/colima#1239). - Use native shasum binary for download assets verification. Fixes [#1163](abiosoft/colima#1163). - Fix download errors when K3s version when empty in config. - Reset formatting after questions prompts in the terminal. Fixes [#1319](abiosoft/colima#1319). #### Other Updates - UDP port forwarding is now supported. Resolves [#1292](abiosoft/colima#1292), [#1300](abiosoft/colima#1300). - Binfmt emulation is configurable via the `--binfmt` flag and `binfmt` config. Defaults to `true` when Rosetta is disabled. - Rename `--cpu` flag to `--cpus` for `colima start`, to align with Lima. `--cpu` still works but deprecated. - Current profile can now be set with `COLIMA_PROFILE` environment variable. Resolves [#1308](abiosoft/colima#1308). ##### Runtime version bumps **NOTE:** container runtime versions can be updated manually by running the `colima update` command. - Docker version updated to `v28.3.3` - Nerdctl version updated to `v2.1.3` - Incus version updated to `v6.14` - K3s version defaults to `v1.33.3+k3s1 ` #### Commits - ci: enable gocritic in golangci by [@PascalBourdier](https://github.com/PascalBourdier) in abiosoft/colima#1223 - faq: update to address potential need for Docker client configuration by [@carljmosca](https://github.com/carljmosca) in abiosoft/colima#1231 - cli: add quotes to k3s-arg example for colima start by [@olamilekan000](https://github.com/olamilekan000) in abiosoft/colima#1230 - chore: update LICENSE, fix copyright license year by [@JasonnnW3000](https://github.com/JasonnnW3000) in abiosoft/colima#1235 - chore: fix copyright license year (Revert [#1235](abiosoft/colima#1235)) by [@windyakin](https://github.com/windyakin) in abiosoft/colima#1237 - chore: log possible error while reading config file by [@olamilekan000](https://github.com/olamilekan000) in abiosoft/colima#1240 - build(deps): bump golang.org/x/term from 0.27.0 to 0.28.0 by [@dependabot](https://github.com/dependabot)\[bot] in abiosoft/colima#1242 - core: use native shasum binary on macOS for download verification by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1245 - core: config initial value for kubernetes version if empty in yaml by [@jason19970210](https://github.com/jason19970210) in abiosoft/colima#1255 - k8s: update latest version for default kubernetes release by [@jason19970210](https://github.com/jason19970210) in abiosoft/colima#1259 - core: respect LIMA\_SSH\_PORT\_FORWARDER env var by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1263 - build(deps): bump golang.org/x/term from 0.28.0 to 0.29.0 by [@dependabot](https://github.com/dependabot)\[bot] in abiosoft/colima#1268 - incus: remove core.https\_address from default config by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1273 - build(deps): bump github.com/spf13/cobra from 1.8.1 to 1.9.1 by [@dependabot](https://github.com/dependabot)\[bot] in abiosoft/colima#1274 - docker: fix cgroupdriver by [@win-t](https://github.com/win-t) in abiosoft/colima#1312 - revert: "docker: fix cgroupdriver" by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1314 - Properly reset formatting at the end of question prompts by [@SukritBhatt](https://github.com/SukritBhatt) in abiosoft/colima#1324 - chore: reorganize dependencies in go.mod by [@alexandear](https://github.com/alexandear) in abiosoft/colima#1325 - github workflows: pin dependencies by [@thypon](https://github.com/thypon) in abiosoft/colima#1326 - vm: support UDP port forwarding by [@stek29](https://github.com/stek29) in abiosoft/colima#1343 - core: update disk images by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1351 - k3s: fix config defaults for Kubernetes values by [@kaisq](https://github.com/kaisq) in abiosoft/colima#1322 - vm: make binfmt emulation configurable by [@winterqt](https://github.com/winterqt) in abiosoft/colima#1315 - docker: replace also localhost with host gateway ip in docker daemon.json by [@fralken](https://github.com/fralken) in abiosoft/colima#1298 - k3s: write kubeconfig to KUBECONFIG defined file by [@Sumoa](https://github.com/Sumoa) in abiosoft/colima#978 - Multiple minor updates by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1352 - build(deps): bump golang.org/x/term from 0.29.0 to 0.30.0 by [@dependabot](https://github.com/dependabot)\[bot] in abiosoft/colima#1281 - chore: expend the path information about current profile being used by [@jason19970210](https://github.com/jason19970210) in abiosoft/colima#1270 - core: add support for COLIMA\_PROFILE environment variable. by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1353 - ci: bump Go version by [@abiosoft](https://github.com/abiosoft) in abiosoft/colima#1354 #### New Contributors - [@PascalBourdier](https://github.com/PascalBourdier) made their first contribution in abiosoft/colima#1223 - [@carljmosca](https://github.com/carljmosca) made their first contribution in abiosoft/colima#1231 - [@olamilekan000](https://github.com/olamilekan000) made their first contribution in abiosoft/colima#1230 - [@JasonnnW3000](https://github.com/JasonnnW3000) made their first contribution in abiosoft/colima#1235 - [@windyakin](https://github.com/windyakin) made their first contribution in abiosoft/colima#1237 - [@jason19970210](https://github.com/jason19970210) made their first contribution in abiosoft/colima#1255 - [@win-t](https://github.com/win-t) made their first contribution in abiosoft/colima#1312 - [@SukritBhatt](https://github.com/SukritBhatt) made their first contribution in abiosoft/colima#1324 - [@alexandear](https://github.com/alexandear) made their first contribution in abiosoft/colima#1325 - [@thypon](https://github.com/thypon) made their first contribution in abiosoft/colima#1326 - [@stek29](https://github.com/stek29) made their first contribution in abiosoft/colima#1343 - [@kaisq](https://github.com/kaisq) made their first contribution in abiosoft/colima#1322 - [@winterqt](https://github.com/winterqt) made their first contribution in abiosoft/colima#1315 - [@fralken](https://github.com/fralken) made their first contribution in abiosoft/colima#1298 - [@Sumoa](https://github.com/Sumoa) made their first contribution in abiosoft/colima#978 **Full Changelog**: abiosoft/colima@v0.8.1...v0.8.2 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever MR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

adam-at-mobomo reviewed Mar 19, 2024

View reviewed changes