Add a multi-platform proposal #26863

luxas · 2016-06-05T20:11:42Z

This is a proposal for the work I've done with multiarch Kubernetes. I've worked quite a lot on it and now I've written it down so others may easily use and learn from it, while at the same time pointing out issues that we should fix in the future.

I would like to get this in before v1.3.

Feel free to comment on it if it's something that's unclear.

@david-mcmahon @ixdy @mikedanese @Pensu @davidopp @bgrant0607 @dlorenc @eparis @vishh @thockin @dchen1107 @lavalamp

This change is

dchen1107 · 2016-06-10T18:51:07Z

@luxas thanks for your proposal, but I don't think this is required for 1.3 release. We are clearing up our 1.3 merging queue, once we are done with that, we are open for prs like this to main branch. Thanks!

luxas · 2016-06-10T19:23:34Z

@dchen1107 It's okay, it would have been nice to have, but agree that it isn't so important if you're branching soon. I guess I was a bit too slow to send a PR for it.

I would be interested in your opinion on this topic though, but understand if you have too much things to do atm and wait until after the release.

vishh · 2016-06-10T20:44:06Z

docs/proposals/multi-platform.md

+
+## Abstract
+
+Kubernetes is written in Go, and Go is ported to multiple platforms.


nit: Go code is portable across platforms.

luxas · 2016-07-07T14:51:01Z

@vishh @david-mcmahon @thockin @ixdy PTAL

Also, from the docker team @tianon and @estesp might be interested

thockin · 2016-07-07T15:50:38Z

This is an interesting read. It's a mix of background info, things that are already done, and some open issues for the future. I'd really like to see those facets split out, so it is easier to figure out how much work is left on this, what's next to do, and what external deps we have. E.g. when Docker delivers multi-arch images, how much of what we've done will change?

luxas · 2016-07-07T18:19:36Z

Thanks
Yes, the initial goal was just to write down things that are relevant to this topic, and that's quite a mix
If you have an outline in mind already, feel free to post it

The kind of problematic thing with creating a proposal like this is,

it isn't such a well-known thing (cross-compilation in all flavours)
it touches many areas
there are external deps (docker 1.10+ and registry 2.3+ for pulling and pushing manifests)
I started working on it before writing a proposal, because at that time, I didn't know what multiarch k8s would look like

So all comments appreciated.

E.g. when Docker delivers multi-arch images, how much of what we've done will change?

Not much. We will still build binary-arch:version images like now, but the binary:version image should be a manifest list, and it should be used in yaml files and everywhere where one is referencing an image. Basically, binary:version is an alias for binary-${GOARCH}:version then.

I'm investigating if we can take advantage of @estesp's project already for v1.4 for this, given the 1.10+ docker requirement (btw, is kubernetes.gcr.io a v2.3+ registry?)

cc @jfrazelle

jessfraz · 2016-07-07T18:25:30Z

Along with this we should scale the testing infrastructure across platforms, we did this for docker, and maybe we can reuse some of the logic for the "go build dashboard". Otherwise stuff will constantly be broken on the other arches

I don't see a problem in having to change things when docker supports multi-arch images. It should be a streamlined change, and +1 to @estesp tool, he is definitely one of the major people working on that effort

jessfraz · 2016-07-07T19:36:31Z

docs/proposals/multi-platform.md

+
+This is especially useful if we want to include the binary in a container. 
+If the binary is statically compiled, we may use `busybox` or even `scratch` as the base image.
+This should be the preferred way of compiling binaries that strictly require C code to be a part of it.


fwiw you can statically cross compile cgo from linux to all other arches, it's not something I am proud of knowing how to do though

Thanks for pointing at a doc flaw.
That's exactly what my the intention is with this section, to describe how to cross-compile statically with CGO code for another architecture.

However, I see that here it says compilation in the header, when it should be cross-compilation, and likewise in other places. Will fix.

luxas · 2016-08-18T21:20:04Z

ping @vishh @david-mcmahon @jfrazelle PTAL

m1093782566 · 2016-08-19T16:33:10Z

@luxas +1 from me

vishh · 2016-08-22T20:18:34Z

docs/proposals/multi-platform.md

+### Support all platforms in a "run everywhere" deployment
+
+The easiest way of running Kubernetes on another architecture is probably by using the docker or the docker-multinode deployment. Of course, you may choose whatever deployment you want, the binaries are easily downloadable from the URL above. But [docker-multinode](https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode) should be a "just works" multi-platform solution with docker as the only real dependency.
+Note: That doesn't mean _everything_ "just works" with the docker-multinode setup, but basic functionality is there.


I'm not confident that docker multinode is an acceptable production solution. It might be worth mentioning that. It is mainly meant for "kick the tires" experience.

No it isn't a production env deployment; and that's why I wrote the basic functionality is there (it nearly passes all tests; I think there were 5 fails of all e2e tests)

I can clarify that's it's meant for a "kick the tires"-cluster even more if you want me to

luxas · 2016-08-25T20:03:25Z

@vishh Please LGTM this now. :)
Personally, it would mean much to get this into v1.4, and I've waited a very long time for it already.
Addressed your comments.

luxas · 2016-08-25T20:48:54Z

@k8s-bot test this please github issue: #IGNORE

vishh · 2016-08-26T00:07:21Z

docs/proposals/multi-platform.md

+ - Relying on platform-specific syscalls (e.g. `syscall.Dup2`)
+
+If someone tries to add a dependency that doesn't satisfy these requirements; he/she should first fix the problem in the repository that is about to be added.
+When the dependency satisfies the Kubernetes requirements for compiling on all platforms, the PR can be merged.


The CI should catch incompatible vendor dependencies.

vishh · 2016-08-26T00:10:54Z

@luxas I'd gate this proposal on requiring testing for additional arch before claiming support for them. I don't like multi-arch blocking progress on amd64 if CI doesn't exist for other architectures. If you can include CI as a pre-req for multi-arch, this proposal lgtm.
Feel free to self-lgtm once you add the CI requirement.

luxas · 2016-08-30T11:41:37Z

@vishh Thanks for the review!
I added a Support levels section about how/when platforms are considered "stable"
Also clarified even more that CI presubmit have to exist.

Self-applying

k8s-bot · 2016-08-30T12:22:41Z

GCE e2e build/test passed for commit b033b25.

k8s-github-robot · 2016-08-30T12:40:39Z

Automatic merge from submit-queue

m1093782566 · 2016-08-30T13:07:15Z

👍

@luxas

…tname Automatic merge from submit-queue Cross-build test/images/serve_hostname  **What this PR does / why we need it**: **cross build** `test/images/serve_hostname` https://github.com/kubernetes/kubernetes/tree/master/test/images/serve_hostname This PR is an effort to achieve multiarch Kubernetes(#26863) **Which issue this PR fixes** : fixes #31238 @luxas @spxtr

@david-mcmahon

Automatic merge from submit-queue Add a multi-platform proposal This is a proposal for the work I've done with multiarch Kubernetes. I've worked quite a lot on it and now I've written it down so others may easily use and learn from it, while at the same time pointing out issues that we should fix in the future. I would like to get this in before `v1.3`. Feel free to comment on it if it's something that's unclear. @david-mcmahon @ixdy @mikedanese @Pensu @davidopp @bgrant0607 @dlorenc @eparis @vishh @thockin @dchen1107 @lavalamp

googlebot added the cla: yes label Jun 5, 2016

k8s-github-robot assigned smarterclayton Jun 5, 2016

k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Jun 5, 2016

luxas added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels Jun 5, 2016

luxas added this to the v1.3 milestone Jun 5, 2016

luxas mentioned this pull request Jun 6, 2016

Standardize 'pause' container #15140

Closed

luxas added the e2e-not-required label Jun 6, 2016

luxas mentioned this pull request Jun 6, 2016

Add a tiny-glibc image kubernetes-retired/contrib#1146

Merged

luxas assigned vishh and david-mcmahon Jun 7, 2016

luxas mentioned this pull request Jun 8, 2016

[Proposal] Make Kubernetes run on ARM devices #17981

Closed

17 tasks

dchen1107 removed this from the v1.3 milestone Jun 10, 2016

vishh reviewed Jun 10, 2016
View reviewed changes

bgrant0607 unassigned smarterclayton Jun 19, 2016

luxas mentioned this pull request Jun 29, 2016

Add kvm driver support kubernetes/minikube#242

Merged

luxas force-pushed the multi_platform branch from fc9f0ce to 79639ef Compare July 6, 2016 12:53

jessfraz reviewed Jul 7, 2016
View reviewed changes

luxas mentioned this pull request Aug 17, 2016

Support Docker manifest lists appc/docker2aci#193

Open

This was referenced Aug 19, 2016

Makefile: Make the ARCH part of the tag name not the image name and add pthread to busybox flannel-io/flannel#489

Merged

[e2e test] Fix e2e test pause image hard code #30964

Merged

luxas mentioned this pull request Aug 22, 2016

Make all Kubernetes binaries statically linked #26028

Closed

vishh reviewed Aug 22, 2016
View reviewed changes

This was referenced Aug 23, 2016

[test/images] cross-build test/images/serve_hostname #31238

Closed

Cross-build test/images/serve_hostname #31241

Merged

luxas added this to the v1.4 milestone Aug 25, 2016

luxas force-pushed the multi_platform branch 2 times, most recently from d7dd7cc to c60490d Compare August 25, 2016 20:01

luxas mentioned this pull request Aug 25, 2016

Getting a Power CI for kubernetes #25730

Closed

vishh reviewed Aug 26, 2016
View reviewed changes

Add a multiarch proposal

b033b25

luxas force-pushed the multi_platform branch from c60490d to b033b25 Compare August 30, 2016 11:40

luxas added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 30, 2016

k8s-github-robot merged commit be848f2 into kubernetes:master Aug 30, 2016

luxas mentioned this pull request Sep 2, 2016

Add heapster to hyperkube #31960

Closed


		## Abstract

		Kubernetes is written in Go, and Go is ported to multiple platforms.

Add a multi-platform proposal #26863

Add a multi-platform proposal #26863

Uh oh!

Conversation

luxas commented Jun 5, 2016 • edited by k8s-oncall Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dchen1107 commented Jun 10, 2016

Uh oh!

luxas commented Jun 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vishh Jun 10, 2016

Choose a reason for hiding this comment

Uh oh!

luxas commented Jul 7, 2016

Uh oh!

thockin commented Jul 7, 2016

Uh oh!

luxas commented Jul 7, 2016

Uh oh!

jessfraz commented Jul 7, 2016

Uh oh!

jessfraz Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

luxas Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

luxas commented Aug 18, 2016

Uh oh!

m1093782566 commented Aug 19, 2016

Uh oh!

vishh Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

luxas Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

luxas commented Aug 25, 2016

Uh oh!

luxas commented Aug 25, 2016

Uh oh!

vishh Aug 26, 2016

Choose a reason for hiding this comment

Uh oh!

vishh commented Aug 26, 2016

Uh oh!

luxas commented Aug 30, 2016

Uh oh!

k8s-bot commented Aug 30, 2016

Uh oh!

k8s-github-robot commented Aug 30, 2016

Uh oh!

m1093782566 commented Aug 30, 2016

Uh oh!

Uh oh!

luxas commented Jun 5, 2016 •

edited by k8s-oncall

Loading

luxas commented Jun 10, 2016 •

edited

Loading