[jjo] add DIND support to contrib/ #3468

jjo · 2018-10-07T23:35:56Z

add contrib/dind with ansible playbook to create "node" containers,
and setup them to mimic host nodes as much as possible see
contrib/dind/README.md
supported node_distro's (docker images): debian, ubuntu, centos, fedora
kubespray core changes: nodes' /etc/hosts editing via blockinfile
and lineinfile need unsafe_writes: yes because /etc/hosts
is bind-mounted by docker, thus can't be handled atomically via
copy + modify + rename

k8s-ci-robot · 2018-10-07T23:36:00Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: ant31

If they are not already assigned, you can assign the PR to them by writing /assign @ant31 in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ant31 · 2018-10-08T08:16:19Z

That's awesome, Thank you !

How complex would be to do the same but on an existing kubernetes cluster instead of using docker directly?

Create 'node-pod' with the kubernetes API (statefullset?), deploy on kubespray on tthose node-pod via kubernetes job?

jjo · 2018-10-08T11:57:47Z

@ant31 wrote:

How complex would be to do the same but on an existing kubernetes cluster instead of using docker directly?
Create 'node-pod' with the kubernetes API (statefullset?), deploy on kubespray on tthose node-pod via kubernetes job?

KINK via kubespray ?
hm it may get constrained by some CNI×CNI and/or CRI×CRI combos,
but feasible (and freakingCool to try) anyway :) !.
Will craft something, although not in this PR as I'd prefer to keep
this one concise and focused on DIND.

ant31 · 2018-10-08T13:27:21Z

yes, KINK, I like the name :)
Plus, it would benefit the kube-scheduler to balanced the node-containers across the main cluster!

, although not in this PR as I'd prefer to keep
this one concise and focused on DIND.

Yes, sure

contrib/dind/roles/dind-cluster/tasks/main.yaml

mattymo · 2018-10-08T14:46:28Z

contrib/dind/roles/dind-cluster/tasks/main.yaml

+    - rsyslog
+    - ssh
+
+- name: Create ubuntu user


Can this user be configurable and then default to ubuntu?

Ditto above(ish), IMO would also require adding variables to select docker "distro"
image for container nodes -- technically easy to add, practically would need better
care on the testing side, to be able to assert this distro "generalization".

BTW thanks for the suggestion, will give it a try w/Debian to see how it'd look

jjo · 2018-10-08T20:38:49Z

@mattymo: added selectable node_distro (debian, ubuntu), PTAL.

ant31 · 2018-10-09T19:30:30Z

contrib/dind/README.md

+(as set in `group_vars/all/all.yaml`), and docker settings.
+
+~~~
+cat > custom.yaml << EOF


Could you just commit this file and eventually comment everything ?
It's better to reduce to the strict minimum actions required

Added as contrib/dind/kubespray-dind.yaml instead (don't
want to pollute kubespray repo root with such file), also added
fedora support (tested to the extent of kubespray deployment finishing ok), PTAL,

jjo · 2018-10-10T01:08:11Z

FYI latest update has all the changes I wanted in for this PR,
including netcheck 100% passing, tested with default node_distro (debian)
and kubespray-dind.yaml settings (weave in particular).

jjo · 2018-10-10T14:59:26Z

@ant31 passing all tests as run from run-test-distros.sh (ansible-playbooks for
docker nodes and kubespray, api, and netcheck), essentially same
README.md steps run for each distro + some checks:

test-debian.out:PASS: [2018-10-10T16:07:05+00:00] debian: dind-nodes
test-debian.out:PASS: [2018-10-10T16:25:40+00:00] debian: kubespray
test-debian.out:PASS: [2018-10-10T16:25:41+00:00] debian: kube-api
test-debian.out:PASS: [2018-10-10T16:25:44+00:00] debian: netcheck
test-debian.out:real    22m45.417s
test-ubuntu.out:PASS: [2018-10-10T16:32:51+00:00] ubuntu: dind-nodes
test-ubuntu.out:PASS: [2018-10-10T16:50:48+00:00] ubuntu: kubespray
test-ubuntu.out:PASS: [2018-10-10T16:50:48+00:00] ubuntu: kube-api
test-ubuntu.out:PASS: [2018-10-10T16:50:49+00:00] ubuntu: netcheck
test-ubuntu.out:real    22m49.893s
test-centos.out:PASS: [2018-10-10T16:55:54+00:00] centos: dind-nodes
test-centos.out:PASS: [2018-10-10T17:18:12+00:00] centos: kubespray
test-centos.out:PASS: [2018-10-10T17:18:13+00:00] centos: kube-api
test-centos.out:PASS: [2018-10-10T17:18:14+00:00] centos: netcheck
test-centos.out:real    25m16.756s
test-fedora.out:PASS: [2018-10-10T17:27:55+00:00] fedora: dind-nodes
test-fedora.out:PASS: [2018-10-10T17:49:19+00:00] fedora: kubespray
test-fedora.out:PASS: [2018-10-10T17:49:20+00:00] fedora: kube-api
test-fedora.out:PASS: [2018-10-10T17:49:38+00:00] fedora: netcheck
test-fedora.out:real    29m20.304s

- add contrib/dind with ansible playbook to create "node" containers, and setup them to mimic host nodes as much as possible (using Ubuntu images), see contrib/dind/README.md - nodes' /etc/hosts editing via `blockinfile` and `lineinfile` need `unsafe_writes: yes` because /etc/hosts are mounted by docker, and thus can't be handled atomically (modify copy + rename)

- add kubespray-dind.yaml (former custom.yaml at README.md) - rework README.md as per above - use some YAML power to share distros' commonality - add fedora support

- create unique /etc/machine-id in each docker node, used as seed for e.g. weave mac addresses - with above, now netchecker 100% passes WoHooOO! 🎉 🎉 🎉 - updated README.md output from (1.12.1, verified netcheck)

…y facts, also verified netcheck Ok \o/

chadswen · 2018-10-12T01:29:22Z

ci check this

ant31 · 2018-10-15T07:45:42Z

Thanks !

I think it would worth the mention it on the repo README, probably a Contrib section with the list of thing you can find inside (azure, terraform, dind...)

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 7, 2018

k8s-ci-robot requested review from chapsuk and mirwan October 7, 2018 23:35

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 7, 2018

jjo force-pushed the jjo-add-contrib-dind branch 2 times, most recently from af5ee24 to bd6942e Compare October 7, 2018 23:52

mattymo reviewed Oct 8, 2018

View reviewed changes

contrib/dind/roles/dind-cluster/tasks/main.yaml Show resolved Hide resolved

mattymo reviewed Oct 8, 2018

View reviewed changes

jjo force-pushed the jjo-add-contrib-dind branch from 3735a82 to 99c402e Compare October 8, 2018 20:37

jjo force-pushed the jjo-add-contrib-dind branch from e9ae59b to f8be0ed Compare October 8, 2018 21:03

ant31 reviewed Oct 9, 2018

View reviewed changes

jjo force-pushed the jjo-add-contrib-dind branch 2 times, most recently from 819578b to eeff9ca Compare October 10, 2018 11:09

jjo force-pushed the jjo-add-contrib-dind branch from 4d6f29f to 5a73680 Compare October 10, 2018 16:02

jjo added 9 commits October 10, 2018 23:22

dind-host role: set node container hostname on creation

a04c4d0

add "Resulting deployment" section with some CLI outputs

728b509

typo

7276242

selectable node_distro: debian, ubuntu

9ecee7e

some fixes for node_distro: ubuntu

aa4a557

cpu optimization: add early pkill -STOP agetty

3426919

typo

6b4d0a8

add centos dind support ;)

8c7be1c

jjo added 8 commits October 10, 2018 23:22

add kubespray-dind.yaml, support fedora

7db09ea

- add kubespray-dind.yaml (former custom.yaml at README.md) - rework README.md as per above - use some YAML power to share distros' commonality - add fedora support

create unique /etc/machine-id and other updates

635981c

- create unique /etc/machine-id in each docker node, used as seed for e.g. weave mac addresses - with above, now netchecker 100% passes WoHooOO! 🎉 🎉 🎉 - updated README.md output from (1.12.1, verified netcheck)

minor typos

48cff6a

fix centos node creation, needs earlier udevadm removal to avoid flak…

74f4e01

…y facts, also verified netcheck Ok \o/

add Q&D test-distros.sh, back to manual /etc/machine-id hack

e4d9d63

run-test-distros.sh cosmetics and minor fixes

8952261

run-test-distros.sh: $rc fix and minor formatting changes

03b6922

run-test-distros.sh output cosmetics

cf05ea5

jjo force-pushed the jjo-add-contrib-dind branch from 9a4664f to cf05ea5 Compare October 11, 2018 02:23

ant31 merged commit 4077934 into kubernetes-sigs:master Oct 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jjo] add DIND support to contrib/ #3468

[jjo] add DIND support to contrib/ #3468

jjo commented Oct 7, 2018 •

edited

Loading

k8s-ci-robot commented Oct 7, 2018

ant31 commented Oct 8, 2018

jjo commented Oct 8, 2018 •

edited

Loading

ant31 commented Oct 8, 2018

mattymo Oct 8, 2018

jjo Oct 8, 2018

jjo Oct 8, 2018

jjo commented Oct 8, 2018

ant31 Oct 9, 2018

jjo Oct 9, 2018

jjo commented Oct 10, 2018

jjo commented Oct 10, 2018 •

edited

Loading

chadswen commented Oct 12, 2018

ant31 commented Oct 15, 2018

[jjo] add DIND support to contrib/ #3468

[jjo] add DIND support to contrib/ #3468

Conversation

jjo commented Oct 7, 2018 • edited Loading

k8s-ci-robot commented Oct 7, 2018

ant31 commented Oct 8, 2018

jjo commented Oct 8, 2018 • edited Loading

ant31 commented Oct 8, 2018

mattymo Oct 8, 2018

Choose a reason for hiding this comment

jjo Oct 8, 2018

Choose a reason for hiding this comment

jjo Oct 8, 2018

Choose a reason for hiding this comment

jjo commented Oct 8, 2018

ant31 Oct 9, 2018

Choose a reason for hiding this comment

jjo Oct 9, 2018

Choose a reason for hiding this comment

jjo commented Oct 10, 2018

jjo commented Oct 10, 2018 • edited Loading

chadswen commented Oct 12, 2018

ant31 commented Oct 15, 2018

jjo commented Oct 7, 2018 •

edited

Loading

jjo commented Oct 8, 2018 •

edited

Loading

jjo commented Oct 10, 2018 •

edited

Loading