Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🌱 hack: implement generator for prowjobs #9937

Merged
merged 10 commits into from
Jan 18, 2024

Conversation

chrischdi
Copy link
Member

@chrischdi chrischdi commented Dec 29, 2023

What this PR does / why we need it:

Implements prowjob-gen a lightweight tool based on go-templates and a configuration file to template prowjob configuration across multiple branches.

Example usage (./../k8s.io/test-infra is the relativ path for me locally to the root of the test-infra directory):

TEST_INFRA_DIR=../../k8s.io/test-infra make generate-test-infra-prowjobs

Example output: kubernetes/test-infra#31581

One change to our configuration would be currently:

Our jobs upgrading to latest kubernetes (currently v1.30) had latest in their job names. With the current implementation those will use 1-30 instead.

Example:

  • Before: periodic-cluster-api-e2e-workload-upgrade-1-29-latest-main
  • After: periodic-cluster-api-e2e-workload-upgrade-1-29-1-30-main

Requires to merge the following to make mdlint happy:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #9257

/area ci

@k8s-ci-robot k8s-ci-robot added area/ci Issues or PRs related to ci cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 29, 2023
@chrischdi chrischdi force-pushed the pr-prowjob-generator branch 2 times, most recently from 33d8ee5 to 2401856 Compare January 9, 2024 13:03
@chrischdi
Copy link
Member Author

/retest

@chrischdi
Copy link
Member Author

 failed to get component source YAML from URL: failed to get https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.6.0/core-components.yaml: got status code 500

/retest

@chrischdi
Copy link
Member Author

/retitle 🌱 [wip] hack: implement generator for prowjobs

@k8s-ci-robot k8s-ci-robot changed the title 🌱 [WIP] hack: implement generator for prowjobs 🌱 [wip] hack: implement generator for prowjobs Jan 9, 2024
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 9, 2024
@chrischdi chrischdi force-pushed the pr-prowjob-generator branch 3 times, most recently from cb7b7d7 to 8e7d2e5 Compare January 10, 2024 18:16
@chrischdi chrischdi changed the title 🌱 [wip] hack: implement generator for prowjobs 🌱 hack: implement generator for prowjobs Jan 10, 2024
@cprivitere
Copy link
Member

cprivitere commented Jan 10, 2024

Any concerns that we're bringing in a library that hasn't been updated since November of 2022? (github.com/Masterminds/sprig)

@chrischdi
Copy link
Member Author

Any concerns that we're bringing in a library that hasn't been updated since November of 2022? (github.com/Masterminds/sprig)

Great point, I did not think about that.

We already use sprig in clusterclass / in our core go.mod file which is why I think this could be considered okay.

@cprivitere
Copy link
Member

cprivitere commented Jan 10, 2024

Any concerns that we're bringing in a library that hasn't been updated since November of 2022? (github.com/Masterminds/sprig)

Great point, I did not think about that.

We already use sprig in clusterclass / in our core go.mod file which is why I think this could be considered okay.

If the group already agrees it's ok, then I guess that's cool.

Was just a bit concerned given their go.mod is based on 1.13 and they're bringing in old versions of semver.

@chrischdi
Copy link
Member Author

chrischdi commented Jan 10, 2024

To add: happy to adjust / remove it if we consider it for it should not being used, definetly not a must. Some of the sprig functions may just make it more easy to write the templates.

@chrischdi
Copy link
Member Author

Note: I removed sprig and replaced it by standard library functions.

One exception: I added a last function which returns the last element of an array or slice.

Also removed the other custom implemented functions.

prowjob-generator allows to generate the periodic and presubmit configuration files in test-infra
from a configuration file which simplifies and automates introducing and chaning tests for branches.
Copy link
Member

@sbueringer sbueringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First round of review. Will continue later

hack/tools/prowjob-gen/config.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/config.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/generator.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/generator.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/generator.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/generator.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/generator.go Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/generator.go Show resolved Hide resolved
hack/tools/prowjob-gen/main_test.go Outdated Show resolved Hide resolved
Copy link
Member

@fabriziopandini fabriziopandini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall lgtm, great to see this making progress!

hack/boilerplate/boilerplate.py Outdated Show resolved Hide resolved
hack/tools/prowjob-gen/config.go Outdated Show resolved Hide resolved
@chrischdi
Copy link
Member Author

@sbueringer PTAL

pending squash

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 17, 2024
@sbueringer
Copy link
Member

sbueringer commented Jan 17, 2024

Thx! Really nice

/lgtm

/assign @fabriziopandini

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 17, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 7add3fe4b7957b2e14925067896068fcde5d40ed

@sbueringer
Copy link
Member

Thx!

/lgtm
/approve

/hold cancel

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 17, 2024
@sbueringer
Copy link
Member

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 17, 2024
@sbueringer
Copy link
Member

sbueringer commented Jan 17, 2024

Sorry wrong tab wanted to approve the test-infra PR :D

/approve cancel

@k8s-ci-robot k8s-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 17, 2024
@sbueringer
Copy link
Member

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 17, 2024
Copy link
Member

@fabriziopandini fabriziopandini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!
/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 18, 2024
@k8s-ci-robot k8s-ci-robot merged commit 26b8ebf into kubernetes-sigs:main Jan 18, 2024
23 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.7 milestone Jan 18, 2024
@sbueringer sbueringer mentioned this pull request Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ci Issues or PRs related to ci cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generate ProwJob YAMLs and automate changes to test jobs
5 participants