diff --git a/.changelog/13023.txt b/.changelog/13023.txt new file mode 100644 index 000000000000..cadf7bb93849 --- /dev/null +++ b/.changelog/13023.txt @@ -0,0 +1,3 @@ +```release-note:improvement +ui: the topology view now properly displays services with mixed connect and non-connect instances. +``` diff --git a/.changelog/17160.txt b/.changelog/17160.txt new file mode 100644 index 000000000000..666a6e8f252c --- /dev/null +++ b/.changelog/17160.txt @@ -0,0 +1,3 @@ +```release-note:bug +Fix a bug that wrongly trims domains when there is an overlap with DC name. +``` diff --git a/.changelog/17565.txt b/.changelog/17565.txt new file mode 100644 index 000000000000..f7cf46c38954 --- /dev/null +++ b/.changelog/17565.txt @@ -0,0 +1,3 @@ +```release-note:feature +reloadable config: Made enable_debug config reloadable and enable pprof command to work when config toggles to true +``` \ No newline at end of file diff --git a/.changelog/17582.txt b/.changelog/17582.txt new file mode 100644 index 000000000000..122b9df98116 --- /dev/null +++ b/.changelog/17582.txt @@ -0,0 +1,3 @@ +```release-note:feature +cli: `consul operator raft list-peers` command shows the number of commits each follower is trailing the leader by to aid in troubleshooting. +``` diff --git a/.changelog/17596.txt b/.changelog/17596.txt new file mode 100644 index 000000000000..1058df1ea3ab --- /dev/null +++ b/.changelog/17596.txt @@ -0,0 +1,3 @@ +```release-note:improvement + debug: change default setting of consul debug command. now default duration is 5ms and default log level is 'TRACE' + ``` \ No newline at end of file diff --git a/.changelog/17739.txt b/.changelog/17739.txt new file mode 100644 index 000000000000..14bbceeaa084 --- /dev/null +++ b/.changelog/17739.txt @@ -0,0 +1,3 @@ +```release-note:bug +http: fixed API endpoint `PUT /acl/token/:AccessorID` (update token), no longer requires `AccessorID` in the request body. Web UI can now update tokens. + ``` diff --git a/.changelog/17754.txt b/.changelog/17754.txt new file mode 100644 index 000000000000..56ab20dc213e --- /dev/null +++ b/.changelog/17754.txt @@ -0,0 +1,3 @@ +```release-note:feature +ui: consul version is displayed in nodes list with filtering and sorting based on versions +``` diff --git a/.changelog/17755.txt b/.changelog/17755.txt new file mode 100644 index 000000000000..7edf7b26e159 --- /dev/null +++ b/.changelog/17755.txt @@ -0,0 +1,3 @@ +```release-note:improvement +mesh: Stop jwt providers referenced by intentions from being deleted. +``` \ No newline at end of file diff --git a/.changelog/17757.txt b/.changelog/17757.txt new file mode 100644 index 000000000000..e207438cf843 --- /dev/null +++ b/.changelog/17757.txt @@ -0,0 +1,3 @@ +```release-note:improvement +connect: Improve transparent proxy support for virtual services and failovers. +``` diff --git a/.changelog/17759.txt b/.changelog/17759.txt new file mode 100644 index 000000000000..0836608ae1f2 --- /dev/null +++ b/.changelog/17759.txt @@ -0,0 +1,3 @@ +```release-note:improvement +extensions: Improve validation and error feedback for `property-override` builtin Envoy extension +``` diff --git a/.changelog/17775.txt b/.changelog/17775.txt new file mode 100644 index 000000000000..8060cfa128ce --- /dev/null +++ b/.changelog/17775.txt @@ -0,0 +1,3 @@ +```release-note:bug +connect: Fix issue where changes to service exports were not reflected in proxies. +``` diff --git a/.changelog/17780.txt b/.changelog/17780.txt new file mode 100644 index 000000000000..b90925a8b9fd --- /dev/null +++ b/.changelog/17780.txt @@ -0,0 +1,3 @@ +```release-note:feature +cli: `consul watch` command uses `-filter` expression to filter response from checks, services, nodes, and service. +``` diff --git a/.changelog/17846.txt b/.changelog/17846.txt new file mode 100644 index 000000000000..bd5a052f851f --- /dev/null +++ b/.changelog/17846.txt @@ -0,0 +1,3 @@ +```release-note:bug +connect/ca: Fixes a bug preventing CA configuration updates in secondary datacenters +``` diff --git a/.changelog/17885.txt b/.changelog/17885.txt new file mode 100644 index 000000000000..2cd690488d92 --- /dev/null +++ b/.changelog/17885.txt @@ -0,0 +1,2 @@ +```release-note:bug +ca: Fixed a bug where the Vault provider was not passing the configured role param for AWS auth diff --git a/.changelog/17888.txt b/.changelog/17888.txt new file mode 100644 index 000000000000..f50fcae09b07 --- /dev/null +++ b/.changelog/17888.txt @@ -0,0 +1,3 @@ +```release-note:improvement +connect: Add capture group labels from Envoy cluster FQDNs to Envoy exported metric labels +``` \ No newline at end of file diff --git a/.changelog/17894.txt b/.changelog/17894.txt new file mode 100644 index 000000000000..5749f995f71a --- /dev/null +++ b/.changelog/17894.txt @@ -0,0 +1,3 @@ +```release-note:bug +connect: Fix incorrect protocol config merging for transparent proxy implicit upstreams. +``` diff --git a/.changelog/17911.txt b/.changelog/17911.txt new file mode 100644 index 000000000000..a17cd6d2bdf2 --- /dev/null +++ b/.changelog/17911.txt @@ -0,0 +1,4 @@ +```release-note:bug +gateway: Fixes a bug where envoy would silently reject RSA keys that are smaller than 2048 bits, +we now reject those earlier in the process when we validate the certificate. +``` diff --git a/.changelog/17939.txt b/.changelog/17939.txt new file mode 100644 index 000000000000..069ac53ee7be --- /dev/null +++ b/.changelog/17939.txt @@ -0,0 +1,4 @@ +```release-note:improvement +http: GET API `operator/usage` endpoint now returns node count +cli: `consul operator usage` command now returns node count +``` \ No newline at end of file diff --git a/.changelog/17978.txt b/.changelog/17978.txt new file mode 100644 index 000000000000..81d5a648da1f --- /dev/null +++ b/.changelog/17978.txt @@ -0,0 +1,3 @@ +```release-note:improvement +mesh: Expose remote jwks cluster configuration through jwt-provider config entry +``` \ No newline at end of file diff --git a/.changelog/18011.txt b/.changelog/18011.txt new file mode 100644 index 000000000000..d6c989f00e9f --- /dev/null +++ b/.changelog/18011.txt @@ -0,0 +1,4 @@ +```release-note:bug +connect: Removes the default health check from the `consul connect envoy` command when starting an API Gateway. +This health check would always fail. +``` diff --git a/.changelog/18024.txt b/.changelog/18024.txt new file mode 100644 index 000000000000..a661e7304c62 --- /dev/null +++ b/.changelog/18024.txt @@ -0,0 +1,3 @@ +```release-note:bug +connect: fix a bug with Envoy potentially starting with incomplete configuration by not waiting enough for initial xDS configuration. +``` \ No newline at end of file diff --git a/.changelog/18068.txt b/.changelog/18068.txt new file mode 100644 index 000000000000..be55ad365877 --- /dev/null +++ b/.changelog/18068.txt @@ -0,0 +1,3 @@ +```release-note:bug +xds: Prevent partial application of non-Required Envoy extensions in the case of failure. +``` \ No newline at end of file diff --git a/.changelog/18080.txt b/.changelog/18080.txt new file mode 100644 index 000000000000..9826b249eb31 --- /dev/null +++ b/.changelog/18080.txt @@ -0,0 +1,3 @@ +```release-note:improvement +Fix some typos in metrics docs +``` \ No newline at end of file diff --git a/.changelog/18112.txt b/.changelog/18112.txt new file mode 100644 index 000000000000..ddd37786f552 --- /dev/null +++ b/.changelog/18112.txt @@ -0,0 +1,3 @@ +```release-note:bug +ca: Fixes a Vault CA provider bug where updating RootPKIPath but not IntermediatePKIPath would not renew leaf signing certificates +``` diff --git a/.changelog/18140.txt b/.changelog/18140.txt new file mode 100644 index 000000000000..fabd9fc2916b --- /dev/null +++ b/.changelog/18140.txt @@ -0,0 +1,3 @@ +```release-note:improvement +hcp: Removes requirement for HCP to provide a management token +``` diff --git a/.changelog/18150.txt b/.changelog/18150.txt new file mode 100644 index 000000000000..492e7ad1b9ff --- /dev/null +++ b/.changelog/18150.txt @@ -0,0 +1,3 @@ +```release-note:improvement +xds: Explicitly enable WebSocket connection upgrades in HTTP connection manager +``` diff --git a/.changelog/18168.txt b/.changelog/18168.txt new file mode 100644 index 000000000000..a68483527e10 --- /dev/null +++ b/.changelog/18168.txt @@ -0,0 +1,3 @@ +```release-note:improvement +hcp: Add dynamic configuration support for the export of server metrics to HCP. +``` \ No newline at end of file diff --git a/.changelog/18184.txt b/.changelog/18184.txt new file mode 100644 index 000000000000..594546ca3259 --- /dev/null +++ b/.changelog/18184.txt @@ -0,0 +1,3 @@ +```release-note:bug +api: Fix client deserialization errors by marking new Enterprise-only prepared query fields as omit empty +``` diff --git a/.changelog/18186.txt b/.changelog/18186.txt new file mode 100644 index 000000000000..dcc75b57653b --- /dev/null +++ b/.changelog/18186.txt @@ -0,0 +1,3 @@ +```release-note:security +Upgrade golang.org/x/net to address [CVE-2023-29406](https://nvd.nist.gov/vuln/detail/CVE-2023-29406) +``` diff --git a/.changelog/18190.txt b/.changelog/18190.txt new file mode 100644 index 000000000000..3468442e2161 --- /dev/null +++ b/.changelog/18190.txt @@ -0,0 +1,5 @@ +```release-note:security +Upgrade to use Go 1.20.6. +This resolves [CVE-2023-29406](https://github.com/advisories/GHSA-f8f7-69v5-w4vx)(`net/http`) for uses of the standard library. +A separate change updates dependencies on `golang.org/x/net` to use `0.12.0`. +``` diff --git a/.changelog/18223.txt b/.changelog/18223.txt new file mode 100644 index 000000000000..067ca64f48e8 --- /dev/null +++ b/.changelog/18223.txt @@ -0,0 +1,3 @@ +```release-note:feature +cli: `consul members` command uses `-filter` expression to filter members based on bexpr. +``` diff --git a/.changelog/18291.txt b/.changelog/18291.txt new file mode 100644 index 000000000000..bb0ec6f89295 --- /dev/null +++ b/.changelog/18291.txt @@ -0,0 +1,3 @@ +```release-note:bug +api-gateway: fix race condition in proxy config generation when Consul is notified of the bound-api-gateway config entry before it is notified of the api-gateway config entry. +``` diff --git a/.changelog/18302.txt b/.changelog/18302.txt new file mode 100644 index 000000000000..c77e7106be91 --- /dev/null +++ b/.changelog/18302.txt @@ -0,0 +1,4 @@ +```release-note:bug +snapshot: fix access denied and handle is invalid when we call snapshot save on windows - skip sync() for folders in windows in +https://github.com/rboyer/safeio/pull/3 +``` diff --git a/.changelog/18303.txt b/.changelog/18303.txt new file mode 100644 index 000000000000..4afc4473b7c9 --- /dev/null +++ b/.changelog/18303.txt @@ -0,0 +1,3 @@ +```release-note:improvement +connect: update supported envoy versions to 1.23.12, 1.24.10, 1.25.9, 1.26.4 +``` diff --git a/.changelog/18319.txt b/.changelog/18319.txt new file mode 100644 index 000000000000..bb9c8cdf2c72 --- /dev/null +++ b/.changelog/18319.txt @@ -0,0 +1,6 @@ +```release-note:improvement +acl: added builtin ACL policy that provides global read-only access (builtin/global-read-only) +``` +```release-note:improvement +acl: allow for a single slash character in policy names +``` diff --git a/.changelog/18325.txt b/.changelog/18325.txt new file mode 100644 index 000000000000..b2870352b6c0 --- /dev/null +++ b/.changelog/18325.txt @@ -0,0 +1,3 @@ +```release-note:bug +mesh: **(Enterprise Only)** Require that `jwt-provider` config entries are created in the `default` namespace. +``` \ No newline at end of file diff --git a/.changelog/18358.txt b/.changelog/18358.txt new file mode 100644 index 000000000000..e29d258c6c92 --- /dev/null +++ b/.changelog/18358.txt @@ -0,0 +1,7 @@ +```release-note:security +Upgrade to use Go 1.20.7. +This resolves vulnerability [CVE-2023-29409](https://nvd.nist.gov/vuln/detail/CVE-2023-29409)(`crypto/tls`). +``` +```release-note:security +Update `golang.org/x/net` to v0.13.0 to address [CVE-2023-3978](https://nvd.nist.gov/vuln/detail/CVE-2023-3978). +``` diff --git a/.changelog/18381.txt b/.changelog/18381.txt new file mode 100644 index 000000000000..f6af62993f59 --- /dev/null +++ b/.changelog/18381.txt @@ -0,0 +1,6 @@ +```release-note:improvement +checks: It is now possible to configure agent TCP checks to use TLS with +optional server SNI and mutual authentication. To use TLS with a TCP check, the +check must enable the `tcp_use_tls` boolean. By default the agent will use the +TLS configuration in the `tls.default` stanza. +``` diff --git a/.changelog/18437.txt b/.changelog/18437.txt new file mode 100644 index 000000000000..2ae3c5bdda23 --- /dev/null +++ b/.changelog/18437.txt @@ -0,0 +1,3 @@ +```release-note:bug +Inherit locality from services when registering sidecar proxies. +``` diff --git a/.changelog/18464.txt b/.changelog/18464.txt new file mode 100644 index 000000000000..b0c15cf0d552 --- /dev/null +++ b/.changelog/18464.txt @@ -0,0 +1,3 @@ +```release-note:bug +UI : Nodes list view was breaking for synthetic-nodes. Fix handles non existence of consul-version meta for node. +``` diff --git a/.changelog/18558.txt b/.changelog/18558.txt new file mode 100644 index 000000000000..9c2b9b44bb31 --- /dev/null +++ b/.changelog/18558.txt @@ -0,0 +1,3 @@ +```release-note:bug +check: prevent go routine leakage when existing Defercheck of same check id is not nil +``` diff --git a/.changelog/18584.txt b/.changelog/18584.txt new file mode 100644 index 000000000000..e7329655ba6e --- /dev/null +++ b/.changelog/18584.txt @@ -0,0 +1,3 @@ +```release-note:improvement +Reduce the frequency of metric exports from Consul to HCP from every 10s to every 1m +``` \ No newline at end of file diff --git a/.changelog/18617.txt b/.changelog/18617.txt new file mode 100644 index 000000000000..1f840d836dee --- /dev/null +++ b/.changelog/18617.txt @@ -0,0 +1,4 @@ +```release-note:improvement +log: Currently consul logs files like this consul-{timestamp}.log. This change makes sure that there is always +consul.log file with the latest logs in it. +``` \ No newline at end of file diff --git a/.changelog/18625.txt b/.changelog/18625.txt new file mode 100644 index 000000000000..8474cac8dc1d --- /dev/null +++ b/.changelog/18625.txt @@ -0,0 +1,5 @@ +```release-note:improvement +Adds flag -append-filename (which works on values version, dc, node and status) to consul snapshot save command. +Adding the flag -append-filename version,dc,node,status will add consul version, consul datacenter, node name and leader/follower +(status) in the file name given in the snapshot save command before the file extension. +``` diff --git a/.changelog/18636.txt b/.changelog/18636.txt new file mode 100644 index 000000000000..ff6990f5abcb --- /dev/null +++ b/.changelog/18636.txt @@ -0,0 +1,3 @@ +```release-note:bug +connect: Fix issue where Envoy endpoints would not populate correctly after a snapshot restore. +``` diff --git a/.changelog/18667.txt b/.changelog/18667.txt new file mode 100644 index 000000000000..c9ef7b455121 --- /dev/null +++ b/.changelog/18667.txt @@ -0,0 +1,3 @@ +```release-note:improvement +api: Add support for listing ACL tokens by service name. +``` diff --git a/.changelog/18681.txt b/.changelog/18681.txt new file mode 100644 index 000000000000..971e9ef81637 --- /dev/null +++ b/.changelog/18681.txt @@ -0,0 +1,3 @@ +```release-note:bug +api: Fix `/v1/agent/self` not returning latest configuration +``` diff --git a/.changelog/18724.txt b/.changelog/18724.txt new file mode 100644 index 000000000000..7fa289eba19f --- /dev/null +++ b/.changelog/18724.txt @@ -0,0 +1,3 @@ +```release-note:bug +telemetry: emit consul version metric on a regular interval. +``` diff --git a/.changelog/18742.txt b/.changelog/18742.txt new file mode 100644 index 000000000000..2d31e5266758 --- /dev/null +++ b/.changelog/18742.txt @@ -0,0 +1,8 @@ +```release-note:security +Upgrade to use Go 1.20.8. This resolves CVEs +[CVE-2023-39320](https://github.com/advisories/GHSA-rxv8-v965-v333) (`cmd/go`), +[CVE-2023-39318](https://github.com/advisories/GHSA-vq7j-gx56-rxjh) (`html/template`), +[CVE-2023-39319](https://github.com/advisories/GHSA-vv9m-32rr-3g55) (`html/template`), +[CVE-2023-39321](https://github.com/advisories/GHSA-9v7r-x7cv-v437) (`crypto/tls`), and +[CVE-2023-39322](https://github.com/advisories/GHSA-892h-r6cr-53g4) (`crypto/tls`) +``` \ No newline at end of file diff --git a/.changelog/18773.txt b/.changelog/18773.txt new file mode 100644 index 000000000000..1d59fe98f0dc --- /dev/null +++ b/.changelog/18773.txt @@ -0,0 +1,3 @@ +```release-note:bug +ca: Vault provider now cleans up the previous Vault issuer and key when generating a new leaf signing certificate [[GH-18779](https://github.com/hashicorp/consul/issues/18779)] +``` diff --git a/.changelog/18797.txt b/.changelog/18797.txt new file mode 100644 index 000000000000..ba40092542a3 --- /dev/null +++ b/.changelog/18797.txt @@ -0,0 +1,3 @@ +```release-note:improvement +command: Adds -since flag in consul debug command which internally calls hcdiag for debug information in the past. +``` diff --git a/.changelog/18831.txt b/.changelog/18831.txt new file mode 100644 index 000000000000..0981fb6a368b --- /dev/null +++ b/.changelog/18831.txt @@ -0,0 +1,3 @@ +```release-note:bug +gateways: Fix a bug where gateway to service mappings weren't being cleaned up properly when externally registered proxies were being deregistered. +``` diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 31495d06491b..d9af3f042a7e 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -8,3 +8,34 @@ # release configuration /.release/ @hashicorp/release-engineering @hashicorp/github-consul-core /.github/workflows/build.yml @hashicorp/release-engineering @hashicorp/github-consul-core + + +# Staff Engineer Review (protocol buffer definitions) +/proto-public/ @hashicorp/consul-core-staff +/proto/ @hashicorp/consul-core-staff + +# Staff Engineer Review (v1 architecture shared components) +/agent/cache/ @hashicorp/consul-core-staff +/agent/consul/fsm/ @hashicorp/consul-core-staff +/agent/consul/leader*.go @hashicorp/consul-core-staff +/agent/consul/server*.go @hashicorp/consul-core-staff +/agent/consul/state/ @hashicorp/consul-core-staff +/agent/consul/stream/ @hashicorp/consul-core-staff +/agent/submatview/ @hashicorp/consul-core-staff +/agent/blockingquery/ @hashicorp/consul-core-staff + +# Staff Engineer Review (raft/autopilot) +/agent/consul/autopilotevents/ @hashicorp/consul-core-staff +/agent/consul/autopilot*.go @hashicorp/consul-core-staff + +# Staff Engineer Review (v2 architecture shared components) +/internal/controller/ @hashicorp/consul-core-staff +/internal/resource/ @hashicorp/consul-core-staff +/internal/storage/ @hashicorp/consul-core-staff +/agent/consul/controller/ @hashicorp/consul-core-staff +/agent/grpc-external/services/resource/ @hashicorp/consul-core-staff + +# Staff Engineer Review (v1 security) +/acl/ @hashicorp/consul-core-staff +/agent/xds/rbac*.go @hashicorp/consul-core-staff +/agent/xds/jwt*.go @hashicorp/consul-core-staff diff --git a/.github/scripts/filter_changed_files_go_test.sh b/.github/scripts/filter_changed_files_go_test.sh new file mode 100755 index 000000000000..4db9e7a8f2bb --- /dev/null +++ b/.github/scripts/filter_changed_files_go_test.sh @@ -0,0 +1,34 @@ +#!/bin/bash + +# Get the list of changed files +files_to_check=$(git diff --name-only origin/$GITHUB_BASE_REF) + +# Define the directories to check +skipped_directories=("docs/" "ui/" "website/" "grafana/") + +# Initialize a variable to track directories outside the skipped ones +other_directories="" +trigger_ci=true + +# # Loop through the changed files and find directories/files outside the skipped ones +# for file_to_check in $files_to_check; do +# file_is_skipped=false +# for dir in "${skipped_directories[@]}"; do +# if [[ "$file_to_check" == "$dir"* ]] || [[ "$file_to_check" == *.md && "$dir" == *"/" ]]; then +# file_is_skipped=true +# break +# fi +# done +# if [ "$file_is_skipped" = "false" ]; then +# other_directories+="$(dirname "$file_to_check")\n" +# trigger_ci=true +# echo "Non doc file(s) changed - triggered ci: $trigger_ci" +# echo -e $other_directories +# echo "trigger-ci=$trigger_ci" >>"$GITHUB_OUTPUT" +# exit 0 ## if file is outside of the skipped_directory exit script +# fi +# done + +# echo "Only doc file(s) changed - triggered ci: $trigger_ci" +echo "Doc file(s) change detection is currently disabled - triggering ci" +echo "trigger-ci=$trigger_ci" >>"$GITHUB_OUTPUT" diff --git a/.github/scripts/get_runner_classes.sh b/.github/scripts/get_runner_classes.sh index 80980b7a8ffc..2e66a4e344c6 100755 --- a/.github/scripts/get_runner_classes.sh +++ b/.github/scripts/get_runner_classes.sh @@ -8,19 +8,19 @@ set -euo pipefail case "$GITHUB_REPOSITORY" in - *-enterprise) - # shellcheck disable=SC2129 - echo "compute-small=['self-hosted', 'linux', 'small']" >> "$GITHUB_OUTPUT" - echo "compute-medium=['self-hosted', 'linux', 'medium']" >> "$GITHUB_OUTPUT" - echo "compute-large=['self-hosted', 'linux', 'large']" >> "$GITHUB_OUTPUT" - # m5d.8xlarge is equivalent to our xl custom runner in OSS - echo "compute-xl=['self-hosted', 'ondemand', 'linux', 'type=m5d.8xlarge']" >> "$GITHUB_OUTPUT" - ;; - *) - # shellcheck disable=SC2129 - echo "compute-small=['custom-linux-s-consul-latest']" >> "$GITHUB_OUTPUT" - echo "compute-medium=['custom-linux-m-consul-latest']" >> "$GITHUB_OUTPUT" - echo "compute-large=['custom-linux-l-consul-latest']" >> "$GITHUB_OUTPUT" - echo "compute-xl=['custom-linux-xl-consul-latest']" >> "$GITHUB_OUTPUT" - ;; +*-enterprise) + # shellcheck disable=SC2129 + echo "compute-small=['self-hosted', 'linux', 'small']" >>"$GITHUB_OUTPUT" + echo "compute-medium=['self-hosted', 'linux', 'medium']" >>"$GITHUB_OUTPUT" + echo "compute-large=['self-hosted', 'linux', 'large']" >>"$GITHUB_OUTPUT" + # m5d.8xlarge is equivalent to our xl custom runner in CE + echo "compute-xl=['self-hosted', 'ondemand', 'linux', 'type=m6a.2xlarge']" >>"$GITHUB_OUTPUT" + ;; +*) + # shellcheck disable=SC2129 + echo "compute-small=['custom-linux-s-consul-latest']" >>"$GITHUB_OUTPUT" + echo "compute-medium=['custom-linux-m-consul-latest']" >>"$GITHUB_OUTPUT" + echo "compute-large=['custom-linux-l-consul-latest']" >>"$GITHUB_OUTPUT" + echo "compute-xl=['custom-linux-xl-consul-latest']" >>"$GITHUB_OUTPUT" + ;; esac diff --git a/.github/scripts/license_checker.sh b/.github/scripts/license_checker.sh new file mode 100755 index 000000000000..6ba026f04fdd --- /dev/null +++ b/.github/scripts/license_checker.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# Copyright (c) HashiCorp, Inc. +# SPDX-License-Identifier: BUSL-1.1 + + +busl_files=$(grep -r 'SPDX-License-Identifier: BUSL' . --exclude-dir .github) + +# If we do not find a file in .changelog/, we fail the check +if [ -n "$busl_files" ]; then + echo "Found BUSL occurrences in the PR branch! (See NET-5258 for details)" + echo -n "$busl_files" + exit 1 +else + echo "Did not find any occurrences of BUSL in the PR branch" + exit 0 +fi diff --git a/.github/workflows/backport-assistant.yml b/.github/workflows/backport-assistant.yml index fffa774d32a5..17462f811261 100644 --- a/.github/workflows/backport-assistant.yml +++ b/.github/workflows/backport-assistant.yml @@ -40,4 +40,4 @@ jobs: curl -s -H "Authorization: token ${{ secrets.PR_COMMENT_TOKEN }}" \ -X POST \ -d "{ \"body\": \"${github_message}\"}" \ - "https://api.github.com/repos/${GITHUB_REPOSITORY}/pull/${{ github.event.pull_request.number }}/comments" + "https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/${{ github.event.pull_request.number }}/comments" diff --git a/.github/workflows/backport-reminder.yml b/.github/workflows/backport-reminder.yml deleted file mode 100644 index b358122dbf0c..000000000000 --- a/.github/workflows/backport-reminder.yml +++ /dev/null @@ -1,29 +0,0 @@ -# Copyright (c) HashiCorp, Inc. -# SPDX-License-Identifier: MPL-2.0 - -# This workflow sends a reminder comment to PRs that have labels starting with -# `backport/` to check that the backport has run successfully. - -name: Backport Assistant Reminder - -on: - pull_request: - types: [ labeled ] - # Runs on PRs to main and all release branches - branches: - - main - - release/* - -jobs: - backport-label-check: - if: "startsWith(github.event.label.name, 'backport/')" - runs-on: ubuntu-latest - - steps: - - name: Comment on PR - run: | - github_message="After merging, confirm that you see linked PRs AND check them for CI errors." - curl -s -H "Authorization: token ${{ secrets.PR_COMMENT_TOKEN }}" \ - -X POST \ - -d "{ \"body\": \"${github_message}\"}" \ - "https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/${{ github.event.pull_request.number }}/comments" diff --git a/.github/workflows/build-artifacts.yml b/.github/workflows/build-artifacts.yml index 2e87d767f5e4..f57ea3527d48 100644 --- a/.github/workflows/build-artifacts.yml +++ b/.github/workflows/build-artifacts.yml @@ -80,7 +80,7 @@ jobs: - name: Set up Docker Buildx uses: docker/setup-buildx-action@f03ac48505955848960e80bbb68046aa35c7b9e7 # pin@v2.4.1 - # NOTE: conditional specific logic as we store secrets in Vault in ENT and use GHA secrets in OSS. + # NOTE: conditional specific logic as we store secrets in Vault in ENT and use GHA secrets in CE. - name: Login to Docker Hub uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a # pin@v2.1.0 with: diff --git a/.github/workflows/build-distros.yml b/.github/workflows/build-distros.yml index 6f5722a82ab9..10c520893341 100644 --- a/.github/workflows/build-distros.yml +++ b/.github/workflows/build-distros.yml @@ -2,7 +2,7 @@ # It is aimed at checking new commits don't introduce any breaking build changes. name: build-distros -on: +on: pull_request: push: branches: @@ -15,6 +15,7 @@ permissions: env: GOTAGS: ${{ endsWith(github.repository, '-enterprise') && 'consulent' || '' }} + GOPRIVATE: github.com/hashicorp # Required for enterprise deps jobs: setup: @@ -26,13 +27,13 @@ jobs: compute-large: ${{ steps.setup-outputs.outputs.compute-large }} compute-xl: ${{ steps.setup-outputs.outputs.compute-xl }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - id: setup-outputs name: Setup outputs run: ./.github/scripts/get_runner_classes.sh check-go-mod: - needs: + needs: - setup uses: ./.github/workflows/reusable-check-go-mod.yml with: @@ -42,21 +43,21 @@ jobs: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} build-386: - needs: - - setup + needs: + - setup - check-go-mod env: XC_OS: "freebsd linux windows" runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - name: Build @@ -67,20 +68,20 @@ jobs: build-amd64: needs: - - setup + - setup - check-go-mod env: XC_OS: "darwin freebsd linux solaris windows" runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - name: Build @@ -91,22 +92,22 @@ jobs: build-arm: needs: - - setup + - setup - check-go-mod runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} env: CGO_ENABLED: 1 GOOS: linux steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - run: | @@ -116,6 +117,26 @@ jobs: - run: CC=arm-linux-gnueabihf-gcc GOARCH=arm GOARM=6 go build -tags "${{ env.GOTAGS }}" - run: CC=aarch64-linux-gnu-gcc GOARCH=arm64 go build -tags "${{ env.GOTAGS }}" + + build-s390x: + if: ${{ endsWith(github.repository, '-enterprise') }} + needs: + - setup + - check-go-mod + runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} + steps: + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 + + # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. + - name: Setup Git + run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" + + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 + with: + go-version-file: 'go.mod' + - name: Build + run: GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -tags "${{ env.GOTAGS }}" + # This is job is required for branch protection as a required gihub check # because GitHub actions show up as checks at the job level and not the # workflow level. This is currently a feature request: @@ -125,18 +146,18 @@ jobs: # - be placed after the fanout of a workflow so that everything fans back in # to this job. # - "need" any job that is part of the fan out / fan in - # - implement the if logic because we have conditional jobs - # (go-test-enteprise) that this job needs and this would potentially get - # skipped if a previous job got skipped. So we use the if clause to make + # - implement the if logic because we have conditional jobs + # (go-test-enteprise) that this job needs and this would potentially get + # skipped if a previous job got skipped. So we use the if clause to make # sure it does not get skipped. - build-distros-success: - needs: + needs: - setup - check-go-mod - build-386 - build-amd64 - build-arm + - build-s390x runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} if: ${{ always() }} steps: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 66b5a14a739c..6ba6d04a2953 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -13,7 +13,9 @@ on: env: PKG_NAME: consul + # TODO(spatel): CE refactor METADATA: oss + GOPRIVATE: github.com/hashicorp # Required for enterprise deps jobs: set-product-version: @@ -25,7 +27,7 @@ jobs: pre-version: ${{ steps.set-product-version.outputs.prerelease-product-version }} shared-ldflags: ${{ steps.shared-ldflags.outputs.shared-ldflags }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - name: set product version id: set-product-version uses: hashicorp/actions-set-product-version@v1 @@ -56,6 +58,7 @@ jobs: echo "Product Date: ${{ needs.set-product-version.outputs.product-date }}" echo "Prerelease Version: ${{ needs.set-product-version.outputs.pre-version }}" echo "Ldflags: ${{ needs.set-product-version.outputs.shared-ldflags }}" + generate-metadata-file: needs: set-product-version runs-on: ubuntu-latest @@ -63,7 +66,7 @@ jobs: filepath: ${{ steps.generate-metadata-file.outputs.filepath }} steps: - name: 'Checkout directory' - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - name: Generate metadata file id: generate-metadata-file uses: hashicorp/actions-generate-metadata@v1 @@ -71,7 +74,7 @@ jobs: version: ${{ needs.set-product-version.outputs.product-version }} product: ${{ env.PKG_NAME }} - - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # pin@3.1.2 + - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2 with: name: metadata.json path: ${{ steps.generate-metadata-file.outputs.filepath }} @@ -82,23 +85,23 @@ jobs: strategy: matrix: include: - - {go: "1.20.4", goos: "linux", goarch: "386"} - - {go: "1.20.4", goos: "linux", goarch: "amd64"} - - {go: "1.20.4", goos: "linux", goarch: "arm"} - - {go: "1.20.4", goos: "linux", goarch: "arm64"} - - {go: "1.20.4", goos: "freebsd", goarch: "386"} - - {go: "1.20.4", goos: "freebsd", goarch: "amd64"} - - {go: "1.20.4", goos: "windows", goarch: "386"} - - {go: "1.20.4", goos: "windows", goarch: "amd64"} - - {go: "1.20.4", goos: "solaris", goarch: "amd64"} + - {go: "1.20.8", goos: "linux", goarch: "386"} + - {go: "1.20.8", goos: "linux", goarch: "amd64"} + - {go: "1.20.8", goos: "linux", goarch: "arm"} + - {go: "1.20.8", goos: "linux", goarch: "arm64"} + - {go: "1.20.8", goos: "freebsd", goarch: "386"} + - {go: "1.20.8", goos: "freebsd", goarch: "amd64"} + - {go: "1.20.8", goos: "windows", goarch: "386"} + - {go: "1.20.8", goos: "windows", goarch: "amd64"} + - {go: "1.20.8", goos: "solaris", goarch: "amd64"} fail-fast: true name: Go ${{ matrix.go }} ${{ matrix.goos }} ${{ matrix.goarch }} build steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - name: Setup with node and yarn - uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # pin@v3.6.0 + uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0 with: node-version: '14' cache: 'yarn' @@ -160,18 +163,69 @@ jobs: echo "RPM_PACKAGE=$(basename out/*.rpm)" >> $GITHUB_ENV echo "DEB_PACKAGE=$(basename out/*.deb)" >> $GITHUB_ENV - - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # pin@3.1.2 + - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2 if: ${{ matrix.goos == 'linux' }} with: name: ${{ env.RPM_PACKAGE }} path: out/${{ env.RPM_PACKAGE }} - - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # pin@3.1.2 + - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2 if: ${{ matrix.goos == 'linux' }} with: name: ${{ env.DEB_PACKAGE }} path: out/${{ env.DEB_PACKAGE }} + build-s390x: + needs: set-product-version + if: ${{ endsWith(github.repository, '-enterprise') }} + runs-on: ubuntu-latest + strategy: + matrix: + include: + - {go: "1.20.8", goos: "linux", goarch: "s390x"} + fail-fast: true + + name: Go ${{ matrix.go }} ${{ matrix.goos }} ${{ matrix.goarch }} build + steps: + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 + + - name: Setup with node and yarn + uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0 + with: + node-version: '14' + cache: 'yarn' + cache-dependency-path: 'ui/yarn.lock' + + - name: Build UI + run: | + CONSUL_VERSION=${{ needs.set-product-version.outputs.product-version }} + CONSUL_DATE=${{ needs.set-product-version.outputs.product-date }} + CONSUL_BINARY_TYPE=${CONSUL_BINARY_TYPE} + CONSUL_COPYRIGHT_YEAR=$(git show -s --format=%cd --date=format:%Y HEAD) + echo "consul_version is ${CONSUL_VERSION}" + echo "consul_date is ${CONSUL_DATE}" + echo "consul binary type is ${CONSUL_BINARY_TYPE}" + echo "consul copyright year is ${CONSUL_COPYRIGHT_YEAR}" + cd ui && make && cd .. + rm -rf agent/uiserver/dist + mv ui/packages/consul-ui/dist agent/uiserver/ + - name: Go Build + env: + PRODUCT_VERSION: ${{ needs.set-product-version.outputs.product-version }} + PRERELEASE_VERSION: ${{ needs.set-product-version.outputs.pre-version }} + CGO_ENABLED: "0" + GOLDFLAGS: "${{needs.set-product-version.outputs.shared-ldflags}}" + uses: hashicorp/actions-go-build@v0.1.7 + with: + product_name: ${{ env.PKG_NAME }} + product_version: ${{ needs.set-product-version.outputs.product-version }} + go_version: ${{ matrix.go }} + os: ${{ matrix.goos }} + arch: ${{ matrix.goarch }} + reproducible: report + instructions: |- + go build -ldflags="$GOLDFLAGS" -o "$BIN_PATH" -trimpath -buildvcs=false + build-darwin: needs: set-product-version runs-on: macos-latest @@ -179,15 +233,15 @@ jobs: matrix: goos: [ darwin ] goarch: [ "amd64", "arm64" ] - go: [ "1.20.4" ] + go: [ "1.20.8" ] fail-fast: true name: Go ${{ matrix.go }} ${{ matrix.goos }} ${{ matrix.goarch }} build steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 - + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 + - name: Setup with node and yarn - uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # pin@v3.6.0 + uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0 with: node-version: '14' cache: 'yarn' @@ -229,13 +283,17 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - arch: ["386", "amd64", "arm", "arm64"] + include: + - { arch: "386" } + - { arch: "arm" } + - { arch: "amd64" } + - { arch: "arm64" } env: repo: ${{github.event.repository.name}} version: ${{needs.set-product-version.outputs.product-version}} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # Strip everything but MAJOR.MINOR from the version string and add a `-dev` suffix # This naming convention will be used ONLY for per-commit dev images @@ -269,7 +327,7 @@ jobs: version: ${{needs.set-product-version.outputs.product-version}} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - uses: hashicorp/actions-docker-build@v1 with: version: ${{env.version}} @@ -289,7 +347,7 @@ jobs: version: ${{needs.set-product-version.outputs.product-version}} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # Strip everything but MAJOR.MINOR from the version string and add a `-dev` suffix # This naming convention will be used ONLY for per-commit dev images @@ -318,7 +376,11 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - arch: ["386", "amd64", "arm", "arm64"] + include: + - { arch: "386" } + - { arch: "arm" } + - { arch: "amd64" } + - { arch: "arm64" } fail-fast: true env: version: ${{ needs.set-product-version.outputs.product-version }} @@ -326,21 +388,24 @@ jobs: name: Verify ${{ matrix.arch }} linux binary steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 + if: ${{ endsWith(github.repository, '-enterprise') || matrix.arch != 's390x' }} - name: Download ${{ matrix.arch }} zip - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # pin@v3.0.2 + if: ${{ endsWith(github.repository, '-enterprise') || matrix.arch != 's390x' }} + uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 with: name: ${{ env.zip_name }} - name: Set up QEMU - uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # pin@v2.1.0 + uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0 if: ${{ matrix.arch == 'arm' || matrix.arch == 'arm64' }} with: # this should be a comma-separated string as opposed to an array platforms: arm,arm64 - name: Run verification for ${{ matrix.arch }} binary + if: ${{ endsWith(github.repository, '-enterprise') || matrix.arch != 's390x' }} run: .github/scripts/verify_artifact.sh ${{ env.zip_name }} v${{ env.version }} verify-darwin: @@ -356,10 +421,10 @@ jobs: name: Verify amd64 darwin binary steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - name: Download amd64 darwin zip - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # pin@v3.0.2 + uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 with: name: ${{ env.zip_name }} @@ -376,14 +441,18 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - arch: ["i386", "amd64", "armhf", "arm64"] + include: + - { arch: "i386" } + - { arch: "armhf" } + - { arch: "amd64" } + - { arch: "arm64" } # fail-fast: true env: version: ${{ needs.set-product-version.outputs.product-version }} name: Verify ${{ matrix.arch }} debian package steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - name: Set package version run: | @@ -394,12 +463,12 @@ jobs: echo "pkg_name=consul_${{ env.pkg_version }}-1_${{ matrix.arch }}.deb" >> $GITHUB_ENV - name: Download workflow artifacts - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # pin@v3.0.2 + uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 with: name: ${{ env.pkg_name }} - name: Set up QEMU - uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # pin@v2.1.0 + uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0 with: platforms: all @@ -413,14 +482,18 @@ jobs: runs-on: ubuntu-latest strategy: matrix: + include: + - { arch: "i386" } + - { arch: "x86_64" } # TODO(eculver): re-enable when there is a smaller verification container available - arch: ["i386", "x86_64"] #, "armv7hl", "aarch64"] + # - { arch: "armv7hl" } + # - { arch: "aarch64" } env: version: ${{ needs.set-product-version.outputs.product-version }} name: Verify ${{ matrix.arch }} rpm steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - name: Set package version run: | @@ -431,12 +504,12 @@ jobs: echo "pkg_name=consul-${{ env.pkg_version }}-1.${{ matrix.arch }}.rpm" >> $GITHUB_ENV - name: Download workflow artifacts - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # pin@v3.0.2 + uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 with: name: ${{ env.pkg_name }} - name: Set up QEMU - uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # pin@v2.1.0 + uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0 with: platforms: all diff --git a/.github/workflows/oss-merge-trigger.yml b/.github/workflows/ce-merge-trigger.yml similarity index 83% rename from .github/workflows/oss-merge-trigger.yml rename to .github/workflows/ce-merge-trigger.yml index 4a4fdaa208e3..4de4751660d1 100644 --- a/.github/workflows/oss-merge-trigger.yml +++ b/.github/workflows/ce-merge-trigger.yml @@ -1,7 +1,7 @@ # Copyright (c) HashiCorp, Inc. # SPDX-License-Identifier: MPL-2.0 -name: Trigger OSS to Enterprise Merge +name: Trigger Community Edition to Enterprise Merge on: pull_request_target: types: @@ -11,8 +11,8 @@ on: - 'release/*.*.x' jobs: - trigger-oss-merge: - # run this only on merge events in OSS repo + trigger-ce-merge: + # run this only on merge events in CE repo if: ${{ github.event.pull_request.merged && github.repository == 'hashicorp/consul' }} runs-on: ubuntu-latest steps: @@ -22,8 +22,9 @@ jobs: GIT_SHA: ${{ github.sha }} GH_PAT: ${{ secrets.ELEVATED_GITHUB_TOKEN }} GIT_ACTOR: ${{ github.actor }} + # TODO(spatel): CE refactor run: | curl -H "Authorization: token $GH_PAT" \ -H 'Accept: application/json' \ -d "{\"event_type\": \"oss-merge\", \"client_payload\": {\"git-ref\": \"${GIT_REF}\", \"git-sha\": \"${GIT_SHA}\", \"git-actor\": \"${GIT_ACTOR}\" }}" \ - "https://api.github.com/repos/hashicorp/consul-enterprise/dispatches" \ No newline at end of file + "https://api.github.com/repos/hashicorp/consul-enterprise/dispatches" diff --git a/.github/workflows/frontend.yml b/.github/workflows/frontend.yml index 5eab231c65a5..defd0b22e380 100644 --- a/.github/workflows/frontend.yml +++ b/.github/workflows/frontend.yml @@ -74,12 +74,12 @@ jobs: ember-build-test: needs: setup - runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} + runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} strategy: matrix: partition: [1, 2, 3, 4] env: - EMBER_TEST_REPORT: test-results/report-oss.xml # outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml # outputs test report for CI test summary EMBER_TEST_PARALLEL: true # enables test parallelization with ember-exam CONSUL_NSPACES_ENABLED: ${{ endsWith(github.repository, '-enterprise') && 1 || 0 }} # NOTE: this should be 1 in ENT. JOBS: 2 # limit parallelism for broccoli-babel-transpiler diff --git a/.github/workflows/go-tests.yml b/.github/workflows/go-tests.yml index 787f92560ec1..3fb2d66661fc 100644 --- a/.github/workflows/go-tests.yml +++ b/.github/workflows/go-tests.yml @@ -1,6 +1,6 @@ name: go-tests -on: +on: pull_request: branches-ignore: - stable-website @@ -15,16 +15,37 @@ on: # Push events on the main branch - main - release/** - + permissions: contents: read env: TEST_RESULTS: /tmp/test-results + GOPRIVATE: github.com/hashicorp # Required for enterprise deps + +# concurrency +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.ref }} + cancel-in-progress: true jobs: + conditional-skip: + runs-on: ubuntu-latest + name: Get files changed and conditionally skip CI + outputs: + trigger-ci: ${{ steps.read-files.outputs.trigger-ci }} + steps: + - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + fetch-depth: 0 + - name: Get changed files + id: read-files + run: ./.github/scripts/filter_changed_files_go_test.sh + setup: + needs: [conditional-skip] name: Setup + if: needs.conditional-skip.outputs.trigger-ci == 'true' runs-on: ubuntu-latest outputs: compute-small: ${{ steps.setup-outputs.outputs.compute-small }} @@ -32,13 +53,13 @@ jobs: compute-large: ${{ steps.setup-outputs.outputs.compute-large }} compute-xl: ${{ steps.setup-outputs.outputs.compute-xl }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 - id: setup-outputs name: Setup outputs run: ./.github/scripts/get_runner_classes.sh check-go-mod: - needs: + needs: - setup uses: ./.github/workflows/reusable-check-go-mod.yml with: @@ -48,16 +69,16 @@ jobs: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} check-generated-protobuf: - needs: - - setup - runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} + needs: + - setup + runs-on: ${{ fromJSON(needs.setup.outputs.compute-medium) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - run: make proto-tools @@ -74,18 +95,18 @@ jobs: name: "Protobuf Lint" - name: Notify Slack if: ${{ failure() }} - run: .github/scripts/notify_slack.sh + run: .github/scripts/notify_slack.sh check-generated-deep-copy: - needs: - - setup + needs: + - setup runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - run: make --always-make deep-copy @@ -99,19 +120,19 @@ jobs: run: .github/scripts/notify_slack.sh lint-enums: - needs: - - setup + needs: + - setup runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - - run: go install github.com/reillywatson/enumcover/cmd/enumcover@master && enumcover ./... + - run: go install github.com/reillywatson/enumcover/cmd/enumcover@master && enumcover ./... - name: Notify Slack if: ${{ failure() }} run: .github/scripts/notify_slack.sh @@ -121,11 +142,11 @@ jobs: - setup runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - run: make lint-container-test-deps @@ -134,16 +155,16 @@ jobs: run: .github/scripts/notify_slack.sh lint-consul-retry: - needs: - - setup + needs: + - setup runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(github.repository, '-enterprise') }} run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - run: go install github.com/hashicorp/lint-consul-retry@master && lint-consul-retry @@ -152,56 +173,69 @@ jobs: run: .github/scripts/notify_slack.sh lint: - needs: - - setup + needs: + - setup uses: ./.github/workflows/reusable-lint.yml with: - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} lint-32bit: - needs: - - setup + needs: + - setup uses: ./.github/workflows/reusable-lint.yml with: go-arch: "386" - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} # create a development build dev-build: - needs: - - setup + needs: + - setup uses: ./.github/workflows/reusable-dev-build.yml - with: - runs-on: ${{ needs.setup.outputs.compute-xl }} + with: + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} + # dev-build-s390x: + # if: ${{ endsWith(github.repository, '-enterprise') }} + # needs: + # - setup + # uses: ./.github/workflows/reusable-dev-build.yml + # with: + # uploaded-binary-name: 'consul-bin-s390x' + # runs-on: ${{ needs.setup.outputs.compute-large }} + # go-arch: "s390x" + # repository-name: ${{ github.repository }} + # secrets: + # elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} + # dev-build-arm64: - # # only run on enterprise because GHA does not have arm64 runners in OSS + # # only run on enterprise because GHA does not have arm64 runners in CE # if: ${{ endsWith(github.repository, '-enterprise') }} - # needs: - # - setup + # needs: + # - setup # uses: ./.github/workflows/reusable-dev-build.yml # with: # uploaded-binary-name: 'consul-bin-arm64' - # runs-on: ${{ needs.setup.outputs.compute-xl }} + # runs-on: ${{ needs.setup.outputs.compute-large }} # go-arch: "arm64" # repository-name: ${{ github.repository }} # secrets: # elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} # go-test-arm64: - # # only run on enterprise because GHA does not have arm64 runners in OSS + # # only run on enterprise because GHA does not have arm64 runners in CE # if: ${{ endsWith(github.repository, '-enterprise') }} - # needs: - # - setup + # needs: + # - setup # - dev-build-arm64 # uses: ./.github/workflows/reusable-unit-split.yml # with: @@ -216,20 +250,20 @@ jobs: # consul-license: ${{secrets.CONSUL_LICENSE}} # datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" - go-test-oss: - needs: - - setup + go-test-ce: + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit-split.yml with: directory: . runner-count: 12 - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} @@ -237,64 +271,85 @@ jobs: go-test-enterprise: if: ${{ endsWith(github.repository, '-enterprise') }} - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit-split.yml with: directory: . runner-count: 12 - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" go-test-race: - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit.yml with: directory: . go-test-flags: 'GO_TEST_FLAGS="-race -gcflags=all=-d=checkptr=0"' package-names-command: "go list ./... | grep -E -v '^github.com/hashicorp/consul/agent(/consul|/local|/routine-leak-checker)?$' | grep -E -v '^github.com/hashicorp/consul(/command|/connect|/snapshot)'" - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" go-test-32bit: - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit.yml with: directory: . go-arch: "386" go-test-flags: 'export GO_TEST_FLAGS="-short"' - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" + # go-test-s390x: + # if: ${{ endsWith(github.repository, '-enterprise') }} + # needs: + # - setup + # - dev-build-s390x + # uses: ./.github/workflows/reusable-unit.yml + # with: + # uploaded-binary-name: 'consul-bin-s390x' + # directory: . + # go-test-flags: 'export GO_TEST_FLAGS="-short"' + # runs-on: ${{ needs.setup.outputs.compute-large }} + # repository-name: ${{ github.repository }} + # go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" + # permissions: + # id-token: write # NOTE: this permission is explicitly required for Vault auth. + # contents: read + # secrets: + # elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} + # consul-license: ${{secrets.CONSUL_LICENSE}} + # datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" + go-test-envoyextensions: needs: - setup @@ -302,12 +357,12 @@ jobs: uses: ./.github/workflows/reusable-unit.yml with: directory: envoyextensions - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} @@ -320,84 +375,88 @@ jobs: uses: ./.github/workflows/reusable-unit.yml with: directory: troubleshoot - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" go-test-api-1-19: - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit.yml with: directory: api - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" + go-version: "1.19" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" go-test-api-1-20: - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit.yml with: directory: api - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" + go-version: "1.20" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" go-test-sdk-1-19: - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit.yml with: directory: sdk - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" + go-version: "1.19" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} datadog-api-key: "${{ !endsWith(github.repository, '-enterprise') && secrets.DATADOG_API_KEY || '' }}" go-test-sdk-1-20: - needs: - - setup + needs: + - setup - dev-build uses: ./.github/workflows/reusable-unit.yml with: directory: sdk - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} go-tags: "${{ github.event.repository.name == 'consul-enterprise' && 'consulent consulprem consuldev' || '' }}" + go-version: "1.20" permissions: id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read + contents: read secrets: elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} consul-license: ${{secrets.CONSUL_LICENSE}} @@ -417,13 +476,14 @@ jobs: # - be placed after the fanout of a workflow so that everything fans back in # to this job. # - "need" any job that is part of the fan out / fan in - # - implement the if logic because we have conditional jobs - # (go-test-enteprise) that this job needs and this would potentially get - # skipped if a previous job got skipped. So we use the if clause to make + # - implement the if logic because we have conditional jobs + # (go-test-enteprise) that this job needs and this would potentially get + # skipped if a previous job got skipped. So we use the if clause to make # sure it does not get skipped. go-tests-success: - needs: + needs: + - conditional-skip - setup - check-generated-deep-copy - check-generated-protobuf @@ -435,7 +495,7 @@ jobs: - lint-32bit # - go-test-arm64 - go-test-enterprise - - go-test-oss + - go-test-ce - go-test-race - go-test-envoyextensions - go-test-troubleshoot @@ -444,8 +504,9 @@ jobs: - go-test-sdk-1-19 - go-test-sdk-1-20 - go-test-32bit + # - go-test-s390x runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} - if: ${{ always() }} + if: always() && needs.conditional-skip.outputs.trigger-ci == 'true' steps: - name: evaluate upstream job results run: | diff --git a/.github/workflows/license-checker.yml b/.github/workflows/license-checker.yml new file mode 100644 index 000000000000..747f81490e6a --- /dev/null +++ b/.github/workflows/license-checker.yml @@ -0,0 +1,27 @@ +# Copyright (c) HashiCorp, Inc. +# SPDX-License-Identifier: MPL-2.0 + +# This workflow checks that the BUSL license is not mentioned anywhere in +# a PR targeting a release that should maintain the MPL-2.0 license. +name: License Checker + +on: + pull_request: + types: [opened, synchronize] + branches: + - release/1.14.* + - release/1.15.* + - release/1.16.* + +jobs: + # checks that the diff does not contain any reference to + # the BUSL license and thus retains the MPL-2.0 license + license-check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + ref: ${{ github.event.pull_request.head.sha }} + fetch-depth: 0 # by default the checkout action doesn't checkout all branches + - name: Check for BUSL text in diff + run: ./.github/scripts/license_checker.sh diff --git a/.github/workflows/nightly-test-1.12.x.yaml b/.github/workflows/nightly-test-1.12.x.yaml index 0f016075e261..c09cc4864b89 100644 --- a/.github/workflows/nightly-test-1.12.x.yaml +++ b/.github/workflows/nightly-test-1.12.x.yaml @@ -42,7 +42,7 @@ jobs: working-directory: ./ui/packages/consul-ui run: make test-node - frontend-build-oss: + frontend-build-ce: runs-on: ubuntu-latest env: JOBS: 2 @@ -64,27 +64,27 @@ jobs: working-directory: ./ui run: make deps - - name: Ember Build OSS - id: build-oss + - name: Ember Build CE + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci - - name: Upload OSS Frontend + - name: Upload CE Frontend uses: actions/upload-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist if-no-files-found: error - frontend-test-oss: + frontend-test-ce: runs-on: ubuntu-latest - needs: [frontend-build-oss] + needs: [frontend-build-ce] strategy: matrix: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 0 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -103,13 +103,13 @@ jobs: working-directory: ./ui run: make deps - - name: Download OSS Frontend + - name: Download CE Frontend uses: actions/download-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist - - name: Ember Test OSS + - name: Ember Test CE id: cache working-directory: ./ui/packages/consul-ui run: node_modules/.bin/ember exam --split=$EMBER_PARTITION_TOTAL --partition=${{ matrix.partition }} --path dist --silent -r xunit @@ -137,7 +137,7 @@ jobs: run: make deps - name: Ember Build ENT - id: build-oss + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci @@ -156,7 +156,7 @@ jobs: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 1 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -218,7 +218,7 @@ jobs: slack-failure-notification: runs-on: ubuntu-latest - needs: [frontend-test-oss, frontend-test-ent] + needs: [frontend-test-ce, frontend-test-ent] if: ${{ failure() }} steps: - name: Slack Notification diff --git a/.github/workflows/nightly-test-1.13.x.yaml b/.github/workflows/nightly-test-1.13.x.yaml index 51a1226b29be..6139eb4bc1e1 100644 --- a/.github/workflows/nightly-test-1.13.x.yaml +++ b/.github/workflows/nightly-test-1.13.x.yaml @@ -42,7 +42,7 @@ jobs: working-directory: ./ui/packages/consul-ui run: make test-node - frontend-build-oss: + frontend-build-ce: runs-on: ubuntu-latest env: JOBS: 2 @@ -64,27 +64,27 @@ jobs: working-directory: ./ui run: make deps - - name: Ember Build OSS - id: build-oss + - name: Ember Build CE + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci - - name: Upload OSS Frontend + - name: Upload CE Frontend uses: actions/upload-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist if-no-files-found: error - frontend-test-oss: + frontend-test-ce: runs-on: ubuntu-latest - needs: [frontend-build-oss] + needs: [frontend-build-ce] strategy: matrix: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 0 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -103,13 +103,13 @@ jobs: working-directory: ./ui run: make deps - - name: Download OSS Frontend + - name: Download CE Frontend uses: actions/download-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist - - name: Ember Test OSS + - name: Ember Test CE id: cache working-directory: ./ui/packages/consul-ui run: node_modules/.bin/ember exam --split=$EMBER_PARTITION_TOTAL --partition=${{ matrix.partition }} --path dist --silent -r xunit @@ -137,7 +137,7 @@ jobs: run: make deps - name: Ember Build ENT - id: build-oss + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci @@ -156,7 +156,7 @@ jobs: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 1 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -218,7 +218,7 @@ jobs: slack-failure-notification: runs-on: ubuntu-latest - needs: [frontend-test-oss, frontend-test-ent] + needs: [frontend-test-ce, frontend-test-ent] if: ${{ failure() }} steps: - name: Slack Notification diff --git a/.github/workflows/nightly-test-1.14.x.yaml b/.github/workflows/nightly-test-1.14.x.yaml index 86f48c37a144..9b310f59065d 100644 --- a/.github/workflows/nightly-test-1.14.x.yaml +++ b/.github/workflows/nightly-test-1.14.x.yaml @@ -42,7 +42,7 @@ jobs: working-directory: ./ui/packages/consul-ui run: make test-node - frontend-build-oss: + frontend-build-ce: runs-on: ubuntu-latest env: JOBS: 2 @@ -64,27 +64,27 @@ jobs: working-directory: ./ui run: make deps - - name: Ember Build OSS - id: build-oss + - name: Ember Build CE + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci - - name: Upload OSS Frontend + - name: Upload CE Frontend uses: actions/upload-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist if-no-files-found: error - frontend-test-oss: + frontend-test-ce: runs-on: ubuntu-latest - needs: [frontend-build-oss] + needs: [frontend-build-ce] strategy: matrix: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 0 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -103,13 +103,13 @@ jobs: working-directory: ./ui run: make deps - - name: Download OSS Frontend + - name: Download CE Frontend uses: actions/download-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist - - name: Ember Test OSS + - name: Ember Test CE id: cache working-directory: ./ui/packages/consul-ui run: node_modules/.bin/ember exam --split=$EMBER_PARTITION_TOTAL --partition=${{ matrix.partition }} --path dist --silent -r xunit @@ -137,7 +137,7 @@ jobs: run: make deps - name: Ember Build ENT - id: build-oss + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci @@ -156,7 +156,7 @@ jobs: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 1 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -218,7 +218,7 @@ jobs: slack-failure-notification: runs-on: ubuntu-latest - needs: [frontend-test-oss, frontend-test-ent] + needs: [frontend-test-ce, frontend-test-ent] if: ${{ failure() }} steps: - name: Slack Notification diff --git a/.github/workflows/nightly-test-1.15.x.yaml b/.github/workflows/nightly-test-1.15.x.yaml index 7fdc9247be72..9048abb4a04e 100644 --- a/.github/workflows/nightly-test-1.15.x.yaml +++ b/.github/workflows/nightly-test-1.15.x.yaml @@ -42,7 +42,7 @@ jobs: working-directory: ./ui/packages/consul-ui run: make test-node - frontend-build-oss: + frontend-build-ce: runs-on: ubuntu-latest env: JOBS: 2 @@ -64,27 +64,27 @@ jobs: working-directory: ./ui run: make deps - - name: Ember Build OSS - id: build-oss + - name: Ember Build CE + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci - - name: Upload OSS Frontend + - name: Upload CE Frontend uses: actions/upload-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist if-no-files-found: error - frontend-test-oss: + frontend-test-ce: runs-on: ubuntu-latest - needs: [frontend-build-oss] + needs: [frontend-build-ce] strategy: matrix: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 0 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -103,13 +103,13 @@ jobs: working-directory: ./ui run: make deps - - name: Download OSS Frontend + - name: Download CE Frontend uses: actions/download-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist - - name: Ember Test OSS + - name: Ember Test CE id: cache working-directory: ./ui/packages/consul-ui run: node_modules/.bin/ember exam --split=$EMBER_PARTITION_TOTAL --partition=${{ matrix.partition }} --path dist --silent -r xunit @@ -137,7 +137,7 @@ jobs: run: make deps - name: Ember Build ENT - id: build-oss + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci @@ -156,7 +156,7 @@ jobs: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 1 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -218,7 +218,7 @@ jobs: slack-failure-notification: runs-on: ubuntu-latest - needs: [frontend-test-oss, frontend-test-ent] + needs: [frontend-test-ce, frontend-test-ent] if: ${{ failure() }} steps: - name: Slack Notification diff --git a/.github/workflows/nightly-test-integrations.yml b/.github/workflows/nightly-test-integrations.yml new file mode 100644 index 000000000000..d4432f27ba86 --- /dev/null +++ b/.github/workflows/nightly-test-integrations.yml @@ -0,0 +1,326 @@ +# Copyright (c) HashiCorp, Inc. +# SPDX-License-Identifier: MPL-2.0 + +name: Nightly test-integrations + +on: + schedule: + # Run nightly at 12AM UTC/8PM EST/5PM PST + - cron: '* 0 * * *' + workflow_dispatch: {} + +env: + TEST_RESULTS_DIR: /tmp/test-results + TEST_RESULTS_ARTIFACT_NAME: test-results + CONSUL_LICENSE: ${{ secrets.CONSUL_LICENSE }} + GOTAGS: ${{ endsWith(github.repository, '-enterprise') && 'consulent' || '' }} + GOTESTSUM_VERSION: "1.10.1" + CONSUL_BINARY_UPLOAD_NAME: consul-bin + # strip the hashicorp/ off the front of github.repository for consul + CONSUL_LATEST_IMAGE_NAME: ${{ endsWith(github.repository, '-enterprise') && github.repository || 'hashicorp/consul' }} + GOPRIVATE: github.com/hashicorp # Required for enterprise deps + +jobs: + setup: + runs-on: ubuntu-latest + name: Setup + outputs: + compute-small: ${{ steps.runners.outputs.compute-small }} + compute-medium: ${{ steps.runners.outputs.compute-medium }} + compute-large: ${{ steps.runners.outputs.compute-large }} + compute-xl: ${{ steps.runners.outputs.compute-xl }} + enterprise: ${{ steps.runners.outputs.enterprise }} + steps: + - name: Checkout code + uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + ref: ${{ inputs.branch }} + - id: runners + run: .github/scripts/get_runner_classes.sh + + dev-build: + needs: [setup] + uses: ./.github/workflows/reusable-dev-build.yml + with: + runs-on: ${{ needs.setup.outputs.compute-large }} + repository-name: ${{ github.repository }} + uploaded-binary-name: 'consul-bin' + secrets: + elevated-github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }} + + generate-envoy-job-matrices: + needs: [setup] + runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} + name: Generate Envoy Job Matrices + outputs: + envoy-matrix: ${{ steps.set-matrix.outputs.envoy-matrix }} + steps: + - name: Checkout code + uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + ref: ${{ inputs.branch }} + - name: Generate Envoy Job Matrix + id: set-matrix + env: + # this is further going to multiplied in envoy-integration tests by the + # other dimensions in the matrix. Currently TOTAL_RUNNERS would be + # multiplied by 8 based on these values: + # envoy-version: ["1.24.10", "1.25.9", "1.26.4", "1.27.0"] + # xds-target: ["server", "client"] + TOTAL_RUNNERS: 4 + JQ_SLICER: '[ inputs ] | [_nwise(length / $runnercount | floor)]' + run: | + NUM_RUNNERS=$TOTAL_RUNNERS + NUM_DIRS=$(find ./test/integration/connect/envoy -mindepth 1 -maxdepth 1 -type d | wc -l) + + if [ "$NUM_DIRS" -lt "$NUM_RUNNERS" ]; then + echo "TOTAL_RUNNERS is larger than the number of tests/packages to split." + NUM_RUNNERS=$((NUM_DIRS-1)) + fi + # fix issue where test splitting calculation generates 1 more split than TOTAL_RUNNERS. + NUM_RUNNERS=$((NUM_RUNNERS-1)) + { + echo -n "envoy-matrix=" + find ./test/integration/connect/envoy -maxdepth 1 -type d -print0 \ + | xargs -0 -n 1 basename \ + | jq --raw-input --argjson runnercount "$NUM_RUNNERS" "$JQ_SLICER" \ + | jq --compact-output 'map(join("|"))' + } >> "$GITHUB_OUTPUT" + + envoy-integration-test: + runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} + needs: + - setup + - generate-envoy-job-matrices + - dev-build + permissions: + id-token: write # NOTE: this permission is explicitly required for Vault auth. + contents: read + strategy: + fail-fast: false + matrix: + envoy-version: ["1.23.12", "1.24.10", "1.25.9", "1.26.4"] + xds-target: ["server", "client"] + test-cases: ${{ fromJSON(needs.generate-envoy-job-matrices.outputs.envoy-matrix) }} + env: + ENVOY_VERSION: ${{ matrix.envoy-version }} + XDS_TARGET: ${{ matrix.xds-target }} + AWS_LAMBDA_REGION: us-west-2 + steps: + - name: Checkout code + uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + ref: ${{ inputs.branch }} + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 + with: + go-version-file: 'go.mod' + + - name: fetch binary + uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 + with: + name: '${{ env.CONSUL_BINARY_UPLOAD_NAME }}' + path: ./bin + - name: restore mode+x + run: chmod +x ./bin/consul + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@2a1a44ac4aa01993040736bd95bb470da1a38365 # v2.9.0 + + - name: Docker build + run: docker build -t consul:local -f ./build-support/docker/Consul-Dev.dockerfile ./bin + + - name: Envoy Integration Tests + env: + GOTESTSUM_JUNITFILE: ${{ env.TEST_RESULTS_DIR }}/results.xml + GOTESTSUM_FORMAT: standard-verbose + COMPOSE_INTERACTIVE_NO_CLI: 1 + LAMBDA_TESTS_ENABLED: "true" + # tput complains if this isn't set to something. + TERM: ansi + run: | + # shellcheck disable=SC2001 + echo "Running $(sed 's,|, ,g' <<< "${{ matrix.test-cases }}" |wc -w) subtests" + # shellcheck disable=SC2001 + sed 's,|,\n,g' <<< "${{ matrix.test-cases }}" + go run gotest.tools/gotestsum@v${{env.GOTESTSUM_VERSION}} \ + --debug \ + --rerun-fails \ + --rerun-fails-report=/tmp/gotestsum-rerun-fails \ + --jsonfile /tmp/jsonfile/go-test.log \ + --packages=./test/integration/connect/envoy \ + -- -timeout=30m -tags integration -run="TestEnvoy/(${{ matrix.test-cases }})" + + # NOTE: ENT specific step as we store secrets in Vault. + - name: Authenticate to Vault + if: ${{ endsWith(github.repository, '-enterprise') }} + id: vault-auth + run: vault-auth + + # NOTE: ENT specific step as we store secrets in Vault. + - name: Fetch Secrets + if: ${{ endsWith(github.repository, '-enterprise') }} + id: secrets + uses: hashicorp/vault-action@v2.5.0 + with: + url: ${{ steps.vault-auth.outputs.addr }} + caCertificate: ${{ steps.vault-auth.outputs.ca_certificate }} + token: ${{ steps.vault-auth.outputs.token }} + secrets: | + kv/data/github/${{ github.repository }}/datadog apikey | DATADOG_API_KEY; + + - name: prepare datadog-ci + if: ${{ !endsWith(github.repository, '-enterprise') }} + run: | + curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" --output "/usr/local/bin/datadog-ci" + chmod +x /usr/local/bin/datadog-ci + + - name: upload coverage + # do not run on forks + if: github.event.pull_request.head.repo.full_name == github.repository + env: + DATADOG_API_KEY: "${{ endsWith(github.repository, '-enterprise') && env.DATADOG_API_KEY || secrets.DATADOG_API_KEY }}" + DD_ENV: ci + run: datadog-ci junit upload --service "$GITHUB_REPOSITORY" $TEST_RESULTS_DIR/results.xml + + upgrade-integration-test: + runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} + needs: + - setup + - dev-build + permissions: + id-token: write # NOTE: this permission is explicitly required for Vault auth. + contents: read + strategy: + fail-fast: false + matrix: + consul-version: ["1.14", "1.15", "1.16"] + env: + CONSUL_LATEST_VERSION: ${{ matrix.consul-version }} + ENVOY_VERSION: "1.24.6" + steps: + - name: Checkout code + uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + ref: ${{ inputs.branch }} + # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. + - name: Setup Git + if: ${{ endsWith(github.repository, '-enterprise') }} + run: git config --global url."https://${{ secrets.ELEVATED_GITHUB_TOKEN }}:@github.com".insteadOf "https://github.com" + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 + with: + go-version-file: 'go.mod' + - run: go env + + # Get go binary from workspace + - name: fetch binary + uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 + with: + name: '${{ env.CONSUL_BINARY_UPLOAD_NAME }}' + path: . + - name: restore mode+x + run: chmod +x consul + - name: Build consul:local image + run: docker build -t ${{ env.CONSUL_LATEST_IMAGE_NAME }}:local -f ./build-support/docker/Consul-Dev.dockerfile . + - name: Build consul-envoy:latest-version image + id: buildConsulEnvoyLatestImage + continue-on-error: true + run: docker build -t consul-envoy:latest-version --build-arg CONSUL_IMAGE=docker.mirror.hashicorp.services/${{ env.CONSUL_LATEST_IMAGE_NAME }}:${{ env.CONSUL_LATEST_VERSION }} --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets + - name: Retry Build consul-envoy:latest-version image + if: steps.buildConsulEnvoyLatestImage.outcome == 'failure' + run: docker build -t consul-envoy:latest-version --build-arg CONSUL_IMAGE=docker.mirror.hashicorp.services/${{ env.CONSUL_LATEST_IMAGE_NAME }}:${{ env.CONSUL_LATEST_VERSION }} --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets + - name: Build consul-envoy:target-version image + id: buildConsulEnvoyTargetImage + continue-on-error: true + run: docker build -t consul-envoy:target-version --build-arg CONSUL_IMAGE=${{ env.CONSUL_LATEST_IMAGE_NAME }}:local --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets + - name: Retry Build consul-envoy:target-version image + if: steps.buildConsulEnvoyTargetImage.outcome == 'failure' + run: docker build -t consul-envoy:target-version --build-arg CONSUL_IMAGE=${{ env.CONSUL_LATEST_IMAGE_NAME }}:local --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets + - name: Build sds image + run: docker build -t consul-sds-server ./test/integration/connect/envoy/test-sds-server/ + - name: Configure GH workaround for ipv6 loopback + if: ${{ !endsWith(github.repository, '-enterprise') }} + run: | + cat /etc/hosts && echo "-----------" + sudo sed -i 's/::1 *localhost ip6-localhost ip6-loopback/::1 ip6-localhost ip6-loopback/g' /etc/hosts + cat /etc/hosts + - name: Upgrade Integration Tests + run: | + mkdir -p "${{ env.TEST_RESULTS_DIR }}" + cd ./test/integration/consul-container/test/upgrade + docker run --rm ${{ env.CONSUL_LATEST_IMAGE_NAME }}:local consul version + go run gotest.tools/gotestsum@v${{env.GOTESTSUM_VERSION}} \ + --raw-command \ + --format=short-verbose \ + --debug \ + --rerun-fails=2 \ + --packages="./..." \ + -- \ + go test \ + -p=4 \ + -tags "${{ env.GOTAGS }}" \ + -timeout=30m \ + -json \ + ./... \ + --follow-log=false \ + --target-image ${{ env.CONSUL_LATEST_IMAGE_NAME }} \ + --target-version local \ + --latest-image docker.mirror.hashicorp.services/${{ env.CONSUL_LATEST_IMAGE_NAME }} \ + --latest-version "${{ env.CONSUL_LATEST_VERSION }}" + ls -lrt + env: + # this is needed because of incompatibility between RYUK container and GHA + GOTESTSUM_JUNITFILE: ${{ env.TEST_RESULTS_DIR }}/results.xml + GOTESTSUM_FORMAT: standard-verbose + COMPOSE_INTERACTIVE_NO_CLI: 1 + # tput complains if this isn't set to something. + TERM: ansi + # NOTE: ENT specific step as we store secrets in Vault. + - name: Authenticate to Vault + if: ${{ endsWith(github.repository, '-enterprise') }} + id: vault-auth + run: vault-auth + + # NOTE: ENT specific step as we store secrets in Vault. + - name: Fetch Secrets + if: ${{ endsWith(github.repository, '-enterprise') }} + id: secrets + uses: hashicorp/vault-action@v2.5.0 + with: + url: ${{ steps.vault-auth.outputs.addr }} + caCertificate: ${{ steps.vault-auth.outputs.ca_certificate }} + token: ${{ steps.vault-auth.outputs.token }} + secrets: | + kv/data/github/${{ github.repository }}/datadog apikey | DATADOG_API_KEY; + + - name: prepare datadog-ci + if: ${{ !endsWith(github.repository, '-enterprise') }} + run: | + curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" --output "/usr/local/bin/datadog-ci" + chmod +x /usr/local/bin/datadog-ci + + - name: upload coverage + # do not run on forks + if: github.event.pull_request.head.repo.full_name == github.repository + env: + DATADOG_API_KEY: "${{ endsWith(github.repository, '-enterprise') && env.DATADOG_API_KEY || secrets.DATADOG_API_KEY }}" + DD_ENV: ci + run: datadog-ci junit upload --service "$GITHUB_REPOSITORY" $TEST_RESULTS_DIR/results.xml + + + test-integrations-success: + needs: + - setup + - dev-build + - generate-envoy-job-matrices + - envoy-integration-test + - upgrade-integration-test + runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} + if: ${{ always() }} + steps: + - name: evaluate upstream job results + run: | + # exit 1 if failure or cancelled result for any upstream job + if printf '${{ toJSON(needs) }}' | grep -E -i '\"result\": \"(failure|cancelled)\"'; then + printf "Tests failed or workflow cancelled:\n\n${{ toJSON(needs) }}" + exit 1 + fi diff --git a/.github/workflows/nightly-test-main.yaml b/.github/workflows/nightly-test-main.yaml index 3fc316a1a354..16160175b681 100644 --- a/.github/workflows/nightly-test-main.yaml +++ b/.github/workflows/nightly-test-main.yaml @@ -42,7 +42,7 @@ jobs: working-directory: ./ui/packages/consul-ui run: make test-node - frontend-build-oss: + frontend-build-ce: runs-on: ubuntu-latest env: JOBS: 2 @@ -64,27 +64,27 @@ jobs: working-directory: ./ui run: make deps - - name: Ember Build OSS - id: build-oss + - name: Ember Build CE + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci - - name: Upload OSS Frontend + - name: Upload CE Frontend uses: actions/upload-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist if-no-files-found: error - frontend-test-oss: + frontend-test-ce: runs-on: ubuntu-latest - needs: [frontend-build-oss] + needs: [frontend-build-ce] strategy: matrix: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 0 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -103,13 +103,13 @@ jobs: working-directory: ./ui run: make deps - - name: Download OSS Frontend + - name: Download CE Frontend uses: actions/download-artifact@v3 with: - name: frontend-oss-${{ env.BRANCH_NAME }} + name: frontend-ce-${{ env.BRANCH_NAME }} path: ./ui/packages/consul-ui/dist - - name: Ember Test OSS + - name: Ember Test CE id: cache working-directory: ./ui/packages/consul-ui run: node_modules/.bin/ember exam --split=$EMBER_PARTITION_TOTAL --partition=${{ matrix.partition }} --path dist --silent -r xunit @@ -137,7 +137,7 @@ jobs: run: make deps - name: Ember Build ENT - id: build-oss + id: build-ce working-directory: ./ui/packages/consul-ui run: make build-ci @@ -156,7 +156,7 @@ jobs: partition: [ 1, 2, 3, 4 ] env: CONSUL_NSPACES_ENABLED: 1 - EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CI test summary + EMBER_TEST_REPORT: test-results/report-ce.xml #outputs test report for CI test summary EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam steps: - uses: actions/checkout@v2 @@ -218,7 +218,7 @@ jobs: slack-failure-notification: runs-on: ubuntu-latest - needs: [frontend-test-oss, frontend-test-ent] + needs: [frontend-test-ce, frontend-test-ent] if: ${{ failure() }} steps: - name: Slack Notification diff --git a/.github/workflows/reusable-dev-build.yml b/.github/workflows/reusable-dev-build.yml index d134508584c0..2db9670655e4 100644 --- a/.github/workflows/reusable-dev-build.yml +++ b/.github/workflows/reusable-dev-build.yml @@ -1,4 +1,4 @@ -name: reusable-dev-build +name: reusable-dev-build on: workflow_call: @@ -25,12 +25,12 @@ jobs: build: runs-on: ${{ fromJSON(inputs.runs-on) }} steps: - - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # pin@v3.3.0 + - uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2 # NOTE: This step is specifically needed for ENT. It allows us to access the required private HashiCorp repos. - name: Setup Git if: ${{ endsWith(inputs.repository-name, '-enterprise') }} run: git config --global url."https://${{ secrets.elevated-github-token }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 with: go-version-file: 'go.mod' - name: Build @@ -38,7 +38,7 @@ jobs: GOARCH: ${{ inputs.goarch }} run: make dev # save dev build to pass to downstream jobs - - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # pin@v3.1.2 + - uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2 with: name: ${{inputs.uploaded-binary-name}} path: ./bin/consul diff --git a/.github/workflows/reusable-unit-split.yml b/.github/workflows/reusable-unit-split.yml index 3b9eb489590a..6c13670e742e 100644 --- a/.github/workflows/reusable-unit-split.yml +++ b/.github/workflows/reusable-unit-split.yml @@ -46,7 +46,7 @@ on: required: true env: TEST_RESULTS: /tmp/test-results - GOTESTSUM_VERSION: 1.8.2 + GOTESTSUM_VERSION: "1.10.1" GOARCH: ${{inputs.go-arch}} TOTAL_RUNNERS: ${{inputs.runner-count}} CONSUL_LICENSE: ${{secrets.consul-license}} diff --git a/.github/workflows/reusable-unit.yml b/.github/workflows/reusable-unit.yml index e7caaae8d6a1..c49a6291fa2e 100644 --- a/.github/workflows/reusable-unit.yml +++ b/.github/workflows/reusable-unit.yml @@ -33,6 +33,10 @@ on: required: false type: string default: "" + go-version: + required: false + type: string + default: "" secrets: elevated-github-token: required: true @@ -42,7 +46,7 @@ on: required: true env: TEST_RESULTS: /tmp/test-results - GOTESTSUM_VERSION: 1.8.2 + GOTESTSUM_VERSION: "1.10.1" GOARCH: ${{inputs.go-arch}} CONSUL_LICENSE: ${{secrets.consul-license}} GOTAGS: ${{ inputs.go-tags}} @@ -57,7 +61,13 @@ jobs: - name: Setup Git if: ${{ endsWith(inputs.repository-name, '-enterprise') }} run: git config --global url."https://${{ secrets.elevated-github-token }}:@github.com".insteadOf "https://github.com" - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # pin@v3.5.0 + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 + if: ${{ inputs.go-version != '' }} + with: + go-version: ${{ inputs.go-version }} + cache: true + - uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1 + if: ${{ inputs.go-version == '' }} with: go-version-file: 'go.mod' cache: true diff --git a/.github/workflows/test-integrations.yml b/.github/workflows/test-integrations.yml index 38b6a44cbe01..57ee1d7b6620 100644 --- a/.github/workflows/test-integrations.yml +++ b/.github/workflows/test-integrations.yml @@ -19,15 +19,35 @@ env: TEST_RESULTS_ARTIFACT_NAME: test-results CONSUL_LICENSE: ${{ secrets.CONSUL_LICENSE }} GOTAGS: ${{ endsWith(github.repository, '-enterprise') && 'consulent' || '' }} - GOTESTSUM_VERSION: "1.9.0" + GOTESTSUM_VERSION: "1.10.1" CONSUL_BINARY_UPLOAD_NAME: consul-bin # strip the hashicorp/ off the front of github.repository for consul - CONSUL_LATEST_IMAGE_NAME: ${{ endsWith(github.repository, '-enterprise') && github.repository || 'consul' }} + CONSUL_LATEST_IMAGE_NAME: ${{ endsWith(github.repository, '-enterprise') && github.repository || 'hashicorp/consul' }} + GOPRIVATE: github.com/hashicorp # Required for enterprise deps + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.ref }} + cancel-in-progress: true jobs: + conditional-skip: + runs-on: ubuntu-latest + name: Get files changed and conditionally skip CI + outputs: + trigger-ci: ${{ steps.read-files.outputs.trigger-ci }} + steps: + - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3 + with: + fetch-depth: 0 + - name: Get changed files + id: read-files + run: ./.github/scripts/filter_changed_files_go_test.sh + setup: + needs: [conditional-skip] runs-on: ubuntu-latest name: Setup + if: needs.conditional-skip.outputs.trigger-ci == 'true' outputs: compute-small: ${{ steps.runners.outputs.compute-small }} compute-medium: ${{ steps.runners.outputs.compute-medium }} @@ -43,7 +63,7 @@ jobs: needs: [setup] uses: ./.github/workflows/reusable-dev-build.yml with: - runs-on: ${{ needs.setup.outputs.compute-xl }} + runs-on: ${{ needs.setup.outputs.compute-large }} repository-name: ${{ github.repository }} uploaded-binary-name: 'consul-bin' secrets: @@ -239,7 +259,7 @@ jobs: # this is further going to multiplied in envoy-integration tests by the # other dimensions in the matrix. Currently TOTAL_RUNNERS would be # multiplied by 8 based on these values: - # envoy-version: ["1.23.10", "1.24.8", "1.25.7", "1.26.2"] + # envoy-version: ["1.23.12", "1.24.10", "1.25.9", "1.26.4"] # xds-target: ["server", "client"] TOTAL_RUNNERS: 4 JQ_SLICER: '[ inputs ] | [_nwise(length / $runnercount | floor)]' @@ -262,7 +282,7 @@ jobs: } >> "$GITHUB_OUTPUT" envoy-integration-test: - runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} + runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} needs: - setup - generate-envoy-job-matrices @@ -273,7 +293,7 @@ jobs: strategy: fail-fast: false matrix: - envoy-version: ["1.23.10", "1.24.8", "1.25.7", "1.26.2"] + envoy-version: ["1.26.4"] xds-target: ["server", "client"] test-cases: ${{ fromJSON(needs.generate-envoy-job-matrices.outputs.envoy-matrix) }} env: @@ -354,7 +374,7 @@ jobs: run: datadog-ci junit upload --service "$GITHUB_REPOSITORY" $TEST_RESULTS_DIR/results.xml compatibility-integration-test: - runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} + runs-on: ${{ fromJSON(needs.setup.outputs.compute-large) }} needs: - setup - dev-build @@ -458,124 +478,10 @@ jobs: DATADOG_API_KEY: "${{ endsWith(github.repository, '-enterprise') && env.DATADOG_API_KEY || secrets.DATADOG_API_KEY }}" DD_ENV: ci run: datadog-ci junit upload --service "$GITHUB_REPOSITORY" $TEST_RESULTS_DIR/results.xml - - upgrade-integration-test: - runs-on: ${{ fromJSON(needs.setup.outputs.compute-xl) }} - needs: - - setup - - dev-build - permissions: - id-token: write # NOTE: this permission is explicitly required for Vault auth. - contents: read - strategy: - fail-fast: false - matrix: - consul-version: [ "1.14", "1.15"] - env: - CONSUL_LATEST_VERSION: ${{ matrix.consul-version }} - ENVOY_VERSION: "1.24.6" - steps: - - uses: actions/checkout@24cb9080177205b6e8c946b17badbe402adc938f # v3.4.0 - - uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # v3.5.0 - with: - go-version-file: 'go.mod' - - run: go env - - # Get go binary from workspace - - name: fetch binary - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2 - with: - name: '${{ env.CONSUL_BINARY_UPLOAD_NAME }}' - path: . - - name: restore mode+x - run: chmod +x consul - - name: Build consul:local image - run: docker build -t ${{ env.CONSUL_LATEST_IMAGE_NAME }}:local -f ./build-support/docker/Consul-Dev.dockerfile . - - name: Build consul-envoy:latest-version image - id: buildConsulEnvoyLatestImage - continue-on-error: true - run: docker build -t consul-envoy:latest-version --build-arg CONSUL_IMAGE=docker.mirror.hashicorp.services/${{ env.CONSUL_LATEST_IMAGE_NAME }}:${{ env.CONSUL_LATEST_VERSION }} --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets - - name: Retry Build consul-envoy:latest-version image - if: steps.buildConsulEnvoyLatestImage.outcome == 'failure' - run: docker build -t consul-envoy:latest-version --build-arg CONSUL_IMAGE=docker.mirror.hashicorp.services/${{ env.CONSUL_LATEST_IMAGE_NAME }}:${{ env.CONSUL_LATEST_VERSION }} --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets - - name: Build consul-envoy:target-version image - id: buildConsulEnvoyTargetImage - continue-on-error: true - run: docker build -t consul-envoy:target-version --build-arg CONSUL_IMAGE=${{ env.CONSUL_LATEST_IMAGE_NAME }}:local --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets - - name: Retry Build consul-envoy:target-version image - if: steps.buildConsulEnvoyTargetImage.outcome == 'failure' - run: docker build -t consul-envoy:target-version --build-arg CONSUL_IMAGE=${{ env.CONSUL_LATEST_IMAGE_NAME }}:local --build-arg ENVOY_VERSION=${{ env.ENVOY_VERSION }} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets - - name: Build sds image - run: docker build -t consul-sds-server ./test/integration/connect/envoy/test-sds-server/ - - name: Configure GH workaround for ipv6 loopback - if: ${{ !endsWith(github.repository, '-enterprise') }} - run: | - cat /etc/hosts && echo "-----------" - sudo sed -i 's/::1 *localhost ip6-localhost ip6-loopback/::1 ip6-localhost ip6-loopback/g' /etc/hosts - cat /etc/hosts - - name: Upgrade Integration Tests - run: | - mkdir -p "${{ env.TEST_RESULTS_DIR }}" - cd ./test/integration/consul-container/test/upgrade - docker run --rm ${{ env.CONSUL_LATEST_IMAGE_NAME }}:local consul version - go run gotest.tools/gotestsum@v${{env.GOTESTSUM_VERSION}} \ - --raw-command \ - --format=short-verbose \ - --debug \ - --rerun-fails=2 \ - --packages="./..." \ - -- \ - go test \ - -p=4 \ - -tags "${{ env.GOTAGS }}" \ - -timeout=30m \ - -json ./... \ - --target-image ${{ env.CONSUL_LATEST_IMAGE_NAME }} \ - --target-version local \ - --latest-image docker.mirror.hashicorp.services/${{ env.CONSUL_LATEST_IMAGE_NAME }} \ - --latest-version "${{ env.CONSUL_LATEST_VERSION }}" - ls -lrt - env: - # this is needed because of incompatibility between RYUK container and GHA - GOTESTSUM_JUNITFILE: ${{ env.TEST_RESULTS_DIR }}/results.xml - GOTESTSUM_FORMAT: standard-verbose - COMPOSE_INTERACTIVE_NO_CLI: 1 - # tput complains if this isn't set to something. - TERM: ansi - # NOTE: ENT specific step as we store secrets in Vault. - - name: Authenticate to Vault - if: ${{ endsWith(github.repository, '-enterprise') }} - id: vault-auth - run: vault-auth - - # NOTE: ENT specific step as we store secrets in Vault. - - name: Fetch Secrets - if: ${{ endsWith(github.repository, '-enterprise') }} - id: secrets - uses: hashicorp/vault-action@v2.5.0 - with: - url: ${{ steps.vault-auth.outputs.addr }} - caCertificate: ${{ steps.vault-auth.outputs.ca_certificate }} - token: ${{ steps.vault-auth.outputs.token }} - secrets: | - kv/data/github/${{ github.repository }}/datadog apikey | DATADOG_API_KEY; - - - name: prepare datadog-ci - if: ${{ !endsWith(github.repository, '-enterprise') }} - run: | - curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" --output "/usr/local/bin/datadog-ci" - chmod +x /usr/local/bin/datadog-ci - - - name: upload coverage - # do not run on forks - if: github.event.pull_request.head.repo.full_name == github.repository - env: - DATADOG_API_KEY: "${{ endsWith(github.repository, '-enterprise') && env.DATADOG_API_KEY || secrets.DATADOG_API_KEY }}" - DD_ENV: ci - run: datadog-ci junit upload --service "$GITHUB_REPOSITORY" $TEST_RESULTS_DIR/results.xml test-integrations-success: needs: + - conditional-skip - setup - dev-build - nomad-integration-test @@ -583,9 +489,8 @@ jobs: - generate-envoy-job-matrices - envoy-integration-test - compatibility-integration-test - - upgrade-integration-test runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }} - if: ${{ always() }} + if: always() && needs.conditional-skip.outputs.trigger-ci == 'true' steps: - name: evaluate upstream job results run: | diff --git a/.golangci.yml b/.golangci.yml index bac9b716a3b4..3e45ef464c14 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -51,11 +51,11 @@ issues: - linters: [unparam] text: "`(t|resp|req|entMeta)` is unused" - # Temp ignore everything in _oss(_test).go and _ent(_test).go. Many of these + # Temp ignore everything in _ce(_test).go and _ent(_test).go. Many of these # could use underscore to ignore the unused arguments, but the "always returns" - # issue will likely remain in oss, and will need to be excluded. + # issue will likely remain in CE, and will need to be excluded. - linters: [unparam] - path: "(_oss.go|_oss_test.go|_ent.go|_ent_test.go)" + path: "(_ce.go|_ce_test.go|_ent.go|_ent_test.go)" linters-settings: govet: diff --git a/.release/security-scan.hcl b/.release/security-scan.hcl index 3352890686ae..3029b33273b0 100644 --- a/.release/security-scan.hcl +++ b/.release/security-scan.hcl @@ -11,6 +11,7 @@ binary { secrets = false go_modules = false osv = true + # TODO(spatel): CE refactor oss_index = true nvd = true } diff --git a/CHANGELOG.md b/CHANGELOG.md index ef4edc700404..a7a2c6e24a8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,143 @@ +## 1.16.1 (August 8, 2023) + +KNOWN ISSUES: + +* connect: Consul versions 1.16.0 and 1.16.1 may have issues when a snapshot restore is performed and the servers are hosting xDS streams. When this bug triggers, it will cause Envoy to incorrectly populate upstream endpoints. This bug only impacts agent-less service mesh and should be fixed in Consul 1.16.2 by [GH-18636](https://github.com/hashicorp/consul/pull/18636). + +SECURITY: + +* Update `golang.org/x/net` to v0.13.0 to address [CVE-2023-3978](https://nvd.nist.gov/vuln/detail/CVE-2023-3978). [[GH-18358](https://github.com/hashicorp/consul/issues/18358)] +* Upgrade golang.org/x/net to address [CVE-2023-29406](https://nvd.nist.gov/vuln/detail/CVE-2023-29406) [[GH-18186](https://github.com/hashicorp/consul/issues/18186)] +* Upgrade to use Go 1.20.6. +This resolves [CVE-2023-29406](https://github.com/advisories/GHSA-f8f7-69v5-w4vx)(`net/http`) for uses of the standard library. +A separate change updates dependencies on `golang.org/x/net` to use `0.12.0`. [[GH-18190](https://github.com/hashicorp/consul/issues/18190)] +* Upgrade to use Go 1.20.7. +This resolves vulnerability [CVE-2023-29409](https://nvd.nist.gov/vuln/detail/CVE-2023-29409)(`crypto/tls`). [[GH-18358](https://github.com/hashicorp/consul/issues/18358)] + +FEATURES: + +* cli: `consul members` command uses `-filter` expression to filter members based on bexpr. [[GH-18223](https://github.com/hashicorp/consul/issues/18223)] +* cli: `consul operator raft list-peers` command shows the number of commits each follower is trailing the leader by to aid in troubleshooting. [[GH-17582](https://github.com/hashicorp/consul/issues/17582)] +* cli: `consul watch` command uses `-filter` expression to filter response from checks, services, nodes, and service. [[GH-17780](https://github.com/hashicorp/consul/issues/17780)] +* reloadable config: Made enable_debug config reloadable and enable pprof command to work when config toggles to true [[GH-17565](https://github.com/hashicorp/consul/issues/17565)] +* ui: consul version is displayed in nodes list with filtering and sorting based on versions [[GH-17754](https://github.com/hashicorp/consul/issues/17754)] + +IMPROVEMENTS: + +* Fix some typos in metrics docs [[GH-18080](https://github.com/hashicorp/consul/issues/18080)] +* acl: added builtin ACL policy that provides global read-only access (builtin/global-read-only) [[GH-18319](https://github.com/hashicorp/consul/issues/18319)] +* acl: allow for a single slash character in policy names [[GH-18319](https://github.com/hashicorp/consul/issues/18319)] +* connect: Add capture group labels from Envoy cluster FQDNs to Envoy exported metric labels [[GH-17888](https://github.com/hashicorp/consul/issues/17888)] +* connect: Improve transparent proxy support for virtual services and failovers. [[GH-17757](https://github.com/hashicorp/consul/issues/17757)] +* connect: update supported envoy versions to 1.23.12, 1.24.10, 1.25.9, 1.26.4 [[GH-18303](https://github.com/hashicorp/consul/issues/18303)] +* debug: change default setting of consul debug command. now default duration is 5ms and default log level is 'TRACE' [[GH-17596](https://github.com/hashicorp/consul/issues/17596)] +* extensions: Improve validation and error feedback for `property-override` builtin Envoy extension [[GH-17759](https://github.com/hashicorp/consul/issues/17759)] +* hcp: Add dynamic configuration support for the export of server metrics to HCP. [[GH-18168](https://github.com/hashicorp/consul/issues/18168)] +* hcp: Removes requirement for HCP to provide a management token [[GH-18140](https://github.com/hashicorp/consul/issues/18140)] +* http: GET API `operator/usage` endpoint now returns node count +cli: `consul operator usage` command now returns node count [[GH-17939](https://github.com/hashicorp/consul/issues/17939)] +* mesh: Expose remote jwks cluster configuration through jwt-provider config entry [[GH-17978](https://github.com/hashicorp/consul/issues/17978)] +* mesh: Stop jwt providers referenced by intentions from being deleted. [[GH-17755](https://github.com/hashicorp/consul/issues/17755)] +* ui: the topology view now properly displays services with mixed connect and non-connect instances. [[GH-13023](https://github.com/hashicorp/consul/issues/13023)] +* xds: Explicitly enable WebSocket connection upgrades in HTTP connection manager [[GH-18150](https://github.com/hashicorp/consul/issues/18150)] + +BUG FIXES: + +* Fix a bug that wrongly trims domains when there is an overlap with DC name. [[GH-17160](https://github.com/hashicorp/consul/issues/17160)] +* api-gateway: fix race condition in proxy config generation when Consul is notified of the bound-api-gateway config entry before it is notified of the api-gateway config entry. [[GH-18291](https://github.com/hashicorp/consul/issues/18291)] +* api: Fix client deserialization errors by marking new Enterprise-only prepared query fields as omit empty [[GH-18184](https://github.com/hashicorp/consul/issues/18184)] +* ca: Fixes a Vault CA provider bug where updating RootPKIPath but not IntermediatePKIPath would not renew leaf signing certificates [[GH-18112](https://github.com/hashicorp/consul/issues/18112)] +* connect/ca: Fixes a bug preventing CA configuration updates in secondary datacenters [[GH-17846](https://github.com/hashicorp/consul/issues/17846)] +* connect: **(Enterprise only)** Fix bug where intentions referencing sameness groups would not always apply to members properly. +* connect: Fix incorrect protocol config merging for transparent proxy implicit upstreams. [[GH-17894](https://github.com/hashicorp/consul/issues/17894)] +* connect: Removes the default health check from the `consul connect envoy` command when starting an API Gateway. +This health check would always fail. [[GH-18011](https://github.com/hashicorp/consul/issues/18011)] +* connect: fix a bug with Envoy potentially starting with incomplete configuration by not waiting enough for initial xDS configuration. [[GH-18024](https://github.com/hashicorp/consul/issues/18024)] +* gateway: Fixes a bug where envoy would silently reject RSA keys that are smaller than 2048 bits, +we now reject those earlier in the process when we validate the certificate. [[GH-17911](https://github.com/hashicorp/consul/issues/17911)] +* http: fixed API endpoint `PUT /acl/token/:AccessorID` (update token), no longer requires `AccessorID` in the request body. Web UI can now update tokens. [[GH-17739](https://github.com/hashicorp/consul/issues/17739)] +* mesh: **(Enterprise Only)** Require that `jwt-provider` config entries are created in the `default` namespace. [[GH-18325](https://github.com/hashicorp/consul/issues/18325)] +* snapshot: fix access denied and handle is invalid when we call snapshot save on windows - skip sync() for folders in windows in +https://github.com/rboyer/safeio/pull/3 [[GH-18302](https://github.com/hashicorp/consul/issues/18302)] +* xds: Prevent partial application of non-Required Envoy extensions in the case of failure. [[GH-18068](https://github.com/hashicorp/consul/issues/18068)] + +## 1.16.0 (June 26, 2023) + +KNOWN ISSUES: + +* connect: Consul versions 1.16.0 and 1.16.1 may have issues when a snapshot restore is performed and the servers are hosting xDS streams. When this bug triggers, it will cause Envoy to incorrectly populate upstream endpoints. This bug only impacts agent-less service mesh and should be fixed in Consul 1.16.2 by [GH-18636](https://github.com/hashicorp/consul/pull/18636). + +BREAKING CHANGES: + +* api: The `/v1/health/connect/` and `/v1/health/ingress/` endpoints now immediately return 403 "Permission Denied" errors whenever a token with insufficient `service:read` permissions is provided. Prior to this change, the endpoints returned a success code with an empty result list when a token with insufficient permissions was provided. [[GH-17424](https://github.com/hashicorp/consul/issues/17424)] +* peering: Removed deprecated backward-compatibility behavior. + Upstream overrides in service-defaults will now only apply to peer upstreams when the `peer` field is provided. + Visit the 1.16.x [upgrade instructions](https://developer.hashicorp.com/consul/docs/upgrading/upgrade-specific) for more information. [[GH-16957](https://github.com/hashicorp/consul/issues/16957)] + +SECURITY: + +* Bump Dockerfile base image to `alpine:3.18`. [[GH-17719](https://github.com/hashicorp/consul/issues/17719)] +* audit-logging: **(Enterprise only)** limit `v1/operator/audit-hash` endpoint to ACL token with `operator:read` privileges. + +FEATURES: + +* api: (Enterprise only) Add `POST /v1/operator/audit-hash` endpoint to calculate the hash of the data used by the audit log hash function and salt. +* cli: (Enterprise only) Add a new `consul operator audit hash` command to retrieve and compare the hash of the data used by the audit log hash function and salt. +* cli: Adds new command - `consul services export` - for exporting a service to a peer or partition [[GH-15654](https://github.com/hashicorp/consul/issues/15654)] +* connect: **(Consul Enterprise only)** Implement order-by-locality failover. +* mesh: Add new permissive mTLS mode that allows sidecar proxies to forward incoming traffic unmodified to the application. This adds `AllowEnablingPermissiveMutualTLS` setting to the mesh config entry and the `MutualTLSMode` setting to proxy-defaults and service-defaults. [[GH-17035](https://github.com/hashicorp/consul/issues/17035)] +* mesh: Support configuring JWT authentication in Envoy. [[GH-17452](https://github.com/hashicorp/consul/issues/17452)] +* server: **(Enterprise Only)** added server side RPC requests IP based read/write rate-limiter. [[GH-4633](https://github.com/hashicorp/consul/issues/4633)] +* server: **(Enterprise Only)** allow automatic license utilization reporting. [[GH-5102](https://github.com/hashicorp/consul/issues/5102)] +* server: added server side RPC requests global read/write rate-limiter. [[GH-16292](https://github.com/hashicorp/consul/issues/16292)] +* xds: Add `property-override` built-in Envoy extension that directly patches Envoy resources. [[GH-17487](https://github.com/hashicorp/consul/issues/17487)] +* xds: Add a built-in Envoy extension that inserts External Authorization (ext_authz) network and HTTP filters. [[GH-17495](https://github.com/hashicorp/consul/issues/17495)] +* xds: Add a built-in Envoy extension that inserts Wasm HTTP filters. [[GH-16877](https://github.com/hashicorp/consul/issues/16877)] +* xds: Add a built-in Envoy extension that inserts Wasm network filters. [[GH-17505](https://github.com/hashicorp/consul/issues/17505)] + +IMPROVEMENTS: + +* * api: Support filtering for config entries. [[GH-17183](https://github.com/hashicorp/consul/issues/17183)] +* * cli: Add `-filter` option to `consul config list` for filtering config entries. [[GH-17183](https://github.com/hashicorp/consul/issues/17183)] +* agent: remove agent cache dependency from service mesh leaf certificate management [[GH-17075](https://github.com/hashicorp/consul/issues/17075)] +* api: Enable setting query options on agent force-leave endpoint. [[GH-15987](https://github.com/hashicorp/consul/issues/15987)] +* audit-logging: **(Enterprise only)** enable error response and request body logging +* ca: automatically set up Vault's auto-tidy setting for tidy_expired_issuers when using Vault as a CA provider. [[GH-17138](https://github.com/hashicorp/consul/issues/17138)] +* ca: support Vault agent auto-auth config for Vault CA provider using AliCloud authentication. [[GH-16224](https://github.com/hashicorp/consul/issues/16224)] +* ca: support Vault agent auto-auth config for Vault CA provider using AppRole authentication. [[GH-16259](https://github.com/hashicorp/consul/issues/16259)] +* ca: support Vault agent auto-auth config for Vault CA provider using Azure MSI authentication. [[GH-16298](https://github.com/hashicorp/consul/issues/16298)] +* ca: support Vault agent auto-auth config for Vault CA provider using JWT authentication. [[GH-16266](https://github.com/hashicorp/consul/issues/16266)] +* ca: support Vault agent auto-auth config for Vault CA provider using Kubernetes authentication. [[GH-16262](https://github.com/hashicorp/consul/issues/16262)] +* command: Adds ACL enabled to status output on agent startup. [[GH-17086](https://github.com/hashicorp/consul/issues/17086)] +* command: Allow creating ACL Token TTL with greater than 24 hours with the -expires-ttl flag. [[GH-17066](https://github.com/hashicorp/consul/issues/17066)] +* connect: **(Enterprise Only)** Add support for specifying "Partition" and "Namespace" in Prepared Queries failover rules. +* connect: update supported envoy versions to 1.23.10, 1.24.8, 1.25.7, 1.26.2 [[GH-17546](https://github.com/hashicorp/consul/issues/17546)] +* connect: update supported envoy versions to 1.23.8, 1.24.6, 1.25.4, 1.26.0 [[GH-5200](https://github.com/hashicorp/consul/issues/5200)] +* fix metric names in /docs/agent/telemetry [[GH-17577](https://github.com/hashicorp/consul/issues/17577)] +* gateway: Change status condition reason for invalid certificate on a listener from "Accepted" to "ResolvedRefs". [[GH-17115](https://github.com/hashicorp/consul/issues/17115)] +* http: accept query parameters `datacenter`, `ap` (enterprise-only), and `namespace` (enterprise-only). Both short-hand and long-hand forms of these query params are now supported via the HTTP API (dc/datacenter, ap/partition, ns/namespace). [[GH-17525](https://github.com/hashicorp/consul/issues/17525)] +* systemd: set service type to notify. [[GH-16845](https://github.com/hashicorp/consul/issues/16845)] +* ui: Update alerts to Hds::Alert component [[GH-16412](https://github.com/hashicorp/consul/issues/16412)] +* ui: Update to use Hds::Toast component to show notifications [[GH-16519](https://github.com/hashicorp/consul/issues/16519)] +* ui: update from