This suite exercises a Cloud Foundry
deployment using the cf
CLI and curl
.
It is scoped to testing user-facing,
end-to-end features.
For example,
one test pushes an app with cf push
,
hits an endpoint on the app with curl
that causes it to crash,
and asserts that we see
a crash event in cf events
.
Tests that won't be introduced here include things like basic CRUD of an object in the Cloud Controller. Such tests belong with the component they relate to.
These tests are not intended for use against production systems. They're meant for acceptance environments used by people developing Cloud Foundry's releases. While these tests attempt to clean up after themselves, there's no guarantee that they won't change the state of your system in an undesirable way. For lightweight system tests that are safe to run against a production environment, please use the CF Smoke Tests.
NOTE: Because we want to parallelize execution, tests should be written to be executable independently. Tests should not depend on state in other tests, and should not modify the CF state in such a way as to impact other tests.
- Install golang >=
1.22
. Set up your golang development environment, per golang.org. - Install the
cf CLI
>=8.5.0
. Make sure that it is accessible in your$PATH
. - Install curl
- Check out a copy of
cf-acceptance-tests
. It uses Go modules, so there is no need to put it in$GOPATH
. - Install a running Cloud Foundry deployment to run these acceptance tests against. For example, bosh-lite.
All go
dependencies required by CATs
are vendored in the vendor
directory.
Make sure to have Golang 1.22+
In order to update a current dependency to a specific version, do the following:
cd cf-acceptance-tests
source .envrc
go get <import_path>@<revision_number>
go mod vendor
If you'd like to add a new dependency just run:
go mod tidy
go mod vendor
You must set the environment variable $CONFIG
which points to a JSON file
that contains several pieces of data
that will be used to configure the acceptance tests,
e.g. telling the tests how to target
your running Cloud Foundry deployment
and what tests to run.
The following can be pasted into a terminal
and will set up a sufficient $CONFIG
to run the core test suites
against a BOSH-Lite
deployment of CF.
See example-cats-config.json
for reference.
Only the following test groups are run by default:
include_apps
include_detect
include_routing
include_v3
include_app_syslog_tcp
api
: Cloud Controller API endpoint, without scheme (HTTP/S) specified.admin_user
: Name of a user in your CF instance with admin credentials. This admin user must have thedoppler.firehose
scope.admin_password
: Password of the admin user above.apps_domain
: A shared domain that tests can use to create subdomains that will route to applications also created in the tests, without scheme (HTTP/S) specified.skip_ssl_validation
: Set to true if using an invalid (e.g. self-signed) cert for traffic routed to your CF instance; this is generally always true for BOSH-Lite deployments of CF.
include_*
parameters are used to specify whether to skip tests based on how a deployment is configured.
-
include_app_syslog_tcp
: Flag to include the app syslog drain over TCP test group. -
include_apps
: Flag to include the apps test group. -
readiness_health_checks_enabled
: Defaults totrue
. Set to false if you are using an environment without readiness health checks. -
include_cnb
: Flag to include tests related to building apps using Cloud Native Buildpacks. Diego must be deployed and the CC API diego_cnb feature flag must be enabled for these tests to pass. -
include_container_networking
: Flag to include tests related to container networking. -
credhub_mode
: Valid values areassisted
ornon-assisted
. See below. -
credhub_location
: Location of CredHub instance; default ishttps://credhub.service.cf.internal:8844
-
credhub_client
: UAA client credential for Service Broker write access to CredHub (required for CredHub tests); default iscredhub_admin_client
. -
credhub_secret
: UAA client secret for Service Broker write access to CredHub (required for CredHub tests). -
include_deployments
: Flag to include tests for the cloud controller rolling deployments. V3 must also be enabled. -
include_detect
: Flag to include tests in the detect group. -
include_docker
: Flag to include tests related to running Docker apps on Diego. Diego must be deployed and the CC API diego_docker feature flag must be enabled for these tests to pass. -
include_http2_routing
: Flag to include the HTTP/2 Routing tests. -
include_internet_dependent
: Flag to include tests that require the deployment to have internet access. -
include_isolation_segments
: Flag to include isolation segment tests. -
include_private_docker_registry
: Flag to run tests that rely on a private docker image. See below. -
include_route_services
: Flag to include the route services tests. Diego must be deployed for these tests to pass. -
include_routing
: Flag to include the routing tests. -
include_routing_isolation_segments
: Flag to include routing isolation segments tests. See below. Cannot be run together with logging isolation segments tests. -
include_security_groups
: Flag to include tests for security groups. See below -
dynamic_asgs_enabled
: Defaults totrue
. Set to false if dynamic ASGs are disabled in the test environment. -
comma_delim_asgs_enabled
: Defaults tofalse
. Set to true if comma delimited ASG destinations are enabled in the test environment. -
include_services
: Flag to include test for the services API. -
include_service_instance_sharing
: Flag to include tests for service instance sharing between spaces.include_services
must be set for these tests to run. Theservice_instance_sharing
feature flag must also be enabled for these tests to pass. -
include_ssh
: Flag to include tests for Diego container ssh feature. -
include_sso
: Flag to include the services tests that integrate with Single Sign On.include_services
must also be set for tests to run. -
include_tasks
: Flag to include the v3 task tests.include_v3
must also be set for tests to run. The CC API task_creation feature flag must be enabled for these tests to pass. -
include_tcp_routing
: Flag to include the TCP Routing tests. These tests are equivalent to the TCP Routing tests from the Routing Acceptance Tests. -
include_user_provided_services
: Flag to include test for user-provided services. -
include_v3
: Flag to include tests for the v3 API. -
include_zipkin
: Flag to include tests for Zipkin tracing.include_routing
must also be set for tests to run. CF must be deployed withrouter.tracing.enable_zipkin
set for tests to pass. -
use_http
: Set to true if you would like CF Acceptance Tests to use HTTP when making api and application requests. (default is HTTPS) -
use_existing_organization
: Set to true when you need to specify an existing organization to use rather than creating a new organization. -
existing_organization
: Name of the existing organization to use. -
use_existing_user
: The admin user configured above will normally be used to create a temporary user (with lesser permissions) to perform actions (such as push applications) during tests, and then delete said user after the tests have run; set this totrue
if you want to use an existing user, configured via the following properties. -
keep_user_at_suite_end
: If using an existing user (see above), set this totrue
unless you are okay having your existing user being deleted at the end. You can also set this totrue
when not using an existing user if you want to leave the temporary user around for debugging purposes after the test teardown. -
existing_user
: Name of the existing user to use. -
existing_user_password
: Password for the existing user to use. -
artifacts_directory
: If set,cf
CLI trace output from test runs will be captured in files and placed in this directory. See below for more. -
default_timeout
: Default time (in seconds) to wait for polling assertions that wait for asynchronous results. -
cf_push_timeout
: Default time (in seconds) to wait forcf push
commands to succeed. -
long_curl_timeout
: Default time (in seconds) to wait for assertions thatcurl
slow endpoints of test applications. -
broker_start_timeout
(only relevant forservices
test group): Time (in seconds) to wait for service broker test app to start. -
async_service_operation_timeout
(only relevant for theservices
test group): Time (in seconds) to wait for an asynchronous service operation to complete. -
test_password
: Used to set the password for the test user. This may be needed if your CF installation has password policies. -
timeout_scale
: Used primarily to scale default timeouts for test setup and teardown actions (e.g. creating an org) as opposed to main test actions (e.g. pushing an app). -
isolation_segment_name
: Name of the isolation segment to use for the isolation segments test. -
isolation_segment_domain
: Domain that will route to the isolated router in the isolation segments and routing isolation segments tests. See below -
private_docker_registry_image
: Name of the private docker image to use when testing private docker registries. See below -
private_docker_registry_username
: Username to access the private docker repository. See below -
private_docker_registry_password
: Password to access the private docker repository. See below -
unallocated_ip_for_security_group
: An unused IP address in the private network used by CF. Defaults to 10.0.244.255. See below -
require_proxied_app_traffic
: Set this totrue
if Diego was configured to require proxied port mappings, i.e. ifcontainers.proxy.enable_unproxied_port_mappings
is set tofalse
. -
staticfile_buildpack_name
See below -
java_buildpack_name
See below -
ruby_buildpack_name
See below -
nginx_buildpack_name
See below -
nodejs_buildpack_name
See below -
go_buildpack_name
See below -
r_buildpack_name
See below -
binary_buildpack_name
See below -
cnb_nodejs_buildpack_name
See below -
include_windows
: Flag to include the tests that run against Windows cells. -
use_windows_test_task
: Flag to include the tasks tests on Windows cells. Default isfalse
. -
use_windows_context_path
: Flag to include the Windows context path routing tests. Default isfalse
. -
windows_stack
: Windows stack to run tests against. Must bewindows
. -
include_service_discovery
: Flag to include test for the service discovery. These tests useapps.internal
domain, which is the default incf-networking-release
. The internal domain is currently not configurable. -
stacks
: An array of stacks to test against. Default is[cflinuxfs4]
. -
include_volume_services
: Flag to include the tests for volume services. The following requirements must be met to run this suite: tcp-routing must be deployed. -
volume_service_name
: The name of the volume service provided by the volume service broker. -
volume_service_plan_name
: The name of the plan of the service provided by the volume service broker. -
volume_service_create_config
: The JSON configuration that is used when volume service is created.
Many tests specify a buildpack when pushing an app, so that on diego the app staging process completes in less time. The default names for the buildpacks are as follows; if you have buildpacks with different names, you can override them by setting different names:
staticfile_buildpack_name: staticfile_buildpack
java_buildpack_name: java_buildpack
ruby_buildpack_name: ruby_buildpack
nginx_buildpack_name: nginx_buildpack
nodejs_buildpack_name: nodejs_buildpack
go_buildpack_name: go_buildpack
r_buildpack_name: r_buildpack
binary_buildpack_name: binary_buildpack
hwc_buildpack_name: hwc_buildpack
For the Cloud Native Buildpacks lifecycle, you can override them by setting different names:
cnb_nodejs_buildpack_name: docker:https://gcr.io/paketo-buildpacks/nodejs:latest
The route_services
test group pushes applications which must be able to reach the load balancer of your Cloud Foundry deployment. This requires configuring application security groups to support this. Your deployment manifest should include the following data if you are running the route_services
group:
...
properties:
...
cc:
...
security_group_definitions:
- name: load_balancer
rules:
- protocol: all
destination: IP_OF_YOUR_LOAD_BALANCER # (e.g. 10.244.0.34 for a standard deployment of Cloud Foundry on BOSH-Lite)
default_running_security_groups: ["load_balancer"]
To run tests that exercise container networking and running application security groups, the include_security_groups
flags must be true.
The Windows ASG tests require an unallocated IP
on the private network used by the CF deployment,
set with the unallocated_ip_for_security_group
config value.
Environments created by bbl on public clouds
can use the default value of 10.0.244.255.
vSphere and Openstack environments may require a custom IP.
To run tests that exercise the use of credentials to access a private docker registry, the include_private_docker_registry
flag must be true, and the following config values must be provided:
private_docker_registry_image
private_docker_registry_username
private_docker_registry_password
These tests assume that the specified private docker image is a private version of the cloudfoundry/diego-docker-app:latest. To upload a private version to your DockerHub account, first create a private repository on DockerHub and log in to docker on the command line. Then run the following commands:
docker pull cloudfoundry/diego-docker-app:latest
docker tag cloudfoundry/diego-docker-app:latest <your-private-repo>:<some-tag>
docker push <your-private-repo>:<some-tag>
The value for the private_docker_registry_image
config value in this case would be ":".
To run tests that involve routing isolation segments, the following config values must be provided:
isolation_segment_name
isolation_segment_domain
Read the documentation here for further setup details.
non-assisted
mode means that apps are responsible for resolving Credhub refs for credentials. When the user binds a service to an app, the service broker will store a credential in Credhub and pass the ref back to the Cloud Controller. When the user restages the app, the Cloud Controller will pass the Credhub ref to the app in theVCAP_SERVICES
environment variable, at which point the app can make a request directly to Credhub to resolve the ref and obtain the credential. This mode is enabled whencc.credential_references.interpolate_service_bindings
is false -- which is the non-default configuration.assisted
mode means that the Credhub ref will be resolved before the app starts running. As before, when the user binds a service to an app, the service broker will store a credential in Credhub and pass the ref back to the Cloud Controller. This time, when the user restages the app, the Cloud Controller will pass the Credhub ref to the Diego runtime, at which point the launcher (from the buildpackapplifecycle or the dockerapplifecycle components) will resolve the Credhub ref, and store the credential inVCAP_SERVICES
for the app to consume. This mode is enabled whencc.credential_references.interpolate_service_bindings
is true -- which is the default configuration.
When a test fails, look for the test group name ([services]
in the example below) in the test output:
• Failure in Spec Setup (BeforeEach) [34.662 seconds]
[services] Service Instance Lifecycle
If you set a value for artifacts_directory
in your $CONFIG
file, then you will be able to capture cf
trace output from failed test runs, this output may be useful in cases where the normal test output is not enough to debug an issue. The cf
trace output for the tests in these specs will be found in CF-TRACE-Applications-*.txt
in the artifacts_directory
.
To execute tests according to your configuration, run the bin/test
script with $CONFIG
set to your
integration_config.json
file path.
It's possible to execute all test groups, and have tests run in parallel across multiple processes. This parallelization can significantly decrease CATs runtime.
However, be careful with this number, as it's effectively "how many apps to push at once", as nearly every example pushes an app.
Use the --procs
flag to set the number of parallel processes, e.g.:
./bin/test --procs=12
For reference here's how many parallel processes the Release Integration team runs with:
Foundation Type | # of Parallel Processes |
---|---|
Vanilla CF | 12 |
BOSH Lite | 4 |
If you are already familiar with CATs you probably know that there are many test groups. You may not wish to run all the tests in all contexts, and sometimes you may want to focus individual test groups to pinpoint a failure. To execute a specific group of acceptance tests, e.g. routing/
, edit your integration_config.json
file and set all include_*
values to false
except for include_routing
then run the following:
./bin/test
To execute tests in a single file use an FDescribe
block around the tests in that file:
var _ = AppsDescribe("Apps", func() {
FDescribe("Focused tests", func() { // Add this line here
// ... rest of file
}) // Close here
})
The test group names correspond to directory names.
To see verbose output from ginkgo
, use the -v
flag.
./bin/test -v
You can of course combine the -v
flag with the --procs=N
flag.
From Ginkgo 2.0, the default timeout for the entire test has been changed to 1 hour (refer to: Ginkgo 2.0 Migration Guide - Timeout Behavior).
Tests may run longer than an hour depending on their environment and parallelism, with the possibility of failure.
Adjust the test timeout as needed with the --timeout
flag:
./bin/test --timeout=24h
For individual timeouts, such as the timeout for cf push
, refer to Test Configuration.
Test Group Name | Description |
---|---|
app_syslog_tcp |
Tests the ability to configure an app syslog drain listener. |
apps |
Tests the core functionalities of Cloud Foundry: staging, running, logging, routing, buildpacks, etc. This test group should always pass against a sound Cloud Foundry deployment. |
credhub |
Tests CredHub-delivered Secure Service credentials in the service binding. CredHub configuration is required to run these tests. In addition to selecting a credhub_mode , credhub_client and credhub_secret values are required for these tests. |
cnb |
Tests our ability to use cloud native buildpacks. |
detect |
Tests the ability of the platform to detect the correct buildpack for compiling an application if no buildpack is explicitly specified. |
docker |
Tests our ability to run docker containers on Diego and that we handle docker metadata correctly. |
internet_dependent |
Tests the feature of being able to specify a buildpack via a Github URL. As such, this depends on your Cloud Foundry application containers having access to the Internet. You should take into account the configuration of the network into which you've deployed your Cloud Foundry, as well as any security group settings applied to application containers. |
isolation_segments |
This test group requires that Diego be deployed with a minimum of 2 cells. One of those cells must have been deployed with a placement_tag . If the deployment has been deployed with a routing isolation segment, isolation_segment_domain must also be set. For more information, please refer to the Isolation Segments documentation. |
route_services |
Tests the Route Services feature of Cloud Foundry. |
routing |
This package contains routing specific acceptance tests (context paths, wildcards, SSL termination, sticky sessions, and zipkin tracing). |
routing_isolation_segments |
Tests that requests to isolated apps are only routed through isolated routers, and vice versa. It requires all of the setup for the isolation segments test suite. Additionally, a minimum of two Gorouter instances must be deployed. One instance must be configured with the property routing_table_sharding_mode: shared-and-segments . The other instance must have the properties routing_table_sharding_mode: segments and isolation_segments: [YOUR_PLACEMENT_TAG_HERE] . The isolation_segment_name in the CATs properties must match the placement_tag and isolation_segment .isolation_segment_domain must be set and traffic to that domain should go to the isolated router. |
security_groups |
Tests the Security Groups feature of Cloud Foundry. |
service_discovery |
Tests the Service Discovery feature for applications running on Cloud Foundry. |
services |
Tests various features related to services, e.g. registering a service broker via the service broker API. Some of these tests exercise special integrations, such as Single Sign-On authentication; you may wish to run some tests in this package but selectively skip others if you haven't configured the required integrations (by setting include_sso parameter to false in your configuration). |
ssh |
Tests communication with Diego apps via ssh, scp, and sftp. |
tasks |
Tests Cloud Foundry's Tasks feature. |
tcp_routing |
Tests TCP Routing Feature of Cloud Foundry. You need to make sure you've set up a TCP domain tcp.<SYSTEM_DOMAIN> as described here. If you are using bbl (BOSH Bootloader), TCP domain is set up for you automatically. |
user_provided_services |
Tests features related to creating and binding user-provided services for holding app credentials securely. |
v3 |
This test group contains tests for the next-generation v3 Cloud Controller API. |
volume_services |
Tests the Volume Services feature of Cloud Foundry. |
This repository uses go mod
to manage go
dependencies.
All go
dependencies required by CATs are vendored in the vendor
directory.
When making changes to the test suite that bring in additional go
packages,
you should use the following workflow:
If you can use the most recent version of a dependency use go mod tidy
,
otherwise use go get <dependency>@<version>
. Both of these require go modules
to be enabled via the envrc. Finally use go mod vendor
to add the
dependencies to the vendor
directory.
For tools and assets, please use the helpers/assets/tools.go via the go mod tool workflow
For additional information refer to the official wiki and the official examples repo
Although the default branch for this repository is main
, we ask that all
pull requests be made against the develop
branch. Please run the unit tests
and make sure they are passing before submitting. Use ./bin/run_units
to run
these unit tests.
Note: it is necessary to run the tests from the root of the repo.
There are a number of conventions we recommend developers of CF acceptance tests adopt:
- When pushing an app:
- set the memory requirement, and use the suite's
DEFAULT_MEMORY_LIMIT
(DEFAULT_WINDOWS_MEMORY_LIMIT
for tests in thewindows
directory) unless the test specifically needs to test a different value, - set the buildpack unless the test specifically needs to test the case where a buildpack is unspecified, and use one of
Config.GetRubyBuildpackName()
,Config.GetJavaBuildpackName()
, etc. unless the test specifically needs to use a buildpack name or URL specific to the test,
For example:
Expect(cf.Cf("push", appName,
"-b", buildpackName, // specify buildpack
"-m", DEFAULT_MEMORY_LIMIT, // specify memory limit
"-d", Config.AppsDomain, // specify app domain
).Wait(Config.CfPushTimeoutDuration())).To(Exit(0))
-
Delete all resources that are created, e.g. apps, routes, quotas, etc. This is in order to leave the system in the same state it was found in. For example, to delete apps and their associated routes:
Expect(cf.Cf("delete", myAppName, "-f", "-r").Wait()).To(Exit(0))
-
Specifically for apps, before tearing them down, print the app guid and recent application logs. There is a helper method
AppReport
provided in theapp_helpers
package for this purpose.AfterEach(func() { app_helpers.AppReport(appName) })
-
Document the purpose of your test groups in this repo's README.md. This is especially important when changing the explicit behavior of existing test groups or adding new test groups.
-
Document all changes to the config object in this repo's README.md.
-
If you add a test that requires a new minimum
cf
CLI version, update theminCliVersion
incats_suite_test.go
.