kube2e

Kubernetes End-to-End tests

This directory contains tests that install each of the 3 Gloo Edge flavors (gateway, ingress, and knative) in a Kubernetes cluster, and run end-to-end tests against them.

Note: All commands should be run from the root directory of the Gloo repository

Background

Kubernetes may be relied on for scheduling, persistence or security. These tests validate that Gloo Edge can successfully operate within a Kubernetes cluster.

How do the tests work?

1. Install Gloo Edge in Kubernetes cluster using Helm
2. Apply Gloo resources using Kubernetes resource clients
3. Execute requests against the Envoy proxy and confirm the expected response. This validates that the Gloo resources have been picked up by the controllers, were been translated correctly into Envoy configuration, the configuration was sent to the Envoy proxy, and the proxy behaves appropriately.

CI

These tests are run by a GitHub action as part of our CI pipeline.

If a test fails, you can retry it from a browser window. If you do this, please make sure to comment on the Pull Request with a link to the failed logs for debugging purposes.

Nightly runs

Tests are also run on a schedule via another GitHub action. The nightly tests use the latest release - specified with the RELEASED_VERSION environment variable.

Extra considerations for running from released builds

The GetTestHelper util method handles installing gloo from either a local or released build. When testing released builds, tests that interact directly with the helm chart need to download the chart using the version stored in testHelper.GetChartVersion()

Adding new tests

The list of tests to run during CI and nightly builds is provided in kube-e2e-test-type matrices in the github workflows. A new test can be added to one or both lists of tests.

Local Development

Setup

For these tests to run, we require the following conditions:

• Gloo Edge Helm chart archive be present in the _test folder,
• glooctl be built in the_output folder
• kind cluster set up and loaded with the images to be installed by the helm chart

Use the CI Install Script

ci/deploy-to-kind-cluster.sh gets run in CI to setup the test environment for the above requirements. It accepts a number of environment variables, to control the creation of a kind cluster and deployment of Gloo resources to that kind cluster.

Example:

CLUSTER_NAME=solo-test-cluster CLUSTER_NODE_VERSION=v1.25.3 VERSION=v1.0.0-solo-test ci/deploy-to-kind-cluster.sh

Verify Your Setup

Before running your tests, it's worthwhile to verify that a cluster was created, and the proper images have been loaded.

CLUSTER_NAME=solo-test-cluster make kind-list-images

You should see the list of images in the cluster, including the ones you just uploaded

Common Setup Errors

Error: validation: chart.metadata.version "solo" is invalid
In newer versions of helm (>3.5), the version used to build the helm chart (ie the VERSION env variable), needs to respect semantic versioning. This error implies that the version provided does not.

Prepend a valid semver to avoid the error. (ie kind can become 0.0.0-kind1)

Run Tests

To run the regression tests, your kubeconfig file must point to a running Kubernetes cluster. kubectl config current-context should run kind-<CLUSTER_NAME>

Use the Make Target

Use the same command that CI relies on:

KUBE2E_TESTS=<test-to-run> make run-ci-regression-tests

Test Environment Variables

The below table contains the environment variables that can be used to configure the test execution.

Common Test Errors

getting Helm chart version: expected a single entry with name [gloo], found: 5
The test helm charts are written to the _test directory, with the index.yaml file containing references to all available charts. The tests require that this file contain only 1 entry. Delete the other entries manually, or run make clean to delete this folder entirely, and then re-build the test helm chart.