vertica-kubernetes

This repository contains the code for a Kubernetes operator that manages Vertica Analytic Database. The operator uses a custom resource definition (CRD) to automate administrative tasks for a Vertica database.

To deploy the operator and a Kubernetes cluster in a local test environment that requires minimal resources, see DEVELOPER.

Prerequisites

• Resources to deploy Kubernetes objects
• Kubernetes (version 1.21.1+)
• kubectl (version 1.21.1+)
• helm (version 3.5.0+)

Installing the CRD

Install the CustomResourceDefinition with a YAML manifest:

kubectl apply -f https://github.com/vertica/vertica-kubernetes/releases/download/v1.1.0/verticadbs.vertica.com-crd.yaml

The operator Helm chart will install the CRD if it is not currently installed.

TLS Requirements

The operator includes an admission controller, a REST endpoint within Kubernetes that uses a webhook to verify that proposed changes to the custom resource are allowed. Kubernetes requires that all webhooks accept TLS certificates. Choose one of the following options to manage TLS certificates for your custom resource:

• cert-manager
• Custom certificates

cert-manager

cert-manager has built-in support in Kubernetes, and generates and manages your certificates. Install the cert-manager with kubectl apply.

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml

Installation might take a few minutes. For steps on how to verify that the install is complete, see the cert-manager documentation.

Custom Certificates

Use custom certificates for more control over your data encryption if your custom resource requires multiple certificates for different client or storage connections.

For detailed configuration instructions, see Installing the Helm Charts in the Vertica documentation. For details about the available configuration options, see the helm parameters.

Installing the Operator

Vertica bundles the operator and admission controller in a Helm chart. Run the following commands to download and install the chart:

helm repo add vertica-charts https://vertica.github.io/charts
helm repo update
helm install vdb-op vertica-charts/verticadb-operator

You can install only one instance of the chart in a namespace. The operator monitors CRs that were defined in its namespace only.

The feature gate NamespaceDefaultLabelName is required for the webhook to work. It is enabled by default on Kubernetes 1.21.0. This feature will add a label 'kubernetes.io/metadata.name=nsName' to each namespace, which is needed in order to use the namespaceSelector in the webhook config. If you are on an older version of Kubernetes, you must manually add the label in order for the webhook to work.

Deploying Vertica

After the operator is installed and running, create a Vertica deployment by generating a custom resource (CR). The operator adds watches to the API server so that it gets notified whenever a CR in the same namespace changes.

Launching a new Vertica deployment with the operator is simple. Below is the minimal required CR configuration:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: vertica-sample
spec:
  communal:
    path: "s3://<bucket-name>/<key-name>"
    endpoint: http://path/to/endpoint
    credentialSecret: s3-creds    
  subclusters:
    - name: defaultsubcluster
      size: 3

In the previous example configuration:

• communal.path must be a bucket that already exists and is empty.
• communal.endpoint is the location that serves the bucket.
• communal.credentialSecret is a Secret in the same namespace that has the access key and secret to authenticate the endpoint. The Secret must use keys with names accesskey and secretkey.
• You must specify at least one subcluster and it must have a name value. If size is omitted, it is set to 3.

After this manifest is applied, the operator creates the necessary objects in Kubernetes, sets up the config directory in each pod, and creates an Eon Mode database in the communal path.

There are many parameters available to fine-tune the deployment. For a complete list, see Parameters. For a detailed custom resource example, see Creating a Custom Resource.

Vertica License

By default, we use the Community Edition (CE) license if no license is provided. The CE license limits the number pods in a cluster to 3, and the dataset size to 1TB. Use your own license to extend the cluster past these limits.

To use your own license, add it to a Secret in the same namespace as the operator. The following command copies the license into a Secret named license:

kubectl create secret generic license --from-file=license.key=/path/to/license.key

Next, specify the name of the Secret in the CR by populating the licenseSecret field:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: vertica-sample
spec:
  licenseSecret: license
  communal:
    path: "s3://<bucket-name>/<key-name>"
    endpoint: http://path/to/endpoint
    credentialSecret: s3-creds   
  subclusters:
    - name: defaultsubcluster
      size: 3

The license is installed automatically if it is set when the CR was initially created. If it is added at a later time, then you must install the license manually with admintools. When a license Secret is specified, the contents of the Secret are mounted as files in /home/dbadmin/licensing/mnt. For example, the Secret created in the previous commands has the following directory in each pod created by the CR:

[dbadmin@demo-sc1-0 ~]$ ls /home/dbadmin/licensing/mnt
license.key

Scale Up/Down

We offer two strategies for scaling, each one improves different types of performance.

• To increase the performance of complex, long-running queries, add nodes to an existing subcluster.
• To increase the throughput of multiple short-term queries (often called "dashboard queries"), improve your cluster's parallelism by adding additional subclusters.

Each subcluster is enumerated in the CR in a list format, and each has a size property. To scale up, add a new subcluster to the CR, or increase an existing subcluster's size.

Below is a sample CRD that defines three subclusters named 'main', 'analytics', and 'ml':

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: vertica-sample
spec:
  communal:
    path: "s3://nimbusdb/mspilchen"
    credentialSecret: s3-creds
    endpoint: http://minio
  subclusters:
    - name: main
      size: 3
      isPrimary: true
    - name: analytics
      size: 5
      isPrimary: false
    - name: ml
      size: 2
      isPrimary: false

After the operator reconciles these changes to the CR, the deployment has a combined total of 10 pods. Each subcluster is deployed as a StatefulSet with its own service object.

To scale down, modify the CR by removing a subcluster from the list element, or decreasing the size property. The admission webhook prevents invalid state transitions, such as 0 subclusters or only secondary subclusters.

The operator automatically rebalances database shards whenever subclusters are added or removed.

Client Connectivity

Each subcluster has a service object for client connections. The service load balances traffic across the pods in the subcluster. Clients connect to the Vertica cluster through one of the subcluster service objects, depending on which subcluster the client is targeting.

For example, suppose we have the following CR:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: verticadb-sample
  namespace: default
spec:
  communal:
    credentialSecret: s3-auth
    path: s3://nimbusdb/db
    endpoint: http://minio
  subclusters:
  - name: defaultsubcluster
    serviceType: ClusterIP
    size: 3
  - name: ml
    serviceType: ClusterIP
    size: 2
  - name: analytics
    serviceType: ClusterIP
    size: 2

This CR uses the following Kubernetes objects:

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/verticadb-sample-analytics              ClusterIP   10.96.21.26     <none>        5433/TCP,5444/TCP   21m
service/verticadb-sample                        ClusterIP   None            <none>        22/TCP              21m
service/verticadb-sample-defaultsubcluster      ClusterIP   10.96.162.159   <none>        5433/TCP,5444/TCP   12h
service/verticadb-sample-ml                     ClusterIP   10.96.71.15     <none>        5433/TCP,5444/TCP   27m
 
NAME                                                  READY   AGE
statefulset.apps/verticadb-sample-analytics           2/2     21m
statefulset.apps/verticadb-sample-defaultsubcluster   3/3     12h
statefulset.apps/verticadb-sample-ml                  2/2     27m

Each service object has its own full qualified domain name (FQDN). The naming convention for each service object is <vdbName>-<subclusterName>. Clients can direct their connections to: verticadb-sample-analytics, verticadb-sample-defaultsubcluster, or verticadb-sample-ml. The actual pod that the client connects with is load balanced.

The previous example includes a headless service object named service/verticadb-sample, which matches the name of the Vertica database. This object exists to provide DNS name resolution for individual pods, and is not intended for client connectivity.

All of the service objects listed above are of type ClusterIP. This is the default service type, and it provides load balancing for connections within the Kubernetes cluster. Use the subclusters[i].serviceType parameter to specify NodePort or LoadBalancer to allow connections from outside of the Kubernetes cluster.

Migrating an Existing Database into Kubernetes

The operator can migrate an existing database into Kubernetes. The operator revives an existing database into a set of Kubernetes objects that mimics the setup of the database. We provide vdb-gen, a standalone program that you can run against a live database to create the CR.

1. The following command runs vdb-gen and generates a CR based on the existing database. The output is written to stdout and redirected to a YAML-formatted file:

   $ vdb-gen --password secret --name mydb 10.44.10.1 vertdb > vdb.yaml
   

   The previous command uses the following flags and values:

   • password: The existing database superuser secret password.
   • name: The new VerticaDB CR object,
   • 10.44.10.1: The IP address of the existing database.
   • vertdb: The name of the existing Eon Mode database.
   • vdb.yaml: The YAML-formatted file that contains the generated custom resource definition.

2. Use admintools -t stop_db to stop the database that currently exists.

3. Apply the manifest that was generated by the CR generator:

   $ kubectl apply -f vdb.yaml
   verticadb.vertica.com/mydb created
   

4. Wait for the operator to construct the StatefulSet, install Vertica in each pod, and run revive. Each of these steps generate events in kubectl. You can use the describe command to see the events for the verticadb:

   $ kubectl describe vdb mydb
   

Upgrading Vertica

The idiomatic way to upgrade in Kubernetes is a rolling update model, which means the new image container is rolled out to the cluster. At any given time, some pods are running the old version, and some are running the new version.

However, Vertica does not support a cluster running mixed releases. The documented steps to upgrade Vertica are summarized below:

1. Stop the entire cluster.
2. Update the RPM at each host.
3. Start the cluster.

Vertica on Kubernetes uses the workflow described in the preceding steps. This is triggered whenever the .spec.image changes in the CR. When the operator detects this, it will enter the upgrade mode, logging events of its progress for monitoring purposes.

Upgrade the Vertica server version and use the kubectl command line tool to monitor the progress. The operator indicates when an upgrade is in progress or complete with the UpgradeInProgress status condition.

1. Update the .spec.image in the CR. This can be driven by a helm upgrade if you have the CR in a chart, or a simple patch of the CR:

   $ kubectl patch verticadb vert-cluster --type=merge --patch '{"spec": {"image": "vertica/vertica-k8s:11.1.1-0"}}'
   

2. Wait for the operator to acknowledge this change and enter the upgrade mode:

   $ kubectl wait --for=condition=UpgradeInProgress=True vdb/vert-cluster –-timeout=180s
   

3. Wait for the operator to leave the upgrade mode:

   $ kubectl wait --for=condition=UpgradeInProgress=True vdb/cluster-name –-timeout=180s
   

You can monitor what part of the upgrade the operator is in by looking at the events it generates.

$ kubectl describe vdb cluster-name
...<snip>...
Events:
  Type    Reason                   Age    From                Message
  ----    ------                   ----   ----                -------
  Normal  ImageChangeStart         5m10s  verticadb-operator  Vertica server image change has been initiated to 'vertica-k8s:11.0.1-0'
  Normal  ClusterShutdownStarted   5m12s  verticadb-operator  Calling 'admintools -t stop_db'
  Normal  ClusterShutdownSucceeded 4m08s  verticadb-operator  Successfully called 'admintools -t stop_db' and it took 56.22132s
  Normal  ClusterRestartStarted    4m25s  verticadb-operator  Calling 'admintools -t start_db' to restart the cluster
  Normal  ClusterRestartSucceeded  25s    verticadb-operator  Successfully called 'admintools -t start_db' and it took 240s
  Normal  ImageChangeSucceeded     5s     verticadb-operator  Vertica server image change has completed successfully.

Vertica recommends that upgrade paths be incremental – meaning you upgrade to each intermediate major and minor release. The operator assumes the images chosen are following this path and doesn't try to validate it.

Persistence

Each pod uses a PV to store local data. The PV is mounted in the container at /home/dbadmin/local-data.

Select a recommended storage format type as the fsType value for your StorageClass.

The local-data directory contains the following subdirectories:

• /home/dbadmin/local-data/<uid>/data/: Stores the local catalogs and any temporary files. There is a symlink to this path to the local.dataPath parameter. Default: /data.
• /home/dbadmin/local-data/<uid>/depot/: Stores the depot for the local node. There is a symlink to this path to the local.depotPath parameter. Default:/depot.
• /home/dbadmin/local-data/<uid>/config/: This is a symlink to /opt/vertica/config. This allows the contents of the configuration directory to persist between restarts.
• /home/dbadmin/local-data/<uid>/log/: This is a symlink to /opt/vertica/log. This allows the log files to persist between restarts.

NOTE: <uid> is the unique ID of the VerticaDB resource that is assigned by Kubernetes.

By default, the PV is selected using the default storage class. Use the local.storageClass parameter to select a specific storage class.

Parameters

The following table describes each configurable parameter in the VerticaDB CRD and their defaults:

kubectl create secret generic s3-creds --from-literal=accesskey=accesskey --from-literal=secretkey=secretkey

communal:  credentialSecret: s3-creds

sidecars:  - name: vlogger    image: image:tag    volumeMounts:      - name: my-custom-vol        mountPath: /path/to/custom-volume

subclusters:  - name: sc1    affinity:      podAntiAffinity:        requiredDuringSchedulingIgnoredDuringExecution:        - labelSelector:            matchExpressions:            - key: app.kubernetes.io/name            operator: In            values:            - vertica          topologyKey: "kubernetes.io/hostname"

subclusters:  - name: sc1    nodeSelector:      disktype: ssd      region: us-east

subclusters:  - name: defaultsubcluster    resources:      requests:        memory: "96Gi"        cpu: 32      limits:        memory: "96Gi"        cpu: 32

db:  superuserSecretPassword: su-passwd 

Additional Details

For additional details on the internals of Vertica, see the Vertica Documentation.

Developers

For details about setting up an environment to develop and run tests, see the developer instructions.

Licensing

vertica-kubernetes is open source code and is under the Apache 2.0 license, but it requires that you install the Vertica server RPM. If you do not have a Vertica server RPM, you can use the free Vertica Community Edition (CE) server RPM. The Vertica Community Edition server RPM is not an open source project, but it is free with certain limits on capacity. For more information about these limitations, see the Vertica Community Edition End User License Agreement.