vertica-kubernetes

This repository contains the code for a Kubernetes operator that manages Vertica Analytic Database. The operator uses a custom resource definition (CRD) to automate administrative tasks for a Vertica database.

Prerequisites

• Kubernetes (version 1.19.3+)
• kubectl (version 1.19.3+).
• helm (version 3.5.0+)

Installing the CRD

Install the CustomResourceDefinition with a YAML manifest:

$ kubectl apply -f https://github.com/vertica/vertica-kubernetes/releases/download/v1.0.0/verticadbs.vertica.com-crd.yaml

Both the webhook and operator Helm chart install the CRD if it is not currently installed.

Installing the Operator

Install the operator with a Helm chart to manage a Vertica database. Run the following commands to download and install the chart:

$ helm repo add vertica-charts https://vertica.github.io/charts
$ helm repo update
$ helm install vdb-op vertica-charts/verticadb-operator

You can install only one instance of the chart in a namespace. The operator monitors CRs that were defined in its namespace only.

Installing the Webhook

A separate install is required to install a webhook for an admission controller. An admission controller is a REST endpoint that you set up within Kubernetes that verifies proposed changes to the custom resource are allowed. Running with the admission controller is optional, but it is highly encouraged to prevent simple errors from being made when modifying the custom resource.

Because Kubernetes requires that the webhook accept TLS certificates, a certificate must be set up prior to installing the webhook. Vertica recommends cert-manager to manage certificates for your Kubernetes resources. Install the cert-manager with kubectl apply.

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

It can take a few minutes for cert-manager install to complete. For steps on how to verify that the install is complete, see the cert-manager documentation.

After cert-manager completes, install the webhook:

$ helm repo add vertica-charts https://vertica.github.io/charts
$ helm repo update
$ helm install -n vertica --create-namespace vdb-webhook vertica-charts/verticadb-webhook

The webhook is cluster-scoped. It is installed into only one namespace (vertica) and is used by operators installed in any namespaces.

Deploying Vertica

After the operator is installed and running, create a Vertica deployment by generating a custom resource (CR). The operator adds watches to the API server so that it gets notified whenever a CR in the same namespace changes.

Launching a new Vertica deployment with the operator is simple. Below is the minimal required CR configuration:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: vertica-sample
spec:
  communal:
    path: "s3://<bucket-name>/<key-name>"
    endpoint: http://path/to/endpoint
    credentialSecret: s3-creds    
  subclusters:
    - name: defaultsubcluster
      size: 3

In the previous example configuration:

• communal.path must be a bucket that already exists and is empty.
• communal.endpoint is the location that serves the bucket.
• communal.credentialSecret is a Secret in the same namespace that has the access key and secret to authenticate the endpoint. The Secret must use keys with names accesskey and secretkey.
• You must specify at least one subcluster and it must have a name value. If size is omitted, it is set to 3.

After this manifest is applied, the operator creates the necessary objects in Kubernetes, sets up the config directory in each pod, and creates an Eon Mode database in the communal path.

There are many parameters available to fine-tune the deployment. For a complete list, see Parameters.

Vertica License

By default, we use the Community Edition (CE) license if no license is provided. The CE license limits the number pods in a cluster to 3, and the dataset size to 1TB. Use your own license to extend the cluster past these limits.

To use your own license, add it to a secret in the same namespace as the operator. The following command copies the license into a secret named license:

$ kubectl create secret generic license –from-file=license.key=/path/to/license.key

Next, specify the name of the secret in the CR by populating the licenseSecret field:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: vertica-sample
spec:
  licenseSecret: license
  communal:
    path: "s3://<bucket-name>/<key-name>"
    endpoint: http://path/to/endpoint
    credentialSecret: s3-creds   
  subclusters:
    - name: defaultsubcluster
      size: 3

The license is installed automatically if it is set when the CR was initially created. If it is added at a later time, then you must install the license manually with admintools. When a license secret is specified, the contents of the secret are mounted as files in /home/dbadmin/licensing/mnt. For example, the secret created in the previous commands has the following directory in each pod created by the CR:

$ [dbadmin@demo-sc1-0 ~]$ ls /home/dbadmin/licensing/mnt
license.key

Scale Up/Down

We offer two strategies for scaling, each one improves different types of performance.

• To increase the performance of complex, long-running queries, add nodes to an existing subcluster.
• To increase the throughput of multiple short-term queries (often called "dashboard queries"), improve your cluster's parallelism by adding additional subclusters.

Each subcluster is enumerated in the CR in a list format, and each has a size property. To scale up, add a new subcluster to the CR, or increase an existing subcluster's size.

Below is a sample CRD that defines three subclusters named 'main', 'analytics', and 'ml':

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: vertica-sample
spec:
  communal:
    path: "s3://nimbusdb/mspilchen"
    credentialSecret: s3-creds
    endpoint: http://minio
  subclusters:
    - name: main
      size: 3
      isPrimary: true
    - name: analytics
      size: 5
      isPrimary: false
    - name: ml
      size: 2
      isPrimary: false

After the operator reconciles these changes to the CR, the deployment has a combined total of 10 pods. Each subcluster is deployed as a StatefulSet with its own service object.

To scale down, modify the CR by removing a subcluster from the list element, or decreasing the size property. The admission webhook prevents invalid state transitions, such as 0 subclusters or only secondary subclusters.

The operator automatically rebalances database shards whenever subclusters are added or removed.

Client Connectivity

Each subcluster has a service object for client connections. The service load balances traffic across the pods in the subcluster. Clients connect to the Vertica cluster through one of the subcluster service objects, depending on which subcluster the client is targeting.

For example, suppose we have the following CR:

apiVersion: vertica.com/v1beta1
kind: VerticaDB
metadata:
  name: verticadb-sample
  namespace: default
spec:
  communal:
    credentialSecret: s3-auth
    path: s3://nimbusdb/db
    endpoint: http://minio
  subclusters:
  - name: defaultsubcluster
    serviceType: ClusterIP
    size: 3
  - name: ml
    serviceType: ClusterIP
    size: 2
  - name: analytics
    serviceType: ClusterIP
    size: 2

This CR uses the following Kubernetes objects:

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/verticadb-sample-analytics              ClusterIP   10.96.21.26     <none>        5433/TCP,5444/TCP   21m
service/verticadb-sample                        ClusterIP   None            <none>        22/TCP              21m
service/verticadb-sample-defaultsubcluster      ClusterIP   10.96.162.159   <none>        5433/TCP,5444/TCP   12h
service/verticadb-sample-ml                     ClusterIP   10.96.71.15     <none>        5433/TCP,5444/TCP   27m
 
NAME                                                  READY   AGE
statefulset.apps/verticadb-sample-analytics           2/2     21m
statefulset.apps/verticadb-sample-defaultsubcluster   3/3     12h
statefulset.apps/verticadb-sample-ml                  2/2     27m

Each service object has its own full qualified domain name (FQDN). The naming convention for each service object is <vdbName>-<subclusterName>. Clients can direct their connections to: verticadb-sample-analytics, verticadb-sample-defaultsubcluster, or verticadb-sample-ml. The actual pod that the client connects with is load balanced.

The previous example includes a headless service object named service/verticadb-sample, which matches the name of the Vertica database. This object exists to provide DNS name resolution for individual pods, and is not intended for client connectivity.

All of the service objects listed above are of type ClusterIP. This is the default service type, and it provides load balancing for connections within the Kubernetes cluster. Use the subclusters[i].serviceType parameter to specify NodePort or LoadBalancer to allow connections from outside of the Kubernetes cluster.

Migrating an Existing Database into Kubernetes

The operator can migrate an existing database into Kubernetes. The operator revives an existing database into a set of Kubernetes objects that mimics the setup of the database. We provide vdb-gen, a standalone program that you can run against a live database to create the CR.

1. The following command runs vdb-gen and generates a CR based on the existing database. The output is written to stdout and redirected to a YAML-formatted file:

   $ vdb-gen --password secret --name mydb 10.44.10.1 vertdb > vdb.yaml
   

   The previous command uses the following flags and values:

   • password: The existing database superuser secret password.
   • name: The new VerticaDB CR object,
   • 10.44.10.1: The IP address of the existing database.
   • vertdb: The name of the existing Eon Mode database.
   • vdb.yaml: The YAML-formatted file that contains the generated custom resource definition.

2. Use admintools -t stop_db to stop the database that currently exists.

3. Apply the manifest that was generated by the CR generator:

   $ kubectl apply -f vdb.yaml
   verticadb.vertica.com/mydb created
   

4. Wait for the operator to construct the StatefulSet, install Vertica in each pod, and run revive. Each of these steps generate events in kubectl. You can use the describe command to see the events for the verticadb:

   $ kubectl describe vdb mydb
   

Upgrading Vertica

The idiomatic way to upgrade in Kubernetes is a rolling update model, which means the new image container is rolled out to the cluster. At any given time, some pods are running the old version, and some are running the new version.

However, Vertica does not support a cluster running mixed releases. The documented steps to upgrade Vertica are summarized below:

1. Stop the entire cluster.
2. Update the RPM at each host.
3. Start the cluster.

Vertica on Kubernetes uses the workflow described in the preceding steps. Because the operator's main purpose is to ensure that Vertica is always running, there is a special autoRestartVertica parameter in the CR that forces the operator to skip its monitoring of the Vertica process. This allows cluster-wide operations like stopping the cluster without operator interference.

The following steps are included automated in the scripts/upgrade-vertica.sh script. To manually upgrade Vertica:

1. Set autoRestartVertica to false in the CRD. This tells the operator to avoid checking if the Vertica process is running:

   $ kubectl patch verticadb vert-cluster --type=merge --patch '{"spec": {"autoRestartVertica": false}}'
   

2. Wait for the operator to acknowledge this state change:

   $ kubectl wait --for=condition=AutoRestartVertica=False vdb/vert-cluster –-timeout=180s
   

3. To prevent mixed Vertica versions, stop the entire cluster:

   $ kubectl exec vert-cluster-sc1-0 -- admintools -t stop_db -F -d vertdb
   

4. Update the container tag in the CR:

   $ kubectl patch verticadb vert-cluster --type=merge --patch '{"spec": {"image": "vertica/vertica-k8s:11.1.1-0"}}'
   

5. Delete the pods so that they can pickup the new image. The simplest way to delete the pods is to delete the StatefulSets so the operator regenerates them:

   $ kubectl delete statefulset -l app.kubernetes.io/instance=vert-cluster -–cascade=forground
   

6. Enable autoRestartVertica to give control back to the operator to restart the Vertica process:

   $ kubectl patch verticadb vert-cluster --type=merge --patch '{"spec": {"autoRestartVertica": true}}'
   

7. Wait for the operator to bring everything back up:

   $ kubectl wait --for=condition=Ready=True pod -l app.kubernetes.io/instance=vert-cluster –-timeout=600s
   

To minimize the number of errors, we validate that the image can only change when autoRestartVertica is false.

Persistence

Each pod uses a PV to store local data. The PV is mounted in the container at /home/dbadmin/local-data.

Select a recommended storage format type as the fsType value for your StorageClass.

The local-data directory contains the following subdirectories:

• /home/dbadmin/local-data/<uid>/data/: Stores the local catalogs and any temporary files. There is a symlink to this path to the local.dataPath parameter. Default: /data.
• /home/dbadmin/local-data/<uid>/depot/: Stores the depot for the local node. There is a symlink to this path to the local.depotPath parameter. Default:/depot.
• /home/dbadmin/local-data/<uid>/config/: This is a symlink to /opt/vertica/config. This allows the contents of the configuration directory to persist between restarts.
• /home/dbadmin/local-data/<uid>/log/: This is a symlink to /opt/vertica/log. This allows the log files to persist between restarts.

NOTE: <uid> is the unique ID of the VerticaDB resource that is assigned by Kubernetes.

By default, the PV is selected using the default storage class. Use the local.storageClass parameter to select a specific storage class.

Parameters

The following table describes each configurable parameter in the VerticaDB CRD and their defaults:

kubectl create secret generic s3-creds 
--from-literal=accesskey=accesskey --from-literal=secretkey=secretkey

communal:
  credentialSecret: s3-creds

sidecars:
  - name: vlogger
    image: image:tag

subclusters:
  - name: sc1
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/name
            operator: In
            values:
            - vertica
          topologyKey: "kubernetes.io/hostname"

subclusters:
  - name: sc1
    nodeSelector:
      disktype: ssd
      region: us-east

subclusters:
  - name: defaultsubcluster
    resources:
      requests:
        memory: "96Gi"
        cpu: 32
      limits:
        memory: "96Gi"
        cpu: 32

db:
  superuserSecretPassword: su-passwd
 

Additional Details

For additional details on the internals of Vertica, see the Vertica Documentation.

Developers

For details about setting up an environment to develop and run tests, see the developer instructions and the Vertica Integrator's Guide.

Licensing

vertica-kubernetes is open source code and is under the Apache 2.0 license, but it requires that you install the Vertica server RPM. If you do not have a Vertica server RPM, you can use the free Vertica Community Edition (CE) server RPM. The Vertica Community Edition server RPM is not an open source project, but it is free with certain limits on capacity. For more information about these limitations, see the Vertica Community Edition End User License Agreement.