faros

Faros

Faros - Greek for Lighthouse

Faros is a GitOps controller that takes a Git repository reference from a Custom Resource Definition (CRD) and applies resources within the repository to a Kubernetes cluster.

Note: This is a proof of concept in the early alpha stage. We are providing no guarantees and recommend that you test and thoroughly understand the project before deploying to production.

Table of Contents

• Introduction
• Installation
  • Deploying to Kubernetes
  • Configuration
    • Ignore Resource types
    • Namespace restriction
    • Leader Election
    • Sync period
• Quick Start
• Project Concepts
  • Owner References and Garbage Collection
  • Three Way Merge
  • Update Strategies
• Communication
• Contributing
• License

Introduction

Faros aims to make it easier for teams to ensure that the desired state of their applications is synchronised between a Kubernetes cluster and Git.

Typically, a team running workloads on Kubernetes will use infrastructure-as-code concepts and keep a copy of their deployment configuration under source control just as they do with their product code. The process of taking this desired state from Git and applying it to the Kubernetes cluster is the problem Faros aims to solve.

By providing Faros with a reference to a Git repository (URL and Git Reference (eg master)), credentials to access the repository and an optional path within the repository, Faros will load all Kubernetes resource definitions from the repository and synchronise these with the Kubernetes cluster.

Faros then watches the child resources (resources created from the repository) and, if they are ever modified, reverts the change back to the state of the Git repository. This allows users to make changes to their deployment configuration exclusively in Git, which in turn enables them to audit and peer review those changes as well as providing a canonical history of what was deployed and when.

Installation

Building

To build faros locally, run

./configure
make build

In order to build all binaries for all supported architectures, you may

./configure
make release

Deploying to Kubernetes

Faros is a Kubebuilder based project, as such we have auto-generated CRDs and Kustomize configuration as examples of how to install the controller in the config folder. To quickly install the controller and CRDs on a cluster you can run make deploy.

You must manually install the CRDs before the controller will be fully functional, it will not install them for you.

A public docker image is available on Quay.

quay.io/pusher/faros

RBAC

If you are using RBAC within your cluster, you must grant the service account used by your Faros instance a superset of all roles you expect it to deploy.

The simplest way to do this is to grant Faros cluster-admin, however, if you wish to be more secure, you can concatenate all rules from each Role and ClusterRole that Faros will manage.

If you do not do so, you will see errors where Faros is attempting to escalate its privileges.

Configuration

The following details the various configuration options that Faros provides at the controller level.

Ignore Resource types

You may not want to have Faros manage all types of Kubernetes resource. The --ignore-resource flag allows you to specify a particular Resource to ignore.

Specify Resources to ignore in the format <resource>.<api-group>/<api-version>, for example to ignore all Kubernetes Jobs specify the following flag:

--ignore-resource=jobs.batch/v1

It is recommended not to manage GitTrack resources using Faros itself, so ignore those using the following flag:

--ignore-resource=gittracks.faros.pusher.com/v1alpha1

Namespace restriction

Faros can be run either as a cluster wide controller or per namespace.

To restrict Faros to watch GitTrack resources only in a single namespace, use the following flag:

--namespace=<namespace>

This will restrict Faros to the namespace you provide. At this point, it will only read GitTrack resources within the defined namespace. If any GitTrack refers to any resource not in the GitTrack's namespace, the resource will be ignored. Therefore a GitTrack in the default namespace cannot manage a resource in the kube-system namespace if the controller is restricted to the default namespace.

When restricted to a namespace, Faros will still manage any non-namespaced Kubernetes resource it finds referenced within its GitTracks. If however, the non-namespaced resource clashes and is defined in another GitTrack within another namespace, Faros will ignore the resource. First owner wins.

Leader Election

Faros can be run in an active-standby HA configuration using Kubernetes leader election. When leader election is enabled, each Pod will attempt to become leader and, whichever is successful, will become the active or master controller. The master will perform all of the reconciliation of Resources.

The standby Pods will sit and wait until the leader goes away and then one standby will be promoted to master.

To enable leader election, set the following flags:

--leader-election=true
--leader-election-id=<name-of-leader-election-configmap>
--leader-election-namespace=<namespace-controller-runs-in>

Sync period

The controller uses Kubernetes informers to cache resources and reduce load on the Kubernetes API server. Every informer has a sync period, after which it will refresh all resources within its cache. At this point, every item in the cache is queued for reconciliation by the controller.

Therefore, by setting the following flag;

--sync-period=5m // Default value of 5m (5 minutes)

You can ensure that every resource will be reconciled at least every 5 minutes.

Server Dry Run

By default, the GitTrackObject controller will attempt to dry run updates to resources before actually applying updates. This helps to prevent unnecessary updates and allows mutating admission controller results to be observed.

Server side dry run currently sits behind a feature gate within Kubernetes, please see the table below for compatibility.

If your version of Kubernets is incompatible, please disable server dry run as below:

--server-dry-run=false // Defaults to true

Metrics

The controller exposes a number of metrics in a prometheus format at a /metrics endpoint. By default, the metrics serving are bound to the address port 8080 on all interfaces.

Change this with the following flag:

--metrics-bind-address=127.0.0.1:8080 // Bind the metrics to localhost only

Serving metrics can be disabled by setting the flag to 0:

--metrics-bind-address=0 // Disable serving all metrics

• faros_gittrack_child_status - Exposes the count of GitTrack child objects by status (applied,discovered,ignored,inSync).

• faros_gittrack_time_to_deploy_seconds_{bucket, count, sum} - Measures the time from updating a repository to the update being propagated to the child object.

• faros_gittrackobject_in_sync - Indicates whether individual children are in sync with their desired state.

• controller_runtime_reconcile_errors_total - Counts the total number of errors produced by the controller.

• controller_runtime_reconcile_queue_length - Counts how many items are currently queued for reconciliation by the controller.

• controller_runtime_reconcile_time_second_{bucket, count, sum} - Measures how long each reconciliation takes within the controller.

Quick Start

If you haven't yet got Faros running on your cluster, see Installation for details on how to get it running.

If you would like to deploy a private Git repository, we recommend doing so using SSH. To allow Faros access to the repository, place an authorised private key within a Secret in the same namespace that you will create the GitTrack in.

Create a GitTrack resource based the example below:

apiVersion: faros.pusher.com/v1alpha1
kind: GitTrack
metadata:
  name: foo
  namespace: bar
spec:
  # Repository accepts any valid Git repository reference, the most common formats
  # are:
  #   https://<server>/<organisation>/<repository>
  #   <user>@<server>:<organisation>/<repository>
  repository: [email protected]:foo-org/k8s-manifests
  # Reference accepts any valid Git reference, this could be a branch name, tag
  # or commit SHA, eg:
  #   master or refs/remotes/origin/master
  #   v1.0.0 or refs/tags/v1.0.0
  #   ec32c240b7f9b440aa727c9d931751fdd0c40b49
  reference: master
  # (Optional) SubPath expects a path to a folder within the repository.
  # Note: Faros loads all .yml/.yaml/.json files recursively within the path.
  subPath: deployments/kube-system
  # (Optional) DeployKey allows you to specify credentials for repository access
  # over SSH or HTTP Basic Auth
  deployKey:
    # SecretName is the name of the secret containing the secret
    secretName: foo-k8s-manifests
    # Key is the Secret's key containing the secret
    key: id_rsa
    # (Optional) Type is the type of credential. Accepted values are "SSH", "HTTPBasicAuth". Defaults to "SSH"
    # When set to "HTTPBasicAuth" the expected secret format is "<username>:<password>".
    type: SSH | HTTPBasicAuth

Deploy the GitTrack to your cluster and watch its status as Faros processes it. Eventually all conditions should have status True and the objectsApplied and objectsInSync fields should be equal.

status:
  conditions:
    - lastTransitionTime: 2018-10-16T17:36:21Z
      lastUpdateTime: 2018-10-16T17:36:21Z
      reason: FileParseSuccess
      status: "True"
      type: FilesParsed
    - lastTransitionTime: 2018-10-16T17:36:21Z
      lastUpdateTime: 2018-10-16T17:36:21Z
      reason: GitFetchSuccess
      status: "True"
      type: FilesFetched
    - lastTransitionTime: 2018-10-16T17:36:21Z
      lastUpdateTime: 2018-10-16T17:36:21Z
      reason: GCSuccess
      status: "True"
      type: ChildrenGarbageCollected
    - lastTransitionTime: 2018-10-16T17:36:21Z
      lastUpdateTime: 2018-10-16T17:36:21Z
      reason: ChildUpdateSuccess
      status: "True"
      type: ChildrenUpToDate
  objectsApplied: 82
  objectsDiscovered: 83
  objectsIgnored: 1
  objectsInSync: 82

Project Concepts

This section outlines some of the underlying concepts that enable this controller to work in the way it does.

Owner References and Garbage Collection

Faros uses Kubernetes OwnerReferences to leverage built-in Garbage Collection.

When Faros manages a resource, it will create a GitTrackObject /ClusterGitTrackObject CRD owned by the source GitTrack and the GTO/CGTO will in turn own the managed resource.

By default, when deleting a GitTrack, the Garbage Collector will in turn delete all dependents (GTOs/CGTOs) and then delete the dependent's dependents (managed resources). Therefore, by deleting a GitTrack, every resource it was managing will be deleted.

If you wish to leave objects in place when deleting a GitTrack, you can use the --cascade=false flag with kubectl.

kubectl delete gittrack --cascade=false <gittrack-name>

If you wish to remove Faros entirely, we recommend deleting all GitTrack and then GitTrackObject and ClusterGitTrackObject resources using the --cascade=false flag.

Three Way Merge

Faros uses a three-way merging strategy to determine the patch to apply when updating resources. In a similar way to kubectl apply, Faros applies a last-applied annotation to resources that it manages.

It uses this annotation, along with the current specification of the Resource and the existing Resource retrieved from the Kubernetes API to calculate the desired change to any Resource.

This strategy means that Faros will not interfere with any changes made by other controllers running on the cluster or changes made outside of Git that do not conflict with the desired state of the Resource in Git.

For example, when using a Horizontal Pod Autoscaler (HPA) with a Deployment, if the Deployment explicitly sets the number of desired replicas (.spec.replicas), each time the HPA scales the Deployment, Faros would undo this action replacing it with the value in the Deployment's spec from Git. However, if the field is unspecified in the Deployment's spec in Git, Faros would ignore the update since it does not cause a clash with the defined desired state.

Update Strategies

Some Kubernetes resources have fields that are immutable, for example the spec of a Job is immutable once created. If you were ever to update the spec of a Resource like this, Faros would encounter an error when attempting to apply the updated Resource.

To change the update behaviour of individual Resources managed by Faros you can set an annotation on the individual Resource within your Git repository and have Faros read this before it attempts to apply the Resource.

Add the annotation faros.pusher.com/update-strategy to your resource with one of the following values:

• update: This is the default value. Faros will update the Resource in place. This is equivalent to a kubectl apply.
• never: Faros will ensure that the Resource exists within Kubernetes but will never update it if the Resource is modified.
• recreate: Faros will first attempt to patch the resource, if this fails it will delete the existing Resource and create a new copy. This is equivalent to a kubectl apply --force.

For example:

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    faros.pusher.com/update-strategy: never
...

This annotation is designed to be used in special cases where individual Resources need special handling by Faros. If you wish to ignore a particular type of Resource altogether (eg. ignoring all Jobs), see Ignore Resource types.

Communication

• Found a bug? Please open an issue.
• Have a feature request. Please open an issue.
• If you want to contribute, please submit a pull request

Contributing

Please see our Contributing guidelines.

License

This project is licensed under Apache 2.0 and a copy of the license is available here.