process-compose

Process Compose

Process Compose is a simple and flexible scheduler and orchestrator to manage non-containerized applications.

Why? Because sometimes you just don't want to deal with docker files, volume definitions, networks and docker registries.

Main use cases would be:

• Processes execution (in parallel or serially)
• Defining processes dependencies and startup order
• Defining recovery policies (restart on-failure, always, no). Manual recovery is also supported.
• Declaring processes arguments
• Declaring processes environment variables

It is heavily inspired by docker-compose, but without the need for containers. The configuration syntax tries to follow the docker-compose specifications, with a few minor additions and lots of subtractions.

Quick Start

Imaginary system diagram:

process-compose.yaml definitions for the system above:

version: "0.5"

environment:
  - 'GLOBAL_ENV_VAR=1'
log_location: /path/to/combined/output/logfile.log
log_level: debug

processes:
  Manager:
    command: "/path/to/manager"
    availability:
      restart: "always"
    depends_on:
      ClientA:
        condition: process_started
      ClientB:
        condition: process_started

  ClientA:
    command: "/path/to/ClientA"
    availability:
      restart: "always"
    depends_on:
      Server_1A:
        condition: process_started
      Server_2A:
        condition: process_started
    environment:
      - 'LOCAL_ENV_VAR=1'

  ClientB:
    command: "/path/to/ClientB -some -arg"
    availability:
      restart: "always"
    depends_on:
      Server_1B:
        condition: process_started
      Server_2B:
        condition: process_started
    environment:
      - 'LOCAL_ENV_VAR=2'

  Server_1A:
    command: "/path/to/Server_1A"
    availability:
      restart: "always"

  Server_2A:
    command: "/path/to/Server_2A"
    availability:
      restart: "always"

  Server_1B:
    command: "/path/to/Server_1B"
    availability:
      restart: "always"

  Server_2B:
    command: "/path/to/Server_2B"
    availability:
      restart: "always"

Finally, run process-compose in the process-compose.yaml directory. Or give it a direct path:

process-compose -f /path/to/process-compose.yaml

Installation

• Go to the releases, download the package for your OS, and copy the binary to somewhere on your PATH.

Documentation

• See examples of workflows for best practices
• See below

List of Features and Planned Features

✅ Mostly implemented

❌ Implementation not started (Your feedback and ⭐ will motivate further development 😃)

✅ Launcher

✅ Parallel

process1:
  command: "sleep 3"
process2:
  command: "sleep 3"

✅ Serial

process1:
  command: "sleep 3"
  depends_on:
    process2: 
      condition: process_completed_successfully # or "process_completed" if you don't care about errors
process2:
  command: "sleep 3"
  depends_on:
    process3: 
      condition: process_completed_successfully # or "process_completed" if you don't care about errors

❌ Instance Number

✅ Define process dependencies

process2:
  depends_on:
  process2: 
    condition: process_completed_successfully # or "process_started" (default)
  process3: 
    condition: process_completed_successfully

✅ Output Handling

✅ Show process name

✅ Different colors per process

✅ StdErr is printed in Red

❌ Silence specific processes

✅ TUI (Terminal User Interface)

✅ Review processes status

✅ Start processes (only completed or disabled)

✅ Stop processes

✅ Review logs

TUI is the default run mode, but it's possible to disable it:

./process-compose -t=false

Control the UI log buffer size:

log_level: info
log_length: 1200 #default: 1000
processes:
  process2:
    command: "ls -R /"

Note: Using a too large buffer will put a significant penalty on your CPU.

✅ Logger

✅ Per Process Log Collection

process2:
  log_location: ./pc.process2.log #if undefined or empty no logs will be saved

✅ Capture StdOut output

✅ Capture StdErr output

✅ Merge into a single file

processes:
  process2:
    command: "chmod 666 /path/to/file"
environment:
  - 'ABC=42'
log_location: ./pc.global.log #if undefined or empty no logs will be saved (if also not defined per process)

❌ Silence specific processes

✅ Process compose console log level

log_level: info # other options: "trace", "debug", "info", "warn", "error", "fatal", "panic"
processes:
  process2:
    command: "chmod 666 /path/to/file"

This setting controls the process-compose log level. The processes log level should be defined inside the process. It is recommended to support its definition with an environment variable that can be defined in process-compose.yaml

❌ Health Checks

❌ Is Alive

❌ Is Ready

❌ Auto Restart if not healthy

✅ Auto Restart on exit

process2:
  availability:
    restart: on-failure # other options: "always", "no" (default)
    backoff_seconds: 2  # default: 1
    max_restarts: 5 # default: 0 (unlimited)

✅ Environment Variables

✅ Per Process

process2:
  environment:
    - 'I_AM_LOCAL_EV=42'

✅ Global

processes:
  process2:
    command: "chmod 666 /path/to/file"
  environment:
    - 'I_AM_LOCAL_EV=42'		
environment:
  - 'I_AM_GLOBAL_EV=42'

Default environment variables:

PC_PROC_NAME - Defines the process name as defined in the process-compose.yaml file.

PC_REPLICA_NUM - Defines the process replica number. Useful for port collision avoidance for processes with multiple replicas.

✅ REST API

A convenient Swagger API is provided: http://localhost:8080/swagger/index.html

✅ Configuration

✅ Support .env file

✅ Override ${var} and $var from environment variables or .env values

❌ Merge 2 or more configuration files with override values

✅ Specify which configuration files to use

process-compose -f "path/to/process-compose-file.yaml"

✅ Auto discover configuration files

The following discovery order is used: compose.yml, compose.yaml, process-compose.yml, process-compose.yaml. If multiple files are present the first one will be used.

✅ Multi-platform

✅ Linux

The default backend is bash. You can define a different backend with a SHELL environment variable.

✅ Windows

The default backend is cmd. You can define a different backend with a SHELL environment variable.

  process1:
    command: "python -c print(str(40+2))" 
    #note that the same command for bash/zsh would look like: "python -c 'print(str(40+2))'"

Using powershell backend had some funky behaviour (like missing command1 && command2 functionality in older versions). If you need to run powershell scripts, use the following syntax:

  process2:
    command: "powershell.exe ./test.ps1 arg1 arg2 argN"

❌ macOS