nerd

nerd

Welcome to the nerd repo! This service offers machine learning capabilities through a simple API, thus allowing other services to be smarter without requiring a huge effort.

Index

• Quick Start
• Requirements
• Deployment
• Use
  • Collectors
    • File
  • Metrics Updates
  • Manual Training
  • Evaluating An Input
  • Listing Available Entities
  • Health
• Testing

Quick Start

If you just want to try nerd out and see what it can do, here is a quick guide for running a test setup with containers:

1. Start a nerd instance:

   Here we're going to run nerd using the filesystem to store its data and the REST API to send it updates. If you'd like a more performant setup, refer to the "Requirements" section for instructions on how to setup Redis, Elasticsearch, Kafka and Zookeeper instead.

   docker run -d --restart=unless-stopped -m 64m \
     --log-opt max-size=5m --log-driver=json-file \
     -p 5400:5400 \
     -e "LOG_LEVEL=INFO" \
     --name nerd qvantel/nerd:0.4.1
   

2. Check that everything is up and running by going to http://localhost:5400 with your browser (if you see a welcome message, everything is good)

   Not seeing anything? You can check the nerd logs with docker logs --tail 100 nerd to see if there are any errors

3. Train a network to detect forged banknotes:
   1. Download the dataset from the UCI ML repo here

      Credit: Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

   2. Shuffle the points:

      sort -R -o shuffled-dataset.txt data_banknote_authentication.txt
      

   3. Load the test data using the built-in file collector:

      If you want, you can add variance,skewness,kurtosis,entropy,class to the beginning of shuffled-dataset.txt and use the -headers flag to properly label the values, otherwise names will be auto-generated

      1. On Linux (or WSL):

         docker run -it --rm \
           --add-host host.docker.internal:host-gateway \
           -v $PWD/shuffled-dataset.txt:/opt/docker/dataset \
           --entrypoint=/opt/docker/fcollect qvantel/nerd:0.4.1 \
             -batch 50 \
             -in 4 \
             -margin 0.4999999 \
             -sep "," \
             -series "banknote-forgery-detection" \
             -targets "http://host.docker.internal:5400" \
             dataset
         

      2. On MacOS:

         docker run -it --rm \
           -v $PWD/shuffled-dataset.txt:/opt/docker/dataset \
           --entrypoint=/opt/docker/fcollect qvantel/nerd:0.4.1 \
             -batch 50 \
             -in 4 \
             -margin 0.4999999 \
             -sep "," \
             -series "banknote-forgery-detection" \
             -targets "http://host.docker.internal:5400" \
             dataset
         

   4. Send a training request:

      If you opted to add the headers in the previous step, use ["variance","skewness","kurtosis","entropy"] as inputs and ["class"] as the output instead of the values bellow

      curl -XPOST -H "Content-Type: application/json" --data @- \
          localhost:5400/api/v1/nets <<EOF
      {
          "errMargin": 0.4999999,
          "inputs": ["value-0", "value-1", "value-2", "value-3"],
          "outputs": ["value-4"],
          "required": 1372,
          "seriesID": "banknote-forgery-detection"
      }
      EOF
      

   5. Check out the resulting net by going to http://localhost:5400/api/v1/series/banknote-forgery-detection/nets
4. Use the network:
   1. With an authentic note (the output should be closer to 0 than 1)

      # (with headers) NET=banknote-forgery-detection-f6217c7e74da371fea775c5a0b11b5b36d9438ed-8d767bf5b72373d12f0efd4406677e9ed076f592-mlp
      NET=banknote-forgery-detection-8921e4a37dabacc06fec3318e908d9fe4eb75b46-7804b6fc74b5c0a74cc0820420fa0edf6b1a117c-mlp
      ENDPOINT=localhost:5400/api/v1/nets/$NET/evaluate
      

      curl -XPOST -H"Content-Type: application/json" --data @- \
          $ENDPOINT <<EOF
      {
          "value-0": 3.2403,
          "value-1": -3.7082,
          "value-2": 5.2804,
          "value-3": 0.41291
      }
      EOF
      

   2. With a forged note (the output should be closer to 1 than 0)

      curl -XPOST -H"Content-Type: application/json" --data @- \
          $ENDPOINT <<EOF
      {
          "value-0": -1.4377,
          "value-1": -1.432,
          "value-2": 2.1144,
          "value-3": 0.42067
      }
      EOF
      

Requirements

This service has the following dependencies:

Kafka

Even though nerd can be used without it (sending updates through the REST API), it's better to use a service like Kafka (maybe nats in the future) to decouple that interaction and benefit from built-in load balancing. When producing metrics updates, the series ID should be used by the partitioning strategy to reduce the chance of triggering training for the same series twice.

For testing, the following commands can be used to start Zookeeper and Kafka:

If running on Linux, include --add-host host.docker.internal:host-gateway in the Kafka docker run command (anywhere between docker run and the image)

docker run -d --restart=unless-stopped \
  --log-driver json-file \
  -p 2181:2181 \
  --name zookeeper zookeeper:3.6.2

docker run -d --restart=unless-stopped \
  --log-driver json-file \
  -p 7203:7203 -p 7204:7204 -p 9092:9092 \
  -e "KAFKA_LISTENERS=PLAINTEXT://:9092" \
  -e "KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:9092" \
  -e "KAFKA_DEFAULT_REPLICATION_FACTOR=1" \
  -e "KAFKA_DELETE_TOPIC_ENABLE=true" \
  -e "KAFKA_ZOOKEEPER_CONNECT=host.docker.internal:2181" \
  -e "KAFKA_BROKER_ID=1" \
  -e "KAFKA_HEAP_OPTS=-Xmx4G -Xms4G" \
  -e "ZOOKEEPER_SESSION_TIMEOUT_MS=30000" \
  --name kafka wurstmeister/kafka:2.12-2.1.1

A Network Parameter Store

Currently, Redis (and the filesystem but that should only be used for testing).

When using Redis with Sentinel, the ML_STORE_PARAMS variable should be used (instead of SD_REDIS) like so:

  -e 'ML_STORE_PARAMS={"group": "<group-name>", "URLs": "<sen1-host>:<sen1-port>,...,<senN-host>:<senN-port>"}'

Where group contains the replica group name and URLs the comma-separated list of Sentinel instance host:port pairs.

For testing, the following command can be used to start up a Redis replica:

docker run -d \
  --log-driver json-file \
  -p 6379:6379 \
  --name redis redis:6.0.10-alpine3.13

A Point Store

Currently, Elasticsearch (and the filesystem but that should only be used for testing).

If Elasticsearch is used:

• The action.auto_create_index setting must be set to .watches,.triggered_watches,.watcher-history-* otherwise it will create non optimal mappings increasing the storage impact.

• Given how index refreshing works, the automatic training request for a series that gets a high number of metrics updates in a very short period of time (less than a second)(possible when the lag is momentarily high for example) might not get issued. To avoid this, it's recommended to include multiple points per update with a lower frequency rather than sending one update per point as it is extracted.

For testing, it is possible to get a working Elasticsearch instance quickly with the following command:

docker run -d \
  --log-driver json-file \
  -p 9200:9200 -p 9300:9300 \
  -e "discovery.type=single-node" \
  -e "action.auto_create_index=.watches,.triggered_watches,.watcher-history-*" \
  --name elasticsearch elasticsearch:7.10.1

Deployment

For a simple deployment, the following command can be used to start up a nerd instance that'll use Kafka, Redis and Elasticsearch (changing the ip:ports for those of the corresponding services in your setup):

If running on Linux, include --add-host host.docker.internal:host-gateway in the docker run command (anywhere between docker run and the image) if you're going to use it as is

docker run -d --restart=unless-stopped -m 64m \
  --log-opt max-size=5m --log-driver=json-file \
  -p 5400:5400 \
  -e "LOG_LEVEL=INFO" \
  -e "SD_ELASTICSEARCH=http://host.docker.internal:9200" \
  -e "SERIES_STORE_TYPE=elasticsearch" \
  -e "SD_KAFKA=host.docker.internal:9092" \
  -e "SD_REDIS=host.docker.internal:6379" \
  -e "ML_STORE_TYPE=redis" \
  --name nerd qvantel/nerd:0.4.1

You can find all available tags here

The following environment variables are available:

* While not strictly required for operation, the default value should be overridden for anything other than testing and even then, not all testing should be done with those values

Use

Once the service has been deployed, it is possible to interact with it either through Kafka or the REST API.

Collectors

These are lightweight components that can be used to import data from other services into nerd. To facilitate their development, nerd exposes the github.com/qvantel/nerd/api/types and github.com/qvantel/nerd/pkg/producer modules which include the types used by the REST and Kafka interfaces as well as ready-made methods for producing messages to them.

File

At the time of writing, the only public collector is the one built into this project under the fcollect command, which imports datasets from plain text files. It can be accessed from the container (as seen in the "Quick Start" section) by changing the entrypoint to /opt/docker/fcollect like so (anything placed after the image will be passed to fcollect as an argument):

docker run -it --rm \
  -v $PWD/shuffled-dataset.txt:/opt/docker/dataset \
  --entrypoint=/opt/docker/fcollect \
  --name fcollect qvantel/nerd:0.4.1 -series "demo" -producer "kafka" -targets "host.docker.internal:9092" -sep "," dataset

Where the -series and -targets flags as well as the path to the dataset (full or relative to /opt/docker inside the container) are required. Additionally, the following flags can be used to change the behaviour of the tool:

Metrics Updates

Metrics updates can be ingested through either the $SERIES_KAFKA_TOPIC topic in Kafka or the /api/v1/series/process endpoint. In both cases the message must conform to the Cloud Events v1 specification where the metadata fields should be filled in as follows:

Additionally, the data fields should be filled in like so:

Example:

{
    "data": {
        "seriesID": "heart-of-gold-lightbulb-usage",
        "errMargin": 0.1,
        "labels": {
            "captain": "Zaphod Beeblebrox"
        },
        "points": [
            {
                "inputs": {
                    "humans": 2,
                    "robots": 1,
                    "aliens": 2
                },
                "outputs": {
                    "lightbulbs-on": 1500
                },
                "timestamp": 777808800
            }
        ],
        "stage": "test"
    },
    "datacontenttype": "application/json",
    "dataschema": "github.com/qvantel/nerd/api/types/",
    "id": "1",
    "source": "test-script",
    "specversion": "1.0",
    "subject": "heart-of-gold",
    "type": "com.qvantel.nerd.metricsupdate"
}

Manual Training

Even though the service will automatically schedule training when it has enough points of a series, it is still possible to manually trigger training from any preexisting series. To do this, just post a training request to the /api/v1/nets endpoint like so (where $URL contains the address of the nerd service):

curl -XPOST -H"Content-Type: application/json" --data @- \
    $URL/api/v1/nets <<EOF
{
    "errMargin": 0.4999999,
    "inputs": ["value-0", "value-1", "value-2", "value-3", "value-4", "value-5", "value-6", "value-7", "value-8"],
    "outputs": ["value-9", "value-10"],
    "required": 699,
    "seriesID": "testloadtestset"
}
EOF

Where, the fields contain the following information:

Evaluating An Input

Once a net has been trained, it can be exploited through the /api/v1/nets/{id}/evaluate endpoint like so (where $URL contains the address of the nerd service and $ID the ID of the network):

NOTE: The network ID is different from that of the series it comes from, as multiple nets could be (and are) created from a single series. The nets for a given series can be found through the /api/v1/nets endpoint (by using the seriesID query param) or the /api/v1/series/{id}/nets endpoint.

curl -XPOST -H"Content-Type: application/json" --data @- \
    $URL/api/v1/nets/$ID/evaluate <<EOF
{
    "value-0": 5000,
    "value-1": 0.07,
    "value-2": 0.1,
    "value-3": 100,
    "value-4": 1,
    "value-5": 0.1,
    "value-6": 0.1,
    "value-7": 0.1,
    "value-8": 1000
}
EOF

Sample response:

{"value-9":0.16796547}

Listing Available Entities

• Nets:
  • Endpoint: /api/v1/nets
  • Method: GET
  • Params:
    • offset: Offset to fetch, 0 by default
    • limit: How many networks to fetch, the service might return more in some cases, 10 by default, 50 maximum
    • seriesID: Filter by series ID (/api/v1/series/{id}/nets can be used to pass the ID as a path param instead)
  • Returns: A types.PagedRes object and a 200 if successful, a types.SimpleRes object and a 400 or 500 if not (depending on the error)
  • Sample response:

{
  "last": true,
  "next": 0,
  "results": [
    {
      "accuracy": 0.9908759,
      "activationFunc": "bipolar-sigmoid",
      "averages": {
        "class": 0.42718446,
        "entropy": -1.2009263,
        "kurtosis": 1.334538,
        "skewness": 2.1060672,
        "variance": 0.47604737
      },
      "deviations": {
        "class": 0.49497,
        "entropy": 2.1664677,
        "kurtosis": 4.235366,
        "skewness": 5.8205276,
        "variance": 2.8741868
      },
      "errMargin": 0.4999999,
      "hLayers": 1,
      "id": "banknote-forgery-detection-f6217c7e74da371fea775c5a0b11b5b36d9438ed-8d767bf5b72373d12f0efd4406677e9ed076f592-mlp",
      "inputs": [
        "entropy",
        "kurtosis",
        "skewness",
        "variance"
      ],
      "learningRate": 0.092,
      "outputs": [
        "class"
      ],
      "type": "mlp"
    }
  ]
}

• Series:
  • Endpoint: /api/v1/series
  • Method: GET
  • Returns: An array of types.BriefSeries objects and a 200 if successful, a types.SimpleRes object and a 500 if not
  • Sample response:

[
  {
    "name": "banknote-forgery-detection",
    "count": 1372
  }
]

• Points:
  • Endpoint: /api/v1/series/{id}/points
  • Method: GET
  • Params:
    • limit: How many points to fetch, 10 by default, 500 maximum
  • Returns: An array of pointstores.Point objects and a 200 if successful, a types.SimpleRes object and a 404 or 500 if not (depending on the error)
  • Sample response:

[
  {
    "@timestamp": 1612706310,
    "class": 0,
    "kurtosis": 2.0938,
    "entropy": 0.20085,
    "skewness": 2.599,
    "stage": "test",
    "subject": "dataset",
    "variance": 2.5367
  },
  {
    "@timestamp": 1612706309,
    "class": 0,
    "kurtosis": -2.4089,
    "entropy": -0.056479,
    "skewness": 5.5788,
    "stage": "test",
    "subject": "dataset",
    "variance": 5.7823
  }
]

Health

• Startup probe:
  • Endpoint: /api/v1/health/startup
  • Method: GET
  • Returns: A types.SimpleRes object and a 200 if successful
  • Sample response:

{
  "result": "ok",
  "message": "The API is up"
}

Testing

NOTE: The tests automatically spin up Docker containers for dependencies like Elasticsearch and Redis so the host must have it installed and the user running them must have the necessary rights

• Unit tests:

  go test -cover ./...
  

• Functional tests:

  These can take a while as they build the nerd image from the Dockerfile

  go test -v --tags=functional github.com/qvantel/nerd/cmd