oaievals-collector

OAIEvals Collector

⭐ Star this repository if you find it valuable and worth maintaining.

👁 Watch this repository to get notified about new releases, issues, etc.

Introduction

The OAIEvals Collector is a Go application specifically designed to collect and store raw evaluation metrics. It provides an HTTP handler & FileSystem Watcher for receiving metric data, a set of utilities for event handling and monitoring, and supports containerized deployment with Docker for ease of use and scalability.

The application integrates seamlessly with InfluxDB for robust and efficient storage of numeric-based time series data, Loki for storing and retrieving log data, providing context and qualitative information around numeric metrics, and now with TimescaleDB for storing and managing event data. All three systems can be used independently or together, depending on the nature of the metrics and the requirements of your system.

The latest additions to the list of integrations are Kafka and MongoDB. Kafka enables the application to efficiently ingest and process a high volume of events in real-time, making it well-suited for systems that need to handle thousands of events per second. Kafka's robustness and scalability complement the existing storage solutions, providing a comprehensive data ingestion and storage solution. MongoDB adds flexible, document-based data modeling and can handle a wide variety of data types.

Demo:

InfluxDB:

InfluxDB Test 1	InfluxDB Test 2

Grafana:

Grafana Test
Basic Visualization via InfluxDB & TimeScaleDB

Kafka:

Kafka Test
Basic Visualization via Kafka UI

Prerequisites

Before you can run the OAIEvals Collector, you'll need to have the following installed:

• Go (version 1.20 or later)
• Docker (version 20.10 or later)
• Docker Compose (version 1.28 or later)

Please ensure you have these prerequisites installed and properly configured before proceeding with the setup of the OAIEvals Collector.

Usage

The OAIEvals Collector is designed to be deployed as a containerized application, with its services orchestrated by Docker Compose. The application can be configured using environment variables.

Preparation:

1. Clone the repository and navigate to the project directory.

Start Services:

1. Run docker-compose up to start the OAIEvals Collector, InfluxDB, Loki, TimescaleDB, Kafka, and MongoDB. Docker will pull the images (if not already present) and build the OAIEvals Collector image.

InfluxDB, Loki, TimescaleDB, Kafka, and MongoDB Setup:

1. While the services are spinning up, navigate to your InfluxDB, Loki, TimescaleDB, Kafka, and MongoDB instances and generate an authentication token (for InfluxDB) and/or get the connection strings. These tokens and strings will be used by the OAIEvals Collector to connect and interact with the databases.

Configuration Setup:

1. Once you've obtained the tokens and connection strings, stop the running Docker Compose services (using CTRL+C or docker-compose down command).
2. Open the .env file (create one based on the provided .env.example if it does not exist), and replace your_token_here in INFLUXDB_TOKEN=your_token_here, LOKI_URL=your_loki_url,KAFKA_BOOTSTRAP_SERVERS=your_kafka_bootstrap_servers, TIMESCALEDB_HOST=your_timescaledb_host, MONGODB_URI=your_mongodb_uri and others with the tokens or endpoints obtained from InfluxDB, Loki, TimescaleDB, Kafka, and MongoDB respectively.

Restart Services:

1. Re-run docker-compose up. Now, the OAIEvals Collector should be able to connect to the InfluxDB, Loki, TimescaleDB, Kafka, and MongoDB with the provided tokens or connection strings.

Dependencies

The OAIEvals Collector relies on InfluxDB, Loki, TimescaleDB, Kafka, and MongoDB for storing collected metrics. Ensure you have instances of these systems available for the application to connect to.

Build, Run, Test

Running with Docker Compose

You can easily run the application along with the desired data storage service (InfluxDB, Loki, TimescaleDB, Kafka, or MongoDB) using Docker Compose. For each data storage service, a dedicated Docker Compose file and corresponding Make commands are provided.

To start the application with a specific service, view the logs, or stop the service, use the appropriate Make command. Here's how you do it for each service:

• TimescaleDB: make up-timescale, make logs-timescale, make down-timescale
• MongoDB: make up-mongo, make logs-mongo, make down-mongo
• InfluxDB: make up-influx, make logs-influx, make down-influx
• Loki: make up-loki, make logs-loki, make down-loki
• Kafka: make up-kafka, make logs-kafka, make down-kafka

For example, make up-timescale starts the application with TimescaleDB, make logs-timescale displays its logs, and make down-timescale stops the service.

Locally

1. Build the application: go build -o oaievals-collector .
2. Run the application: ./oaievals-collector

Test the Application:

You can test the OAIEvals Collector by sending a POST request with a payload representing an event. For instance:

curl -X POST \
     -H "Content-Type: application/json" \
     -d '[
         {
           "run_id": "2307080128125Q6U7IFP",
           "event_id": 3,
           "sample_id": "abstract-causal-reasoning-text.dev.484",
           "type": "match",
           "data": {
             "correct": false,
             "expected": "ANSWER: off",
             "picked": "undetermined.",
             "sampled": "undetermined."
           },
           "created_by": "",
           "created_at": "2023-07-08 01:28:13.704853+00:00"
         }
       ]' http://localhost:8081/events

Contributing

Feel free to create an issue or propose a pull request. Follow the Code of Conduct.