Documentation
¶
Overview ¶
Package base defines Iter8's experiment, task and metric data structures. It contains the core logic for running an experiment.
Index ¶
- Constants
- Variables
- func BoolPointer(b bool) *bool
- func CompletePath(prefix string, suffix string) string
- func NormalizeMetricName(m string) (string, error)
- func RunExperiment(reuseResult bool, driver Driver) error
- func SetupWithMock(t *testing.T)
- func StringPointer(s string) *string
- func Uniq(list interface{}) []interface{}
- type ABNMetricsInputs
- type AggregationType
- type Driver
- type Experiment
- type ExperimentResult
- type ExperimentSpec
- type HTTPParam
- type HistBucket
- type Insights
- type KubeDriver
- type Metric
- type MetricMeta
- type MetricType
- type ProviderSpec
- type Report
- type SLO
- type SLOLimits
- type SLOResults
- type Task
- type TaskMeta
- type VersionInfo
Constants ¶
const ( // CounterMetricType corresponds to Prometheus Counter metric type CounterMetricType MetricType = "Counter" // GaugeMetricType corresponds to Prometheus Gauge metric type GaugeMetricType MetricType = "Gauge" // HistogramMetricType corresponds to a Histogram metric type HistogramMetricType MetricType = "Histogram" // SampleMetricType corresponds to a Sample metric type SampleMetricType MetricType = "Sample" // SummaryMetricType corresponds to a Summary metric type SummaryMetricType MetricType = "Summary" // CountAggregator corresponds to aggregation of type count CountAggregator AggregationType = "count" // MeanAggregator corresponds to aggregation of type mean MeanAggregator AggregationType = "mean" // StdDevAggregator corresponds to aggregation of type stddev StdDevAggregator AggregationType = "stddev" // MinAggregator corresponds to aggregation of type min MinAggregator AggregationType = "min" // MaxAggregator corresponds to aggregation of type max MaxAggregator AggregationType = "max" // PercentileAggregator corresponds to aggregation of type max PercentileAggregator AggregationType = "percentile" // PercentileAggregatorPrefix corresponds to prefix for percentiles PercentileAggregatorPrefix = "p" )
const (
// AssessTaskName is the name of the task this file implements
AssessTaskName = "assess"
)
const (
// CollectABNMetricsTaskName is the name of the task to read A/B(/n) metric values
CollectABNMetricsTaskName = "abnmetrics"
)
const (
// CollectGRPCTaskName is the name of this task which performs load generation and metrics collection for gRPC services.
CollectGRPCTaskName = "grpc"
)
const (
// CollectHTTPTaskName is the name of this task which performs load generation and metrics collection.
CollectHTTPTaskName = "http"
)
const (
// CustomMetricsTaskName is the name of this task which fetches metrics templates, constructs metric specs, and then fetches metrics for each version from metric provider databases
CustomMetricsTaskName = "custommetrics"
)
const (
// NotifyTaskName is the task name
NotifyTaskName = "notify"
)
const (
// ReadinessTaskName is the task name
ReadinessTaskName = "ready"
)
const (
// RunTaskName is the name of the run task which performs running of a shell script
RunTaskName = "run"
)
Variables ¶
var MajorMinor = "v0.11"
MajorMinor is the minor version of Iter8 set this manually whenever the major or minor version changes
var Version = "v0.11.0"
Version is the semantic version of Iter8 (with the `v` prefix) Version is intended to be set using LDFLAGS at build time
Functions ¶
func BoolPointer ¶ added in v0.9.3
BoolPointer takes bool as input, creates a new variable with the input value, and returns a pointer to the variable
func CompletePath ¶
CompletePath is a helper function for converting file paths, specified relative to the caller of this function, into absolute ones. CompletePath is useful in tests and enables deriving the absolute path of experiment YAML files.
func NormalizeMetricName ¶ added in v0.8.29
NormalizeMetricName normalizes percentile values in metric names
func RunExperiment ¶ added in v0.9.3
RunExperiment runs an experiment
func SetupWithMock ¶ added in v0.9.3
SetupWithMock mocks an HTTP endpoint and registers and cleanup function
func StringPointer ¶ added in v0.8.8
StringPointer takes string as input, creates a new variable with the input value, and returns a pointer to the variable
Types ¶
type ABNMetricsInputs ¶ added in v0.11.12
type ABNMetricsInputs struct { Endpoint *string `json:"endpoint" yaml:"endpoint"` // Application is name of application to evaluate Application string `json:"application" yaml:"application"` }
ABNMetricsInputs is the inputs for for the abnmetrics task
type AggregationType ¶ added in v0.8.29
type AggregationType string
AggregationType identifies the type of the metric aggregator.
type Driver ¶ added in v0.9.3
type Driver interface { // Read the experiment Read() (*Experiment, error) // Write the experiment Write(e *Experiment) error // GetRevision returns the experiment revision GetRevision() int }
Driver enables interacting with experiment result stored externally
type Experiment ¶
type Experiment struct { // Spec is the sequence of tasks that constitute this experiment Spec ExperimentSpec `json:"spec" yaml:"spec"` // Result is the current results from this experiment. // The experiment may not have completed in which case results may be partial. Result *ExperimentResult `json:"result" yaml:"result"` // contains filtered or unexported fields }
Experiment struct containing spec and result
func BuildExperiment ¶ added in v0.9.3
func BuildExperiment(driver Driver) (*Experiment, error)
BuildExperiment builds an experiment
func (*Experiment) Completed ¶ added in v0.9.3
func (exp *Experiment) Completed() bool
Completed returns true if the experiment is complete
func (*Experiment) NoFailure ¶ added in v0.9.3
func (exp *Experiment) NoFailure() bool
NoFailure returns true if no task in the experiment has failed
func (*Experiment) SLOs ¶
func (exp *Experiment) SLOs() bool
SLOs returns true if all versions satisfy SLOs
type ExperimentResult ¶
type ExperimentResult struct { // Revision of this experiment Revision int `json:"revision,omitempty" yaml:"revision,omitempty"` // StartTime is the time when the experiment run started StartTime time.Time `json:"startTime" yaml:"startTime"` // NumLoops is the number of iterations this experiment has been running for NumLoops int `json:"numLoops" yaml:"numLoops"` // NumCompletedTasks is the number of completed tasks NumCompletedTasks int `json:"numCompletedTasks" yaml:"numCompletedTasks"` // Failure is true if any of its tasks failed Failure bool `json:"failure" yaml:"failure"` // Insights produced in this experiment Insights *Insights `json:"insights,omitempty" yaml:"insights,omitempty"` // Iter8Version is the version of Iter8 CLI that created this result object Iter8Version string `json:"iter8Version" yaml:"iter8Version"` }
ExperimentResult defines the current results from the experiment
type ExperimentSpec ¶ added in v0.8.29
type ExperimentSpec []Task
ExperimentSpec specifies the set of tasks in this experiment
func (*ExperimentSpec) UnmarshalJSON ¶ added in v0.8.29
func (s *ExperimentSpec) UnmarshalJSON(data []byte) error
UnmarshalJSON will unmarshal an experiment spec from bytes This is a custom JSON unmarshaler
type HTTPParam ¶ added in v0.11.3
type HTTPParam struct { // Name is the name of the HTTP parameter Name string `json:"name" yaml:"name"` // Value is the value of the HTTP parameter Value string `json:"value" yaml:"value"` }
HTTPParam defines an HTTP parameter
type HistBucket ¶ added in v0.8.29
type HistBucket struct { // Lower endpoint of a histogram bucket Lower float64 `json:"lower" yaml:"lower"` // Upper endpoint of a histogram bucket Upper float64 `json:"upper" yaml:"upper"` // Count is the frequency count of the bucket Count uint64 `json:"count" yaml:"count"` }
HistBucket is a single bucket in a histogram
type Insights ¶
type Insights struct { // NumVersions is the number of app versions detected by Iter8 NumVersions int `json:"numVersions" yaml:"numVersions"` // VersionNames is list of version identifiers if known VersionNames []VersionInfo `json:"versionNames" yaml:"versionNames"` // MetricsInfo identifies the metrics involved in this experiment MetricsInfo map[string]MetricMeta `json:"metricsInfo,omitempty" yaml:"metricsInfo,omitempty"` // NonHistMetricValues: // the outer slice must be the same length as the number of app versions // the map key must match name of a metric in MetricsInfo // the inner slice contains the list of all observed metric values for given version and given metric; float value [i]["foo/bar"][k] is the [k]th observation for version [i] for the metric bar under backend foo. // this struct is meant exclusively for metrics of type other than histogram NonHistMetricValues []map[string][]float64 `json:"nonHistMetricValues,omitempty" yaml:"nonHistMetricValues,omitempty"` // HistMetricValues: // the outer slice must be the same length as the number of app versions // the map key must match name of a histogram metric in MetricsInfo // the inner slice contains the list of all observed histogram buckets for a given version and given metric; value [i]["foo/bar"][k] is the [k]th observed bucket for version [i] for the hist metric `bar` under backend `foo`. HistMetricValues []map[string][]HistBucket `json:"histMetricValues,omitempty" yaml:"histMetricValues,omitempty"` // SummaryMetricValues: // the outer slice must be the same length as the number of tracks // the map key must match the name of the summary metric in MetricsInfo SummaryMetricValues []map[string]summarymetrics.SummaryMetric // SLOs involved in this experiment SLOs *SLOLimits `json:"SLOs,omitempty" yaml:"SLOs,omitempty"` // SLOsSatisfied indicator matrices that show if upper and lower SLO limits are satisfied SLOsSatisfied *SLOResults `json:"SLOsSatisfied,omitempty" yaml:"SLOsSatisfied,omitempty"` }
Insights records the number of versions in this experiment, metric values and SLO indicators for each version, metrics metadata for all metrics, and SLO definitions for all SLOs
func (*Insights) GetMetricsInfo ¶ added in v0.8.30
func (in *Insights) GetMetricsInfo(nm string) (*MetricMeta, error)
GetMetricsInfo gets metric meta for the given normalized metric name
func (*Insights) ScalarMetricValue ¶ added in v0.8.29
ScalarMetricValue gets the value of the given scalar metric for the given version
func (*Insights) TrackVersionStr ¶ added in v0.11.12
TrackVersionStr creates a string of version name/track for display purposes
type KubeDriver ¶ added in v0.10.4
type KubeDriver struct { // EnvSettings provides generic Kubernetes options *cli.EnvSettings // contains filtered or unexported fields }
KubeDriver embeds Kube configuration, and enables interaction with a Kubernetes cluster through Kube APIs
func NewFakeKubeDriver ¶ added in v0.10.4
func NewFakeKubeDriver(s *cli.EnvSettings, objects ...runtime.Object) *KubeDriver
NewFakeKubeDriver creates and returns a new KubeDriver with fake clients
func NewKubeDriver ¶ added in v0.10.4
func NewKubeDriver(s *cli.EnvSettings) *KubeDriver
NewKubeDriver creates and returns a new KubeDriver
type Metric ¶ added in v0.9.3
type Metric struct { // Name is the name of the metric Name string `json:"name" yaml:"name"` // Description is the description of the metric Description *string `json:"description,omitempty" yaml:"description,omitempty"` // Type is the type of the metric, either gauge or counter Type string `json:"type" yaml:"type"` // Units is the unit of the metric, which can be omitted for unitless metrics Units *string `json:"units,omitempty" yaml:"units,omitempty"` // Params is the set of HTTP parameters that need to be sent Params *[]HTTPParam `json:"params,omitempty" yaml:"params,omitempty"` // Body is the HTTP request body that needs to be sent Body *string `json:"body,omitempty" yaml:"body,omitempty"` // JqExpression is the jq expression that can extract the value from the HTTP // response JqExpression string `json:"jqExpression" yaml:"jqExpression"` }
Metric defines how to construct HTTP requests and parse HTTP responses when querying a database for a metric. Metric struct also includes metadata such as the name and description of the metric.
type MetricMeta ¶
type MetricMeta struct { // Description is a human readable description of the metric Description string `json:"description" yaml:"description"` // Units for this metric (if any) Units *string `json:"units,omitempty" yaml:"units,omitempty"` // Type of the metric. Example: counter Type MetricType `json:"type" yaml:"type"` }
MetricMeta describes a metric
type ProviderSpec ¶ added in v0.11.3
type ProviderSpec struct { // URL is the database endpoint URL string `json:"url" yaml:"url"` // Method is the HTTP method that needs to be used Method string `json:"method" yaml:"method"` // Headers is the set of HTTP headers that need to be sent Headers map[string]string `json:"headers" yaml:"headers"` // Metrics is the set of metrics that can be obtained Metrics []Metric `json:"metrics" yaml:"metrics"` }
ProviderSpec specifies how to get metrics from a provider
type Report ¶ added in v0.11.10
type Report struct { // Timestamp is when the report was created // For example: 2022-08-09 15:10:36.569745 -0400 EDT m=+12.599643189 TimeStamp string `json:"timeStamp" yaml:"timeStamp"` // Completed is whether or not the experiment has completed Completed bool `json:"completed" yaml:"completed"` // NoTaskFailures is whether or not the experiment had any tasks that failed NoTaskFailures bool `json:"noTaskFailures" yaml:"noTaskFailures"` // NumTasks is the number of tasks in the experiment NumTasks int `json:"numTasks" yaml:"numTasks"` // NumCompletedTasks is the number of completed tasks in the experiment NumCompletedTasks int `json:"numCompletedTasks" yaml:"numCompletedTasks"` // NumLoops is the current loop of the experiment NumLoops int `json:"numLoops" yaml:"numLoops"` // Experiment is the experiment struct Experiment *Experiment `json:"experiment" yaml:"experiment"` }
Report is the data that is given to the payload template
type SLO ¶
type SLO struct { // Metric is the fully qualified metric name in the backendName/metricName format Metric string `json:"metric" yaml:"metric"` // Limit is the acceptable limit for this metric Limit float64 `json:"limit" yaml:"limit"` }
SLO is a service level objective
type SLOLimits ¶ added in v0.11.0
type SLOLimits struct { // Upper limits for metrics Upper []SLO `json:"upper,omitempty" yaml:"upper,omitempty"` // Lower limits for metrics Lower []SLO `json:"lower,omitempty" yaml:"lower,omitempty"` }
SLOLimits specify upper or lower limits for metrics
type SLOResults ¶ added in v0.11.0
type SLOResults struct { // Upper limits for metrics // Upper[i][j] specifies if upper SLO i is satisfied by version j Upper [][]bool `json:"upper,omitempty" yaml:"upper,omitempty"` // Lower limits for metrics // Lower[i][j] specifies if lower SLO i is satisfied by version j Lower [][]bool `json:"lower,omitempty" yaml:"lower,omitempty"` }
SLOResults specify the results of SLO evaluations
type Task ¶
type Task interface {
// contains filtered or unexported methods
}
Task is the building block of an experiment spec An experiment spec is a sequence of tasks
type TaskMeta ¶ added in v0.8.29
type TaskMeta struct { // Task is the name of the task Task *string `json:"task,omitempty" yaml:"task,omitempty"` // Run is the script used in a run task // Specify either Task or Run but not both Run *string `json:"run,omitempty" yaml:"run,omitempty"` // If is the condition used to determine if this task needs to run // If the condition is not satisfied, then it is skipped in an experiment // Example: SLOs() If *string `json:"if,omitempty" yaml:"if,omitempty"` }
TaskMeta provides common fields used across all tasks
type VersionInfo ¶ added in v0.11.12
type VersionInfo struct { // Version name Version string `json:"version" yaml:"version"` // Track identifier assigned to version Track string `json:"track" yaml:"track"` }
VersionInfo is basic information about a version
Source Files
¶
Directories
¶
Path | Synopsis |
---|---|
Package internal provides gRPC code used for testing load-test-grpc
|
Package internal provides gRPC code used for testing load-test-grpc |
helloworld/helloworld
Package helloworld implements the helloworld grpc service.
|
Package helloworld implements the helloworld grpc service. |
Package log enables logging for Iter8.
|
Package log enables logging for Iter8. |
Package summarymetrics provides a summary metric implementation
|
Package summarymetrics provides a summary metric implementation |