dataobj

package
v3.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2025 License: AGPL-3.0 Imports: 30 Imported by: 0

Documentation

Overview

Package dataobj holds utilities for working with data objects.

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrBuilderFull  = errors.New("builder full")
	ErrBuilderEmpty = errors.New("builder empty")
)

ErrBuilderFull is returned by Builder.Append when the buffer is full and needs to flush; call Builder.Flush to flush it.

Functions

This section is empty.

Types

type AndPredicate added in v3.5.0

type AndPredicate[P Predicate] struct{ Left, Right P }

An AndPredicate is a Predicate which requires both its Left and Right predicate to be true.

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

A Builder builds data objects from a set of incoming log data. Log data is appended to a builder by calling Builder.Append. Buffered log data is flushed manually by calling Builder.Flush.

Methods on Builder are not goroutine-safe; callers are responsible for synchronizing calls.

func NewBuilder

func NewBuilder(cfg BuilderConfig) (*Builder, error)

NewBuilder creates a new Builder which stores data objects for the specified tenant in a bucket.

NewBuilder returns an error if BuilderConfig is invalid.

func (*Builder) Append

func (b *Builder) Append(stream logproto.Stream) error

Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBuilderFull if the builder is full.

Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.

func (*Builder) Flush

func (b *Builder) Flush(output *bytes.Buffer) (FlushStats, error)

Flush flushes all buffered data to the buffer provided. Calling Flush can result in a no-op if there is no buffered data to flush.

Builder.Reset is called after a successful Flush to discard any pending data and allow new data to be appended.

func (*Builder) RegisterMetrics

func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error

RegisterMetrics registers metrics about builder to report to reg. All metrics will have a tenant label set to the tenant ID of the Builder.

If multiple Builders for the same tenant are running in the same process, reg must contain additional labels to differentiate between them.

func (*Builder) Reset

func (b *Builder) Reset()

Reset discards pending data and resets the builder to an empty state.

func (*Builder) UnregisterMetrics

func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)

UnregisterMetrics unregisters metrics about builder from reg.

type BuilderConfig

type BuilderConfig struct {
	// TargetPageSize configures a target size for encoded pages within the data
	// object. TargetPageSize accounts for encoding, but not for compression.
	TargetPageSize flagext.Bytes `yaml:"target_page_size"`

	// TargetObjectSize configures a target size for data objects.
	TargetObjectSize flagext.Bytes `yaml:"target_object_size"`

	// TargetSectionSize configures the maximum size of data in a section. Sections
	// which support this parameter will place overflow data into new sections of
	// the same type.
	TargetSectionSize flagext.Bytes `yaml:"target_section_size"`

	// BufferSize configures the size of the buffer used to accumulate
	// uncompressed logs in memory prior to sorting.
	BufferSize flagext.Bytes `yaml:"buffer_size"`

	// SectionStripeMergeLimit configures the number of stripes to merge at once when
	// flushing stripes into a section. MergeSize must be larger than 1. Lower
	// values of MergeSize trade off lower memory overhead for higher time spent
	// merging.
	SectionStripeMergeLimit int `yaml:"section_stripe_merge_limit"`
}

BuilderConfig configures a data object Builder.

func (*BuilderConfig) RegisterFlagsWithPrefix

func (cfg *BuilderConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)

RegisterFlagsWithPrefix registers flags with the given prefix.

func (*BuilderConfig) Validate

func (cfg *BuilderConfig) Validate() error

Validate validates the BuilderConfig.

type FlushStats added in v3.5.0

type FlushStats struct {
	MinTimestamp time.Time
	MaxTimestamp time.Time
}

type LabelFilterPredicate added in v3.5.0

type LabelFilterPredicate struct {
	Name string
	Keep func(name, value string) bool
}

A LabelFilterPredicate is a StreamsPredicate that requires that labels with the provided name pass a Keep function.

The name is is provided to the keep function to allow the same function to be used for multiple filter predicates.

Uses of LabelFilterPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.

type LabelMatcherPredicate added in v3.5.0

type LabelMatcherPredicate struct{ Name, Value string }

A LabelMatcherPredicate is a StreamsPredicate which requires a label named Name to exist with a value of Value.

type LogMessageFilterPredicate added in v3.5.0

type LogMessageFilterPredicate struct {
	Keep func(line []byte) bool
}

A LogMessageFilterPredicate is a LogsPredicate that requires the log message of the entry to pass a Keep function.

type LogsPredicate added in v3.5.0

type LogsPredicate interface {
	Predicate
	// contains filtered or unexported methods
}

LogsPredicate is a Predicate that can be used to filter logs in a LogsReader.

type LogsReader added in v3.5.0

type LogsReader struct {
	// contains filtered or unexported fields
}

LogsReader reads the set of logs from an Object.

func NewLogsReader added in v3.5.0

func NewLogsReader(obj *Object, sectionIndex int) *LogsReader

NewLogsReader creates a new LogsReader that reads from the logs section of the given object.

func (*LogsReader) Close added in v3.5.0

func (r *LogsReader) Close() error

Close closes the LogsReader and releases any resources it holds. Closed LogsReaders can be reused by calling LogsReader.Reset.

func (*LogsReader) MatchStreams added in v3.5.0

func (r *LogsReader) MatchStreams(ids iter.Seq[int64]) error

MatchStreams provides a sequence of stream IDs for the logs reader to match. LogsReader.Read will only return logs for the provided stream IDs.

MatchStreams may be called multiple times to match multiple sets of streams.

MatchStreams may only be called before reading begins or after a call to LogsReader.Reset.

func (*LogsReader) Read added in v3.5.0

func (r *LogsReader) Read(ctx context.Context, s []Record) (int, error)

Read reads up to the next len(s) records from the reader and stores them into s. It returns the number of records read and any error encountered. At the end of the logs section, Read returns 0, io.EOF.

func (*LogsReader) Reset added in v3.5.0

func (r *LogsReader) Reset(obj *Object, sectionIndex int)

Reset resets the LogsReader with a new object and section index to read from. Reset allows reusing a LogsReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the LogsReader without needing a new object.

func (*LogsReader) SetPredicate added in v3.5.0

func (r *LogsReader) SetPredicate(p LogsPredicate) error

SetPredicate sets the predicate to use for filtering logs. LogsReader.Read will only return logs for which the predicate passes.

A predicate may only be set before reading begins or after a call to LogsReader.Reset.

type Metadata added in v3.5.0

type Metadata struct {
	StreamsSections int // Number of streams sections in the Object.
	LogsSections    int // Number of logs sections in the Object.
}

Metadata holds high-level metadata about an Object.

type MetadataFilterPredicate added in v3.5.0

type MetadataFilterPredicate struct {
	Key  string
	Keep func(key, value string) bool
}

A MetadataFilterPredicate is a LogsPredicate that requires that metadata with the provided key pass a Keep function.

The key is provided to the keep function to allow the same function to be used for multiple filter predicates.

Uses of MetadataFilterPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.

type MetadataMatcherPredicate added in v3.5.0

type MetadataMatcherPredicate struct{ Key, Value string }

A MetadataMatcherPredicate is a LogsPredicate that requires a metadata key named Key to exist with a value of Value.

type NotPredicate added in v3.5.0

type NotPredicate[P Predicate] struct{ Inner P }

A NotPredicate is a Predicate which requires its Inner predicate to be false.

type Object added in v3.5.0

type Object struct {
	// contains filtered or unexported fields
}

An Object is a representation of a data object.

func FromBucket added in v3.5.0

func FromBucket(bucket objstore.Bucket, path string) *Object

FromBucket opens an Object from the given storage bucket and path.

func FromReaderAt added in v3.5.0

func FromReaderAt(r io.ReaderAt, size int64) *Object

FromReadSeeker opens an Object from the given ReaderAt. The size argument specifies the size of the data object in bytes.

func (*Object) Metadata added in v3.5.0

func (o *Object) Metadata(ctx context.Context) (Metadata, error)

Metadata returns the metadata of the Object. Metadata returns an error if the object cannot be read.

type OrPredicate added in v3.5.0

type OrPredicate[P Predicate] struct{ Left, Right P }

An OrPredicate is a Predicate which requires either its Left or Right predicate to be true.

type Predicate added in v3.5.0

type Predicate interface {
	// contains filtered or unexported methods
}

Predicate is an expression used to filter entries in a data object.

type Record added in v3.5.0

type Record struct {
	StreamID  int64         // StreamID associated with the log record.
	Timestamp time.Time     // Timestamp of the log record.
	Metadata  labels.Labels // Set of metadata associated with the log record.
	Line      []byte        // Line of the log record.
}

A Record is an individual log record in a data object.

type Stream added in v3.5.0

type Stream struct {
	// ID of the stream. Stream IDs are unique across all sections in an object,
	// but not across multiple objects.
	ID int64

	// MinTime and MaxTime denote the range of timestamps across all entries in
	// the stream.
	MinTime, MaxTime time.Time

	// UncompressedSize is the total size of all the log lines and structured metadata values in the stream
	UncompressedSize int64

	// Labels of the stream.
	Labels labels.Labels
}

A Stream is an individual stream in a data object.

type StreamsPredicate added in v3.5.0

type StreamsPredicate interface {
	Predicate
	// contains filtered or unexported methods
}

StreamsPredicate is a Predicate that can be used to filter streams in a StreamsReader.

type StreamsReader added in v3.5.0

type StreamsReader struct {
	// contains filtered or unexported fields
}

StreamsReader reads the set of streams from an Object.

func NewStreamsReader added in v3.5.0

func NewStreamsReader(obj *Object, sectionIndex int) *StreamsReader

NewStreamsReader creates a new StreamsReader that reads from the streams section of the given object.

func (*StreamsReader) Close added in v3.5.0

func (r *StreamsReader) Close() error

Close closes the StreamsReader and releases any resources it holds. Closed StreamsReaders can be reused by calling StreamsReader.Reset.

func (*StreamsReader) Read added in v3.5.0

func (r *StreamsReader) Read(ctx context.Context, s []Stream) (int, error)

Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.

func (*StreamsReader) Reset added in v3.5.0

func (r *StreamsReader) Reset(obj *Object, sectionIndex int)

Reset resets the StreamsReader with a new object and section index to read from. Reset allows reusing a StreamsReader without allocating a new one.

Any set predicate is cleared when Reset is called.

Reset may be called with a nil object and a negative section index to clear the StreamsReader without needing a new object.

func (*StreamsReader) SetPredicate added in v3.5.0

func (r *StreamsReader) SetPredicate(p StreamsPredicate) error

SetPredicate sets the predicate to use for filtering logs. LogsReader.Read will only return logs for which the predicate passes.

SetPredicate returns an error if the predicate is not supported by LogsReader.

A predicate may only be set before reading begins or after a call to StreamsReader.Reset.

type TimeRangePredicate added in v3.5.0

type TimeRangePredicate[P Predicate] struct {
	StartTime, EndTime time.Time
	IncludeStart       bool // Whether StartTime is inclusive.
	IncludeEnd         bool // Whether EndTime is inclusive.
}

A TimeRangePredicate is a Predicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.

Directories

Path Synopsis
internal
dataset
Package dataset contains utilities for working with datasets.
Package dataset contains utilities for working with datasets.
encoding
Package encoding provides utilities for encoding and decoding data objects.
Package encoding provides utilities for encoding and decoding data objects.
result
Package result provides utilities for dealing with iterators that can fail during iteration.
Package result provides utilities for dealing with iterators that can fail during iteration.
sections/logs
Package logs defines types used for the data object logs section.
Package logs defines types used for the data object logs section.
sections/streams
Package streams defines types used for the data object streams section.
Package streams defines types used for the data object streams section.
streamio
Package streamio defines interfaces shared by other packages for streaming binary data.
Package streamio defines interfaces shared by other packages for streaming binary data.
util/bitmask
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length.
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length.
util/bufpool
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
util/sliceclear
Package sliceclear provides a way to clear and truncate the length of a slice.
Package sliceclear provides a way to clear and truncate the length of a slice.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
JackTT - Gopher 🇻🇳