Documentation
¶
Overview ¶
Package dataobj holds utilities for working with data objects.
Index ¶
- Variables
- type AndPredicate
- type Builder
- type BuilderConfig
- type FlushStats
- type LabelFilterPredicate
- type LabelMatcherPredicate
- type LogMessageFilterPredicate
- type LogsPredicate
- type LogsReader
- type Metadata
- type MetadataFilterPredicate
- type MetadataMatcherPredicate
- type NotPredicate
- type Object
- type OrPredicate
- type Predicate
- type Record
- type Stream
- type StreamsPredicate
- type StreamsReader
- type TimeRangePredicate
Constants ¶
This section is empty.
Variables ¶
var ( ErrBuilderFull = errors.New("builder full") ErrBuilderEmpty = errors.New("builder empty") )
ErrBuilderFull is returned by Builder.Append when the buffer is full and needs to flush; call Builder.Flush to flush it.
Functions ¶
This section is empty.
Types ¶
type AndPredicate ¶ added in v3.5.0
type AndPredicate[P Predicate] struct{ Left, Right P }
An AndPredicate is a Predicate which requires both its Left and Right predicate to be true.
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
A Builder builds data objects from a set of incoming log data. Log data is appended to a builder by calling Builder.Append. Buffered log data is flushed manually by calling Builder.Flush.
Methods on Builder are not goroutine-safe; callers are responsible for synchronizing calls.
func NewBuilder ¶
func NewBuilder(cfg BuilderConfig) (*Builder, error)
NewBuilder creates a new Builder which stores data objects for the specified tenant in a bucket.
NewBuilder returns an error if BuilderConfig is invalid.
func (*Builder) Append ¶
Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBuilderFull if the builder is full.
Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.
func (*Builder) Flush ¶
func (b *Builder) Flush(output *bytes.Buffer) (FlushStats, error)
Flush flushes all buffered data to the buffer provided. Calling Flush can result in a no-op if there is no buffered data to flush.
Builder.Reset is called after a successful Flush to discard any pending data and allow new data to be appended.
func (*Builder) RegisterMetrics ¶
func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
RegisterMetrics registers metrics about builder to report to reg. All metrics will have a tenant label set to the tenant ID of the Builder.
If multiple Builders for the same tenant are running in the same process, reg must contain additional labels to differentiate between them.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset discards pending data and resets the builder to an empty state.
func (*Builder) UnregisterMetrics ¶
func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
UnregisterMetrics unregisters metrics about builder from reg.
type BuilderConfig ¶
type BuilderConfig struct {
// TargetPageSize configures a target size for encoded pages within the data
// object. TargetPageSize accounts for encoding, but not for compression.
TargetPageSize flagext.Bytes `yaml:"target_page_size"`
// TargetObjectSize configures a target size for data objects.
TargetObjectSize flagext.Bytes `yaml:"target_object_size"`
// TargetSectionSize configures the maximum size of data in a section. Sections
// which support this parameter will place overflow data into new sections of
// the same type.
TargetSectionSize flagext.Bytes `yaml:"target_section_size"`
// BufferSize configures the size of the buffer used to accumulate
// uncompressed logs in memory prior to sorting.
BufferSize flagext.Bytes `yaml:"buffer_size"`
// SectionStripeMergeLimit configures the number of stripes to merge at once when
// flushing stripes into a section. MergeSize must be larger than 1. Lower
// values of MergeSize trade off lower memory overhead for higher time spent
// merging.
SectionStripeMergeLimit int `yaml:"section_stripe_merge_limit"`
}
BuilderConfig configures a data object Builder.
func (*BuilderConfig) RegisterFlagsWithPrefix ¶
func (cfg *BuilderConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
RegisterFlagsWithPrefix registers flags with the given prefix.
func (*BuilderConfig) Validate ¶
func (cfg *BuilderConfig) Validate() error
Validate validates the BuilderConfig.
type FlushStats ¶ added in v3.5.0
type LabelFilterPredicate ¶ added in v3.5.0
A LabelFilterPredicate is a StreamsPredicate that requires that labels with the provided name pass a Keep function.
The name is is provided to the keep function to allow the same function to be used for multiple filter predicates.
Uses of LabelFilterPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.
type LabelMatcherPredicate ¶ added in v3.5.0
type LabelMatcherPredicate struct{ Name, Value string }
A LabelMatcherPredicate is a StreamsPredicate which requires a label named Name to exist with a value of Value.
type LogMessageFilterPredicate ¶ added in v3.5.0
A LogMessageFilterPredicate is a LogsPredicate that requires the log message of the entry to pass a Keep function.
type LogsPredicate ¶ added in v3.5.0
type LogsPredicate interface {
Predicate
// contains filtered or unexported methods
}
LogsPredicate is a Predicate that can be used to filter logs in a LogsReader.
type LogsReader ¶ added in v3.5.0
type LogsReader struct {
// contains filtered or unexported fields
}
LogsReader reads the set of logs from an Object.
func NewLogsReader ¶ added in v3.5.0
func NewLogsReader(obj *Object, sectionIndex int) *LogsReader
NewLogsReader creates a new LogsReader that reads from the logs section of the given object.
func (*LogsReader) Close ¶ added in v3.5.0
func (r *LogsReader) Close() error
Close closes the LogsReader and releases any resources it holds. Closed LogsReaders can be reused by calling LogsReader.Reset.
func (*LogsReader) MatchStreams ¶ added in v3.5.0
func (r *LogsReader) MatchStreams(ids iter.Seq[int64]) error
MatchStreams provides a sequence of stream IDs for the logs reader to match. LogsReader.Read will only return logs for the provided stream IDs.
MatchStreams may be called multiple times to match multiple sets of streams.
MatchStreams may only be called before reading begins or after a call to LogsReader.Reset.
func (*LogsReader) Read ¶ added in v3.5.0
Read reads up to the next len(s) records from the reader and stores them into s. It returns the number of records read and any error encountered. At the end of the logs section, Read returns 0, io.EOF.
func (*LogsReader) Reset ¶ added in v3.5.0
func (r *LogsReader) Reset(obj *Object, sectionIndex int)
Reset resets the LogsReader with a new object and section index to read from. Reset allows reusing a LogsReader without allocating a new one.
Any set predicate is cleared when Reset is called.
Reset may be called with a nil object and a negative section index to clear the LogsReader without needing a new object.
func (*LogsReader) SetPredicate ¶ added in v3.5.0
func (r *LogsReader) SetPredicate(p LogsPredicate) error
SetPredicate sets the predicate to use for filtering logs. LogsReader.Read will only return logs for which the predicate passes.
A predicate may only be set before reading begins or after a call to LogsReader.Reset.
type Metadata ¶ added in v3.5.0
type Metadata struct {
StreamsSections int // Number of streams sections in the Object.
LogsSections int // Number of logs sections in the Object.
}
Metadata holds high-level metadata about an Object.
type MetadataFilterPredicate ¶ added in v3.5.0
A MetadataFilterPredicate is a LogsPredicate that requires that metadata with the provided key pass a Keep function.
The key is provided to the keep function to allow the same function to be used for multiple filter predicates.
Uses of MetadataFilterPredicate are not eligible for page filtering and should only be used when a condition cannot be expressed by other basic predicates.
type MetadataMatcherPredicate ¶ added in v3.5.0
type MetadataMatcherPredicate struct{ Key, Value string }
A MetadataMatcherPredicate is a LogsPredicate that requires a metadata key named Key to exist with a value of Value.
type NotPredicate ¶ added in v3.5.0
type NotPredicate[P Predicate] struct{ Inner P }
A NotPredicate is a Predicate which requires its Inner predicate to be false.
type Object ¶ added in v3.5.0
type Object struct {
// contains filtered or unexported fields
}
An Object is a representation of a data object.
func FromBucket ¶ added in v3.5.0
FromBucket opens an Object from the given storage bucket and path.
func FromReaderAt ¶ added in v3.5.0
FromReadSeeker opens an Object from the given ReaderAt. The size argument specifies the size of the data object in bytes.
type OrPredicate ¶ added in v3.5.0
type OrPredicate[P Predicate] struct{ Left, Right P }
An OrPredicate is a Predicate which requires either its Left or Right predicate to be true.
type Predicate ¶ added in v3.5.0
type Predicate interface {
// contains filtered or unexported methods
}
Predicate is an expression used to filter entries in a data object.
type Record ¶ added in v3.5.0
type Record struct {
StreamID int64 // StreamID associated with the log record.
Timestamp time.Time // Timestamp of the log record.
Metadata labels.Labels // Set of metadata associated with the log record.
Line []byte // Line of the log record.
}
A Record is an individual log record in a data object.
type Stream ¶ added in v3.5.0
type Stream struct {
// ID of the stream. Stream IDs are unique across all sections in an object,
// but not across multiple objects.
ID int64
// MinTime and MaxTime denote the range of timestamps across all entries in
// the stream.
MinTime, MaxTime time.Time
// UncompressedSize is the total size of all the log lines and structured metadata values in the stream
UncompressedSize int64
// Labels of the stream.
Labels labels.Labels
}
A Stream is an individual stream in a data object.
type StreamsPredicate ¶ added in v3.5.0
type StreamsPredicate interface {
Predicate
// contains filtered or unexported methods
}
StreamsPredicate is a Predicate that can be used to filter streams in a StreamsReader.
type StreamsReader ¶ added in v3.5.0
type StreamsReader struct {
// contains filtered or unexported fields
}
StreamsReader reads the set of streams from an Object.
func NewStreamsReader ¶ added in v3.5.0
func NewStreamsReader(obj *Object, sectionIndex int) *StreamsReader
NewStreamsReader creates a new StreamsReader that reads from the streams section of the given object.
func (*StreamsReader) Close ¶ added in v3.5.0
func (r *StreamsReader) Close() error
Close closes the StreamsReader and releases any resources it holds. Closed StreamsReaders can be reused by calling StreamsReader.Reset.
func (*StreamsReader) Read ¶ added in v3.5.0
Read reads up to the next len(s) streams from the reader and stores them into s. It returns the number of streams read and any error encountered. At the end of the stream section, Read returns 0, io.EOF.
func (*StreamsReader) Reset ¶ added in v3.5.0
func (r *StreamsReader) Reset(obj *Object, sectionIndex int)
Reset resets the StreamsReader with a new object and section index to read from. Reset allows reusing a StreamsReader without allocating a new one.
Any set predicate is cleared when Reset is called.
Reset may be called with a nil object and a negative section index to clear the StreamsReader without needing a new object.
func (*StreamsReader) SetPredicate ¶ added in v3.5.0
func (r *StreamsReader) SetPredicate(p StreamsPredicate) error
SetPredicate sets the predicate to use for filtering logs. LogsReader.Read will only return logs for which the predicate passes.
SetPredicate returns an error if the predicate is not supported by LogsReader.
A predicate may only be set before reading begins or after a call to StreamsReader.Reset.
type TimeRangePredicate ¶ added in v3.5.0
type TimeRangePredicate[P Predicate] struct { StartTime, EndTime time.Time IncludeStart bool // Whether StartTime is inclusive. IncludeEnd bool // Whether EndTime is inclusive. }
A TimeRangePredicate is a Predicate which requires the timestamp of the entry to be within the range of StartTime and EndTime.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
dataset
Package dataset contains utilities for working with datasets.
|
Package dataset contains utilities for working with datasets. |
|
encoding
Package encoding provides utilities for encoding and decoding data objects.
|
Package encoding provides utilities for encoding and decoding data objects. |
|
result
Package result provides utilities for dealing with iterators that can fail during iteration.
|
Package result provides utilities for dealing with iterators that can fail during iteration. |
|
sections/logs
Package logs defines types used for the data object logs section.
|
Package logs defines types used for the data object logs section. |
|
sections/streams
Package streams defines types used for the data object streams section.
|
Package streams defines types used for the data object streams section. |
|
streamio
Package streamio defines interfaces shared by other packages for streaming binary data.
|
Package streamio defines interfaces shared by other packages for streaming binary data. |
|
util/bitmask
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length.
|
Package bitmask provides an API for creating and manipulating bitmasks of arbitrary length. |
|
util/bufpool
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets.
|
Package bufpool offers a pool of *bytes.Buffer objects that are placed into exponentially sized buckets. |
|
util/sliceclear
Package sliceclear provides a way to clear and truncate the length of a slice.
|
Package sliceclear provides a way to clear and truncate the length of a slice. |