README
¶
Audit archive Format Overview
Archive is a utility (part of the audit package) to capture and store large number of artifacts related to a NATS deployment.
For example, capture all 'monitor' endpoint for all servers in a given cluster. Or capture all details of all streams in a given account, or capture the streams state and configuration.
Primary design choices for archive:
- Single file: easy to share over email, web, chat, etc.
- Compressed: most artifacts have a lot of redundancy
- Indexed: can be queried in a generic way
- Default to JSON: easy for humans to grok, and for programs to process
An "archive" is a ZIP file that conforms to a specific schema convention.
The archive package provides Reader and Writer classes.
Artifact tagging and manifest
Rather than relying on path convention and contrived splitting, each artifact is added to the archive with a unique set of tags.
For example, the capture of a server VARZ endpoint may be tagged with:
cluster:awseast_barserver:bar_1type:server_varz
A filename for is derived from it's tags, and may look something like:
capture/clusters/awseast_bar/bar_1/variables.json
In addition to capture artifacts, the archive contains a few special files.
The most important special file is manifest.json which stores tags for each file in the archive.
The manifest allows programmatic discovery of the archive content, and can be used to build indices at read time.
An example usage is: list all streams discovered under a given account A.
Rather than having to infer this information from iterating and parsing files paths, the manifest can be traversed filtering by account:A and type:stream_info, and all matching files in the archive can be processed.
Archive organization
n.b. Do not rely on path parsing and path conventions, query using the manifest instead.
n.b.: If a server does not belong to a cluster, then cluster_name is unclustered for all the paths below.
Server-specific artifacts
${prefix}/clusters/${cluster_name}/${server_name}/${artifact_type}.json
Example: snapshot of health endpoint
Account artifacts
Each account artifact can appear multiple times as it is captured through different servers.
${prefix}/accounts/${account_name}/servers/${cluster_name}__${server_name}/${artifact_type}.json"
Example: account connections
Stream artifacts
Each stream artifact can appear multiple times as it is captured through different replicas.
${prefix}/accounts/${account_name}/streams/${stream_name}/replicas/${server_name}/${artifact_type}.json
Example: stream details
Server profiles
${prefix}/profiles/${cluster_name}/${server_name}__${profile_name}.prof
Example: memory profile
Special files
In addition to artifacts, each archive contains a few special files.
Capture metadata
${prefix}/capture_metadata.json
Contains information such as the date of capture, the username, the tool version, and more.
Manifest
${prefix}/manifest.json
Contains a list of all artifacts and a list of tags associated with each one.
Capture log
${prefix}/capture.log
A log file for the process that created the archive, in case it contains useful information about artifacts (and lack of thereof).
Documentation
¶
Index ¶
- Variables
- type Reader
- func (r *Reader) Close() error
- func (r *Reader) GetAccountNames() []string
- func (r *Reader) GetAccountStreamNames(accountName string) []string
- func (r *Reader) GetClusterNames() []string
- func (r *Reader) GetClusterServerNames(clusterName string) []string
- func (r *Reader) GetStreamServerNames(accountName, streamName string) []string
- func (r *Reader) Load(v any, queryTags ...*Tag) error
- type Tag
- func TagAccount(accountName string) *Tag
- func TagAccountConnections() *Tag
- func TagAccountInfo() *Tag
- func TagAccountJetStream() *Tag
- func TagAccountLeafs() *Tag
- func TagAccountSubs() *Tag
- func TagArtifactType(artifactType string) *Tag
- func TagCluster(clusterName string) *Tag
- func TagNoCluster() *Tag
- func TagProfileName(profileType string) *Tag
- func TagServer(serverName string) *Tag
- func TagServerAccounts() *Tag
- func TagServerConnections() *Tag
- func TagServerGateways() *Tag
- func TagServerHealth() *Tag
- func TagServerJetStream() *Tag
- func TagServerLeafs() *Tag
- func TagServerProfile() *Tag
- func TagServerRoutes() *Tag
- func TagServerSubs() *Tag
- func TagServerVars() *Tag
- func TagSpecial(special string) *Tag
- func TagStream(streamName string) *Tag
- func TagStreamInfo() *Tag
- type TagLabel
- type Writer
Constants ¶
This section is empty.
Variables ¶
var ErrMultipleMatches = fmt.Errorf("multiple files matched the given query")
ErrMultipleMatches is returned if multiple artifact matched the input combination of tags
var ErrNoMatches = fmt.Errorf("no file matched the given query")
ErrNoMatches is returned if no artifact matched the input combination of tags
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader encapsulates a reader for the actual underlying archive, and also provides indices for faster and more convenient iteration and querying of the archive content
func NewReader ¶
NewReader creates a new reader for the file at the given archivePath. Reader expect the file to comply to format and content created by a Writer in this same package. During creation, Reader creates in-memory indices to speed up subsequent queries.
func (*Reader) GetAccountNames ¶
GetAccountNames list the unique names of accounts found in the archive The list of names is sorted alphabetically
func (*Reader) GetAccountStreamNames ¶
GetAccountStreamNames list the unique stream names found in the archive for the given account The list of names is sorted alphabetically
func (*Reader) GetClusterNames ¶
GetClusterNames list the unique names of clusters found in the archive The list of names is sorted alphabetically
func (*Reader) GetClusterServerNames ¶
GetClusterServerNames list the unique server names found in the archive for the given cluster The list of names is sorted alphabetically
func (*Reader) GetStreamServerNames ¶
GetStreamServerNames list the unique server names found in the archive for the given stream in the given account The list of names is sorted alphabetically
type Tag ¶
func TagAccount ¶
func TagAccountConnections ¶
func TagAccountConnections() *Tag
func TagAccountInfo ¶
func TagAccountInfo() *Tag
func TagAccountJetStream ¶
func TagAccountJetStream() *Tag
func TagAccountLeafs ¶
func TagAccountLeafs() *Tag
func TagAccountSubs ¶
func TagAccountSubs() *Tag
func TagArtifactType ¶
func TagCluster ¶
func TagNoCluster ¶
func TagNoCluster() *Tag
func TagProfileName ¶
func TagServerAccounts ¶
func TagServerAccounts() *Tag
func TagServerConnections ¶
func TagServerConnections() *Tag
func TagServerGateways ¶
func TagServerGateways() *Tag
func TagServerHealth ¶
func TagServerHealth() *Tag
func TagServerJetStream ¶
func TagServerJetStream() *Tag
func TagServerLeafs ¶
func TagServerLeafs() *Tag
func TagServerProfile ¶
func TagServerProfile() *Tag
func TagServerRoutes ¶
func TagServerRoutes() *Tag
func TagServerSubs ¶
func TagServerSubs() *Tag
func TagServerVars ¶
func TagServerVars() *Tag
func TagSpecial ¶
func TagStreamInfo ¶
func TagStreamInfo() *Tag
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer encapsulates a zip writer for the underlying archive file, but also tracks metadata used by the Reader to construct indices
func NewWriter ¶
NewWriter creates a new writer for the file at the given archivePath. Writer creates a ZIP file whose content has additional structure and metadata. If archivePath is an existing file, it will be overwritten.
func (*Writer) Add ¶
Add serializes the given artifact to JSON and adds it to the archive, it creates a file name based on the provided tags and ensures uniqueness. The artifact is also added to the manifest for indexing, enabling tag-based querying in via Reader
Source Files