README
¶
⚠ This is prerelease software.
ACE Data Tool
ACE Data Tool is the primary way for researchers, data scientists, and AI/ML practitioners to work with large data sets in the ACT3 ecosystem.
ACE Data Tool:
- Uses the concept of a data bottle to encapsulate data sets
- Allows for the identification and documentation of metadata associated with a data bottle
- Facilitates reuse of a data set used while conducting research or an experiment
- Documents the lineage (starting point, any modifications, and current state) of a data set, when connected to the Telemetry Server
- Makes heavy use of the OCI data model and content addressable storage to avoid duplicate work
Data Tool provides command groups that can be used via the CLI to:
- Create (initialize) data bottles
- Add metadata to bottles
- Commit and push bottles to OCI registries and to the ACE Telemetry Server
- Pull and use bottles for experiments, research, and AL/ML projects
- Mirror container images from one registry to another, especially secure air-gapped environments
Data Tool is designed to help researchers and AI/ML practitioners:
- Practice responsible AI by providing a variety of metadata options for data bottles
- Automate tracking of data lineage by using the ACE Telemetry Server
Documentation
The documentation for ACE Data Tool is organized as follows:
- Quick Start Guide: provides documentation for downloading, installing, and configuring Data Tool
- User Guide: provides an overview of Data Tool, introduces new users to key concepts, and explains basic usage
- Bottle Creator Guide: documents the process and known best practices for creating high quality bottles
- Mirror User Guide: documents specific considerations and known best practices for using Data Tool's mirror features
- Labels and Selectors Guide: provides an overview of the relationship between labels created in ACE Data Tool and selectors used in ACE Telemetry Server to build queries and locate data bottles
How to Contribute
See the Developer Guide for contributing code to the ACE Data Tool repository.
Support
- Mattermost channel: create a post in the ACT3 Data Tool channel for assistance
- FAQ: see the frequently asked (and answered) questions for ACE Data Tool
Approved for public release: distribution unlimited. Case Number: AFRL-2024-1542
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
ace-dt
Package main is the entrypoint for ace-dt, which provides a set of tools that facilitate data set transfer to and from OCI registry storage, focused on OCI objects called bottles, but additionally working with general OCI objects.
|
Package main is the entrypoint for ace-dt, which provides a set of tools that facilitate data set transfer to and from OCI registry storage, focused on OCI objects called bottles, but additionally working with general OCI objects. |
|
ace-dt/cli
Package cli exports the ACE Data Tool command.
|
Package cli exports the ACE Data Tool command. |
|
ace-dt/internal/cli
Package cli contains all the subcommands for the CLI
|
Package cli contains all the subcommands for the CLI |
|
ace-dt/internal/cli/bottle
Package bottle provides command and subcommands for manipulating bottles and bottle metadata.
|
Package bottle provides command and subcommands for manipulating bottles and bottle metadata. |
|
ace-dt/internal/cli/git
Package git defines general git operations backed by oras.
|
Package git defines general git operations backed by oras. |
|
ace-dt/internal/cli/internal/flag
Package flag defines a command flag for specifying a telemetry URL.
|
Package flag defines a command flag for specifying a telemetry URL. |
|
ace-dt/internal/cli/internal/functesting
Package functesting provides helper utilities for facilitating functional and integration tests.
|
Package functesting provides helper utilities for facilitating functional and integration tests. |
|
ace-dt/internal/cli/internal/ui
Package ui provides UI helpers for the CLI.
|
Package ui provides UI helpers for the CLI. |
|
ace-dt/internal/cli/mirror
Package mirror provides commands for mirroring OCI and Bottle objects.
|
Package mirror provides commands for mirroring OCI and Bottle objects. |
|
ace-dt/internal/cli/oci
Package oci defines commands for general purpose oci manipulation back by oras.
|
Package oci defines commands for general purpose oci manipulation back by oras. |
|
ace-dt/internal/cli/pypi
Package pypi declares pypi related commands, such as serving an OCI backed pypi registry, and performing a sync.
|
Package pypi declares pypi related commands, such as serving an OCI backed pypi registry, and performing a sync. |
|
ace-dt/internal/cli/security
Package security contains the ace-dt security subcommands.
|
Package security contains the ace-dt security subcommands. |
|
Package docs embeds relevant documentation to be surfaced in the ace-dt CLI.
|
Package docs embeds relevant documentation to be surfaced in the ace-dt CLI. |
|
internal
|
|
|
actions
Package actions contains all the major functional actions that can be taken by the cli.
|
Package actions contains all the major functional actions that can be taken by the cli. |
|
actions/bottle
Package bottle defines bottle command and subcommand actions.
|
Package bottle defines bottle command and subcommand actions. |
|
actions/git
Package git facilitates executing actions for git repositories stored in an OCI registry.
|
Package git facilitates executing actions for git repositories stored in an OCI registry. |
|
actions/internal/format
Package format and console layout utilities
|
Package format and console layout utilities |
|
actions/mirror
Package mirror defines commands for performing mirror operations for OCI objects and bottles.
|
Package mirror defines commands for performing mirror operations for OCI objects and bottles. |
|
actions/oci
Package oci defines general purpose oci push and pull operations backed by oras.
|
Package oci defines general purpose oci push and pull operations backed by oras. |
|
actions/pypi
Package pypi facilitates executing the program as a proxy server for translating pypi requests into OCI registry operations.
|
Package pypi facilitates executing the program as a proxy server for translating pypi requests into OCI registry operations. |
|
actions/security
Package security defines the security actions.
|
Package security defines the security actions. |
|
archive
Package archive implements methods for working with archives, including archiving and compressing tar archives using tar and zstd libraries.
|
Package archive implements methods for working with archives, including archiving and compressing tar archives using tar and zstd libraries. |
|
bottle
Package bottle implements the local bottle handling functionality, providing facilities for:
|
Package bottle implements the local bottle handling functionality, providing facilities for: |
|
bottle/label
Package label handles part label files
|
Package label handles part label files |
|
bottle/status
Package status handles inspecting the filesystem for changes
|
Package status handles inspecting the filesystem for changes |
|
cache
Package cache implements local cached storage of part data.
|
Package cache implements local cached storage of part data. |
|
cache/bicbackend
Package bicbackend implements the management and storage of blob info cache data, including boltdb and memory implementations.
|
Package bicbackend implements the management and storage of blob info cache data, including boltdb and memory implementations. |
|
descriptor
Package descriptor implements a comparable descriptor and related de-duplication utilities
|
Package descriptor implements a comparable descriptor and related de-duplication utilities |
|
gen
Package main is a fake package for generating code.
|
Package main is a fake package for generating code. |
|
git
Package git implements tools for transferring git repositories to/from OCI compliant registiries.
|
Package git implements tools for transferring git repositories to/from OCI compliant registiries. |
|
git/cache
Package cache facilitates caching git and git-lfs objects.
|
Package cache facilitates caching git and git-lfs objects. |
|
git/cmd
Package cmd facilitates execution of git and git-lfs commands.
|
Package cmd facilitates execution of git and git-lfs commands. |
|
git/oci
Package oci facilitates transferring of git and git-lfs OCI artifacts.
|
Package oci facilitates transferring of git and git-lfs OCI artifacts. |
|
httplogger
Package httplogger provides functionality to attach a logger to an http request's context.
|
Package httplogger provides functionality to attach a logger to an http request's context. |
|
mirror
Package mirror implements the logic for the mirror commands.
|
Package mirror implements the logic for the mirror commands. |
|
mirror/blockbuf
Package blockbuf implements a functionality similar to "mbuffer".
|
Package blockbuf implements a functionality similar to "mbuffer". |
|
mirror/encoding
Package encoding implements the protocol used to serialize and deserialize data across the wire.
|
Package encoding implements the protocol used to serialize and deserialize data across the wire. |
|
mirror/multiplex
Package multiplex implements a multiplex and demultiplexer for streams of binary data
|
Package multiplex implements a multiplex and demultiplexer for streams of binary data |
|
mirror/testing
Package testing helps generate test data for use in testing telemetry
|
Package testing helps generate test data for use in testing telemetry |
|
oci
Package oci implements oci specific functionality, primarily related to interpreting and serializing manifests and manifest lists.
|
Package oci implements oci specific functionality, primarily related to interpreting and serializing manifests and manifest lists. |
|
orasutil
Package orasutil is a collection of oras extensions
|
Package orasutil is a collection of oras extensions |
|
print
Package print implements the pretty printing of OCI artifacts
|
Package print implements the pretty printing of OCI artifacts |
|
push
Package push provides cross-repository blob mounting features for use during a push operation.
|
Package push provides cross-repository blob mounting features for use during a push operation. |
|
python
Package python implements tools for dealing with python distribution packages
|
Package python implements tools for dealing with python distribution packages |
|
ref
Package ref provides functionality for interpreting and manipulating OCI references, including delineated data for scheme, registry, repository, name, tag, and digest that identifies an OCI object (or bottle).
|
Package ref provides functionality for interpreting and manipulating OCI references, including delineated data for scheme, registry, repository, name, tag, and digest that identifies an OCI object (or bottle). |
|
registry
Package registry provides options for remote OCI registry config and caching.
|
Package registry provides options for remote OCI registry config and caching. |
|
registry/cache
Package cache implements tools for OCI registry caching.
|
Package cache implements tools for OCI registry caching. |
|
security
Package security contains the logic for fetching artifact details, attaching SBOM's, etc.
|
Package security contains the logic for fetching artifact details, attaching SBOM's, etc. |
|
sign
Package sign provides an interface abstraction and implementation for creating/verifying signatures of OCI or docker manifest digests.
|
Package sign provides an interface abstraction and implementation for creating/verifying signatures of OCI or docker manifest digests. |
|
storage
Package storage defines a storage provider interface for abstracting storage operations during transfers.
|
Package storage defines a storage provider interface for abstracting storage operations during transfers. |
|
transfer/bottle
Package bottle provides functions for managing transfer of bottle objects to and from an OCI registry, including configuring a pulled bottle, and establishing local metadata and file structure.
|
Package bottle provides functions for managing transfer of bottle objects to and from an OCI registry, including configuring a pulled bottle, and establishing local metadata and file structure. |
|
ui
Package ui defines unified user interface to decouple the information source from the presentation
|
Package ui defines unified user interface to decouple the information source from the presentation |
|
ui/tracker
Package tracker provides progress tracking
|
Package tracker provides progress tracking |
|
util
Package util implements general utility functionality for operations such as:
|
Package util implements general utility functionality for operations such as: |
|
pkg
|
|
|
apis
Package apis implements shared functionality intended for api use, including configuration schema manipulation.
|
Package apis implements shared functionality intended for api use, including configuration schema manipulation. |
|
apis/config.dt.act3-ace.io/v1alpha1
Package v1alpha1 provides the ServerConfiguration used for configuring the Telemetry server.
|
Package v1alpha1 provides the ServerConfiguration used for configuring the Telemetry server. |
|
cache
Package cache implements local cached storage of part data.
|
Package cache implements local cached storage of part data. |
|
conf
Package conf provides configuration management for ace-dt and ace-dt library consumers.
|
Package conf provides configuration management for ace-dt and ace-dt library consumers. |
|
registry
Package registry provides options for remote OCI registry config and caching.
|
Package registry provides options for remote OCI registry config and caching. |
|
telemetry
Package telemetry provides utilities for telemetry bottle resolution and event updating.
|
Package telemetry provides utilities for telemetry bottle resolution and event updating. |
|
transfer/bottle
Package bottle provides functions for managing transfer of bottle objects to and from an OCI registry, including configuring a pulled bottle, and establishing local metadata and file structure.
|
Package bottle provides functions for managing transfer of bottle objects to and from an OCI registry, including configuring a pulled bottle, and establishing local metadata and file structure. |
Click to show internal directories.
Click to hide internal directories.