README
¶
BagIns - BagIt Library in Go
This is a library for working with BagIt <http://en.wikipedia.org/wiki/BagIt>_
hierarchal file packages.
It uses the Go Language <http://golang.org/>_ to leverage its speed and
concurrency features to take advantage of manipulating bags in a cloud based
enviorment.
I'm fairly new to the Go Language so excuse the prototype nature of the code in general.
GoDocs For This Project: http://godoc.org/github.com/APTrust/bagins
Installation
This package is "go get-able" so simply import the package via 'github.com/APTrust/bagins' or use the go get command directly as follows::
go get github.com/APTrust/bagins
Modifying The Code
To keep this project "go get-able" you will need to create the top level directories under your go source directory and then checkout the code into it like so::
Switch to your Go source directory:
cd $GOPATH/src
Create the parent directories:
mkdir -p github.com/APTrust
Switch to the directory you just created:
cd github.com/APTrust
Checkout the source code via git:
git clone https://github.com/APTrust/bagins.git
Usage
The Library is still under development, there is basic code for formatting and writing Manifest and Tag files as well as utilities for file checksumming files. Soon I'll tie it all together with a basic bag creation code. I'll post examples of it here when complete.
Command Line Executable
This library includes a command line executable in the directory github.com/APtrust/bagins/bagmaker
Assuming you have checked out the code to your GOPATH, to build and compile this just execute::
>go install $GOPATH/src/github.com/APTrust/bagins/bagmaker/bagmaker.go
If you have GOBIN set this will deposite a compile file called bagmaker into your GOBIN directory.
Usage: ./bagmaker -dir -name -payload [-algo ]
Flags:
-algo <value> Checksum algorithm to use. md5, sha1, sha224, sha256,
sha512, or sha384.
-dir <value> Directory to create the bag.
-name <value> Name for the bag root directory.
-payload <value> Directory of files to parse into the bag
Documentation
¶
Overview ¶
Package for working with files stored using the BagIt specification (see below).
It facilitates the creation of bags, adding files to the bag payload and managing checksums for the file manifest as well as data stored in tag files.
For more information on Bag tagfiles see http://tools.ietf.org/html/draft-kunze-bagit-09#section-2.3
Index ¶
- Constants
- type Bag
- func (b *Bag) AddCustomTagfile(sourcePath string, destPath string, includeInTagManifests bool) error
- func (b *Bag) AddDir(src string) (errs []error)
- func (b *Bag) AddFile(src string, dst string) error
- func (b *Bag) AddTagfile(name string) error
- func (b *Bag) BagInfo() (*TagFile, error)
- func (b *Bag) GetManifest(manifestType, algorithm string) *Manifest
- func (b *Bag) GetManifests(manifestType string) []*Manifest
- func (b *Bag) ListFiles() ([]string, error)
- func (b *Bag) ListTagFiles() []string
- func (b *Bag) Path() string
- func (b *Bag) Save() (errs []error)
- func (b *Bag) TagFile(name string) (*TagFile, error)
- func (b *Bag) UnparsedTagFiles() ([]string, error)
- type Manifest
- type Payload
- type TagField
- type TagFieldList
- type TagFile
Constants ¶
const ( PayloadManifest = "payload_manifest" TagManifest = "tag_manifest" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bag ¶
type Bag struct {
Manifests []*Manifest
// contains filtered or unexported fields
}
Represents the basic structure of a bag which is controlled by methods.
func NewBag ¶
func NewBag(location string, name string, hashNames []string, createTagManifests bool) (*Bag, error)
Creates a new bag under the location directory and creates a bag root directory
with the provided name. Returns an error if the location does not exist or if the
bag already exist.
This constructor will automatically create manifests with the
specified hash algorithms. Supported algorithms include:
"md5", "sha1", "sha256", "sha512", "sha224" and "sha384"
If param createTagManifests is true, this will also create tag manifests
with the specified algorithms.
example:
NewBag("archive/bags", "bag-34323", ["sha256", "md5"], true)
func ReadBag ¶
Reads the directory provided as the root of a new bag and attemps to parse the file contents into payload, manifests and tagfiles.
func (*Bag) AddCustomTagfile ¶
func (b *Bag) AddCustomTagfile(sourcePath string, destPath string, includeInTagManifests bool) error
AddCustomTagfile adds a tag file of ANY format into the bag at the specified path without making any attempt to validate or even read the contents of the custom tag file.
The sourcePath param describes where the file should be copied from. The destPath param describes what the file's relative path in the bag should be, while includeInTagManifests describes whether the custom tag file should be included in the bag's tag manifests.
The destPath parameter cannot start with "data/" because that would put it in the payload directory, and it cannot start with a slash or contain "..".
Example:
bag.AddCustomTagfile("/home/june/cleaver.xml", "customtags/cleaver-meta.xml", true)
That says put "/home/june/cleaver.xml" into the bag at "customtags/cleaver-meta.xml" and record it in the tagmanifests with the appropriate checksums.
func (*Bag) AddDir ¶
Performans a Bag.AddFile on all files found under the src location including all subdirectories. example:
errs := b.AddDir("/tmp/mypreservationfiles")
func (*Bag) AddFile ¶
Adds a file specified by src parameter to the data directory under
the relative path and filename provided in the dst parameter.
example:
err := b.AddFile("/tmp/myfile.txt", "myfile.txt")
func (*Bag) AddTagfile ¶
Adds a tagfile to the bag with the filename provided,
creating whatever subdirectories are needed if supplied
as part of name parameter.
example:
err := b.AddTagfile("baginfo.txt")
Note that this is for adding tag files that adhere to
the "Text Tag File Format" described in section 2.2.4
of the BagIt spec at http://tools.ietf.org/html/draft-kunze-bagit-13.
For this type of tag file, you add name-value pairs to
the tag file's Data attribute, and this library ensures
that the data is written to the file according to the
specification.
The spec also allows you to add non-standard tag files
in ANY format. For that, see AddCustomTagfile.
func (*Bag) BagInfo ¶
Convienence method to return the bag-info.txt tag file if it exists. Since this is optional it will not be created by default and will return an error if you have not defined or added it yourself via Bag.AddTagfile
func (*Bag) GetManifest ¶
Returns the manifest with the specified algorithm and type, or nil. For example, GetManifest(PayloadManifest, "sha256") returns either a reference to manifest-sha256.txt or nil. GetManifest(TagManifest, "md5") returns a reference to tagmanifest-md5.txt or nil.
func (*Bag) GetManifests ¶
Returns the manifests of the specified type, or an empty slice. For example, GetManifests(PayloadManifest) returns all of the payload manifests.
func (*Bag) ListFiles ¶
Walks the bag directory and subdirectories and returns the filepaths found inside and any errors skipping files in the payload directory.
func (*Bag) ListTagFiles ¶
Lists all the current tag files the bag is tracking. These are the tag files that the bag has actually parsed. The bag may have any number of unparsed (and perhaps unreadable) tag files as well. For those, see UnparsedTagFiles()
func (*Bag) Save ¶
This method writes all the relevant tag and manifest files to finish off the bag.
func (*Bag) TagFile ¶
Finds a tagfile in by its relative path to the bag root directory.
example:
tf, err := b.TagFile("bag-info.txt")
func (*Bag) UnparsedTagFiles ¶
Returns a list of unparsed tag files, which includes any file not a manifest, not in the data directory, and not among the tag files passed into ReadBag().
type Manifest ¶
type Manifest struct {
Data map[string]string // Key is file path, value is checksum
// contains filtered or unexported fields
}
Manifest represents information about a BagIt manifest file. As of BagIt spec 0.97 this means only manifest-<algo>.txt and tagmanifest-<algo>.txt files.
For more information see:
manifest: http://tools.ietf.org/html/draft-kunze-bagit-09#section-2.1.3 tagmanifest: http://tools.ietf.org/html/draft-kunze-bagit-09#section-2.2.1
func NewManifest ¶
Returns a pointer to a new manifest or returns an error if improperly named.
func ReadManifest ¶
Opens a manifest file, parses attemps to parse the hashtype from the filename, parses the file contents and returns a pointer to a Manifest. Error slice may comprise multiple parsing errors when attempting to read data for fault tolerance.
func (*Manifest) Algorithm ¶
Returns the name of the manifest's hashing algorithm. "sha256", "md5", etc.
func (*Manifest) Name ¶
Returns a sting of the filename for this manifest file based on Path, BaseName and Algo
func (*Manifest) RunChecksums ¶
Calculates a checksum for files listed in the manifest and compares it to the value stored in manifest file. Returns an error for each file that fails the fixity check.
type Payload ¶
type Payload struct {
// contains filtered or unexported fields
}
Payloads describes a filepath location to serve as the data directory of a Bag and methods around managing content inside of it.
func NewPayload ¶
Returns a new Payload struct managing the path provied.
func (*Payload) Add ¶
func (p *Payload) Add(srcPath string, dstPath string, manifests []*Manifest) (map[string]string, error)
Adds the file at srcPath to the payload directory as dstPath and returns a checksum value as calulated by the provided hash. This function also writes the checksums into the proper manifests, so you don't have to.
Param manifests should be a slice of payload manifests, which you can get from a bag by calling:
bag.GetManifests(PayloadManifest)
Returns the checksums in the form of a map whose keys are the algorithms and values are the digests.
If you have an md5 manifest and a sha256 manifest, you'll get back a map that looks like this:
checksums["md5"] = "0a0a0a0a" checksums["sha256"] = "0b0b0b0b"
func (*Payload) AddAll ¶
func (p *Payload) AddAll(src string, manifests []*Manifest) (checksums map[string]map[string]string, errs []error)
Performs an add on every file under the directory supplied to the method. Returns a map of filenames and fixity values based on the hash function in the manifests.
Param manifests should be a slice of payload manifests, which you can get from a bag by calling:
bag.GetManifests(PayloadManifest)
If you have an md5 manifest and a sha256 manifest, you'll get back a map that looks like this:
checksums["file1.txt"] = { "md5": "0a0a0a0a", "sha256": "0b0b0b0b" } checksums["file2.xml"] = { "md5": "1a1a1a1a", "sha256": "1b1b1b1b" } checksums["file3.jpg"] = { "md5": "2a2a2a2a", "sha256": "2b2b2b2b" }
func (*Payload) OctetStreamSum ¶
Returns the octetstream sum and number of files of all the files in the payload directory. See the BagIt specification "Oxsum" field of the bag-info.txt file for more information.
type TagField ¶
type TagField struct {
// contains filtered or unexported fields
}
Represents a tag field as referenced in the standard BagIt tag file and used in bag-info.txt. It represents a standard key value pair with the label with corresponding value. For more information see http://tools.ietf.org/html/draft-kunze-bagit-09#section-2.2.2
func NewTagField ¶
Creates and returns a pointer to a new TagField
type TagFieldList ¶
type TagFieldList struct {
// contains filtered or unexported fields
}
Represents an ordered list of tag fields as specified for use with bag-info.txt in the bag it standard. It supports ordered, repeatable fields. http://tools.ietf.org/html/draft-kunze-bagit-09#section-2.2.2
func NewTagFieldList ¶
func NewTagFieldList() *TagFieldList
Returns a pointer to a new TagFieldList.
func (*TagFieldList) AddField ¶
func (fl *TagFieldList) AddField(field TagField)
Adds a Field to the end of the tag field list.
func (*TagFieldList) Fields ¶
func (fl *TagFieldList) Fields() []TagField
Returns a slice copy of the current tag fields.
func (*TagFieldList) RemoveField ¶
func (fl *TagFieldList) RemoveField(i int) error
Removes a field from the tag field list at the specified index. Returns an error if index out of bounds.
func (*TagFieldList) SetFields ¶
func (fl *TagFieldList) SetFields(fields []TagField)
Sets the tag field slice to use for the tag field list.
type TagFile ¶
type TagFile struct {
Data *TagFieldList // key value pairs of data for the tagfile.
// contains filtered or unexported fields
}
Represents a tag file object in the bag with its related fields.
func NewTagFile ¶
Creates a new tagfile object and returns it or returns an error if improperly formatted. The name argument represents the filepath of the tagfile, which must end in txt
func ReadTagFile ¶
Reads a tagfile, parsing the contents as tagfile field data and returning the TagFile object. name is the filepath to the tag file. It throws an error if contents cannot be properly parsed.
Source Files
Directories