README
¶
Open Science Data Federation (OSDF) Client
The OSDF Client downloads files from the Open Science Data Federation through a series of caches. The OSDF is used by multiple organizations to effeciently transfer input and output data.
The client consists of two parts: a command-line tool named StashCP, and an HTCondor file transfer plugin.
When configured, this plugin will allow the user to specify transfer_input_files with the stash:// protocol which will be downloaded through the OSDF caches. An example of a submit file:
...
transfer_input_files = stash:///osgconnect/public/dweitzel/blast/queries/query1
...
Note: This repo is the continuation of opensciencegrid/stashcp; visit that repo for old issues and releases.
Building
Building is performed with the goreleaser tool. To build a snapshot (not release):
$ goreleaser --rm-dist --snapshot
The binaries will be located in ./dist directory.
Testing and Usage
Run this simple command to download a test file
$ ./stashcp /osgconnect/public/dweitzel/blast/queries/query1 ./
Configuration
stashcp is affected by the environment variables:
| Environment Variable | Description |
|---|---|
OSG_DISABLE_PROXY_FALLBACK |
Do not disable using proxies. By default, stashcp will try to use an HTTP proxy when connecting to a cache. If this environment variable is set (no value necessary, only if it's set), then stashcp will not fallback to no proxy if the proxy download fails. |
STASHCP_MINIMUM_DOWNLOAD_SPEED |
The lower limit a download will be cancelled, in bytes per second |
STASH_NAMESPACE_URL |
The URL to download the namespace and cache information. Default: https://topology.opensciencegrid.org/stashcache/namespaces |
Documentation
¶
Index ¶
- Variables
- func AddError(err error) bool
- func ByteCountSI(b int64) string
- func DoStashCPSingle(sourceFile string, destination string, methods []string, recursive bool) (bytesTransferred int64, err error)
- func DownloadHTTP(transfer TransferDetails, dest string, token string) (int64, error)
- func ErrorsRetryable() bool
- func Find(slice []string, val string) (int, bool)
- func GetBestStashcache(cacheListName string) ([]string, error)
- func GetErrors() string
- func HasPort(host string) bool
- func IsDir(dirUrl *url.URL, token string, namespace Namespace) (bool, error)
- func IsRetryable(err error) bool
- func UploadFile(src string, dest *url.URL, token string, namespace Namespace) (int64, error)
- type Cache
- type ConnectionSetupError
- type Namespace
- type NamespaceFull
- type OptionsStruct
- type ProgressReader
- type SlowTransferError
- type TransferDetails
- type TransferResults
Constants ¶
This section is empty.
Variables ¶
var CacheOverride bool
CacheOverride
var CachesJsonLocation string
var NearestCache string
Nearest cache
var NearestCacheList []string
List of caches, in order from closest to furthest
var UploadClient = &http.Client{}
Functions ¶
func ByteCountSI ¶
func DoStashCPSingle ¶
func DoStashCPSingle(sourceFile string, destination string, methods []string, recursive bool) (bytesTransferred int64, err error)
Start the transfer, whether read or write back
func DownloadHTTP ¶
func DownloadHTTP(transfer TransferDetails, dest string, token string) (int64, error)
DownloadHTTP - Perform the actual download of the file
func ErrorsRetryable ¶
func ErrorsRetryable() bool
ErrorsRetryable returns if the errors in the stack are retryable later
func Find ¶
Find takes a slice and looks for an element in it. If found it will return it's key, otherwise it will return -1 and a bool of false. From https://golangcode.com/check-if-element-exists-in-slice/
func GetBestStashcache ¶
func IsRetryable ¶
IsRetryable will return true if the error is retryable
Types ¶
type Cache ¶
type Cache struct {
AuthEndpoint string `json:"auth_endpoint"`
Endpoint string `json:"endpoint"`
Resource string `json:"resource"`
}
Cache
type ConnectionSetupError ¶
ConnectionSetupError is an error that is returned when a connection to the remote server fails
func (*ConnectionSetupError) Error ¶
func (e *ConnectionSetupError) Error() string
func (*ConnectionSetupError) Is ¶
func (e *ConnectionSetupError) Is(target error) bool
func (*ConnectionSetupError) Unwrap ¶
func (e *ConnectionSetupError) Unwrap() error
type Namespace ¶
type Namespace struct {
Caches []Cache `json:"caches"`
Path string `json:"path"`
ReadHTTPS bool `json:"readhttps"`
UseTokenOnRead bool `json:"usetokenonread"`
WriteBackHost string `json:"writebackhost"`
DirListHost string `json:"dirlisthost"`
}
Namespace holds the structure of stash namespaces
func GetNamespaces ¶
GetNamespaces returns the list of namespaces
func MatchNamespace ¶
MatchNamespace matches the namespace passed in to the namespaces in the list
func (*Namespace) GetCacheHosts ¶
func (*Namespace) MatchCaches ¶
MatchCaches compares the caches passed in (presumably from an ordered list of caches) to the caches for the namespace, and returns the intersection of the two
type NamespaceFull ¶
type OptionsStruct ¶
var Options OptionsStruct
type ProgressReader ¶
type ProgressReader struct {
// contains filtered or unexported fields
}
ProgressReader wraps the io.Reader to get progress Adapted from https://stackoverflow.com/questions/26050380/go-tracking-post-request-progress
func (*ProgressReader) Close ¶
func (pr *ProgressReader) Close() error
Close implments the close function of io.Closer
type SlowTransferError ¶
type SlowTransferError struct {
BytesTransferred int64
BytesPerSecond int64
BytesTotal int64
Duration time.Duration
}
SlowTransferError is an error that is returned when a transfer takes longer than the configured timeout
func (*SlowTransferError) Error ¶
func (e *SlowTransferError) Error() string
func (*SlowTransferError) Is ¶
func (e *SlowTransferError) Is(target error) bool
type TransferDetails ¶
type TransferDetails struct {
// Url is the url.URL of the cache and port
Url url.URL
// Proxy specifies if a proxy should be used
Proxy bool
}
func NewTransferDetails ¶
func NewTransferDetails(cache Cache, https bool) []TransferDetails
NewTransferDetails creates the TransferDetails struct with the given cache
Source Files
Directories