jobs

package
v0.5.0-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2024 License: Apache-2.0 Imports: 25 Imported by: 0

README

jobs

Table of Contents

  1. Description
  2. Structure and Organisation
  3. Class Diagram
  4. Functionality
  5. Data Types
  6. Testing
  7. Proposed Functionality/Requirements
  8. References

Specification

Description

This pacakge manages local jobs and their allocation, including relation to execution environments, etc. It will manage jobs through whatever executor it's running (Vontainer, VM, Direct_exe, Java etc).

Structure and Organisation

Here is quick overview of the contents of this directory:

TBD

Class Diagram

The class diagram for the jobs package is shown below.

Source file

jobs Class Diagram

Rendered from source file
!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/dms/jobs"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml
Functionality

TBD

Note: the functionality of DMS is being currently developed. See the proposed section for the suggested design of interfaces and methods.

Data Types

TBD

Note: the functionality of DMS is being currently developed. See the proposed section for the suggested data types.

Testing

TBD

Proposed Functionality / Requirements
List of issues

All issues that are related to the implementation of dms package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

Interfaces & Methods
proposed Job interface
type Job_interface interface {
	// extends graph.Vertex interface in order to be able to build 
	// job structure as a special instantiation of a graph
	// commented it since it not shown in the class diagram
    //graph.Vertex

	// additional methods
	getPods() []jobs.Pod
}

getPods: will fetch list of pods currently running on the machine

proposed Pod interface
type Pod_interface interface {
	combineCapabilities() dms.Capability
}

combineCapabilities: will combine the capability requirements of different jobs to calculate total capabality needed for a Pod.

type JobLink_interface interface {
	// extends graph.Edge interface
	// commented as it is not shown in class diagram
    // dms.graph.Edge

    validateRelations()
}

validateRelations: It validates the JobLink properties provided.

proposed Allocation interface
type Allocation_interface interface {
	// start the allocation execution via the executor package
    start()                
    
    // send message to another actor
    sendMessage(telemetry.Message)
    
    // register the allocation with the node
    register()
}

start: starts the allocation execution

sendMessage: sends a message to another actor (Node/Allocation)

register: registers the allocation with the node that it is running on

Data types
  • proposed dms.jobs.Job: Nunet job which will be sent to the network wrapped as a BidRequest. If needed it will have child jobs to be executed. The relation between parent and child job needs to be specified.
type Job struct {
	// JobID is the unique identifier of the job 
	JobId 				types.ID
    
    // all information that is needed to convert a job description into ExecutionRequest
	// in principle, the logic of inputs and outputs can be expressed 
	// within graph structure, when (and if) we develop general interface 
	// of storage.StorageProvider and make it extend Vertex interface
	// until that we need to use special structures outside Graph interface
	inputs 		Slice[types.SpecConfig]
	outputs		Slice[types.SpecConfig]

    RequiredCapability dms.Capability

    // child jobs that have to be executed as part of this job
    // specified using JobLink structure
    children Slice[dms.jobs.JobLink] // in graph.Vertex type this is outEdge

    // extend graph.Vertex type 
    // given that, all children jobs will be expressed via the this interface
	// just expanding on that here
	// clearly saying that a job can have only one parent but many children
    // commented it as we have not included this in the class diagram
    // dms.graph.Vertex
}

  • proposed dms.jobs.JobLink: specifies the properties that relate parent and child job.
type JobLink struct {
	// extends Edge struct
	// commented as it is not shown in class diagram
    // dms.graph.Edge // but types Vertexes on both end of the link to Jobs...

	// RelationProperties captures the relation between the parent and the child job
    RelationProperties types.SpecConfig
	
    // Child job that is linked to the parent job
    RelatedJob dms.jobs.Job
}
  • proposed dms.jobs.Pod: collection of jobs that need to be executed on the same machine.
type Pod struct {
	// identifer of the Pod
    ID types.ID 
	
    // Combined capability required by the Pod
    podCapability dms.Capability
	
    // Jobs that are part of the Pod
    jobs Slice[dms.jobs.Job]
}
  • proposed dms.jobs.Allocation: maps the job to the process on a executor. Each Allocation is an Actor.
type Allocation struct {
    // allocation is an actor (so extends Actor struct)
    // commented since it is not shown in class diagram
    // dms.orchestrator.Actor 
	
    // identifier for the allocation
    ID      dms.jobs.AllocationID
	
    // identifier of the job that had led to this allocation
    jobID   dms.jobs.Job.JobID

    // identifier of the node that is executing the allocation
	NodeID  dms.node.NodeID
	
    // identifier of the node that created the job
    Source  types.ID

    // Mailbox of the Allocation
    Mailbox    dms.orchestrator.Mailbox
}

  • proposed dms.jobs.AllocationID: identifier for Allocation objects.
type AllocationID struct {
	// UUID is the unique identifier of the Allocation
    UUID types.ID.UUID

	// CID is a Content identifier that can be used in DHT or otherwise;
	// CIDs may be especially useful for routing messages directly to allocations via Kademlia DHT
	// it is not clear if we are going to use it now
	CID string 
}

References

Allocation as an Actor: As per initial specification of NuNet ontology / nomenclature, Allocation is considered as an Actor. That makes a running job a first class citizen of NuNet's Actor model, so being able to send and receive messages and maintain state.

Documentation

Index

Constants

View Source
const (
	BidRequestTopic    = "/nunet/deployment"
	BidRequestBehavior = "/dms/deployment/request"
	BidRequestTimeout  = 5 * time.Second
	BidReplyBehavior   = "/dms/deployment/bid"

	VerifyEdgeConstraintBehavior = "/dms/deployment/constraint/edge"
	VerifyEdgeConstraintTimeout  = 5 * time.Second

	CommitDeploymentBehavior     = "/dms/deployment/commit"
	CommitDeploymentTimeout      = 3 * time.Second
	AllocationDeploymentBehavior = "/dms/deployment/allocate"
	AllocationDeploymentTimeout  = 5 * time.Second
	RevertDeploymentBehavior     = "/dms/deployment/revert"
	AllocationStartBehavior      = "/dms/deployment/start"
	AllocationStartTimeout       = 5 * time.Second

	MinEnsembleDeploymentTime = 15 * time.Second

	MaxBidMultiplier = 8

	SubnetCreateBehavior          = "/dms/node/subnet/create"
	SubnetDestroyBehavior         = "/dms/node/subnet/destroy"
	SubnetAddPeerBehavior         = "/dms/node/subnet/add-peer"
	SubnetRemovePeerBehavior      = "/dms/node/subnet/remove-peer"
	SubnetAcceptPeerBehavior      = "/dms/node/subnet/accept-peer"
	SubnetMapPortBehavior         = "/dms/node/subnet/map-port"
	SubnetDNSAddRecordBehavior    = "/dms/node/subnet/dns/add-record"
	SubnetUnmapPortBehavior       = "/dms/node/subnet/unmap-port"
	SubnetDNSRemoveRecordBehavior = "/dms/node/subnet/dns/remove-record"
)
View Source
const MaxPermutations = 1_000_000

Variables

View Source
var (
	ErrProvisioningFailed = errors.New("failed to provision the ensemble")
	ErrDeploymentFailed   = errors.New("failed to create deployment")
	ErrTODO               = errors.New("TODO")
)

Functions

func DeploymentStatusString

func DeploymentStatusString(d DeploymentStatus) string

Types

type Allocation

type Allocation struct {
	ID string

	Actor actor.Actor

	Job Job
	// contains filtered or unexported fields
}

Allocation represents an allocation

func NewAllocation

func NewAllocation(actor actor.Actor, details AllocationDetails, resourceManager types.ResourceManager) (*Allocation, error)

NewAllocation creates a new allocation given the actor.

func (*Allocation) Run

func (a *Allocation) Run(ctx context.Context) error

Run creates the executor based on the execution engine configuration.

func (*Allocation) Start

func (a *Allocation) Start() error

Start the actor of the allocation.

func (*Allocation) Status

func (a *Allocation) Status(_ context.Context) Status

Status returns information about the allocated/usage of resources and execution status of workload.

func (*Allocation) Stop

func (a *Allocation) Stop(ctx context.Context) error

Stop stops the running executor

type AllocationConfig

type AllocationConfig struct {
	Executor    AllocationExecutor // the executor of the allocation
	Resources   types.Resources    // the HW resources required by the allocation
	Execution   types.SpecConfig   // the allocation execution configuration
	DNSName     string             // the internal DNS name of the allocation
	Keys        []string           // names of the authorized ssh keys for the allocation
	Provision   []string           // names of provisioning scripts to run (in order)
	HealthCheck string             // name of the script to run for health checks
}

AllocationConfig is the configuration of an allocation

type AllocationDeploymentConfig

type AllocationDeploymentConfig struct {
	Executor         AllocationExecutor
	Resources        types.Resources
	Execution        types.SpecConfig
	ProvisionScripts map[string][]byte
}

type AllocationDeploymentRequest

type AllocationDeploymentRequest struct {
	EnsembleID  string
	NodeID      string
	Allocations map[string]AllocationDeploymentConfig
}

type AllocationDeploymentResponse

type AllocationDeploymentResponse struct {
	OK          bool
	Error       string
	Allocations map[string]actor.Handle
	Tokens      ucan.TokenList // tokens granted for allocation capabilities
}

type AllocationDetails

type AllocationDetails struct {
	Job      Job
	NodeID   string
	SourceID string
}

AllocationDetails encapsulates the dependencies to the constructor.

type AllocationExecutor

type AllocationExecutor string

AllocationExecutor is the executor reoquired for the allocation

const (
	ExecutorFirecracker AllocationExecutor = "firecracker"
	ExecutorDocker      AllocationExecutor = "docker"
	ExecutorNull        AllocationExecutor = "null"
)

type AllocationManifest

type AllocationManifest struct {
	ID       string       // allocation unique id
	NodeID   string       // allocation node
	Handle   actor.Handle // handle of the allocation control actor
	DNSName  string       // (internal) DNS name of the allocation
	PrivAddr string       // (VPN) private IP address of the allocation peer
	Ports    map[int]int  // port mapping, public -> private
}

type AllocationStartRequest

type AllocationStartRequest struct{}

type AllocationStartResponse

type AllocationStartResponse struct {
	OK    bool
	Error string
}

type AllocationStatus

type AllocationStatus string

AllocationStatus is a representation of the execution status

type Bid

type Bid struct {
	V1 *BidV1
}

Bid is the version struct for Bids in response to a bid request

func (*Bid) EnsembleID

func (b *Bid) EnsembleID() string

func (*Bid) Handle

func (b *Bid) Handle() actor.Handle

func (*Bid) Location

func (b *Bid) Location() Location

func (*Bid) NodeID

func (b *Bid) NodeID() string

func (*Bid) Peer

func (b *Bid) Peer() string

func (*Bid) Sign

func (b *Bid) Sign(key did.Provider) error

func (*Bid) SignatureData

func (b *Bid) SignatureData() ([]byte, error)

func (*Bid) Validate

func (b *Bid) Validate() error

type BidRequest

type BidRequest struct {
	V1 *BidRequestV1
}

BidRequest is a versioned bid request

type BidRequestV1

type BidRequestV1 struct {
	NodeID      string               // unique identifier for a node, within the context of an ensemble
	Executors   []AllocationExecutor // list of required executors to support the allocation(s)
	Resources   types.Resources      // (aggregate) required hardware resources
	Location    LocationConstraints  // node location constraints
	PublicPorts struct {
		Static  []int // statically configured public ports
		Dynamic int   // number of dynamic ports
	}
}

BidRequestV1 is v1 of bid requests for a node to use for deployment

type BidV1

type BidV1 struct {
	EnsembleID string       // unique identifier for the ensemble
	NodeID     string       // unique identifier for a node; matches the id of the BidRequest to which this bid pertains
	Peer       string       // the peer ID of the node
	Location   Location     // the location of the node
	Handle     actor.Handle // the handle of the actor submitting the bid
	Signature  []byte
}

BidV1 is v1 of the bid structure

type CommitDeploymentRequest

type CommitDeploymentRequest struct {
	EnsembleID string
	NodeID     string
}

type CommitDeploymentResponse

type CommitDeploymentResponse struct {
	OK    bool
	Error string
}

type Coordinate

type Coordinate struct {
	// contains filtered or unexported fields
}

func (*Coordinate) Empty

func (c *Coordinate) Empty() bool

type DeploymentStatus

type DeploymentStatus int
const (
	DeploymentStatusPreparing DeploymentStatus = iota
	DeploymentStatusGenerating
	DeploymentStatusCommitting
	DeploymentStatusProvisioning
	DeploymentStatusRunning
	DeploymentStatusFailed
)

type EdgeConstraint

type EdgeConstraint struct {
	S, T string // (named) nodes connected by the edge
	RTT  uint   // maximum edge RTT in milliseconds
	BW   uint   // minimum edge bandwidth in Kbps
}

EdgeConstraint is a constraint for a network edge between two nodes

type EnsembleBidRequest

type EnsembleBidRequest struct {
	ID            string       // unique identifier of an ensemble (in the context of the orchestrator)
	Request       []BidRequest // list of node bid requests
	PeerExclusion []string     // list of peers to exclude from bidding
}

EnsembleBidRequest is a request for a bids pertaining to an ensemble

Note: At the moment, we embed a bid request for each node This is fine for small deployments, and a small network, which is what we have. For large deployments however, this won't scale and we will have to create aggregate bid requests for related group of nodes and also handle them with bid request aggregators who control multiple nodes.

func (*EnsembleBidRequest) Validate

func (b *EnsembleBidRequest) Validate() error

type EnsembleConfig

type EnsembleConfig struct {
	V1 *EnsembleConfigV1
}

EnsembleConfig is the versioned structure that contains the ensemble configuration

func (*EnsembleConfig) Allocation

func (e *EnsembleConfig) Allocation(allocID string) (AllocationConfig, bool)

func (*EnsembleConfig) Allocations

func (e *EnsembleConfig) Allocations() map[string]AllocationConfig

func (*EnsembleConfig) Clone

func (e *EnsembleConfig) Clone() EnsembleConfig

func (*EnsembleConfig) EdgeConstraints

func (e *EnsembleConfig) EdgeConstraints() []EdgeConstraint

func (*EnsembleConfig) Node

func (e *EnsembleConfig) Node(nodeID string) (NodeConfig, bool)

func (*EnsembleConfig) Nodes

func (e *EnsembleConfig) Nodes() map[string]NodeConfig

func (*EnsembleConfig) Validate

func (e *EnsembleConfig) Validate() error

config validation

type EnsembleConfigV1

type EnsembleConfigV1 struct {
	Allocations map[string]AllocationConfig // (named) allocations in the ensemble
	Nodes       map[string]NodeConfig       // (named) nodes in the ensemble
	Edges       []EdgeConstraint            // network edge constraints
	Supervisor  SupervisorConfig            // supervision structure
	Keys        map[string]string           // (named) ssh public keys relevant to the allocation
	Scripts     map[string][]byte           // (named) provisioning scripts
}

EnsembleConfigV1 is version 1 of the configuration for an ensemble

type EnsembleManifest

type EnsembleManifest struct {
	ID           string                        // ensemble globally unique id
	Orchestrator actor.Handle                  // orchestrator actor
	Allocations  map[string]AllocationManifest // allocation name -> manifest
	Nodes        map[string]NodeManifest       // node name -> manifest
}

func (*EnsembleManifest) Clone

func (mf *EnsembleManifest) Clone() EnsembleManifest

type GeoLocator

type GeoLocator struct {
	// contains filtered or unexported fields
}

func NewGeoLocator

func NewGeoLocator() (*GeoLocator, error)

func (*GeoLocator) Coordinate

func (geo *GeoLocator) Coordinate(loc Location) (Coordinate, error)

type Job

type Job struct {
	ID               string
	Resources        types.Resources
	Execution        types.SpecConfig
	ProvisionScripts map[string][]byte
}

type Location

type Location struct {
	Region  string // geographic region of the location (required)
	Country string // country (code or name) of the location (optional)
	City    string // city of the location; optional but country must be specified if not empty
	ASN     uint   // Autonomous System Number for the location (optional)
	ISP     string // Internet Service Provider name for the location (optional)
}

Location is a geographical location on Planet Earth

func (*Location) Includes

func (l *Location) Includes(other Location) bool

type LocationConstraints

type LocationConstraints struct {
	Accept []Location // acceptable location constraints (disjunction)
	Reject []Location // negative location constraints (conjunction); eg !USA for GPDR purposes
}

LocationConstraints provides the node location placement constraints

type NodeConfig

type NodeConfig struct {
	Allocations []string            // the list of (named) allocations in the node
	Ports       []PortConfig        // the port mapping configuration for the node
	Location    LocationConstraints // the geographical location constraints for the node
	Peer        string              // (optional) a fixed peer for the node

}

NodeConfig is the configuration of a distinct DMS node

type NodeManifest

type NodeManifest struct {
	ID          string       // node unique id
	Peer        string       // peer where the node is running
	Handle      actor.Handle // handle of the control actor for the node
	PubAddrss   []string     // public IP4/6 address of the node peer
	Location    Location     // location of the peer
	Allocations []string     // allocations in the nod
}

type Orchestrator

type Orchestrator struct {
	// contains filtered or unexported fields
}

func NewOrchestrator

func NewOrchestrator(actor actor.Actor, network network.Network, cfg EnsembleConfig) (*Orchestrator, error)

func (*Orchestrator) Config

func (o *Orchestrator) Config() EnsembleConfig

func (*Orchestrator) Deploy

func (o *Orchestrator) Deploy(expiry time.Time) error

func (*Orchestrator) ID

func (o *Orchestrator) ID() string

func (*Orchestrator) Manifest

func (o *Orchestrator) Manifest() EnsembleManifest

func (*Orchestrator) Shutdown

func (o *Orchestrator) Shutdown() error

func (*Orchestrator) Status

func (o *Orchestrator) Status() DeploymentStatus

func (*Orchestrator) Stop

func (o *Orchestrator) Stop()

type PortConfig

type PortConfig struct {
	Public     int    // the public port 0 for any
	Private    int    // the private mapping
	Allocation string // the allocation where the port is mapped
}

PortConfig is the configuration for a port mapping a public port to a private port in an allocation

type RestartAllocationRequest

type RestartAllocationRequest struct {
	AllocationID string
}

type RestartAllocationResponse

type RestartAllocationResponse struct {
	OK    bool
	Error string
}

type RevertDeploymentMessage

type RevertDeploymentMessage struct {
	EnsembleID     string
	AllocationsIDs []string
}

type Status

type Status struct {
	JobResources types.Resources
	Status       AllocationStatus
}

Status holds the status of an allocation.

type SubnetAcceptPeerRequest

type SubnetAcceptPeerRequest struct {
	SubnetID string
	PeerID   string
	IP       string
}

type SubnetAcceptPeerResponse

type SubnetAcceptPeerResponse struct {
	OK    bool
	Error string
}

type SubnetAddPeerRequest

type SubnetAddPeerRequest struct {
	SubnetID string
	PeerID   string
	IP       string
}

type SubnetAddPeerResponse

type SubnetAddPeerResponse struct {
	OK    bool
	Error string
}

type SubnetCreateRequest

type SubnetCreateRequest struct {
	SubnetID     string
	IP           string
	RoutingTable map[string]string
}

type SubnetCreateResponse

type SubnetCreateResponse struct {
	OK    bool
	Error string
}

type SubnetDNSAddRecordRequest

type SubnetDNSAddRecordRequest struct {
	SubnetID   string
	DomainName string
	IP         string
}

type SubnetDNSAddRecordResponse

type SubnetDNSAddRecordResponse struct {
	OK    bool
	Error string
}

type SubnetDNSRemoveRecordRequest

type SubnetDNSRemoveRecordRequest struct {
	SubnetID   string
	DomainName string
}

type SubnetDNSRemoveRecordResponse

type SubnetDNSRemoveRecordResponse struct {
	OK    bool
	Error string
}

type SubnetDestroyRequest

type SubnetDestroyRequest struct {
	SubnetID string
}

type SubnetDestroyResponse

type SubnetDestroyResponse struct {
	OK    bool
	Error string
}

type SubnetMapPortRequest

type SubnetMapPortRequest struct {
	SubnetID   string
	Protocol   string
	SourceIP   string
	SourcePort string
	DestIP     string
	DestPort   string
}

type SubnetMapPortResponse

type SubnetMapPortResponse struct {
	OK    bool
	Error string
}

type SubnetRemovePeerRequest

type SubnetRemovePeerRequest struct {
	SubnetID string
	PeerID   string
}

type SubnetRemovePeerResponse

type SubnetRemovePeerResponse struct {
	OK    bool
	Error string
}

type SubnetUnmapPortRequest

type SubnetUnmapPortRequest struct {
	SubnetID   string
	Protocol   string
	SourceIP   string
	SourcePort string
	DestIP     string
	DestPort   string
}

type SubnetUnmapPortResponse

type SubnetUnmapPortResponse struct {
	OK    bool
	Error string
}

type SupervisorConfig

type SupervisorConfig struct {
	Strategy    SupervisorStrategy // the strategy for the supervision group
	Allocations []string           // allocations in this supervision group
	Children    []SupervisorConfig // allocation children for recursive groups
}

SupervisorConfig is the supervisory structure configuration for the ensemble

type SupervisorStrategy

type SupervisorStrategy string

SupervisoryStrategy is the name of a supervision strategy

const (
	StrategyOneForOne  SupervisorStrategy = "OneForOne"
	StrategyAllForOne  SupervisorStrategy = "AllForOne"
	StrategyRestForOne SupervisorStrategy = "RestForOne"
)

type VerifyEdgeConstraintRequest

type VerifyEdgeConstraintRequest struct {
	EnsembleID string // the ensemble identifier
	S, T       string // the peer IDs of the edge S->T
	RTT        uint   //  maximum RTT in ms (if > 0)
	BW         uint   // minim BW in Kbps
}

type VerifyEdgeConstraintResponse

type VerifyEdgeConstraintResponse struct {
	OK    bool
	Error string
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
JackTT - Gopher 🇻🇳