migrations

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 7, 2025 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Overview

package migrations implements a long-running migrations protocol for Kwil. This allows networks to upgrade to new networks over long periods of time, without any downtime.

The process is as follows:

  1. A network votes to create a new network. If enough votes are attained, the process is started.

  2. Once the process is started, each validator should create a new node to run the new network, which will connect to their current node. This new node will forward all changes from the old network to the new network.

  3. The two networks will run in parallel until the old network reaches the scheduled shutdown block. At this point, the new network will take over and the old network will be shut down.

The old network cannot deploy databases, drop them, transfer balances, vote on any resolutions, or change their validator power.

For more information on conflict resolution, see https://github.com/kwilteam/kwil-db/wiki/Long%E2%80%90Running-Network-Migrations

Index

Constants

View Source
const (
	ListenerName = "migrations"
)

Changeset Extension polls the changesets from the old chain during migration. The changesets received are broadcasted to the eventstore under ChangesetMigrationEvent type for voting. The changesets are then applied to the new chain's database once the resolution is approved.

View Source
const (
	MaxChunkSize = 1 * 1000 * 1000 // around 1MB
)
View Source
const MigrationVersion int = 0 // for migration metadata

Variables

View Source
var (
	ErrNoMoreChunksToRead = errors.New("no more chunks to read")
	ErrChangesetNotFound  = errors.New("changeset not found")
)
View Source
var (
	ErrNoActiveMigration = fmt.Errorf("no active migration")
)
View Source
var MigrationResolution = resolutions.ResolutionConfig{
	ConfirmationThreshold: big.NewRat(2, 3),
	ExpirationPeriod:      24 * 7 * time.Hour,
	ResolveFunc: func(ctx context.Context, app *common.App, resolution *resolutions.Resolution, block *common.BlockContext) error {
		return startMigration(ctx, app, resolution, block)
	},
}

MigrationResolution is the definition for the network migration vote type in Kwil's voting system.

Functions

func ChangesetsDir

func ChangesetsDir(migrationDir string) string

ChangesetsDir returns the directory where changesets are stored, relative to the migration directory.

func CleanupResolutionsAfterMigration

func CleanupResolutionsAfterMigration(ctx context.Context, db sql.DB) error

CleanupResolutionsAtStartup is called at startup to clean up the resolutions table. It does the below things: - Remove all the pending migration, changeset, validator join and validator remove resolutions - Fix the expiry heights of all the pending resolutions (how to handle this for offline migrations? we have no way to know the last height of the old chain)

func SnapshotDir

func SnapshotDir(migrationDir string) string

func Start

func Start(ctx context.Context, service *common.Service, eventStore listeners.EventStore) error

Types

type Accounts

type Accounts interface {
	GetBlockSpends() []*accounts.Spend
}

accounts tracks all the spends that have occurred in the block.

type BlockSpends

type BlockSpends struct {
	Spends []*accounts.Spend
}

func (BlockSpends) MarshalBinary

func (bs BlockSpends) MarshalBinary() ([]byte, error)

func (BlockSpends) Prefix

func (bs BlockSpends) Prefix() byte

func (*BlockSpends) UnmarshalBinary

func (bs *BlockSpends) UnmarshalBinary(bts []byte) error

type ChangesetMetadata

type ChangesetMetadata struct {
	Height     int64   `json:"height"`
	Chunks     int64   `json:"chunks"`
	ChunkSizes []int64 `json:"chunk_sizes"`
}

type Database

type Database interface {
	sql.TxMaker
	sql.ReadTxMaker
	sql.SnapshotTxMaker
}

It should connect to the same Postgres database as kwild, but should be a different connection pool.

type MigrationConfig

type MigrationConfig struct {
	// StartHeight is the block height at which the migration started on the old chain.
	StartHeight uint64

	// End height is the block height at which the migration ended on the old chain.
	EndHeight uint64

	// ListenAddress is the address of the kwild server to receive changesets from.
	ListenAddress string
}

func (*MigrationConfig) Map

func (c *MigrationConfig) Map() map[string]string

type MigrationDeclaration

type MigrationDeclaration struct {
	// ActivationPeriod is the amount of blocks before the migration is activated.
	// It starts after the migration is approved via the voting system.
	// The intention is to allow validators to prepare for the migration.
	ActivationPeriod uint64
	// Duration is the amount of blocks the migration will take to complete.
	Duration uint64
	// Timestamp is the time the migration was created. It is set by the migration
	// creator. The primary purpose of it is to guarantee uniqueness of the serialized
	// MigrationDeclaration, since that is a requirement for the voting system.
	Timestamp string
}

MigrationDeclaration creates a new migration. It is used to agree on terms of a migration, and is voted on using Kwil's vote store.

func (MigrationDeclaration) MarshalBinary

func (md MigrationDeclaration) MarshalBinary() ([]byte, error)

MarshalBinary marshals the MigrationDeclaration into a binary format.

func (*MigrationDeclaration) UnmarshalBinary

func (md *MigrationDeclaration) UnmarshalBinary(data []byte) error

UnmarshalBinary unmarshals the MigrationDeclaration from a binary format.

type Migrator

type Migrator struct {
	DB Database

	Logger log.Logger
	// contains filtered or unexported fields
}

migrator is responsible for managing the migrations. It is responsible for tracking any in-process migrations, snapshotting at the appropriate height, persisting changesets for the migration for each block as it occurs, and making that data available via RPC for the new node. Similarly, if the local process is the new node, it is responsible for reading changesets from the external node and applying them to the local database. The changesets are stored from the start height of the migration to the end height (both inclusive).

func SetupMigrator

func SetupMigrator(ctx context.Context, db Database, snapshotter Snapshotter, accounts Accounts, dir string, migrationParams config.MigrationParams, validators Validators, logger log.Logger) (*Migrator, error)

SetupMigrator initializes the migrator instance with the necessary dependencies.

func (*Migrator) GetChangeset

func (m *Migrator) GetChangeset(height int64, index int64) ([]byte, error)

GetChangeset gets the changeset at the given height and index.

func (*Migrator) GetChangesetMetadata

func (m *Migrator) GetChangesetMetadata(height int64) (*ChangesetMetadata, error)

GetChangesetMetadata gets the metadata for the changeset at the given height.

func (*Migrator) GetGenesisSnapshotChunk

func (m *Migrator) GetGenesisSnapshotChunk(chunkIdx uint32) ([]byte, error)

GetGenesisSnapshotChunk gets the snapshot chunk of Index at the given height.

func (*Migrator) GetMigrationMetadata

func (m *Migrator) GetMigrationMetadata(ctx context.Context, status types.MigrationStatus) (*types.MigrationMetadata, error)

GetMigrationMetadata gets the metadata for the genesis snapshot, as well as the available changesets.

func (*Migrator) NotifyHeight

func (m *Migrator) NotifyHeight(ctx context.Context, block *common.BlockContext, db Database, consensusTx sql.Executor) error

NotifyHeight notifies the migrator that a new block has been committed. It is called at the end of the block being applied, but before the block is committed to the database, in between tx.PreCommit and tx.Commit. consensusTx is needed to read the migration state from the database if any migration is active.

func (*Migrator) PersistLastChangesetHeight

func (m *Migrator) PersistLastChangesetHeight(ctx context.Context, tx sql.Executor, height int64) error

func (*Migrator) StoreChangesets

func (m *Migrator) StoreChangesets(height int64, changes <-chan any) error

storeChangeset persists a changeset to the migrations/changesets directory.

type NamespaceManager

type NamespaceManager interface {
	ListPostgresSchemasToDump() ([]string, error)
}

type Snapshotter

type Snapshotter interface {
	CreateSnapshot(ctx context.Context, height uint64, snapshotID string, schemas, excludedTables []string, excludedTableData []string) error
	LoadSnapshotChunk(height uint64, format uint32, chunkIdx uint32) ([]byte, error)
	ListSnapshots() []*snapshotter.Snapshot
}

snapshotter creates snapshots of the state at the migration height.

type Validators

type Validators interface {
	GetValidators() []*types.Validator
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
JackTT - Gopher 🇻🇳