Documentation
¶
Overview ¶
package migrations implements a long-running migrations protocol for Kwil. This allows networks to upgrade to new networks over long periods of time, without any downtime.
The process is as follows:
A network votes to create a new network. If enough votes are attained, the process is started.
Once the process is started, each validator should create a new node to run the new network, which will connect to their current node. This new node will forward all changes from the old network to the new network.
The two networks will run in parallel until the old network reaches the scheduled shutdown block. At this point, the new network will take over and the old network will be shut down.
The old network cannot deploy databases, drop them, transfer balances, vote on any resolutions, or change their validator power.
For more information on conflict resolution, see https://github.com/kwilteam/kwil-db/wiki/Long%E2%80%90Running-Network-Migrations
Index ¶
- Constants
- Variables
- func ChangesetsDir(migrationDir string) string
- func CleanupResolutionsAfterMigration(ctx context.Context, db sql.DB) error
- func SnapshotDir(migrationDir string) string
- func Start(ctx context.Context, service *common.Service, eventStore listeners.EventStore) error
- type Accounts
- type BlockSpends
- type ChangesetMetadata
- type Database
- type MigrationConfig
- type MigrationDeclaration
- type Migrator
- func (m *Migrator) GetChangeset(height int64, index int64) ([]byte, error)
- func (m *Migrator) GetChangesetMetadata(height int64) (*ChangesetMetadata, error)
- func (m *Migrator) GetGenesisSnapshotChunk(chunkIdx uint32) ([]byte, error)
- func (m *Migrator) GetMigrationMetadata(ctx context.Context, status types.MigrationStatus) (*types.MigrationMetadata, error)
- func (m *Migrator) NotifyHeight(ctx context.Context, block *common.BlockContext, db Database, ...) error
- func (m *Migrator) PersistLastChangesetHeight(ctx context.Context, tx sql.Executor, height int64) error
- func (m *Migrator) StoreChangesets(height int64, changes <-chan any) error
- type NamespaceManager
- type Snapshotter
- type Validators
Constants ¶
const (
ListenerName = "migrations"
)
Changeset Extension polls the changesets from the old chain during migration. The changesets received are broadcasted to the eventstore under ChangesetMigrationEvent type for voting. The changesets are then applied to the new chain's database once the resolution is approved.
const (
MaxChunkSize = 1 * 1000 * 1000 // around 1MB
)
const MigrationVersion int = 0 // for migration metadata
Variables ¶
var ( ErrNoMoreChunksToRead = errors.New("no more chunks to read") ErrChangesetNotFound = errors.New("changeset not found") )
var (
ErrNoActiveMigration = fmt.Errorf("no active migration")
)
var MigrationResolution = resolutions.ResolutionConfig{ ConfirmationThreshold: big.NewRat(2, 3), ExpirationPeriod: 24 * 7 * time.Hour, ResolveFunc: func(ctx context.Context, app *common.App, resolution *resolutions.Resolution, block *common.BlockContext) error { return startMigration(ctx, app, resolution, block) }, }
MigrationResolution is the definition for the network migration vote type in Kwil's voting system.
Functions ¶
func ChangesetsDir ¶
ChangesetsDir returns the directory where changesets are stored, relative to the migration directory.
func CleanupResolutionsAfterMigration ¶
CleanupResolutionsAtStartup is called at startup to clean up the resolutions table. It does the below things: - Remove all the pending migration, changeset, validator join and validator remove resolutions - Fix the expiry heights of all the pending resolutions (how to handle this for offline migrations? we have no way to know the last height of the old chain)
func SnapshotDir ¶
Types ¶
type BlockSpends ¶
func (BlockSpends) MarshalBinary ¶
func (bs BlockSpends) MarshalBinary() ([]byte, error)
func (BlockSpends) Prefix ¶
func (bs BlockSpends) Prefix() byte
func (*BlockSpends) UnmarshalBinary ¶
func (bs *BlockSpends) UnmarshalBinary(bts []byte) error
type ChangesetMetadata ¶
type Database ¶
type Database interface { sql.TxMaker sql.ReadTxMaker sql.SnapshotTxMaker }
It should connect to the same Postgres database as kwild, but should be a different connection pool.
type MigrationConfig ¶
type MigrationConfig struct { // StartHeight is the block height at which the migration started on the old chain. StartHeight uint64 // End height is the block height at which the migration ended on the old chain. EndHeight uint64 // ListenAddress is the address of the kwild server to receive changesets from. ListenAddress string }
func (*MigrationConfig) Map ¶
func (c *MigrationConfig) Map() map[string]string
type MigrationDeclaration ¶
type MigrationDeclaration struct { // ActivationPeriod is the amount of blocks before the migration is activated. // It starts after the migration is approved via the voting system. // The intention is to allow validators to prepare for the migration. ActivationPeriod uint64 // Duration is the amount of blocks the migration will take to complete. Duration uint64 // Timestamp is the time the migration was created. It is set by the migration // creator. The primary purpose of it is to guarantee uniqueness of the serialized // MigrationDeclaration, since that is a requirement for the voting system. Timestamp string }
MigrationDeclaration creates a new migration. It is used to agree on terms of a migration, and is voted on using Kwil's vote store.
func (MigrationDeclaration) MarshalBinary ¶
func (md MigrationDeclaration) MarshalBinary() ([]byte, error)
MarshalBinary marshals the MigrationDeclaration into a binary format.
func (*MigrationDeclaration) UnmarshalBinary ¶
func (md *MigrationDeclaration) UnmarshalBinary(data []byte) error
UnmarshalBinary unmarshals the MigrationDeclaration from a binary format.
type Migrator ¶
migrator is responsible for managing the migrations. It is responsible for tracking any in-process migrations, snapshotting at the appropriate height, persisting changesets for the migration for each block as it occurs, and making that data available via RPC for the new node. Similarly, if the local process is the new node, it is responsible for reading changesets from the external node and applying them to the local database. The changesets are stored from the start height of the migration to the end height (both inclusive).
func SetupMigrator ¶
func SetupMigrator(ctx context.Context, db Database, snapshotter Snapshotter, accounts Accounts, dir string, migrationParams config.MigrationParams, validators Validators, logger log.Logger) (*Migrator, error)
SetupMigrator initializes the migrator instance with the necessary dependencies.
func (*Migrator) GetChangeset ¶
GetChangeset gets the changeset at the given height and index.
func (*Migrator) GetChangesetMetadata ¶
func (m *Migrator) GetChangesetMetadata(height int64) (*ChangesetMetadata, error)
GetChangesetMetadata gets the metadata for the changeset at the given height.
func (*Migrator) GetGenesisSnapshotChunk ¶
GetGenesisSnapshotChunk gets the snapshot chunk of Index at the given height.
func (*Migrator) GetMigrationMetadata ¶
func (m *Migrator) GetMigrationMetadata(ctx context.Context, status types.MigrationStatus) (*types.MigrationMetadata, error)
GetMigrationMetadata gets the metadata for the genesis snapshot, as well as the available changesets.
func (*Migrator) NotifyHeight ¶
func (m *Migrator) NotifyHeight(ctx context.Context, block *common.BlockContext, db Database, consensusTx sql.Executor) error
NotifyHeight notifies the migrator that a new block has been committed. It is called at the end of the block being applied, but before the block is committed to the database, in between tx.PreCommit and tx.Commit. consensusTx is needed to read the migration state from the database if any migration is active.
func (*Migrator) PersistLastChangesetHeight ¶
type NamespaceManager ¶
type Snapshotter ¶
type Snapshotter interface { CreateSnapshot(ctx context.Context, height uint64, snapshotID string, schemas, excludedTables []string, excludedTableData []string) error LoadSnapshotChunk(height uint64, format uint32, chunkIdx uint32) ([]byte, error) ListSnapshots() []*snapshotter.Snapshot }
snapshotter creates snapshots of the state at the migration height.