diff --git a/.github/README.md b/.github/README.md index 8692905..588d343 100644 --- a/.github/README.md +++ b/.github/README.md @@ -1,85 +1,50 @@ -
-

@supervysor

-
- ![banner](../assets/banner.png) -## Content - -- [What is the supervysor?](#what-is-the-supervysor) -- [How does it work?](#how-does-it-work) -- [Requirements](#requirements) -- [Installation](#installation) -- [Usage](#usage) -- [Examples](#examples) - -## What is the supervysor? - -Participating in a KYVE data pool such as Cosmoshub or Osmosis requires running two nodes: the KYVE protocol node and the data source node (e.g., full node of Cosmoshub, Osmosis, etc.). However, running these full nodes in parallel can result in high storage requirements (approximately 10TB for Osmosis), leading to increased operational costs and inefficient resource utilization. This inefficiency arises because the node begins synchronizing from the start, even though it only requires storage for a certain range of blocks. Additionally, the node lacks information about the progress of the KYVE pool and the already validated data, making pruning impractical when running a node as a KYVE data source. - -However, if the synchronization process is halted, the node cannot fulfill its responsibilities as data source effectively. To overcome this challenge, the supervysor is introduced as a solution. The supervysor manages the data source node process based on the requirements of a KYVE data pool. It ensures that the node synchronizes only up to the necessary extent and continues to provide data even when the synchronization process is paused. - -By implementing the supervysor, the synchronization process is optimized, reducing unnecessary storage usage and operational costs. The node can focus on synchronizing up to the required point, thus efficiently utilizing resources while fulfilling its role as a data source for the KYVE pool. - -## How does it work? - -### Structure - -The supervysor is a process manager that is wrapped around a Tendermint node or the Cosmovisor. After the initial start, the node-height and the pool-height of the KYVE data pool are queried at a specified interval, after which the difference between the two values is calculated. If the difference is higher than ```height_difference_max``` , the node is set to the `Ghost Mode`. In this mode, the synchronization process is stopped by making the address book inaccessible and by starting the node without seeds and with a modified laddr. This ensures that the node cannot reach other peers and thus cannot synchronize new blocks. If the difference is smaller than ```height_difference_min```, the address book is made accessible again and the node is started with specified seeds so that peers can be found and the synchronization process can continue. If the difference is smaller than ```height_difference_max``` and larger than ```height_difference_min``` the current mode is kept. In both modes, the endpoints are accessible to the protocol node, so the required data remains accessible even if the node does not synchronize. -

- +Run your KYVE Protocol node as efficient as possible

-To keep memory requirements as low as possible, we need to specify both a maximum value for how far the data source node can synchronize beyond the current pool height and the matching pruning settings to make sure that data can only be pruned after validation. Derived from this, these values were calculated as followed: - -* `height_difference_max = max_bundle_size / upload_interval * 60 * 60 * 24 * 2` (maximum bundles for 2 days) -* `height_difference_min = height_difference_max / 2` (maximum bundles for 1 day) - -These values ensure that -* the data source node will always be 1 day ahead to the latest pool-height, -* the data source node will not sync to the latest height, because it will stop syncing when the required blocks for the next 2 days are stored locally, -* the data source node has a time window of 1 day to connect to peers to continue syncing before the pool catches up. - -### Pruning - -Aside from the optimized syncing process, pruning already validated data is the second role of the supervysor to fulfill its goal of reducing disk storage requirements. Therefore, a custom pruning method is used, which relies on the provided Tendermint functionality of pruning all blocks until a specified height. In the context of the supervysor, this until-height should always be lower than the latest validated height of the KYVE data pool to ensure no data is pruned that needs validation. Unfortunately, the node has to be stopped to execute the pruning process, while a pruning-interval needs specification in hours. During this interval, the supervysor halts the current node process, prunes all validated blocks, and restarts the node. Due to the required time to connect with peers and to prevent the pool from catching up with the node, the pruning process is only initiated if the node is in GhostMode. If the node is in NormalMode, even if the interval reaches the pruning threshold, pruning will be enabled immediately after the node enters GhostMode. Additionally, it is recommended to set the pruning-interval to a value of at least six hours to ensure there is enough time to find peers before the pool catches up. - -## Requirements - -The supervysor manages the process of the data source node. First of all, it should be ensured that this node can run successfully, which can be tested by trying to sync the first `n` blocks. In addition, to successfully participate in a KYVE data pool, it is necessary to create a protocol validator and join a data pool. Further information can be found here: https://docs.kyve.network/validators/protocol_nodes/overview - -Make sure your Go version is at least ```1.20```. -## Installation - -To install the latest version of `supervysor`, run the following command: +
+ License: Apache-2.0 -```bash -go install github.com/KYVENetwork/supervysor/cmd/supervysor@latest -``` + License: Apache-2.0 -To install a previous version, you can specify the version: + License: Apache-2.0 -```bash -go install github.com/KYVENetwork/supervysor/cmd/supervysor@v0.1.0 -``` + License: Apache-2.0 +
-_Optional:_ If you have issues to successfully run the `go install` command, make sure to export the following to your environment: +
+ + Twitter + + + Discord + + + Telegram + +
-```bash -env GIT_TERMINAL_PROMPT=1 -``` +
-Run `supervysor version` to check the installed version. +> [!IMPORTANT] +> In this README you will find information on contribution guidelines and +> detailed documentation about the low level implementation of the supervysor. +> +> You can find the complete documentation on installation and usage +> here: **[https://docs.kyve.network/tools/supervysor/overview](https://docs.kyve.network/tools/supervysor/overview)** +> +# Build from Source -You can also install from source by pulling the supervysor repository and switching to the correct version and building +You can install the supervysor from source by pulling the supervysor repository and switching to the correct version and building as follows: ```bash git clone git@github.com:KYVENetwork/supervysor.git cd supervysor git checkout tags/vx.x.x -b vx.x.x -make supervysor +make ``` This will build supervysor in `/build` directory. Afterwards you may want to put it into your machine's PATH like @@ -89,60 +54,55 @@ as follows: cp build/supervysor ~/go/bin/supervysor ``` -## Usage +# How to contribute -To use the supervysor, you first need to initialize it: +Generally, you can contribute to the supervysor via Pull Requests. The following branch conventions are required: -```bash -supervysor init ---binary string 'path to chain binaries (e.g. ~/go/bin/osmosisd)' ---chain-id string 'KYVE chain-id' ---home string 'path to home directory (e.g. ~/.osmosisd)' ---metrics string 'exposing Prometheus metrics ("true" or "false")' ---pool-id int 'KYVE pool-id' ---seeds string 'seeds for the node to connect' ---pruning-interval int 'block-pruning interval (hours) (default 24)' ---fallback-endpoints string 'additional endpoints to query KYVE pool height [optional]' -``` +- **feat/\***: for a new feature +- **fix/\***: for fixing a bug +- **refactor/\***: for improving code maintainability without changing any logic +- **docs/\***: for upgrading documentation around the supervysor +- **test/\***: for addition or changes in tests -This command creates a config file at ```~/.supervysor/config.toml``` which is editable and required to start the supervysor. +For committing new changes [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) have +to be used. -To start the supervysor after the successful initialisation, run the following command: +Once the Pull Request is ready, it can be opened against the `main` branch. Once the owners have approved +your Pull Request, your changes get merged. -```bash -supervysor start -``` +# Releases -Then the supervysor starts the chain binaries or cosmovisor to manage the syncing process depending on the required data of the KYVE pool. +For creating release versions [Semantic Versioning](https://semver.org/) is used. A release is created +by manually creating the release over GitHub. For that the latest commit gets tagged with the new version, +which will also be the title of the release. Once the release is created it gets automatically published to +https://pkg.go.dev/github.com/KYVENetwork/supervysor. -## Examples +# What is the supervysor? -### 1. Run a Cosmovisor Osmosis node with the supervysor +Participating in a KYVE data pool such as CosmosHub or Osmosis requires running two nodes: the KYVE protocol node and the data source node (e.g., full node of CosmosHub, Osmosis, etc.). However, running these full nodes in parallel can result in high storage requirements (approximately >10TB for Osmosis), leading to increased operational costs and inefficient resource utilization. This inefficiency arises because the node begins synchronizing from the start, even though it only requires storage for a certain range of blocks. Additionally, the node lacks information about the progress of the KYVE pool and the already validated data, making pruning impractical when running a node as a KYVE data source. -To run an Osmosis node with the Cosmovisor you have to download and set up the correct binaries. You can see a more detailed -introduction [here](https://docs.osmosis.zone/networks/join-mainnet/). +However, if the synchronization process is halted, the node cannot fulfill its responsibilities as data source effectively. To overcome this challenge, the supervysor is introduced as a solution. The supervysor manages the data source node process based on the requirements of a KYVE data pool. It ensures that the node synchronizes only up to the necessary extent and continues to provide data even when the synchronization process is paused and prunes data that already has been validated. -Verify the correct installation and setup with the successful start of the node syncing ~ 2,000 blocks: +By implementing the supervysor, the synchronization process and the disk storage requirements are optimized, reducing unnecessary operational costs. The node can focus on synchronizing up to the required point, thus efficiently utilizing resources while fulfilling its role as a data source for the KYVE pool. -```bash -cosmovisor run start [flags] -``` +# Structure + +The supervysor is a process manager that is wrapped around a Tendermint node or the Cosmovisor. After the initial start, the node-height and the pool-height of the KYVE data pool are queried at a specified interval, after which the difference between the two values is calculated. If the difference is higher than ```height_difference_max``` , the node is set to the `Ghost Mode`. In this mode, the synchronization process is stopped by making the address book inaccessible and by starting the node without seeds and with a modified laddr. This ensures that the node cannot reach other peers and thus cannot synchronize new blocks. If the difference is smaller than ```height_difference_min```, the address book is made accessible again and the node is started with specified seeds so that peers can be found and the synchronization process can continue. If the difference is smaller than ```height_difference_max``` and larger than ```height_difference_min``` the current mode is kept. In both modes, the endpoints are accessible to the protocol node, so the required data remains accessible even if the node does not synchronize. -With your node being able to run using Cosmovisor, you can stop the process and install the supervysor to start optimize this process for KYVE purposes. After the [installation](#installation), you can initialize the supervysor with the following command: +

+ +

-```bash -supervysor init \ ---binary '/root/go/bin/cosmovisor' \ ---chain-id 'kyve-1' \ ---home '/root/.osmosisd' \ ---pool-id 1 \ ---seeds '6bcdbcfd5d2c6ba58460f10dbcfde58278212833@osmosis.artifact-staking.io:26656,ade4d8bc8cbe014af6ebdf3cb7b1e9ad36f412c0@seeds.polkachu.com:12556' -``` +To keep memory requirements as low as possible, we need to specify a maximum value for how far the data source node can synchronize beyond the current pool height, that is calculated as followed: -After the successful initialization you can start your node with: +* `height_difference_max = max_bundle_size / upload_interval * 60 * 60 * 24 * 2` (maximum bundles for 2 days) +* `height_difference_min = height_difference_max / 2` (maximum bundles for 1 day) -```bash -supervysor start -``` +These values ensure that +* the data source node will always be 1 day ahead to the latest pool-height, +* the data source node will not sync to the latest height, because it will stop syncing when the required blocks for the next 2 days are stored locally, +* the data source node has a time window of 1 day to connect to peers to continue syncing before the pool catches up. + +## Pruning -The supervysor then will start an Osmosis node as data source for the pool with the ID 27 of the Korellia network. \ No newline at end of file +Aside from the optimized syncing process, pruning already validated data is the second role of the supervysor to fulfill its goal of reducing disk storage requirements. Therefore, a custom pruning method is used, which relies on the provided Tendermint functionality of pruning all blocks and the state until a specified height. In the context of the supervysor, this until-height should always be lower than the latest validated height of the KYVE data pool to ensure no data is pruned that needs validation. Unfortunately, the node has to be stopped to execute the pruning process, while a pruning-interval needs specification in hours. During this interval, the supervysor halts the current node process, prunes blocks and state until the already validated height, and restarts the node. Due to the required time to connect with peers and to prevent the pool from catching up with the node, the pruning process is only initiated if the node is in GhostMode. If the node is in NormalMode, even if the interval reaches the pruning threshold, pruning will be enabled immediately after the node enters GhostMode. Additionally, it is recommended to set the pruning-interval to a value of at least six hours to ensure there is enough time to find peers before the pool catches up. \ No newline at end of file diff --git a/Makefile b/Makefile index 3213ca1..a8369a8 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ -VERSION := v0.2.3 +VERSION := v0.3.0 ldflags := $(LDFLAGS) -ldflags += -X main.Version=$(VERSION) +ldflags += -X main.version=$(VERSION) ldflags := $(strip $(ldflags)) BUILD_FLAGS := -ldflags '$(ldflags)' diff --git a/cmd/supervysor/backup.go b/cmd/supervysor/commands/backup.go similarity index 90% rename from cmd/supervysor/backup.go rename to cmd/supervysor/commands/backup.go index 8de3e6b..fe603ae 100644 --- a/cmd/supervysor/backup.go +++ b/cmd/supervysor/commands/backup.go @@ -1,22 +1,19 @@ -package main +package commands import ( "fmt" "path/filepath" + "github.com/KYVENetwork/supervysor/cmd/supervysor/commands/helpers" + + "github.com/KYVENetwork/supervysor/utils" + "github.com/KYVENetwork/supervysor/store" "github.com/KYVENetwork/supervysor/backup" - "github.com/KYVENetwork/supervysor/cmd/supervysor/helpers" "github.com/spf13/cobra" ) -var ( - compressionType string - destPath string - maxBackups int -) - func init() { backupCmd.Flags().StringVar(&home, "home", "", "path to home directory (e.g. /root/.osmosisd)") if err := backupCmd.MarkFlagRequired("home"); err != nil { @@ -28,12 +25,16 @@ func init() { backupCmd.Flags().StringVar(&compressionType, "compression", "", "compression type to compress backup directory ['tar.gz', 'zip', '']") backupCmd.Flags().IntVar(&maxBackups, "max-backups", 0, "number of kept backups (set 0 to keep all)") + + backupCmd.Flags().BoolVar(&optOut, "opt-out", false, "disable the collection of anonymous usage data") } var backupCmd = &cobra.Command{ Use: "backup", Short: "Backup data directory", Run: func(cmd *cobra.Command, args []string) { + utils.TrackBackupEvent(optOut) + backupDir, err := helpers.GetBackupDir() if err != nil { logger.Error("failed to get ksync home directory", "err", err) diff --git a/cmd/supervysor/helpers/helpers.go b/cmd/supervysor/commands/helpers/helpers.go similarity index 78% rename from cmd/supervysor/helpers/helpers.go rename to cmd/supervysor/commands/helpers/helpers.go index 2f09d30..7374344 100644 --- a/cmd/supervysor/helpers/helpers.go +++ b/cmd/supervysor/commands/helpers/helpers.go @@ -4,8 +4,12 @@ import ( "fmt" "net/http" "os" + "os/exec" "path/filepath" "strconv" + "strings" + + "cosmossdk.io/log" "github.com/spf13/viper" @@ -15,6 +19,8 @@ import ( cfg "github.com/tendermint/tendermint/config" ) +var logger = log.NewLogger(os.Stdout) + func CreateDestPath(backupDir string, latestHeight int64) (string, error) { if err := os.Mkdir(filepath.Join(backupDir, strconv.FormatInt(latestHeight, 10)), 0o755); err != nil { return "", fmt.Errorf("error creating backup directory: %v", err) @@ -76,6 +82,45 @@ func GetBackupDir() (string, error) { return backupDir, nil } +func GetHomePathFromBinary(binaryPath string) string { + cmdPath, err := exec.LookPath(binaryPath) + if err != nil { + logger.Error(fmt.Sprintf("failed to lookup binary path: %s", err.Error())) + os.Exit(1) + } + + startArgs := make([]string, 0) + + // if we run with cosmovisor we start with the cosmovisor run command + if strings.HasSuffix(binaryPath, "cosmovisor") { + startArgs = append(startArgs, "run") + } + + out, err := exec.Command(cmdPath, startArgs...).Output() + if err != nil { + logger.Error("failed to get output of binary") + os.Exit(1) + } + + // here we search for a specific line in the binary output when simply + // executed without arguments. In the output, the default home path + // is printed, which is parsed and used by KSYNC + for _, line := range strings.Split(string(out), "\n") { + if strings.Contains(line, "--home") { + if strings.Count(line, "\"") != 2 { + logger.Error(fmt.Sprintf("did not found default home path in help line: %s", line)) + os.Exit(1) + } + + return strings.Split(line, "\"")[1] + } + } + + logger.Error("did not found default home path in entire binary output") + os.Exit(1) + return "" +} + func GetSupervysorDir() (string, error) { home, err := os.UserHomeDir() if err != nil { diff --git a/cmd/supervysor/init.go b/cmd/supervysor/commands/init.go similarity index 74% rename from cmd/supervysor/init.go rename to cmd/supervysor/commands/init.go index d67c6b9..ddc4c34 100644 --- a/cmd/supervysor/init.go +++ b/cmd/supervysor/commands/init.go @@ -1,47 +1,34 @@ -package main +package commands import ( "errors" "fmt" "os" + "strings" - "golang.org/x/exp/slices" + "github.com/KYVENetwork/supervysor/cmd/supervysor/commands/helpers" - "github.com/KYVENetwork/supervysor/cmd/supervysor/helpers" - "github.com/KYVENetwork/supervysor/types" + "github.com/KYVENetwork/supervysor/utils" + + "golang.org/x/exp/slices" "github.com/KYVENetwork/supervysor/settings" + "github.com/KYVENetwork/supervysor/types" "github.com/pelletier/go-toml/v2" "github.com/spf13/cobra" ) -var ( - abciEndpoint string - binary string - chainId string - fallbackEndpoints string - home string - metrics bool - metricsPort int - poolId int - seeds string - pruningInterval int - - cfg types.SupervysorConfig -) - func init() { - initCmd.Flags().StringVar(&binary, "binary", "", "path to chain binaries or cosmovisor (e.g. /root/go/bin/cosmovisor)") + initCmd.Flags().StringVarP(&binary, "binary", "b", "", "path to chain binaries or cosmovisor (e.g. /root/go/bin/cosmovisor)") if err := initCmd.MarkFlagRequired("binary"); err != nil { panic(fmt.Errorf("flag 'binary-path' should be required: %w", err)) } initCmd.Flags().StringVar(&home, "home", "", "path to home directory (e.g. /root/.osmosisd)") - if err := initCmd.MarkFlagRequired("home"); err != nil { - panic(fmt.Errorf("flag 'home-path' should be required: %w", err)) - } + + initCmd.Flags().StringVar(&config, "config", "", "path to config directory (default: ~/.supervysor/)") initCmd.Flags().IntVar(&poolId, "pool-id", 0, "KYVE pool-id") if err := initCmd.MarkFlagRequired("pool-id"); err != nil { @@ -55,11 +42,15 @@ func init() { initCmd.Flags().StringVar(&chainId, "chain-id", "kyve-1", "KYVE chain-id") + initCmd.Flags().BoolVar(&optOut, "opt-out", false, "disable the collection of anonymous usage data") + initCmd.Flags().StringVar(&fallbackEndpoints, "fallback-endpoints", "", "additional endpoints to query KYVE pool height") initCmd.Flags().IntVar(&pruningInterval, "pruning-interval", 24, "block-pruning interval (hours)") - initCmd.Flags().BoolVar(&metrics, "metrics", true, "exposing Prometheus metrics (true or false)") + initCmd.Flags().BoolVar(&statePruning, "state-pruning", true, "state pruning enabled") + + initCmd.Flags().BoolVar(&metrics, "metrics", false, "exposing Prometheus metrics (true or false)") initCmd.Flags().IntVar(&metricsPort, "metrics-port", 26660, "port for metrics server") @@ -76,25 +67,31 @@ var initCmd = &cobra.Command{ return fmt.Errorf("not supported chain-id") } + // if no home path was given get the default one if home == "" { - logger.Error("home directory can not be empty") - return fmt.Errorf("empty home directory path") + home = helpers.GetHomePathFromBinary(binary) } if pruningInterval <= 6 { logger.Error("pruning-interval should be higher than 6 hours") } + utils.TrackInitEvent(chainId, optOut) + if err := settings.InitializeSettings(binary, home, poolId, false, seeds, chainId, fallbackEndpoints); err != nil { logger.Error("could not initialize settings", "err", err) return err } logger.Info("successfully initialized settings") - configPath, err := helpers.GetSupervysorDir() - if err != nil { - logger.Error("could not get supervysor directory path", "err", err) - return err + if config == "" { + configPath, err = helpers.GetSupervysorDir() + if err != nil { + logger.Error("could not get supervysor directory path", "err", err) + return err + } + } else { + configPath = config } if _, err = os.Stat(configPath + "/config.toml"); err == nil { @@ -109,7 +106,7 @@ var initCmd = &cobra.Command{ } logger.Info("initializing supverysor...") - config := types.SupervysorConfig{ + supervysorConfig := types.SupervysorConfig{ ABCIEndpoint: abciEndpoint, BinaryPath: binary, ChainId: chainId, @@ -123,9 +120,10 @@ var initCmd = &cobra.Command{ PoolId: poolId, PruningInterval: pruningInterval, Seeds: seeds, + StatePruning: statePruning, StateRequests: false, } - b, err := toml.Marshal(config) + b, err := toml.Marshal(supervysorConfig) if err != nil { logger.Error("could not unmarshal config", "err", err) return err @@ -137,7 +135,7 @@ var initCmd = &cobra.Command{ return err } - _, err = getSupervysorConfig() + _, err = getSupervysorConfig(configPath) if err != nil { logger.Error("could not load config file", "err", err) return err @@ -153,13 +151,12 @@ var initCmd = &cobra.Command{ } // getSupervysorConfig returns the supervysor config.toml file. -func getSupervysorConfig() (*types.SupervysorConfig, error) { - configPath, err := helpers.GetSupervysorDir() - if err != nil { - return nil, fmt.Errorf("could not get supervysor directory path: %s", err) +func getSupervysorConfig(configPath string) (*types.SupervysorConfig, error) { + if !strings.HasSuffix(configPath, "/config.toml") { + configPath += "/config.toml" } - data, err := os.ReadFile(configPath + "/config.toml") + data, err := os.ReadFile(configPath) if err != nil { return nil, fmt.Errorf("could not find config. Please initialize again: %s", err) } diff --git a/cmd/supervysor/commands/prune.go b/cmd/supervysor/commands/prune.go new file mode 100644 index 0000000..64d3231 --- /dev/null +++ b/cmd/supervysor/commands/prune.go @@ -0,0 +1,38 @@ +package commands + +import ( + "fmt" + + "github.com/KYVENetwork/supervysor/utils" + + "github.com/KYVENetwork/supervysor/store" + "github.com/spf13/cobra" +) + +func init() { + pruneCmd.Flags().StringVar(&home, "home", "", "path to home directory (e.g. /root/.osmosisd)") + if err := pruneCmd.MarkFlagRequired("home"); err != nil { + panic(fmt.Errorf("flag 'home' should be required: %w", err)) + } + + pruneCmd.Flags().Int64Var(&untilHeight, "until-height", 0, "prune blocks until specified height (excluding)") + if err := pruneCmd.MarkFlagRequired("until-height"); err != nil { + panic(fmt.Errorf("flag 'until-height' should be required: %w", err)) + } + + pruneCmd.Flags().BoolVar(&statePruning, "state-pruning", true, "state pruning enabled") + + pruneCmd.Flags().BoolVar(&optOut, "opt-out", false, "disable the collection of anonymous usage data") +} + +var pruneCmd = &cobra.Command{ + Use: "prune", + Short: "Prune blocks and optionally state from base height until a specific height", + Run: func(cmd *cobra.Command, args []string) { + utils.TrackPruneEvent(optOut) + + if err := store.Prune(home, untilHeight, statePruning, logger); err != nil { + logger.Error(err.Error()) + } + }, +} diff --git a/cmd/supervysor/commands/root.go b/cmd/supervysor/commands/root.go new file mode 100644 index 0000000..c0f2f05 --- /dev/null +++ b/cmd/supervysor/commands/root.go @@ -0,0 +1,83 @@ +package commands + +import ( + "io" + "os" + "path/filepath" + "time" + + "github.com/KYVENetwork/supervysor/types" + + "github.com/KYVENetwork/supervysor/cmd/supervysor/commands/helpers" + + "cosmossdk.io/log" + "github.com/rs/zerolog" + "github.com/spf13/cobra" +) + +var ( + abciEndpoint string + binary string + binaryFlags string + cfgFlag string + chainId string + config string + fallbackEndpoints string + home string + metrics bool + metricsPort int + optOut bool + poolId int + pruningInterval int + seeds string + statePruning bool + untilHeight int64 + + cfg types.SupervysorConfig + + err error + configPath string + + compressionType string + destPath string + maxBackups int +) + +var logger = log.NewLogger(os.Stdout) + +var supervysor = &cobra.Command{ + Use: "supervysor", + Short: "Supervysor helps sync a Tendermint node used as a KYVE data source.", + Version: types.Version, +} + +func Execute() { + logsDir, err := helpers.GetLogsDir() + if err != nil { + panic(err) + } + logFilePath := filepath.Join(logsDir, time.Now().Format("20060102_150405")+".log") + + file, err := os.OpenFile(logFilePath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0o777) + if err != nil { + panic(err) + } + + writer := io.MultiWriter(os.Stdout) + customConsoleWriter := zerolog.ConsoleWriter{Out: writer} + customConsoleWriter.FormatCaller = func(i interface{}) string { + return "\x1b[36m[supervysor]\x1b[0m" + } + multiLogger := io.MultiWriter(customConsoleWriter, file) + logger = log.NewCustomLogger(zerolog.New(multiLogger).With().Timestamp().Logger()) + + supervysor.AddCommand(initCmd) + supervysor.AddCommand(startCmd) + supervysor.AddCommand(versionCmd) + supervysor.AddCommand(pruneCmd) + supervysor.AddCommand(backupCmd) + + if err = supervysor.Execute(); err != nil { + os.Exit(1) + } +} diff --git a/cmd/supervysor/start.go b/cmd/supervysor/commands/start.go similarity index 61% rename from cmd/supervysor/start.go rename to cmd/supervysor/commands/start.go index 6d23eb1..1551eb0 100644 --- a/cmd/supervysor/start.go +++ b/cmd/supervysor/commands/start.go @@ -1,11 +1,14 @@ -package main +package commands import ( "fmt" "path/filepath" "time" - "github.com/KYVENetwork/supervysor/cmd/supervysor/helpers" + "github.com/KYVENetwork/supervysor/cmd/supervysor/commands/helpers" + + "github.com/KYVENetwork/supervysor/utils" + "github.com/prometheus/client_golang/prometheus" "github.com/spf13/cobra" @@ -14,21 +17,41 @@ import ( "github.com/KYVENetwork/supervysor/pool" ) +func init() { + startCmd.Flags().StringVar(&cfgFlag, "config", "", "path to config directory (e.g. ~/.supervysor/)") + + startCmd.Flags().StringVar(&binaryFlags, "flags", "", "flags for the underlying binary (e.g. '--address, ')") + + startCmd.Flags().BoolVar(&optOut, "opt-out", false, "disable the collection of anonymous usage data") +} + // The startCmd of the supervysor launches and manages the node process using the specified binary. // It periodically retrieves the heights of the node and the associated KYVE pool, and dynamically adjusts // the sync mode of the node based on these heights. var startCmd = &cobra.Command{ - Use: "start", - Short: "Start a supervysed Tendermint node", - DisableFlagParsing: true, - RunE: func(cmd *cobra.Command, flags []string) error { + Use: "start", + Short: "Start a supervysed Tendermint node", + RunE: func(cmd *cobra.Command, args []string) error { + if cfgFlag == "" { + configPath, err = helpers.GetSupervysorDir() + if err != nil { + logger.Error("could not get supervysor directory path", "err", err) + return err + } + } else { + configPath = cfgFlag + } + // Load initialized config. - config, err := getSupervysorConfig() + supervysorConfig, err := getSupervysorConfig(configPath) if err != nil { logger.Error("could not load config", "err", err) return err } - metrics := config.Metrics + + utils.TrackStartEvent(supervysorConfig.ChainId, optOut) + + metrics := supervysorConfig.Metrics // Create Prometheus registry reg := prometheus.NewRegistry() @@ -36,17 +59,17 @@ var startCmd = &cobra.Command{ if metrics { go func() { - err := helpers.StartMetricsServer(reg, config.MetricsPort) + err := helpers.StartMetricsServer(reg, supervysorConfig.MetricsPort) if err != nil { panic(err) } }() } - e := executor.NewExecutor(&logger, config) + e := executor.NewExecutor(&logger, supervysorConfig) // Start data source node initially. - if err := e.InitialStart(flags); err != nil { + if err := e.InitialStart(binaryFlags); err != nil { logger.Error("initial start failed", "err", err) return err } @@ -56,7 +79,7 @@ var startCmd = &cobra.Command{ if metrics { go func() { for { - dbSize, err := helpers.GetDirectorySize(filepath.Join(config.HomePath, "data")) + dbSize, err := helpers.GetDirectorySize(filepath.Join(supervysorConfig.HomePath, "data")) if err != nil { logger.Error("could not get data directory size; will not expose metrics", "err", err) } else { @@ -83,7 +106,7 @@ var startCmd = &cobra.Command{ m.NodeHeight.Set(float64(nodeHeight)) } - poolHeight, err := pool.GetPoolHeight(config.ChainId, config.PoolId, config.FallbackEndpoints) + poolHeight, err := pool.GetPoolHeight(supervysorConfig.ChainId, supervysorConfig.PoolId, supervysorConfig.FallbackEndpoints) if err != nil { logger.Error("could not get pool height", "err", err) if shutdownErr := e.Shutdown(); shutdownErr != nil { @@ -95,11 +118,11 @@ var startCmd = &cobra.Command{ m.PoolHeight.Set(float64(poolHeight)) } - logger.Info("fetched heights successfully", "node", nodeHeight, "pool", poolHeight, "max-height", poolHeight+config.HeightDifferenceMax, "min-height", poolHeight+config.HeightDifferenceMin) + logger.Info("fetched heights successfully", "node", nodeHeight, "pool", poolHeight, "max-height", poolHeight+supervysorConfig.HeightDifferenceMax, "min-height", poolHeight+supervysorConfig.HeightDifferenceMin) - if config.PruningInterval != 0 { - logger.Info("current pruning count", "pruning-count", fmt.Sprintf("%.2f", pruningCount), "pruning-threshold", config.PruningInterval) - if pruningCount > float64(config.PruningInterval) && nodeHeight > 0 { + if supervysorConfig.PruningInterval != 0 { + logger.Info("current pruning count", "pruning-count", fmt.Sprintf("%.2f", pruningCount), "pruning-threshold", supervysorConfig.PruningInterval) + if pruningCount > float64(supervysorConfig.PruningInterval) && nodeHeight > 0 { if currentMode == "ghost" { pruneHeight := poolHeight if nodeHeight < poolHeight { @@ -107,7 +130,7 @@ var startCmd = &cobra.Command{ } logger.Info("pruning blocks after node shutdown", "until-height", pruneHeight) - err = e.PruneBlocks(config.HomePath, pruneHeight-1, flags) + err = e.PruneBlocks(supervysorConfig.HomePath, pruneHeight-1, supervysorConfig.StatePruning, binaryFlags) if err != nil { logger.Error("could not prune blocks", "err", err) return err @@ -116,7 +139,7 @@ var startCmd = &cobra.Command{ if nodeHeight < poolHeight { logger.Info("pruning blocks after node shutdown", "until-height", nodeHeight) - err = e.PruneBlocks(config.HomePath, nodeHeight-1, flags) + err = e.PruneBlocks(supervysorConfig.HomePath, nodeHeight-1, supervysorConfig.StatePruning, binaryFlags) if err != nil { logger.Error("could not prune blocks", "err", err) return err @@ -131,18 +154,18 @@ var startCmd = &cobra.Command{ heightDiff := nodeHeight - poolHeight if metrics { - m.MaxHeight.Set(float64(poolHeight + config.HeightDifferenceMax)) - m.MinHeight.Set(float64(poolHeight + config.HeightDifferenceMin)) + m.MaxHeight.Set(float64(poolHeight + supervysorConfig.HeightDifferenceMax)) + m.MinHeight.Set(float64(poolHeight + supervysorConfig.HeightDifferenceMin)) } - if heightDiff >= config.HeightDifferenceMax { + if heightDiff >= supervysorConfig.HeightDifferenceMax { if currentMode != "ghost" { logger.Info("enabling GhostMode") } else { logger.Info("keeping GhostMode") } // Data source node has synced far enough, enable or keep Ghost Mode - if err = e.EnableGhostMode(flags); err != nil { + if err = e.EnableGhostMode(binaryFlags); err != nil { logger.Error("could not enable Ghost Mode", "err", err) if shutdownErr := e.Shutdown(); shutdownErr != nil { @@ -151,7 +174,7 @@ var startCmd = &cobra.Command{ return err } currentMode = "ghost" - } else if heightDiff < config.HeightDifferenceMax && heightDiff > config.HeightDifferenceMin { + } else if heightDiff < supervysorConfig.HeightDifferenceMax && heightDiff > supervysorConfig.HeightDifferenceMin { // No threshold reached, keep current mode logger.Info("keeping current Mode", "mode", currentMode, "height-difference", heightDiff) } else { @@ -161,7 +184,7 @@ var startCmd = &cobra.Command{ logger.Info("keeping NormalMode") } // Difference is < HeightDifferenceMin, Data source needs to catch up, enable or keep Normal Mode - if err = e.EnableNormalMode(flags); err != nil { + if err = e.EnableNormalMode(binaryFlags); err != nil { logger.Error("could not enable Normal Mode", "err", err) if shutdownErr := e.Shutdown(); shutdownErr != nil { @@ -176,8 +199,8 @@ var startCmd = &cobra.Command{ logger.Info("node has not reached pool height yet, can not use it as data source") } } - pruningCount = pruningCount + float64(config.Interval)/60/60 - time.Sleep(time.Second * time.Duration(config.Interval)) + pruningCount = pruningCount + float64(supervysorConfig.Interval)/60/60 + time.Sleep(time.Second * time.Duration(supervysorConfig.Interval)) } }, } diff --git a/cmd/supervysor/commands/version.go b/cmd/supervysor/commands/version.go new file mode 100644 index 0000000..20e37ef --- /dev/null +++ b/cmd/supervysor/commands/version.go @@ -0,0 +1,21 @@ +package commands + +import ( + "github.com/KYVENetwork/supervysor/types" + "github.com/KYVENetwork/supervysor/utils" + "github.com/spf13/cobra" +) + +func init() { + versionCmd.Flags().BoolVar(&optOut, "opt-out", false, "disable the collection of anonymous usage data") +} + +var versionCmd = &cobra.Command{ + Use: "version", + Short: "Print the version of supervysor", + Run: func(cmd *cobra.Command, args []string) { + utils.TrackVersionEvent(optOut) + + logger.Info(types.Version) + }, +} diff --git a/cmd/supervysor/main.go b/cmd/supervysor/main.go index cb4581c..a443204 100644 --- a/cmd/supervysor/main.go +++ b/cmd/supervysor/main.go @@ -1,56 +1,16 @@ package main import ( - "io" - "os" - "path/filepath" - "time" - - "cosmossdk.io/log" - "github.com/rs/zerolog" - "github.com/spf13/cobra" - - "github.com/KYVENetwork/supervysor/cmd/supervysor/helpers" + cmd "github.com/KYVENetwork/supervysor/cmd/supervysor/commands" + "github.com/KYVENetwork/supervysor/types" ) -var logger = log.NewLogger(os.Stdout) +var version string -var Version = "" - -var supervysor = &cobra.Command{ - Use: "supervysor", - Short: "Supervysor helps sync a Tendermint node used as a KYVE data source.", - Version: Version, +func init() { + types.Version = version } -// main initializes logger including file logging and all supervysor commands. func main() { - logsDir, err := helpers.GetLogsDir() - if err != nil { - panic(err) - } - logFilePath := filepath.Join(logsDir, time.Now().Format("20060102_150405")+".log") - - file, err := os.OpenFile(logFilePath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0o777) - if err != nil { - panic(err) - } - - writer := io.MultiWriter(os.Stdout) - customConsoleWriter := zerolog.ConsoleWriter{Out: writer} - customConsoleWriter.FormatCaller = func(i interface{}) string { - return "\x1b[36m[supervysor]\x1b[0m" - } - multiLogger := io.MultiWriter(customConsoleWriter, file) - logger = log.NewCustomLogger(zerolog.New(multiLogger).With().Timestamp().Logger()) - - supervysor.AddCommand(initCmd) - supervysor.AddCommand(startCmd) - supervysor.AddCommand(versionCmd) - supervysor.AddCommand(pruneCmd) - supervysor.AddCommand(backupCmd) - - if err = supervysor.Execute(); err != nil { - os.Exit(1) - } + cmd.Execute() } diff --git a/cmd/supervysor/prune.go b/cmd/supervysor/prune.go deleted file mode 100644 index 7fc7ff2..0000000 --- a/cmd/supervysor/prune.go +++ /dev/null @@ -1,32 +0,0 @@ -package main - -import ( - "fmt" - - "github.com/KYVENetwork/supervysor/store" - "github.com/spf13/cobra" -) - -var untilHeight int64 - -func init() { - pruneCmd.Flags().StringVar(&home, "home", "", "home directory") - if err := pruneCmd.MarkFlagRequired("home"); err != nil { - panic(fmt.Errorf("flag 'home' should be required: %w", err)) - } - - pruneCmd.Flags().Int64Var(&untilHeight, "until-height", 0, "prune blocks until this height (excluding)") - if err := pruneCmd.MarkFlagRequired("until-height"); err != nil { - panic(fmt.Errorf("flag 'until-height' should be required: %w", err)) - } -} - -var pruneCmd = &cobra.Command{ - Use: "prune-blocks", - Short: "Prune blocks until a specific height", - Run: func(cmd *cobra.Command, args []string) { - if err := store.PruneBlocks(home, untilHeight, logger); err != nil { - logger.Error(err.Error()) - } - }, -} diff --git a/cmd/supervysor/version.go b/cmd/supervysor/version.go deleted file mode 100644 index b519fd7..0000000 --- a/cmd/supervysor/version.go +++ /dev/null @@ -1,15 +0,0 @@ -package main - -import ( - "fmt" - - "github.com/spf13/cobra" -) - -var versionCmd = &cobra.Command{ - Use: "version", - Short: "Print the version of supervysor", - Run: func(cmd *cobra.Command, args []string) { - logger.Info(fmt.Sprintf("version: %s", Version)) - }, -} diff --git a/executor/executor.go b/executor/executor.go index fdd1f3d..a7c98f3 100644 --- a/executor/executor.go +++ b/executor/executor.go @@ -23,7 +23,7 @@ func NewExecutor(logger *log.Logger, cfg *types.SupervysorConfig) *Executor { } // InitialStart initiates the node by starting it in the initial mode. -func (e *Executor) InitialStart(flags []string) error { +func (e *Executor) InitialStart(flags string) error { e.Logger.Info("starting initially") process, err := node.StartNode(e.Cfg, e.Logger, &e.Process, true, false, flags) if err != nil { @@ -41,7 +41,7 @@ func (e *Executor) InitialStart(flags []string) error { // EnableGhostMode activates the Ghost Mode by starting the node in GhostMode if it is not already enabled. // If not, it shuts down the node running in NormalMode, initiates the GhostMode and updates the process ID // and GhostMode upon success. -func (e *Executor) EnableGhostMode(flags []string) error { +func (e *Executor) EnableGhostMode(flags string) error { if !e.Process.GhostMode { if err := node.ShutdownNode(&e.Process); err != nil { e.Logger.Error("could not shutdown node", "err", err) @@ -69,7 +69,7 @@ func (e *Executor) EnableGhostMode(flags []string) error { // EnableNormalMode enables the Normal Mode by starting the node in NormalMode if it is not already enabled. // If the GhostMode is active, it shuts down the node, starts the NormalMode with the provided parameters // and updates the process ID and GhostMode upon success. -func (e *Executor) EnableNormalMode(flags []string) error { +func (e *Executor) EnableNormalMode(flags string) error { if e.Process.GhostMode { if err := node.ShutdownNode(&e.Process); err != nil { e.Logger.Error("could not shutdown node", "err", err) @@ -94,12 +94,12 @@ func (e *Executor) EnableNormalMode(flags []string) error { return nil } -func (e *Executor) PruneBlocks(homePath string, pruneHeight int, flags []string) error { +func (e *Executor) PruneBlocks(homePath string, pruneHeight int, statePruning bool, flags string) error { if err := e.Shutdown(); err != nil { e.Logger.Error("could not shutdown node process", "err", err) return err } - err := store.PruneBlocks(homePath, int64(pruneHeight)-1, e.Logger) + err := store.Prune(homePath, int64(pruneHeight)-1, statePruning, e.Logger) if err != nil { e.Logger.Error("could not prune blocks, exiting") return err diff --git a/go.mod b/go.mod index 09b62d4..2500de2 100644 --- a/go.mod +++ b/go.mod @@ -5,13 +5,16 @@ go 1.20 require ( cosmossdk.io/log v1.1.0 github.com/golangci/golangci-lint v1.52.2 + github.com/google/uuid v1.3.0 github.com/pelletier/go-toml/v2 v2.0.5 github.com/prometheus/client_golang v1.12.1 github.com/rs/zerolog v1.29.1 + github.com/segmentio/analytics-go v3.1.0+incompatible github.com/spf13/cobra v1.7.0 github.com/spf13/viper v1.12.0 github.com/tendermint/tendermint v0.34.14 github.com/tendermint/tm-db v0.6.7 + golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e mvdan.cc/gofumpt v0.5.0 ) @@ -34,6 +37,7 @@ require ( github.com/beorn7/perks v1.0.1 // indirect github.com/bkielbasa/cyclop v1.2.0 // indirect github.com/blizzy78/varnamelen v0.8.0 // indirect + github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 // indirect github.com/bombsimon/wsl/v3 v3.4.0 // indirect github.com/breml/bidichk v0.2.4 // indirect github.com/breml/errchkjson v0.3.1 // indirect @@ -154,6 +158,7 @@ require ( github.com/sashamelentyev/interfacebloat v1.1.0 // indirect github.com/sashamelentyev/usestdlibvars v1.23.0 // indirect github.com/securego/gosec/v2 v2.15.0 // indirect + github.com/segmentio/backo-go v1.0.1 // indirect github.com/shazow/go-diff v0.0.0-20160112020656-b6b7b6733b8c // indirect github.com/sirupsen/logrus v1.9.0 // indirect github.com/sivchari/containedctx v1.0.2 // indirect @@ -181,6 +186,7 @@ require ( github.com/ultraware/funlen v0.0.3 // indirect github.com/ultraware/whitespace v0.0.5 // indirect github.com/uudashr/gocognit v1.0.6 // indirect + github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c // indirect github.com/yagipy/maintidx v1.0.0 // indirect github.com/yeya24/promlinter v0.2.0 // indirect gitlab.com/bosi/decorder v0.2.3 // indirect @@ -189,7 +195,6 @@ require ( go.uber.org/multierr v1.6.0 // indirect go.uber.org/zap v1.24.0 // indirect golang.org/x/crypto v0.5.0 // indirect - golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect golang.org/x/exp/typeparams v0.0.0-20230224173230-c95f2b4c22f2 // indirect golang.org/x/mod v0.10.0 // indirect golang.org/x/net v0.9.0 // indirect diff --git a/go.sum b/go.sum index a110eb8..c127d67 100644 --- a/go.sum +++ b/go.sum @@ -117,6 +117,8 @@ github.com/bkielbasa/cyclop v1.2.0 h1:7Jmnh0yL2DjKfw28p86YTd/B4lRGcNuu12sKE35sM7 github.com/bkielbasa/cyclop v1.2.0/go.mod h1:qOI0yy6A7dYC4Zgsa72Ppm9kONl0RoIlPbzot9mhmeI= github.com/blizzy78/varnamelen v0.8.0 h1:oqSblyuQvFsW1hbBHh1zfwrKe3kcSj0rnXkKzsQ089M= github.com/blizzy78/varnamelen v0.8.0/go.mod h1:V9TzQZ4fLJ1DSrjVDfl89H7aMnTvKkApdHeyESmyR7k= +github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 h1:DDGfHa7BWjL4YnC6+E63dPcxHo2sUxDIu8g3QgEJdRY= +github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4= github.com/bombsimon/wsl/v3 v3.4.0 h1:RkSxjT3tmlptwfgEgTgU+KYKLI35p/tviNXNXiL2aNU= github.com/bombsimon/wsl/v3 v3.4.0/go.mod h1:KkIB+TXkqy6MvK9BDZVbZxKNYsE1/oLRJbIFtf14qqo= github.com/breml/bidichk v0.2.4 h1:i3yedFWWQ7YzjdZJHnPo9d/xURinSq3OM+gyM43K4/8= @@ -394,6 +396,8 @@ github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLe github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= +github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g= @@ -651,7 +655,6 @@ github.com/otiai10/copy v1.2.0/go.mod h1:rrF5dJ5F0t/EWSYODDu4j9/vEeYHMkc8jt0zJCh github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE= github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs= github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo= -github.com/otiai10/mint v1.3.1 h1:BCmzIS3n71sGfHB5NMNDB3lHYPz8fWSkCAErHed//qc= github.com/otiai10/mint v1.3.1/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc= github.com/pact-foundation/pact-go v1.0.4/go.mod h1:uExwJY4kCzNPcHRj+hCR/HBbOOIwwtUjcrb0b5/5kLM= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= @@ -754,6 +757,10 @@ github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg github.com/seccomp/libseccomp-golang v0.9.1/go.mod h1:GbW5+tmTXfcxTToHLXlScSlAvWlF4P2Ca7zGrPiEpWo= github.com/securego/gosec/v2 v2.15.0 h1:v4Ym7FF58/jlykYmmhZ7mTm7FQvN/setNm++0fgIAtw= github.com/securego/gosec/v2 v2.15.0/go.mod h1:VOjTrZOkUtSDt2QLSJmQBMWnvwiQPEjg0l+5juIqGk8= +github.com/segmentio/analytics-go v3.1.0+incompatible h1:IyiOfUgQFVHvsykKKbdI7ZsH374uv3/DfZUo9+G0Z80= +github.com/segmentio/analytics-go v3.1.0+incompatible/go.mod h1:C7CYBtQWk4vRk2RyLu0qOcbHJ18E3F1HV2C/8JvKN48= +github.com/segmentio/backo-go v1.0.1 h1:68RQccglxZeyURy93ASB/2kc9QudzgIDexJ927N++y4= +github.com/segmentio/backo-go v1.0.1/go.mod h1:9/Rh6yILuLysoQnZ2oNooD2g7aBnvM7r/fNVxRNWfBc= github.com/shazow/go-diff v0.0.0-20160112020656-b6b7b6733b8c h1:W65qqJCIOVP4jpqPQ0YvHYKwcMEMVWIzWC5iNQQfBTU= github.com/shazow/go-diff v0.0.0-20160112020656-b6b7b6733b8c/go.mod h1:/PevMnwAxekIXwN8qQyfc5gl2NlkB3CQlkizAbOkeBs= github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk= @@ -875,6 +882,8 @@ github.com/vishvananda/netlink v1.1.0/go.mod h1:cTgwzPIzzgDAYoQrMm0EdrjRUBkTqKYp github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU= github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= +github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c h1:3lbZUMbMiGUW/LMkfsEABsc5zNT9+b1CvsJx47JzJ8g= +github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c/go.mod h1:UrdRz5enIKZ63MEE3IF9l2/ebyx59GyGgPi+tICQdmM= github.com/yagipy/maintidx v1.0.0 h1:h5NvIsCz+nRDapQ0exNv4aJ0yXSI0420omVANTv3GJM= github.com/yagipy/maintidx v1.0.0/go.mod h1:0qNf/I/CCZXSMhsRsrEPDZ+DkekpKLXAJfsTACwgXLk= github.com/yeya24/promlinter v0.2.0 h1:xFKDQ82orCU5jQujdaD8stOHiv8UN68BSdn2a8u8Y3o= diff --git a/node/helpers/helpers.go b/node/helpers/helpers.go index c928e73..0edf9e8 100644 --- a/node/helpers/helpers.go +++ b/node/helpers/helpers.go @@ -6,6 +6,7 @@ import ( "net" "os" "path/filepath" + "strings" "cosmossdk.io/log" ) @@ -133,3 +134,34 @@ func MoveAddressBook(activateGhostMode bool, addrBookPath string, log log.Logger return nil } + +func SplitArgs(argsString string) []string { + // Split the string by spaces + split := strings.Fields(argsString) + + var args []string + var currentArg string + + for _, part := range split { + // If the current part starts with "--" or "-", consider it as a new argument + if strings.HasPrefix(part, "--") || strings.HasPrefix(part, "-") { + // If there was a previous argument, add it to the result + if currentArg != "" { + args = append(args, currentArg) + } + + // Start a new argument + currentArg = part + } else { + // If the current part doesn't start with "--" or "-", append it to the current argument + currentArg += " " + part + } + } + + // Add the last argument to the result + if currentArg != "" { + args = append(args, currentArg) + } + + return args +} diff --git a/node/node.go b/node/node.go index b40cb0c..eedfa1c 100644 --- a/node/node.go +++ b/node/node.go @@ -67,7 +67,7 @@ func GetNodeHeight(log log.Logger, p *types.ProcessType, abciEndpoint string) (i // StartNode starts the node process in Normal Mode and returns the os.Process object representing // the running process. It checks if the node is being started initially or not, moves the // address book if necessary, and sets the appropriate command arguments based on the binaryPath. -func StartNode(cfg *types.SupervysorConfig, log log.Logger, p *types.ProcessType, initial bool, restart bool, flags []string) (*os.Process, error) { +func StartNode(cfg *types.SupervysorConfig, log log.Logger, p *types.ProcessType, initial bool, restart bool, flags string) (*os.Process, error) { addrBookPath := filepath.Join(cfg.HomePath, "config", "addrbook.json") if !initial { @@ -109,7 +109,9 @@ func StartNode(cfg *types.SupervysorConfig, log log.Logger, p *types.ProcessType return nil, fmt.Errorf("empty home path in config") } - args = append(args, flags...) + parsedFlags := helpers.SplitArgs(flags) + + args = append(args, parsedFlags...) cmd := exec.Command(cmdPath, args...) cmd.Stdout = os.Stdout @@ -154,7 +156,7 @@ func StartNode(cfg *types.SupervysorConfig, log log.Logger, p *types.ProcessType // representing the running process. It moves the address book, checks if the node is already running // or in Ghost Mode ands sets the appropriate command arguments based on the binaryPath. // It starts the node without seeds and with a changed laddr, so the node can't continue syncing. -func StartGhostNode(cfg *types.SupervysorConfig, log log.Logger, p *types.ProcessType, restart bool, flags []string) (*os.Process, error) { +func StartGhostNode(cfg *types.SupervysorConfig, log log.Logger, p *types.ProcessType, restart bool, flags string) (*os.Process, error) { addrBookPath := filepath.Join(cfg.HomePath, "config", "addrbook.json") if err := helpers.MoveAddressBook(true, addrBookPath, log); err != nil { @@ -196,7 +198,9 @@ func StartGhostNode(cfg *types.SupervysorConfig, log log.Logger, p *types.Proces args = append(args, "--home", cfg.HomePath) } - args = append(args, flags...) + parsedFlags := helpers.SplitArgs(flags) + + args = append(args, parsedFlags...) cmd := exec.Command(cmdPath, args...) cmd.Stdout = os.Stdout diff --git a/store/prune-blocks.go b/store/prune.go similarity index 52% rename from store/prune-blocks.go rename to store/prune.go index 9befa4a..a3f8d31 100644 --- a/store/prune-blocks.go +++ b/store/prune.go @@ -4,13 +4,14 @@ import ( "fmt" "os" + "github.com/KYVENetwork/supervysor/cmd/supervysor/commands/helpers" + "cosmossdk.io/log" - "github.com/KYVENetwork/supervysor/cmd/supervysor/helpers" dbm "github.com/tendermint/tm-db" ) -func PruneBlocks(home string, untilHeight int64, logger log.Logger) error { +func Prune(home string, untilHeight int64, statePruning bool, logger log.Logger) error { config, err := helpers.LoadConfig(home) if err != nil { return fmt.Errorf("failed to load config: %w", err) @@ -44,7 +45,25 @@ func PruneBlocks(home string, untilHeight int64, logger log.Logger) error { os.Exit(0) } - logger.Info(fmt.Sprintf("Pruned %d blocks, new base height is %d", blocks, blockStore.Base())) + if statePruning { + stateStoreDB, stateStore, err := GetStateDBs(config) + defer func(stateStoreDB dbm.DB) { + err = stateStoreDB.Close() + if err != nil { + logger.Error(err.Error()) + os.Exit(0) + } + }(stateStoreDB) + + if err = stateStore.PruneStates(base, untilHeight); err != nil { + logger.Error(err.Error()) + os.Exit(0) + } + + logger.Info(fmt.Sprintf("Pruned %d blocks and the state until %d, new base height is %d", blocks, untilHeight, blockStore.Base())) + } else { + logger.Info(fmt.Sprintf("Pruned %d blocks until %d, new base height is %d", blocks, untilHeight, blockStore.Base())) + } return nil } diff --git a/types/constants.go b/types/constants.go index 11fc38d..540bb03 100644 --- a/types/constants.go +++ b/types/constants.go @@ -2,20 +2,27 @@ package types var ( KaonEndpoints = []string{ - "https://api-eu-1.kaon.kyve.network", - "https://api-us-1.kaon.kyve.network", + "https://api.kaon.kyve.network", } KorelliaEndpoints = []string{ "https://api.korellia.kyve.network", - "https://api-eu-1.korellia.kyve.network", - "https://api-explorer.korellia.kyve.network", } MainnetEndpoints = []string{ - "https://api-eu-1.kyve.network", - "https://api-us-1.kyve.network", + "https://api.kyve.network", } ) +var Version string + const ( BackoffMaxRetries = 15 + SegmentKey = "oLhjq9j6pOrIB7TjNHxWWB1ILhK5Fwn6" +) + +const ( + BACKUP = "BACKUP" + INIT = "INIT" + PRUNE = "PRUNE" + START = "START" + VERSION = "VERSION" ) diff --git a/types/types.go b/types/types.go index c0bff5d..cdfb1ec 100644 --- a/types/types.go +++ b/types/types.go @@ -20,6 +20,7 @@ type SupervysorConfig struct { PoolId int PruningInterval int Seeds string + StatePruning bool StateRequests bool } diff --git a/utils/events.go b/utils/events.go new file mode 100644 index 0000000..2c721ba --- /dev/null +++ b/utils/events.go @@ -0,0 +1,193 @@ +package utils + +import ( + "fmt" + "os" + "path/filepath" + "runtime" + "runtime/debug" + "strings" + "time" + + "github.com/KYVENetwork/supervysor/cmd/supervysor/commands/helpers" + + "github.com/KYVENetwork/supervysor/types" + "github.com/google/uuid" + "github.com/segmentio/analytics-go" +) + +var ( + startId = uuid.New().String() + client = analytics.New(types.SegmentKey) +) + +func getContext() *analytics.Context { + version := "local" + build, _ := debug.ReadBuildInfo() + + if strings.TrimSpace(build.Main.Version) != "" { + version = strings.TrimSpace(build.Main.Version) + } + + timezone, _ := time.Now().Zone() + locale := os.Getenv("LANG") + + return &analytics.Context{ + App: analytics.AppInfo{ + Name: "supervysor", + Version: version, + }, + Location: analytics.LocationInfo{}, + OS: analytics.OSInfo{ + Name: fmt.Sprintf("%s %s", runtime.GOOS, runtime.GOARCH), + }, + Locale: locale, + Timezone: timezone, + } +} + +func getUserId() (string, error) { + supervysorDir, err := helpers.GetSupervysorDir() + if err != nil { + return "", fmt.Errorf("could not find .supervysor directory: %s", err) + } + + userId := uuid.New().String() + + idFile := filepath.Join(supervysorDir, "id") + if _, err = os.Stat(idFile); os.IsNotExist(err) { + if err := os.WriteFile(idFile, []byte(userId), 0o755); err != nil { + return "", err + } + } else { + data, err := os.ReadFile(idFile) + if err != nil { + return "", err + } + userId = string(data) + } + + return userId, nil +} + +func TrackBackupEvent(optOut bool) { + if optOut { + return + } + + userId, err := getUserId() + if err != nil { + return + } + + err = client.Enqueue(analytics.Track{ + UserId: userId, + Event: types.BACKUP, + Context: getContext(), + }) + + if err != nil { + return + } + + err = client.Close() + _ = err +} + +func TrackInitEvent(chainId string, optOut bool) { + if optOut { + return + } + + userId, err := getUserId() + if err != nil { + return + } + + err = client.Enqueue(analytics.Track{ + UserId: userId, + Event: types.INIT, + Properties: analytics.NewProperties().Set("chain_id", chainId), + Context: getContext(), + }) + + if err != nil { + return + } + + err = client.Close() + _ = err +} + +func TrackPruneEvent(optOut bool) { + if optOut { + return + } + + userId, err := getUserId() + if err != nil { + return + } + + err = client.Enqueue(analytics.Track{ + UserId: userId, + Event: types.PRUNE, + Context: getContext(), + }) + + if err != nil { + return + } + + err = client.Close() + _ = err +} + +func TrackStartEvent(chainId string, optOut bool) { + if optOut { + return + } + + userId, err := getUserId() + if err != nil { + return + } + + err = client.Enqueue(analytics.Track{ + UserId: userId, + Event: types.START, + Properties: analytics.NewProperties().Set("chain_id", chainId).Set("start_id", startId), + Context: getContext(), + }) + + if err != nil { + return + } + + err = client.Close() + _ = err +} + +func TrackVersionEvent(optOut bool) { + if optOut { + return + } + + userId, err := getUserId() + if err != nil { + return + } + + err = client.Enqueue(analytics.Track{ + UserId: userId, + Event: types.VERSION, + Context: getContext(), + }) + + if err != nil { + return + } + + err = client.Close() + _ = err +}