Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove the iterative verifier #139

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Remove the iterative verifier #139

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a5ef20c
Removed iterative verifier
shuhaowu Sep 25, 2019
176cddf
Updated documentation to refer to InlineVerifier
shuhaowu Oct 8, 2019
c459035
Fixed typos in documentation
shuhaowu Oct 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 0 additions & 246 deletions compression_verifier.go
View file
Open in desktop

This file was deleted.

70 changes: 1 addition & 69 deletions config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ import (

const (
VerifierTypeChecksumTable = "ChecksumTable"
VerifierTypeIterative = "Iterative"
VerifierTypeInline = "Inline"
VerifierTypeNoVerification = "NoVerification"
)
Expand Down Expand Up @@ -179,58 +178,6 @@ func (c *InlineVerifierConfig) Validate() error {
return nil
}

type IterativeVerifierConfig struct {
// List of tables that should be ignored by the IterativeVerifier.
IgnoredTables []string

// List of columns that should be ignored by the IterativeVerifier.
// This is in the format of table_name -> [list of column names]
IgnoredColumns map[string][]string

// The number of concurrent verifiers. Note that a single table can only be
// assigned to one goroutine and currently multiple goroutines per table
// is not supported.
Concurrency int

// The maximum expected downtime during cutover, in the format of
// time.ParseDuration.
MaxExpectedDowntime string

// Map of the table and column identifying the compression type
// (if any) of the column. This is used during verification to ensure
// the data was successfully copied as some compression algorithms can
// output different compressed data with the same input data.
//
// The data structure is a map of table names to a map of column names
// to the compression algorithm.
// ex: {books: {contents: snappy}}
//
// Currently supported compression algorithms are:
// 1. Snappy (https://google.github.io/snappy/) as "SNAPPY"
//
// Optional: defaults to empty map/no compression
//
// Note that the IterativeVerifier is in the process of being deprecated.
// If this is specified, ColumnCompressionConfig should also be filled out in
// the main Config.
TableColumnCompression TableColumnCompressionConfig
}

func (c *IterativeVerifierConfig) Validate() error {
if c.MaxExpectedDowntime != "" {
_, err := time.ParseDuration(c.MaxExpectedDowntime)
if err != nil {
return err
}
}

if c.Concurrency == 0 {
c.Concurrency = 4
}

return nil
}

// SchemaName => TableName => ColumnName => CompressionAlgorithm
// Example: blog1 => articles => body => snappy
// (SELECT body FROM blog1.articles => returns compressed blob)
Expand Down Expand Up @@ -379,19 +326,12 @@ type Config struct {

// The verifier to use during the run. Valid choices are:
// ChecksumTable
// Iterative
// NoVerification
//
// If it is left blank, the Verifier member variable on the Ferry will be
// used. If that member variable is nil, no verification will be done.
VerifierType string

// Only useful if VerifierType == Iterative.
// This specifies the configurations to the IterativeVerifier.
//
// This option is in the process of being deprecated.
IterativeVerifierConfig IterativeVerifierConfig

// Only useful if VerifierType == Inline.
// This specifies the configurations to the InlineVerifierConfig.
InlineVerifierConfig InlineVerifierConfig
Expand All @@ -415,10 +355,6 @@ type Config struct {
// uncompressed data is equal.
// - This column signals to the InlineVerifier that it needs to decompress
// the data to compare identity.
//
// Note: a similar option exists in IterativeVerifier. However, the
// IterativeVerifier is being deprecated and this will be the correct place
// to specify it if you don't need the IterativeVerifier.
CompressedColumnsForVerification ColumnCompressionConfig

// This config is also for inline verification for the same special case of
Expand Down Expand Up @@ -452,11 +388,7 @@ func (c *Config) ValidateConfig() error {
return fmt.Errorf("StateToResumeFrom version mismatch: resume = %s, current = %s", c.StateToResumeFrom.GhostferryVersion, VersionString)
}

if c.VerifierType == VerifierTypeIterative {
if err := c.IterativeVerifierConfig.Validate(); err != nil {
return fmt.Errorf("IterativeVerifierConfig invalid: %v", err)
}
} else if c.VerifierType == VerifierTypeInline {
if c.VerifierType == VerifierTypeInline {
if err := c.InlineVerifierConfig.Validate(); err != nil {
return fmt.Errorf("InlineVerifierConfig invalid: %v", err)
}
Expand Down
4 changes: 2 additions & 2 deletions docs/source/copydbinterruptresume.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ Some other considerations/notes:
runs.
* To test resuming errored runs further, see :ref:`prodtesting`.

* Verifiers are not resumable, including the IterativeVerifier. This may change
in the future.
* ChecksumTableVerifier is not resumable, but the InlineVerifier is as long as
cut over didn't begin.
* While we are confident that the algorithm to be correct, this is still a
highly experimental feature. USE AT YOUR OWN RISK.

Loading