Skip to content

Commit

Permalink
Add forking detection to TBC (#101)
Browse files Browse the repository at this point in the history 
* Add testing framework

* Somewhat working chain

* Code is not ready to remove threading

* Implent get data

* be a bit less verbose

* Use normal cache element count

* And have a pass condition

* check balance

* go vet

* go mod tidy

* no ulimit check during tests

* Push before rebase

* Fix possible race conditions in tbcfork_test

* Tidy up parts of tbcfork_test

* Add forking support fake btc server

* create for and advance both heads

* We need to start panicing on fork situations

* add some comments to mark special spots

* added test case configuration for bitcoind inclusion in tests

* add comment

* removed unused val

* Encode/decode difficulty in blockheader

* imperitive forking in tests

* no syncBlocks anymore since we generate them with current timestamps

* comment out likely unused test

* tbc: fix loop unconditionally exited after one interation (SA4004)

* Add panic to catch unsupported fork

* removed unused test data

* fixup rebase drama, need to reenable TestLevelDB

* tbc: use time.Since instead of time.Now().Sub (S1012)

* finish fork scenario 1

* added more test cases

* NewBuffer works in surprising ways

* Detect forks when downloading block headers

* XXX

* deal with extending forks

* First attempt at making BlockHeadersBest singular

* tbcapi: update for BlockHeadersBest -> BlockHeaderBest

* Add type to detect how block headers were inserted

* Reap peers if not synced or sending crap block headers

* Simplify and return hint to caller how to handle forks; caller now also resumes fork and chain extensions (in case fork id deep)

* Remove debug panic and annotate code

* Moar testing

* tbcd: fix unnecessary use of fmt.Sprintf (S1039)

* Yay working forks

* tbcd: fix only first constant having explicit type (SA9004)

* tbcd/level: use element from range, fix empty if statement

* lower limits because latest ubuntu update

* be less loud

* skip test that causes an issue due to port bindings

* Remove cached last block header

* skipping balance tests for forking

* tbc: BlockHeadersBest -> BlockHeaderBest in rpc_test

* tbcapi: best block headers -> best block header

* Work around most fork situations

* mostly working but ugly interleaved start/stop downloading/indexing

* Make peers wanted a setting and abort indexing a bit earlier

* Ok this seems to work

* remove clipped for now and only index if enabled

* Fix broken test

* Make fork choices more explicit and rename BlockHeaderInsert -> BlockHeaderGenesisInsert

* make PeersWanted a runtime setting

* Return canonical and last inserted block header in BlockHeaderInsert to pretty print and keep things sane

* Rename 'last' to canonical/best

* remove unused h2b80 function

* Remove unused headerTime headerHash functions

* Remove unsued structure blockPeer

* Test err and use Error instead of Fatal

* Remove unused function getEndpointWithRetries

* Remove unused function submitBlock

* Unexport map with fork type names

* deleted unused test variable, increased for loop time sleep

* Update database/tbcd/database.go

Co-authored-by: Joshua Sing <joshua@bloq.com>

* Update database/tbcd/database.go

Co-authored-by: Joshua Sing <joshua@bloq.com>

* Make Infof into debugf

* Remove loud infof

* Move loud fork proclamation to debugf

* Remove leftover debug

* Update service/tbc/tbc_test.go

Co-authored-by: Joshua Sing <joshua@bloq.com>

* tbc: nest short err assignments

* tbcd/level: nest short err assignments

* tbc: use errors.Is for error comparison

* tbc/crawler: nest short err assignments

* tbcd/level: deduplicate cbh/lbh assignment

---------

Co-authored-by: ClaytonNorthey92 <clayton.northey@gmail.com>
Co-authored-by: Joshua Sing <joshua@bloq.com>
  • Loading branch information
@marcopeereboom @ClaytonNorthey92 @joshuasing
3 people authored Jun 17, 2024
1 parent bbeed8b commit 41a0009
Show file tree
Hide file tree
Showing 15 changed files with 2,269 additions and 726 deletions.
54 changes: 25 additions & 29 deletions api/tbcapi/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ This document provides details on the RPC protocol and available commands for th
* [📥 Response](#-response-1)
* [Payload](#payload-3)
* [Example Response](#example-response-1)
* [👉 Best Block Headers](#-best-block-headers)
* [👉 Best Block Header](#-best-block-header)
* [🗂 Raw Data](#-raw-data-1)
* [📤 Request](#-request-2)
* [Example Request](#example-request-2)
Expand Down Expand Up @@ -330,16 +330,16 @@ Response for a request with **id** `68656d69` and **height** `43111`:

---

## 👉 Best Block Headers
## 👉 Best Block Header

Retrieve the best block headers.

### 🗂 Raw Data

| Type | `command` value |
|----------|------------------------------------------|
| Request | `tbcapi-block-headers-best-raw-request` |
| Response | `tbcapi-block-headers-best-raw-response` |
| Type | `command` value |
|----------|-----------------------------------------|
| Request | `tbcapi-block-header-best-raw-request` |
| Response | `tbcapi-block-header-best-raw-response` |

#### 📤 Request

Expand All @@ -350,7 +350,7 @@ Retrieve the best block headers:
```json
{
"header": {
"command": "tbcapi-block-headers-best-raw-request",
"command": "tbcapi-block-header-best-raw-request",
"id": "68656d69"
}
}
Expand All @@ -361,7 +361,7 @@ Retrieve the best block headers:
##### Payload

- **`height`**: The best-known height.
- **`block_headers`**: An array of the best-known block headers encoded as hexadecimal strings.
- **`block_header`**: The best-known block header encoded as a hexadecimal string.

##### Example Response

Expand All @@ -370,24 +370,22 @@ Response for a request with **id** `68656d69` and **best height** `2182000`:
```json
{
"header": {
"command": "tbcapi-block-headers-best-raw-response",
"command": "tbcapi-block-header-best-raw-response",
"id": "68656d69"
},
"payload": {
"height": 2182000,
"block_headers": [
"0420002075089ac1ab1cab70cf6e6b774a86703a8d7127c0ebed1d3dfa2c00000000000086105509ec4a79457a400451290ad2a019fec4c76b47512623f1bb17a0ced76f38d82662bef4001b07d86700"
]
"block_header": "0420002075089ac1ab1cab70cf6e6b774a86703a8d7127c0ebed1d3dfa2c00000000000086105509ec4a79457a400451290ad2a019fec4c76b47512623f1bb17a0ced76f38d82662bef4001b07d86700"
}
}
```

#### 🗂 Serialized Data

| Type | `command` value |
|----------|--------------------------------------|
| Request | `tbcapi-block-headers-best-request` |
| Response | `tbcapi-block-headers-best-response` |
| Type | `command` value |
|----------|-------------------------------------|
| Request | `tbcapi-block-header-best-request` |
| Response | `tbcapi-block-header-best-response` |

#### 📤 Request

Expand All @@ -398,7 +396,7 @@ Retrieve the best block headers:
```json
{
"header": {
"command": "tbcapi-block-headers-best-request",
"command": "tbcapi-block-header-best-request",
"id": "68656d69"
}
}
Expand All @@ -409,7 +407,7 @@ Retrieve the best block headers:
##### Payload

- **`height`**: The best-known height.
- **`block_headers`**: An array of best-known [block headers](#block-header).
- **`block_header`**: The best-known [block header](#block-header).

##### Example Response

Expand All @@ -418,21 +416,19 @@ Response for a request with **id** `68656d69` and **height** `2587400`:
```json
{
"header": {
"command": "tbcapi-block-headers-best-response",
"command": "tbcapi-block-header-best-response",
"id": "68656d69"
},
"payload": {
"height": 2587400,
"block_headers": [
{
"version": 536887296,
"prev_hash": "000000000000002bbbbec8f126dc76a82109d898383bca5013a2386c8675ce34",
"merkle_root": "b9d74efdafb5436330b47478b2df28251057da5a9bc11c5509410950253d4f0e",
"timestamp": 1713461092,
"bits": "192e17d5",
"nonce": 3365605040
}
]
"block_header": {
"version": 536887296,
"prev_hash": "000000000000002bbbbec8f126dc76a82109d898383bca5013a2386c8675ce34",
"merkle_root": "b9d74efdafb5436330b47478b2df28251057da5a9bc11c5509410950253d4f0e",
"timestamp": 1713461092,
"bits": "192e17d5",
"nonce": 3365605040
}
}
}
```
Expand Down
36 changes: 18 additions & 18 deletions api/tbcapi/tbcapi.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ const (
CmdBlockHeadersByHeightRequest = "tbcapi-block-headers-by-height-request"
CmdBlockHeadersByHeightResponse = "tbcapi-block-headers-by-height-response"

CmdBlockHeadersBestRawRequest = "tbcapi-block-headers-best-raw-request"
CmdBlockHeadersBestRawResponse = "tbcapi-block-headers-best-raw-response"
CmdBlockHeaderBestRawRequest = "tbcapi-block-header-best-raw-request"
CmdBlockHeaderBestRawResponse = "tbcapi-block-header-best-raw-response"

CmdBlockHeadersBestRequest = "tbcapi-block-headers-best-request"
CmdBlockHeadersBestResponse = "tbcapi-block-headers-best-response"
CmdBlockHeaderBestRequest = "tbcapi-block-header-best-request"
CmdBlockHeaderBestResponse = "tbcapi-block-header-best-response"

CmdBalanceByAddressRequest = "tbcapi-balance-by-address-request"
CmdBalanceByAddressResponse = "tbcapi-balance-by-address-response"
Expand Down Expand Up @@ -88,20 +88,20 @@ type BlockHeadersByHeightResponse struct {
Error *protocol.Error `json:"error,omitempty"`
}

type BlockHeadersBestRawRequest struct{}
type BlockHeaderBestRawRequest struct{}

type BlockHeadersBestRawResponse struct {
Height uint64 `json:"height"`
BlockHeaders []api.ByteSlice `json:"block_headers"`
Error *protocol.Error `json:"error,omitempty"`
type BlockHeaderBestRawResponse struct {
Height uint64 `json:"height"`
BlockHeader api.ByteSlice `json:"block_header"`
Error *protocol.Error `json:"error,omitempty"`
}

type BlockHeadersBestRequest struct{}
type BlockHeaderBestRequest struct{}

type BlockHeadersBestResponse struct {
Height uint64 `json:"height"`
BlockHeaders []*BlockHeader `json:"block_headers"`
Error *protocol.Error `json:"error,omitempty"`
type BlockHeaderBestResponse struct {
Height uint64 `json:"height"`
BlockHeader *BlockHeader `json:"block_header"`
Error *protocol.Error `json:"error,omitempty"`
}

type BalanceByAddressRequest struct {
Expand Down Expand Up @@ -192,10 +192,10 @@ var commands = map[protocol.Command]reflect.Type{
CmdBlockHeadersByHeightRawResponse: reflect.TypeOf(BlockHeadersByHeightRawResponse{}),
CmdBlockHeadersByHeightRequest: reflect.TypeOf(BlockHeadersByHeightRequest{}),
CmdBlockHeadersByHeightResponse: reflect.TypeOf(BlockHeadersByHeightResponse{}),
CmdBlockHeadersBestRawRequest: reflect.TypeOf(BlockHeadersBestRawRequest{}),
CmdBlockHeadersBestRawResponse: reflect.TypeOf(BlockHeadersBestRawResponse{}),
CmdBlockHeadersBestRequest: reflect.TypeOf(BlockHeadersBestRequest{}),
CmdBlockHeadersBestResponse: reflect.TypeOf(BlockHeadersBestResponse{}),
CmdBlockHeaderBestRawRequest: reflect.TypeOf(BlockHeaderBestRawRequest{}),
CmdBlockHeaderBestRawResponse: reflect.TypeOf(BlockHeaderBestRawResponse{}),
CmdBlockHeaderBestRequest: reflect.TypeOf(BlockHeaderBestRequest{}),
CmdBlockHeaderBestResponse: reflect.TypeOf(BlockHeaderBestResponse{}),
CmdBalanceByAddressRequest: reflect.TypeOf(BalanceByAddressRequest{}),
CmdBalanceByAddressResponse: reflect.TypeOf(BalanceByAddressResponse{}),
CmdUtxosByAddressRawRequest: reflect.TypeOf(UtxosByAddressRawRequest{}),
Expand Down
10 changes: 6 additions & 4 deletions cmd/hemictl/hemictl.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -241,14 +241,16 @@ func tbcdb() error {
fmt.Printf("height: %v\n", bh.Height)

case "blockheadersbest":
bhs, err := s.DB().BlockHeadersBest(ctx)
bhb, err := s.DB().BlockHeaderBest(ctx)
if err != nil {
return fmt.Errorf("block headers best: %w", err)
}
for k := range bhs {
fmt.Printf("hash (%v): %v\n", k, bhs[k])
fmt.Printf("height (%v): %v\n", k, bhs[k].Height)
hash, err := chainhash.NewHash(bhb.Hash)
if err != nil {
return fmt.Errorf("block headers best chainhash: %w", err)
}
fmt.Printf("hash : %v\n", hash)
fmt.Printf("height: %v\n", bhb.Height)

case "blockheadersbyheight":
height := args["height"]
Expand Down
6 changes: 6 additions & 0 deletions cmd/tbcd/tbcd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,12 @@ var (
Help: "bitcoin network; mainnet or testnet3",
Print: config.PrintAll,
},
"TBC_PEERS_WANTED": config.Config{
Value: &cfg.PeersWanted,
DefaultValue: 64,
Help: "number of wanted p2p peers",
Print: config.PrintAll,
},
"TBC_PROMETHEUS_ADDRESS": config.Config{
Value: &cfg.PrometheusListenAddress,
DefaultValue: "",
Expand Down
39 changes: 32 additions & 7 deletions database/tbcd/database.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"encoding/hex"
"errors"
"fmt"
"math/big"
"time"

"github.com/btcsuite/btcd/chaincfg/chainhash"
Expand All @@ -19,6 +20,26 @@ import (
"github.com/hemilabs/heminetwork/database"
)

type InsertType int

const (
ITInvalid InsertType = 0 // Invalid insert
ITChainExtend InsertType = 1 // Normal insert, does not require further action.
ITChainFork InsertType = 2 // Chain forked, unwind and rewind indexes.
ITForkExtend InsertType = 3 // Extended a fork, does not require further action.
)

var itStrings = map[InsertType]string{
ITInvalid: "invalid",
ITChainExtend: "chain extended",
ITChainFork: "chain forked",
ITForkExtend: "fork extended",
}

func (it InsertType) String() string {
return itStrings[it]
}

type Database interface {
database.Database

Expand All @@ -28,10 +49,13 @@ type Database interface {
MetadataPut(ctx context.Context, key, value []byte) error

// Block header
BlockHeaderBest(ctx context.Context) (*BlockHeader, error) // return canonical
BlockHeaderByHash(ctx context.Context, hash []byte) (*BlockHeader, error)
BlockHeadersBest(ctx context.Context) ([]BlockHeader, error)
BlockHeaderGenesisInsert(ctx context.Context, bh [80]byte) error

// Block headers
BlockHeadersByHeight(ctx context.Context, height uint64) ([]BlockHeader, error)
BlockHeadersInsert(ctx context.Context, bhs []BlockHeader) error
BlockHeadersInsert(ctx context.Context, bhs [][80]byte) (InsertType, *BlockHeader, *BlockHeader, error)

// Block
BlocksMissing(ctx context.Context, count int) ([]BlockIdentifier, error)
Expand All @@ -58,12 +82,13 @@ type Database interface {
UtxosByScriptHash(ctx context.Context, sh ScriptHash, start uint64, count uint64) ([]Utxo, error)
}

// BlockHeader contains the first 80 raw bytes of a bitcoin block and its
// location information (hash+height).
// BlockHeader contains the first 80 raw bytes of a bitcoin block plus its
// location information (hash+height) and the cumulative difficulty.
type BlockHeader struct {
Hash database.ByteArray
Height uint64
Header database.ByteArray
Hash database.ByteArray
Height uint64
Header database.ByteArray
Difficulty big.Int
}

func (bh BlockHeader) String() string {
Expand Down
Loading

0 comments on commit 41a0009

Please sign in to comment.