V0.7.0-beta.2 SQLite backend

What's Changed

• SQLite-powered backend by @digizeph in #30

Deploy with Docker

version: '3'
services:
  bgpkit-broker:
    image: bgpkit/broker:0.7.0-beta.2
    ports:
      - "40064:40064"
    volumes:
      - ./data:/bgpkit-broker

Then run docker-compose up

Full Changelog: v0.7.0-beta.1...v0.7.0-beta.2

V0.7.0-beta.1

What's Changed

• apply broker filters to latest() call by @digizeph in #29
• add example Hashicorp Nomad deployment job spec: https://github.com/bgpkit/bgpkit-broker/blob/main/deployment/nomad_api_raw.hcl

Full Changelog: v0.7.0-alpha.3...v0.7.0-beta.1

v0.7.0-alpha.3

What's Changed

• by default load preconfigured collectors by @digizeph in #23
• update health check endpoint by @digizeph in #24
• configure default bootstrap length by @digizeph in #25

Full Changelog: v0.7.0-alpha.1...v0.7.0-alpha.3

V0.7.0 Alpha 1: All-in-one CLI

What's Changed

• bgpkit-broker cli with crawler + api + search by @digizeph in #22

This release adds the all-in-one command-line application implementation: bgpkit-broker.

This application allows users to quickly spin up a new self-host instance with a single command-line execution.

This pull request also brings about a number of refactoring efforts that helps overall structure of this repository.

bgpkit-broker CLI Tool

bgpkit-broker is a command-line application that packages many functionalities to allow users to self-host a BGPKIT Broker instance with ease.

bgpkit-broker has the following subcommands

A library and command-line to provide indexing and searching functionalities for public BGP data archive files over time.


Usage: bgpkit-broker [OPTIONS] <COMMAND>

Commands:
  serve      Serve the Broker content via RESTful API
  update     Update the Broker database
  config     Print out current configuration
  bootstrap  Bootstrap the Broker database
  backup     Export broker database to parquet file
  search     Search MRT files in Broker db
  help       Print this message or the help of the given subcommand(s)

Options:
      --no-log             disable logging
      --bootstrap-parquet  bootstrap from parquet file instead of DuckDB file
  -h, --help               Print help
  -V, --version            Print version

serve

bgpkit-broker serve is the main command to start the BGPKIT Broker service. It will start a web server that serves the API endpoints. It will also periodically update the local database unless the --no-update flag is set.

Serve the Broker content via RESTful API

Usage: bgpkit-broker serve [OPTIONS]

Options:
  -i, --update-interval <UPDATE_INTERVAL>  update interval in seconds [default: 300]
      --no-log                             disable logging
      --bootstrap-parquet                  bootstrap from parquet file instead of DuckDB file
  -h, --host <HOST>                        host address [default: 0.0.0.0]
  -p, --port <PORT>                        port number [default: 40064]
  -r, --root <ROOT>                        root path, useful for configuring docs UI [default: /]
      --no-updater                         disable updater service
      --no-api                             disable API service
  -h, --help                               Print help
  -V, --version                            Print version

update

bgpkit-broker update triggers a local database update manually. This command cannot be run at the same time as serve because the active API will lock the database file.

Update the Broker database

Usage: bgpkit-broker update [OPTIONS]

Options:
      --no-log             disable logging
      --bootstrap-parquet  bootstrap from parquet file instead of DuckDB file
  -h, --help               Print help
  -V, --version            Print version

search

bgpkit-broker search queries for MRT files using the default production API unless specified otherwise.

Search MRT files in Broker db

Usage: bgpkit-broker search [OPTIONS]

Options:
      --no-log                       disable logging
  -t, --ts-start <TS_START>          Start timestamp
      --bootstrap-parquet            bootstrap from parquet file instead of DuckDB file
  -T, --ts-end <TS_END>              End timestamp
  -p, --project <PROJECT>            filter by route collector projects, i.e. `route-views` or `riperis`
  -c, --collector-id <COLLECTOR_ID>  filter by collector IDs, e.g. 'rrc00', 'route-views2. use comma to separate multiple collectors
  -d, --data-type <DATA_TYPE>        filter by data types, i.e. 'update', 'rib'
      --page <PAGE>                  page number
      --page-size <PAGE_SIZE>        page size
  -u, --url <URL>                    
  -j, --json                         print out search results in JSON format instead of Markdown table
  -h, --help                         Print help
  -V, --version                      Print version

config

bgpkit-broker config displays current configuration, e.g. local database path, update interval, etc.

backup

bgpkit-broker update runs a database backup and export the database to a duckdb file and a parquet file. This can be run while serve is running.

Full Changelog: v0.6.1...v0.7.0-alpha.1

V0.6.1 reqwest for http queries

What's Changed

• switch http lib to reqwest from ureq by @digizeph in #20

Full Changelog: v0.6.0...v0.6.1

V0.6.0 New endpoint

What's Changed

• support anyhow error by @digizeph in #16
  • errors can now be caught and handled by anyhow in downstream libraries/binaries.
• update default URL; fix latest query by @digizeph in #19
  • Update default query endpoint, see docs at https://api.bgpkit.com/docs. The query parameters did not change.
  • Fix query for the /latest endpoint.

Full Changelog: v0.5.0...v0.6.0

V0.5.0 Major overhaul

Breaking Change

This version changes the core interface of how BgpkitBroker objects are constructed. It will not be compatible with V0.4 and before.

Added features

• Implement Display for BrokerItem and QueryResult by @digizeph in #10
• replace reqwest with ureq by @digizeph in #12
• Add .latest() function call by @digizeph in #14

Major Overhaul (#13)

The main noticeable changes include:

1. new easier-to-use constructor with default values for query and broker URL;
2. factory-pattern-style constructor allowing assigning query params directly;
3. simplified internal iterator implementation;
4. main query function renamed to .query().
5. added .latest() function to gather information about the latest MRT file for each collector we keep track of.
6. improved documentation

Full Changelog: v0.4.1...v0.5.0

V0.4.1 Fix collector filtering

What's Changed

• Fix collector filtering by @digizeph in #4

Full Changelog: v0.4.0...v0.4.1

V0.4.0 V2 API Support

Breaking Change

This version no longer support the V1 backend API. If you have a V1 API running, please consider switching to V2 API, or use hosted https://api.broker.bgpkit.com/v2 as the broker_url.

What's Changed

• Support V2 broker API by @digizeph in #3

Full Changelog: v0.3.2...v0.4.0

V0.3.2

Fixes

Fixed broker returning items ordering issue during iteration.