Skip to content
This repository has been archived by the owner on Feb 5, 2020. It is now read-only.

Latest commit

 

History

History
132 lines (96 loc) · 8.14 KB

README.md

File metadata and controls

132 lines (96 loc) · 8.14 KB

Choria Network Broker

This repository is now archived, code is now part of github.com/choria-io/go-choria

This is a NATS compatible Network Broker for use by the Choria Orchestration System.

Please review the official documentation at choria.io for installation and usage.

Motivation

Running a middleware broker for Choria is quite the undertaking, while NATS is really easy to operate it does have a plethora of settings and using the wrong ones can adversely affect your network.

The Choria Network Broker is a managed NATS broker that integrates into the choria broker command. It ships as part of the normal Choria package and exist within the single binary.

It sets up the NATS server in ways thats suitable for use by Choria with sane defaults enabled.

Features:

  • Works by default without any broker specific configuration in your Choria broker
  • Secure by default - only accepts TLS connections with certificates signed by the known CA
  • Support NATS Accounts technology for large scale multi tenancy
  • Supports Clustering using a simple comma separated list of peers - TLS by default
  • Support Gateways enabling communication between NATS clusters - an alternative to Choria Federation
  • Support Leafnodes enabling joining older or unauthenticated clients to a secure multi tenant network
  • Exports statistics using the popular Prometheus format via the normal Choria statistics port

Configuration

The broker is configured using the Choria daemon configuration, below a reference of the settings it supports.

Choria core settings that affect the broker:

Setting Description
plugin.choria.broker_network Enables the network broker when running choria broker run
loglevel The logging level to use
plugin.choria.stats_port The port Choria listens on for metrics, when >0 the broker enables statistics
plugin.choria.stats_address The network address to listen on for metrics requests

It also uses the build.maxBrokerClients build time configuration in Choria to configure it's maximum connection limit, this defaults to 50 000.

Basic Broker Settings

Setting Description Default
plugin.choria.network.listen_address The network address to listen on ::
plugin.choria.network.client_port The port to listen on for network clients 4222
plugin.choria.network.write_deadline The time to allow for writes to network clients to complete before considering them slow 5s
plugin.choria.network.client_hosts List of hosts - ip addresses or cidrs - that are allowed to use clients all
plugin.choria.network.client_tls_force_required Force TLS on for client connections regardless of build settings false
plugin.choria.network.tls_timeout Sets the timeout for establishing TLS connections 2

Cluster Settings

Network Clusters are suitable for creating a cluster of up to 5 nodes on a local LAN. These form a full Mesh and provides scalability and HA.

They are based on NATS technology and you can read more about them at NATS.io

Setting Description Default
plugin.choria.network.peer_port The port to listen on for broker cluster peers 5222
plugin.choria.network.peer_user Username to connect to cluster peers with unset
plugin.choria.network.peer_password Password to use when connecting to cluster peers unset
plugin.choria.network.peers Comma separated List of cluster peers to connect to unset

Gateway Settings

Gateways allow you to combine multiple Clusters into a single large cluster. This allow you to span your collective across multiple data centers without the need for the much harder to configure federation brokers.

By default if the broker is compiled with TLS the Gateway will use the same TLS settings for the connection - you can customize it on a per remote basis.

They are based on NATS technology and you can read more about them at NATS.io

Setting Description Default
plugin.choria.network.gateway_port The port to listen to for Gateway connections, disabled when 0 0
plugin.choria.network.gateway_name Unique name for the cluster listening on the port CHORIA
plugin.choria.network.gateway_remotes A comma sep list of remote names to activate ""
plugin.choria.network.gateway_remote.C1.urls A comma sep list of host:port combinations to connect to for the remote C1 cluster
plugin.choria.network.gateway_remote.C1.tls.cert Path to a custom certificate for this remote only
plugin.choria.network.gateway_remote.C1.tls.key Path to a custom private key for this remote only
plugin.choria.network.gateway_remote.C1.tls.ca Path to a custom ca for this remote only
plugin.choria.network.gateway_remote.C1.tls.disable Disables the TLS configuration that would have inherited from the Choria Security system false
plugin.choria.network.gateway_remote.C1.tls.verify Disables full TLS verify for this remote only true

Leafnode Settings

Leafnodes exist to take unauthenticated or unsecured connections and forge them into a specific Account (see below). They allow older Choria agents and clients to take part of a multi tenant or account secured network.

By default if the broker is compiled with TLS the leafnode will use the same TLS settings for the connection - you can customize it on a per remote basis.

They are based on NATS technology and you can read more about them at NATS.io

Setting Description Default
plugin.choria.network.leafnode_port The port to listen to for Gateway connections, disabled when 0 0
plugin.choria.network.leafnode_remotes A comma sep list of remote names to activate ""
plugin.choria.network.leafnode_remote.C1.url A host:port combination to connect to for the remote C1 leafnode
plugin.choria.network.leafnode_remote.C1.account The local account name to use when connecting to the remote
plugin.choria.network.leafnode_remote.C1.credential The local credential file to use when connecting to the remote
plugin.choria.network.leafnode_remote.C1.tls.cert Path to a custom certificate for this remote only
plugin.choria.network.leafnode_remote.C1.tls.key Path to a custom private key for this remote only
plugin.choria.network.leafnode_remote.C1.tls.ca Path to a custom ca for this remote only
plugin.choria.network.leafnode_remote.C1.tls.disable Disables the TLS configuration that would have inherited from the Choria Security system false
plugin.choria.network.leafnode_remote.C1.tls.verify Disables full TLS verify for this remote only true

Accounts

Accounts are based on NATS technology, you can read more about them at NATS.io

Setting Description Default
plugin.choria.network.operator_account The operator account that is managing this cluster
plugin.choria.network.system_account The system account to use, when set enables server events

Statistics

When Statistics are enabled in Choria by setting plugin.choria.stats_port to nonzero the Choria Broker expose the following Prometheus statistics:

Statistic Description
choria_network_connections Current connections on the network broker
choria_network_total_connections Total connections received since start
choria_network_routes Current active routes to other brokers
choria_network_remotes Current active connections to other brokers
choria_network_leafnode_remotes Current active connections to leaf nodes
choria_network_in_msgs Messages received by the network broker
choria_network_out_msgs Messages sent by the network broker
choria_network_in_bytes Total size of messages received by the network broker
choria_network_out_bytes Total size of messages sent by the network broker
choria_network_slow_consumers Total number of clients who were considered slow consumers
choria_network_subscriptions Number of active subscriptions to subjects on this broker