Nesoi is a coordination and configuration manager for distributed applications. It allows clients to find services to communicate with, and services to find its configuration. It also allows clients to register webhooks that will be notified when something changes.
Nesoi has three concepts:
- Applications stores configuration for a service.
- Service holds information about instances of a service.
- Host communicate endpoints for an instance of a service.
The normal use-case is this: When an application is bootstrapped it
reads out its configuration from an app-resource. When done, it
registers itself as a host for a service that it provides. Clients
find endpoints for a service by inspecting /srv/NAME.
Applications are located under the /app tree. A simple example
of creating an application configuration:
$ curl -X PUT -d '{"name":"dm", "config":{}}' http://localhost:6553/app/dm
Each app object needs a name and config attribute.
To list all available applications, issue a GET to /app:
$ curl http://localhost:6553/app
{
"apps": ["dm"]
}
When your service has been bootstrapped, register the host instance with the service using a simple command like this:
$ curl -X PUT -d '{"endpoints":{"http":"http://localhost:5432/"}}' http://localhost:6553/srv/dm/host1
{
"endpoints": {
"http": "http://localhost:5432/"
},
"updated_at": 1319523542
}
endpoints property is required. Hosts, and apps can of course be
deleted using DELETE.
To make sure that your application instance gets notified about changes to the configuration it can register itself with as a web-hook.
First find the subscriptions resource:
$ curl --verbose -X HEAD http://localhost:6553/app/dm
...
< HTTP/1.1 200 OK
< Date: Mon, 29 Aug 2011 10:00:50 GMT
< Link: http://localhost:6553/app/dm/web-hooks, rel="Subscriptions"
POST your endpoint information to the Subscriptions resource:
$ curl -X POST -d '{"name":"dm-host1", "endpoint":"http://localhost:4322/web-hooks"}' http://localhost:6553/app/dm/web-hooks
{
"endpoint": "http://localhost:4322/web-hooks",
"name": "dm-host1"
}
When the configuration for dm changes, a HTTP POST will be made
with a JSON object to the specified endpoint. Endpoints have to be
explicitly removed using DELETE. Make a note of the Location
header when registering the web-hook.
Requirements:
- Twisted (core and web)
- txgossip
Nesoi is installed as a twistd plugin, so you'll have to start it
using the twistd command-line tool. See twistd --help for more
information on generic options and such.
The nesoi service accepts the following options:
--listen-address IPlisten address (required)--listen-port PORTlisten port (required)--data-file FILEwhere to store config data (required)--seed IP:PORTanother nesoi instance to comminicate with
Example:
twistd nesoi --listen-address 10.2.2.2 --listen-port 6553 --seed 10.2.2.1:6553
Nesoi is in its foundation a distributed key-value store. Each resource pretty much maps to a key-value pair (except for the collection resources that maps to many key-value pairs of course).
Nesoi is in itself distributed. All Nesoi instances communicate
using a gossip protocol (using txgossip). Instances gossip with
each other about state changes to their local key-value stores.
Eventually all data has propagated to all nodes in the system.
Each value in the key-value store is annotated with a timestamp. This timestamp is used to resolve conflicts. A newer value always wins. As an effect of this, Nesoi assumes that all nodes running Nesoi instances have synchronized clocks.
Each Nesoi cluster has a leader. This leader is responsible for sending out the watcher notifications.
The API is quite simple.
Application configurations live under /app. You can retreive a
configuration using GET /app/<appname>. To update a create or
update a configuration use PUT /app/<appname>.
The data pushed to a /app/<appname> must be a JSON object holding a
config property.
To get a list of all application configurations do a GET /app.
Instances of applications register themselves as a service running
on a host. They do this but issuing a PUT to /srv/<appname>/<host>.
The data pushed to a /srv/<appname>/<host> must be a JSON object
holding a endpoints property.
Services can update their state by issuing further PUTs. Also, when
an instance shuts down it SHOULD delete it itself from the
registry using a DELETE on /srv/<appname>/<host>.
Nesoi implements webhooks [1] to allow clients to monitor changes to
a resource. See [2] for more information. The Notification-Type is
currently ignored. Hooks will be informed about all changes.
When registering a webhook, POST a json object with the following
attributes to the subscription resource:
- A client name (
name). Used to identify the endpoint from a service point of view. Normally constructed from hostname and service name. - An endpoint (
endpoint). URI where notification should be posted.
When something happens in Nesoi that triggers a notification, a
HTTP POST will be sent to the registered endpoint.
The payload of the body is a json object with the following
attributes:
- Name of webhook (
name) - URI to the resource that was changed (
uri)
Web-hooks can be attached to application configurations
(/app/<appname>/web-hooks) and service (/srv/<appname>/web-hooks).
An example:
{
"name": "node1-test",
"uri": "/app/test"
}
[1] http://wiki.webhooks.org [2] http://wiki.webhooks.org/w/page/13385128/RESTful%20WebHooks
To ease configuration management, Nesoi can help with distributing configuration data to all instances of a service.
The configuration manager (a person, or a piece of software) creates a
/app/NAME resource using the REST interface. When the configuration
is changed, he or she simply updates the resource with the new config.
Service instances are configured with information about where to reach
Nesoi. When starting up, the instance fetches the /app/NAME
resource. It also installes a watch on /app/NAME allowing the
service to be informed when the configuration is changed.
When a service instance has found its configuratio and initialized itself it should announce its presence to the rest of the distributed system. It does this by create (or updating) a service information resource at '/srv/APP/HOST'. The resource holds information about where the service endpoints (communication endspoints) are. Other information may also be included.
The service instance resource contains metadata about when the data was last updated. This information can be used to communicate some kind of status about the service instance. For example, a protocol can be put in place that the service instance should update its resource once every minute. If the resource has not been updated for two minutes, it should be considered dead.
If a leader among all service instances has to communicated to users
of the service, this can be done by setting a master field to true
in the resource. When a instance looses an election, it should right
away update its resource with master set to false. If a client
finds several service instances with that states that they are master,
the one with the latest "updated_at" time should be trusted.
When a service instance is removed from the system, the service instance resource should be removed with it.
When a client wants to talk to a service it fetches a representation
of the /srv/NAME resource. The resource will enumerate all known
service instances and their communication endpoints.
A client may also register a watch on /srv/NAME so that it will
get notified about changes to service instances.