This documentation page describes a development version, for production systems please use the stable version instead.
This tutorial will guide you through bringing up your Astarte instance, creating a realm and streaming your first data from a device simulator (or a real device) before your cup of tea is ready.
This tutorial will guide you through bringing up your Astarte instance, creating a realm and streaming your first data from a device simulator (or a real device) before your cup of tea is ready.
before-you-begin
diff --git a/astarte/1.2/062-trigger_delivery_policies.html b/astarte/1.2/062-trigger_delivery_policies.html index cbdd9763..8490b5d1 100644 --- a/astarte/1.2/062-trigger_delivery_policies.html +++ b/astarte/1.2/062-trigger_delivery_policies.html @@ -126,7 +126,7 @@Error handlers: a non-empty list of handlers. Each handler acts on groups of delivery errors and describes the strategy Astarte should take when they occur. In pseudo-BNF syntax, an handler is described by the following grammar:
handler ::= "{"
- "on" ":" "client_error" | "server_error" | "any_error" | [<int>] ","
+ "on" ":" "client_error" | "server_error" | "any_error" | [<int>] ","
"strategy" ":" "discard" | "retry"
"}"There are two possible strategies: either discard the event or retry. If the strategy is retry, events will be requeued in the event queue.
The default retry strategy is discarding events.
diff --git a/astarte/1.2/Astarte.epub b/astarte/1.2/Astarte.epub
index 6aced164..ae3bef27 100644
Binary files a/astarte/1.2/Astarte.epub and b/astarte/1.2/Astarte.epub differ
diff --git a/astarte/1.2/dist/search_items-BD27FAF1.js b/astarte/1.2/dist/search_items-8EDBF5DF.js
similarity index 94%
rename from astarte/1.2/dist/search_items-BD27FAF1.js
rename to astarte/1.2/dist/search_items-8EDBF5DF.js
index 3c17a8d9..3d4f0336 100644
--- a/astarte/1.2/dist/search_items-BD27FAF1.js
+++ b/astarte/1.2/dist/search_items-8EDBF5DF.js
@@ -1 +1 @@
-searchNodes=[{"type":"extras","title":"Introduction","doc":"This documentation page describes a development version, for production systems please use the stable version instead. Astarte is a collection of components written in Elixir meant to orchestrate and pilot a number of 3rd party components. These components include: One or more ingresses (the most popular implementation being an MQTT broker) An AMQP broker for handling messages and queues between Astarte's services A Cassandra-like Database for ingesting and retrieving data (currently Cassandra and ScyllaDB are both supported) These components are never directly exposed to Astarte's end user, who requires no knowledge whatsoever of the mentioned frameworks - they are rather orchestrated and managed directly by Astarte's services. It is, however, responsability of Astarte's administrators to make sure these services are made available the way they are meant to. For more details on this topic and, in general, on how to deal with Astarte's installation and maintenance, please refer to the Administrator Guide .","ref":"001-intro_architecture.html"},{"type":"extras","title":"Design Principles","doc":"Astarte has a strongly opinionated design aimed at the generic IoT / data-driven use case. As such, and unlike other platforms, it strives to streamline a very simple user workflow for ingesting, distributing and retrieving data, built on a set of concepts and principles.","ref":"010-design_principles.html"},{"type":"extras","title":"Design Principles - Declarative vs. Explicit Data Management","doc":"Astarte does not allow exchanging raw data - it rather forces the user to describe data before it is sent into the platform. Data is described with a mechanism named Interfaces, explained in detail in the user guide . Through Interfaces, Astarte creates and maintains a data model autonomously, sparing the user from the complexity of dealing with Databases and Data Management in general.","ref":"010-design_principles.html#declarative-vs-explicit-data-management"},{"type":"extras","title":"Design Principles - AMQP as internal API mechanism","doc":"Astarte services use a Protobuf-based API to exchange data over AMQP in a gRPC like fashion. As such, as long as a service conforms with the policies defined by the queues, it is possible to extend Astarte in virtually any language that can deliver a compliant AMQP client.","ref":"010-design_principles.html#amqp-as-internal-api-mechanism"},{"type":"extras","title":"Design Principles - Device ID","doc":"Astarte identifies each device with a 128 bit Device ID which has to be unique within its Realm. As a best practice, it is advised to generate such an ID from hardware unique IDs or using dedicated hardware modules, to make it consistent across device reflashes. It is advised to use a cryptographic hash function (such as sha256) when generating it using a software module. Astarte will use URL encoded base64 (without padding) strings like V_zv6ThCCtXWveQ8mPjsKg in its representation. Although not required, it is strongly advised to use UUIDs as Astarte Device IDs. In fact, Astarte Device ID's specification is 100% compatible with UUIDs Base64 encoded adhering to RFC 7515. In the same fashion, UUIDv5 can be used to generate a deterministic Device ID from any kind of input data. Astarte Clients which generate Astarte Device IDs (such as astartectl or Astarte Dashboard) will always generate a Device ID out of UUIDv4 (random ID) or UUIDv5 (deterministic ID). This detail is relevant not only for identifying and querying the device, but also for the Pairing mechanism , as a device's credentials are associated to its Device ID. Note: currently, Astarte accepts Device IDs longer than 128 bit, which are then truncated to 128 bit internally. This behaviour exists for compatibility reasons but it's not supported and will likely change in future releases - hence, refrain from using anything which is not a 128-bit Device ID. Note: As much as Device IDs should effectively be unique per-realm and this configuration will always be supported, some future optional optimizations might be available on top of the assumption that Device IDs are globally unique to an Astarte installation. Given the Device ID format has a 2<sup>-128</sup> chance of collision, it is safe to assume that as long as best practices for Device ID generation are followed, Device IDs will always be globally unique.","ref":"010-design_principles.html#device-id"},{"type":"extras","title":"Design Principles - Device interaction","doc":"Astarte assumes devices are capable of exchanging data over a transport/protocol supporting SSL/TLS (e.g.: MQTT). This is a strong requirement, as Astarte identifies devices through client SSL certificates when it comes to data exchange. Each transport implementation must be capable of mapping interfaces and out-of-band messages on top of it. Astarte itself does not care about the implementation detail of the transport itself, as the transport is in charge of converting its input to an AMQP message following Astarte's internal API specification. Astarte's official reference and recommended design is MQTT using VerneMQ and its Astarte plugin. Device SDK and code generation Device SDKs can take advantage of the interface design to dynamically generate code for exchanging data with Astarte. This way, developers using Device SDKs are spared from knowing details about the underlying transports and protocols, and can use a data-driven API. However, there are some limitations and requirements: The SDK requires SSL support - Astarte does not allow exchanging data over unencrypted channels and its design builds on the assumption that everything runs on top of SSL. If your device isn't capable of SSL, you are probably looking for Gateway support in Astarte. As much as the SDK can implement virtually any transport protocol, it is required that the SDK supports at least HTTP(s) for Pairing.","ref":"010-design_principles.html#device-interaction"},{"type":"extras","title":"Design Principles - Realms and multitenancy","doc":"Astarte is natively multitenant through the concept of Realms. Each Realm is a logical portion of Astarte, and usually represents an organization or, in general, a set of devices physically/logically isolated. Realms build upon the concept of keyspaces in Cassandra. Each Realm has its very own keyspace and has no shared data with other Realms. In fact, it is even possible to have a dedicated Cassandra cluster for a single realm in complex installations.","ref":"010-design_principles.html#realms-and-multitenancy"},{"type":"extras","title":"Design Principles - Message Ordering","doc":"In Astarte, transports are given the task to deliver messages in a well-known AMQP structure. The ordering of such messages is then preserved on a set of criterias: There is no such thing as "in-order" among devices. A message X sent to device A can be processed after a message Y sent to device B even if Y was ingested in the AMQP queue before X. This is intentional and by design. All messages to a specific device A are always guaranteed to be processed in the very same order of the transport ingestion. Ordering is not dependent on the message timestamp, which can be set by different sources (depending on the interface's definition of timestamp). For example, interface A has explicit timestamping while interface B doesn't. Message X from A has an earlier timestamp than message Y from B, but if message Y has been ingested before X, Y will be processed before X regardless. Responsibility of message ordering before entering AMQP is entirely up to the transport, and different transports might have different behaviors when it comes to message ordering. Astarte provides this guarantee right after the transport itself. Message ordering concerns only pipelines in the DUP , including but not limited to data ingestion in the Database and Simple Triggers.","ref":"010-design_principles.html#message-ordering"},{"type":"extras","title":"Design Principles - Triggers","doc":"Triggers are rules which are "triggered" whenever one or more conditions are satisfied. Every satisfied condition generates an ordered event for the Trigger Engine to be processed. They are one of the core concepts in Astarte and are the preferred way to handle push interactions between Astarte and connected applications. More details about triggers can be found in the dedicated section .","ref":"010-design_principles.html#triggers"},{"type":"extras","title":"Components","doc":"Astarte is a distributed system interacting over AMQP, as explained in Design Principles . This is an overview of its main internal services.","ref":"020-components.html"},{"type":"extras","title":"Components - Pairing","doc":"Pairing takes care of Device Authentication and Authorization. It interacts with Astarte's CA and orchestrates the way devices connect and interact with Transports. It also handles Device Registration. Agent, Device and Pairing interaction is described in detail here .","ref":"020-components.html#pairing"},{"type":"extras","title":"Components - Data Updater Plant (DUP)","doc":"Data Updater Plant is a replicable, scalable component which takes care of the ingestion pipeline. It gathers data from devices and orchestrates data flow amongst other components. It is, arguably, the most critical component of the system and the most resource hungry - the way DUP is deployed, replicated and configured has a tremendous impact on Astarte's performances, especially when dealing with massive data flows.","ref":"020-components.html#data-updater-plant-dup"},{"type":"extras","title":"Components - Trigger Engine","doc":"Trigger Engine takes care of processing Triggers. It is a purely computational component which handles every Trigger's pipeline and triggers actions accordingly.","ref":"020-components.html#trigger-engine"},{"type":"extras","title":"Components - AppEngine","doc":"AppEngine is Astarte's main API endpoint for end users. AppEngine exposes a RESTful API to retrieve and send data from/to devices, according to their interfaces. Every direct device interaction can be done from here. It also exposes Channels, a WebSocket-based solution for listening to device events in real-time with Triggers' same mechanism and semantics.","ref":"020-components.html#appengine"},{"type":"extras","title":"Components - Realm Management","doc":"Realm Management is an administrator-like API for configuring a Realm. It is used for managing Interfaces and Triggers.","ref":"020-components.html#realm-management"},{"type":"extras","title":"Components - Housekeeping","doc":"Housekeeping is the equivalent of a superadmin API. It is usually not accessible to the end user but rather to Astarte's administrator who, in most cases, might deny overall outside access. It allows to manage and create Realms, and perform cluster-wide maintenance actions.","ref":"020-components.html#housekeeping"},{"type":"extras","title":"Interfaces","doc":"Interfaces are a core concept of Astarte which defines how data is exchanged between Astarte and its peers. They are not to be intended as OOP interfaces, but rather as the following definition: In computing, an interface is a shared boundary across which two or more separate components of a computer system exchange information. In Astarte each interface has an owner, can represent either a continuous data stream or a snapshot of a set of properties, and can be either aggregated into an object or be an independent set of individual members. If you are already familiar with interface's basic concepts, you might want to jump directly to the Interface Schema .","ref":"030-interface.html"},{"type":"extras","title":"Interfaces - Versioning","doc":"Interfaces are versioned, each interface having both a major version and a minor version number. The concept behind these two version numbers mimics Semantic Versioning : arbitrary changes can happen exclusively between different major versions (e.g. removing members, changing types, etc...), whereas minor versions allow incremental additive changes only (e.g. adding members). Several different major versions of the same interface can coexist at the same time in Astarte, although a Device can hold only a single version of an interface at a time (even though interfaces can be updated over time). Interfaces, internally, are univocally identified by their name and their major version.","ref":"030-interface.html#versioning"},{"type":"extras","title":"Interfaces - Format","doc":"Interfaces are described using a JSON document. Each interface is identified by an unique interface name of maximum 128 characters, which must be a Reverse Domain Name . As a convention, the interface name usually contains its author's URI Reverse Internet Domain Name. An example skeleton looks like this: { "interface_name": "com.test.MyInterfaceName", "version_major": 1, "version_minor": 0, [...] } Valid values and variables are listed in the Interface Schema . Name limitations A valid interface name consists of a Reverse Domain Name containing alphanumeric characters, hyphens and dots. By design, both the top level domain and last domain component can not contain hyphens, although hypens are allowed in other parts of the interface name (e.g.: org.astarte-platform.Values is a valid interface name). Interface names have to be fully-defined Reverse Domain Names. Values will not be accepted as an Astarte interface name, whereas org.astarte-platform.Values is a valid one. Interface's uniqueness is case insensitive - this means you cannot install two interfaces with the same name and different casing (e.g.: org.astarte-platform.MyValues and org.astarte-platform.Myvalues ). This also applies to Major versioning: interfaces sharing the same name with a different major version cannot have different casing. Although not enforced, naming conventions for Astarte Interfaces require lowercasing for anything but the last part of the Interface name, which should be CamelCase. Valid examples are: org.astarte-platform.conventions.ValidInterfaceName org.astarte-platform.ValidInterfaceName org.astarte-platform.conventions.satisfied.ValidInterfaceName Non-valid examples are: org.astarte-platform.Conventions.ValidInterfaceName org.astarte-platform.validInterfaceName org.astarte-platform.Conventions.satisfied.ValidInterfaceName","ref":"030-interface.html#format"},{"type":"extras","title":"Interfaces - Interface Type","doc":"Interfaces have a well-known, predefined type, which can be either property or datastream . Every Device in Astarte can have any number of interfaces of any different types. Datastream datastream represents a mutable, ordered stream of data, with no concept of persistent state or synchronization. As a rule of thumb, datastream interfaces should be used when dealing with values such as sensor samples, commands and events. datastream are stored as time series in the database, making them suitable for time span filtering and any other common time series operation, and they are not idempotent in the REST API semantics. Due to their nature, datastream interfaces have a number of additional properties which fine tune their behavior. Properties properties represent a persistent, stateful, synchronized state with no concept of history or timestamping. properties are useful, for example, when dealing with settings, states or policies/rules. properties are stored in a key-value fashion, and grouped according to their interface, and they are idempotent in the REST API semantics. Rather than being able to act on a stream like in the datastream case, properties can be retrieved, or can be used as a trigger whenever they change. Values in a properties interface can be unset (or deleted according to the http jargon): to allow such a thing, the interface must have its allow_unset property set to true . Please refer to the JSON Schema for further details.","ref":"030-interface.html#interface-type"},{"type":"extras","title":"Interfaces - Ownership","doc":"Astarte's design mandates that each interface has an owner. The owner of an interface has a write-only access to it, whereas other actors have read-only access. Interface ownership can be either device or server : the owner is the actor producing the data, whereas the other actor consumes data.","ref":"030-interface.html#ownership"},{"type":"extras","title":"Interfaces - Mappings","doc":"Every interface must have an array of mappings. Mappings are designed around REST controller semantics: each mapping describes an endpoint which is resolved to a path, it is strongly typed, and can have additional options. Just like in REST controllers, Endpoints can be parametrized to build REST-like collection and trees. Parameters are identified by %{parameterName} , with each endpoint supporting any number of parameters (see Limitations ). This is how a parametrized mapping looks like: [...] "mappings": [ { "endpoint": "/%{itemIndex}/value", "type": "integer", "reliability": "unique", "retention": "discard" }, [...] In this example, /0/value , /1/value or /test/value all map to a valid endpoint, while /te/st/value can't be resolved by any endpoint. Supported data types The following types are supported: double : A double-precision floating-point number as specified by binary64, by the IEEE 754 standard (NaNs and other non numerical values are not supported). integer : A signed 32 bit integer. boolean : Either true or false , adhering to JSON boolean type. longinteger : A signed 64 bit integer (please note that longinteger is represented as a string by default in JSON-based APIs.). string : An UTF-8 string, at most 65536 bytes long. binaryblob : An arbitrary sequence of any byte that should be shorter than 64 KiB. ( binaryblob is represented as a base64 string by default in JSON-based APIs.). datetime : A UTC timestamp, internally represented as milliseconds since 1st Jan 1970 using a signed 64 bits integer. ( datetime is represented as an ISO 8601 string by default in JSON based APIs.) doublearray , integerarray , booleanarray , longintegerarray , stringarray , binaryblobarray , datetimearray : A list of values, represented as a JSON Array. Arrays can have up to 1024 items and each item must respect the limits of its scalar type ( i.e. each string in a stringarray must be at most 65535 bytes long, each binary blob in a binaryblobarray must be shorter than 64 KiB. Make sure that the differences between two distinct interface names are not limited to the casing or the presence of hyphens. This situation leads to a collision in the interface names which brings to an error in the interface installation process. Limitations A valid interface must resolve a path univocally to a single endpoint. Take the following example: [...] "mappings": [ { "endpoint": "/%{itemIndex}/value", "type": "integer" }, { "endpoint": "/myPath/value", "type": "integer" }, [...] In such a case, the interface isn't valid and is rejected, due to the fact that path /myPath/value is ambiguous and could be resolved to two different endpoints. Any endpoint configuration must not generate paths that are prefix of other paths, for this reason the following example is also invalid: [...] "mappings": [ { "endpoint": "/some/thing", "type":"integer" }, { "endpoint": "/some/%{param}/value", "type": "integer" }, [...] In case the interface's aggregation is object , additional restrictions apply. Endpoints in the same interface must all have the same depth, and the same number of parameters. If the interface is parametrized, every endpoint must have the same parameter name at the same level. This is an example of a valid aggregated interface mapping: [...] "mappings": [ { "endpoint": "/%{itemIndex}/value", "type": "integer" }, { "endpoint": "/%{itemIndex}/otherValue", "type": "string" }, [...] Additional limitations (which stem from the MQTT protocol specification) can be outlined. When using parametric endpoints, the actual values used in place of parameter placeholders must fulfill the following requirements: endpoint parameters must be non-empty UTF-8 encoded strings; endpoint parameters must not contain the following characters: + and # . Those characters are treated as wildcards for MQTT topics and therefore must be avoided; endpoint parameters must not contain the / character.","ref":"030-interface.html#mappings"},{"type":"extras","title":"Interfaces - Aggregation","doc":"In a real world scenario, such as an array of sensors, there are usually two main cases. A sensor might have one or more independent values which are sampled individually and sent whenever they become available independently. Or a sensor might sample at the same time a number of values, which might as well have some form of correlation. In Astarte, this concept is mapped to interface aggregation . In case aggregation is individual , each mapping is treated as an independent value and is managed individually. In case aggregation is object , Astarte expects the owner to send all of the interface's mappings at the same time, packed in a single message. In this case, all of the mappings share some core properties such as the timestamp. Aggregation is a powerful mechanism that can be used to map interfaces to real world "objects" . Moreover, aggregated interfaces can also be parametrized, although with some limitations . Endpoints and aggregation Since Astarte 0.11, Aggregations cannot have endpoints with depth 1. This was an erroneously allowed behavior in Astarte 0.10 which is kept for retrocompatibility - however, new interfaces should ensure each endpoint in an aggreate has at least depth 2, as support for depth 1 will be removed in a future release. This change has been done to be consistent with AppEngine API design, and to ensure that path / is not ambiguous. This is the correct way to set up a valid endpoint structure for an aggregate: [...] "mappings": [ { "endpoint": "/objects/value", "type": "integer" }, { "endpoint": "/objects/otherValue", "type": "string" }, [...] The following structure, instead, is deprecated: [...] "mappings": [ { "endpoint": "/value", "type": "integer" }, { "endpoint": "/otherValue", "type": "string" }, [...]","ref":"030-interface.html#aggregation"},{"type":"extras","title":"Interfaces - Datastream-specific features","doc":"datastream interfaces are highly tunable, depending on the kind of data they are representing: it is possible to fine tune several aspects of how data is stored, transferred and indexed. The following properties can be set at mapping level. NOTE: In case the interface is aggregated, additional properties must be the same for each mapping. explicit_timestamp : By default, Astarte associates a timestamp to data whenever it is collected (or - when the message hits the data collection stage). However, when setting this property to true , Astarte expects the owner to attach a valid timestamp each time it produces data. In that case, the provided timestamp is used for indexing. reliability : Each mapping can be unreliable (default), guaranteed , unique . This defines whether data should be considered delivered when the transport successfully sends the data regardless of the outcome ( unreliable ), when data has been received at least once by the recipient ( guaranteed ) or when data has been received exactly once by the recipient ( unique ). When using reliable data, consider you might incur in additional resource usage on both the transport and the device's end. retention : Each mapping can have a discard (default), volatile , stored retention. This defines whether data should be discarded if the transport is temporarily uncapable of delivering it ( discard ), should be kept in a cache in memory ( volatile ) or on disk ( stored ), and guaranteed to be delivered in the timeframe defined by the expiry . expiry : Meaningful only when retention is stored . Defines how many seconds a specific data entry should be kept before giving up and erasing it from the persistent cache. A value <= 0 means the persistent cache never expires, and is the default. database_retention_policy : Useful only with datastream. Defines whether data should expire from the database after a given interval. Valid values are: no_ttl and use_ttl. database_retention_ttl : Useful when database_retention_policy is "use_ttl" . Defines how many seconds a specific data entry should be kept before erasing it from the database.","ref":"030-interface.html#datastream-specific-features"},{"type":"extras","title":"Interfaces - Best practices","doc":"When creating interface drafts, or for testing purposes in general, it is recommended to use 0 as the major version, to make maintenance and testing easier. Currently, Astarte allows only interfaces with major_version == 0 to be deleted, and this limitation will probably be never lifted to prevent data loss. When sending real time commands in datastream interfaces, discard is usually the best option. Even though it does not guarantee delivery, it prevents users from unwillingly sending the same command over and over if the recipient isn't available, causing a queue of commands to be sent to the recipient when it gets back online. In general, retention should be used to keep track of low traffic/important events","ref":"030-interface.html#best-practices"},{"type":"extras","title":"Interface Schema","doc":"The schema contains the following objects: Interface (root object) Mapping","ref":"040-interface_schema.html"},{"type":"extras","title":"Interface Schema - Interface","doc":"This schema describes how an Astarte interface should be declared Properties Type Description Required interface_name string The name of the interface. This has to be an unique, alphanumeric reverse internet domain name, shorther than 128 characters. ✔ Yes version_major integer A Major version qualifier for this interface. Interfaces with the same id and different version_major number are deemed incompatible. It is then acceptable to redefine any property of the interface when changing the major version number. ✔ Yes version_minor integer A Minor version qualifier for this interface. Interfaces with the same id and major version number and different version_minor number are deemed compatible between each other. When changing the minor number, it is then only possible to insert further mappings. Any other modification might lead to incompatibilities and undefined behavior. ✔ Yes type string Identifies the type of this Interface. Currently two types are supported: datastream and properties. datastream should be used when dealing with streams of non-persistent data, where a single path receives updates and there's no concept of state. properties, instead, are meant to be an actual state and as such they have only a change history, and are retained. ✔ Yes ownership string Identifies the quality of the interface. Interfaces are meant to be unidirectional, and this property defines who's sending or receiving data. device means the device/gateway is sending data to Astarte, consumer means the device/gateway is receiving data from Astarte. Bidirectional mode is not supported, you should instantiate another interface for that. ✔ Yes aggregation string Identifies the aggregation of the mappings of the interface. Individual means every mapping changes state or streams data independently, whereas an object aggregation treats the interface as an object, making all the mappings changes interdependent. Choosing the right aggregation might drastically improve performances. No, default: "individual" description string An optional description of the interface. No doc string A string containing documentation that will be injected in the generated client code. No mappings Astarte Mapping Schema [1-1024] Mappings define the endpoint of the interface, where actual data is stored/streamed. They are defined as relative URLs (e.g. /my/path) and can be parametrized (e.g.: /%{myparam}/path). A valid interface must have no mappings clash, which means that every mapping must resolve to a unique path or collection of paths (including parametrization). Every mapping acquires type, quality and aggregation of the interface. ✔ Yes Additional properties are allowed. astarte.interface.schema.interface_name ✔ The name of the interface. This has to be an unique, alphanumeric reverse internet domain name, shorther than 128 characters. Type : string Required : Yes Minimum Length : >= 1 Allowed Pattern : ^([a-zA-Z][a-zA-Z0-9]*.([a-zA-Z0-9][a-zA-Z0-9-]*.)*)?[a-zA-Z][a-zA-Z0-9]*$ astarte.interface.schema.version_major ✔ A Major version qualifier for this interface. Interfaces with the same id and different version_major number are deemed incompatible. It is then acceptable to redefine any property of the interface when changing the major version number. Type : integer Required : Yes astarte.interface.schema.version_minor ✔ A Minor version qualifier for this interface. Interfaces with the same id and major version number and different version_minor number are deemed compatible between each other. When changing the minor number, it is then only possible to insert further mappings. Any other modification might lead to incompatibilities and undefined behavior. Type : integer Required : Yes astarte.interface.schema.type ✔ Identifies the type of this Interface. Currently two types are supported: datastream and properties. datastream should be used when dealing with streams of non-persistent data, where a single path receives updates and there's no concept of state. properties, instead, are meant to be an actual state and as such they have only a change history, and are retained. Type : string Required : Yes Allowed values : "datastream" "properties" astarte.interface.schema.ownership ✔ Identifies the quality of the interface. Interfaces are meant to be unidirectional, and this property defines who's sending or receiving data. device means the device/gateway is sending data to Astarte, consumer means the device/gateway is receiving data from Astarte. Bidirectional mode is not supported, you should instantiate another interface for that. Type : string Required : Yes Allowed values : "device" "server" astarte.interface.schema.aggregation Identifies the aggregation of the mappings of the interface. Individual means every mapping changes state or streams data independently, whereas an object aggregation treats the interface as an object, making all the mappings changes interdependent. Choosing the right aggregation might drastically improve performances. Type : string Required : No, default: "individual" Allowed values : "individual" "object" astarte.interface.schema.description An optional description of the interface. Type : string Required : No astarte.interface.schema.doc A string containing documentation that will be injected in the generated client code. Type : string Required : No astarte.interface.schema.mappings ✔ Mappings define the endpoint of the interface, where actual data is stored/streamed. They are defined as relative URLs (e.g. /my/path) and can be parametrized (e.g.: /%{myparam}/path). A valid interface must have no mappings clash, which means that every mapping must resolve to a unique path or collection of paths (including parametrization). Every mapping acquires type, quality and aggregation of the interface. Type : Astarte Mapping Schema [1-1024] Each element in the array must be unique. Required : Yes","ref":"040-interface_schema.html#interface"},{"type":"extras","title":"Interface Schema - Mapping","doc":"Identifies a mapping for an interface. A mapping must consist at least of an endpoint and a type. Properties Type Description Required endpoint string The template of the path. This is a UNIX-like path (e.g. /my/path) and can be parametrized. Parameters are in the %{name} form, and can be used to create interfaces which represent dictionaries of mappings. When the interface aggregation is object, an object is composed by all the mappings for one specific parameter combination. ✔ Yes type string Defines the type of the mapping. ✔ Yes reliability string Useful only with datastream. Defines whether the sent data should be considered delivered when the transport successfully sends the data (unreliable), when we know that the data has been received at least once (guaranteed) or when we know that the data has been received exactly once (unique). unreliable by default. When using reliable data, consider you might incur in additional resource usage on both the transport and the device's end. No, default: "unreliable" explicit_timestamp boolean Allow to set a custom timestamp, otherwise a timestamp is added when the message is received. If true explicit timestamp will also be used for sorting. This feature is only supported on datastreams. No, default: false retention string Useful only with datastream. Defines whether the sent data should be discarded if the transport is temporarily uncapable of delivering it (discard) or should be kept in a cache in memory (volatile) or on disk (stored), and guaranteed to be delivered in the timeframe defined by the expiry. discard by default. No, default: "discard" expiry integer Useful when retention is stored. Defines after how many seconds a specific data entry should be kept before giving up and erasing it from the persistent cache. A value <= 0 means the persistent cache never expires, and is the default. No, default: 0 database_retention_policy string Useful only with datastream. Defines whether data should expire from the database after a given interval. Valid values are: no_ttl and use_ttl. No, default: "no_ttl" database_retention_ttl integer Useful when database_retention_policy is "use_ttl" . Defines how many seconds a specific data entry should be kept before erasing it from the database. No allow_unset boolean Used only with properties. Used with producers, it generates a method to unset the property. Used with consumers, it generates code to call an unset method when an empty payload is received. No, default: false description string An optional description of the mapping. No doc string A string containing documentation that will be injected in the generated client code. No Additional properties are allowed. astarte.mapping.schema.endpoint ✔ The template of the path. This is a UNIX-like path (e.g. /my/path) and can be parametrized. Parameters are in the %{name} form, and can be used to create interfaces which represent dictionaries of mappings. When the interface aggregation is object, an object is composed by all the mappings for one specific parameter combination. Refer to the mapping section of the interface page for further details on allowed parameter substitution values. Type : string Required : Yes Minimum Length : >= 2 Allowed Pattern : ^(/(%{([a-zA-Z_][a-zA-Z0-9_]*)}|[a-zA-Z_][a-zA-Z0-9_]*)){1,64}$ astarte.mapping.schema.type ✔ Defines the type of the mapping. Type : string Required : Yes Allowed values : "double" "integer" "boolean" "longinteger" "string" "binaryblob" "datetime" "doublearray" "integerarray" "booleanarray" "longintegerarray" "stringarray" "binaryblobarray" "datetimearray" astarte.mapping.schema.reliability Useful only with datastream. Defines whether the sent data should be considered delivered when the transport successfully sends the data (unreliable), when we know that the data has been received at least once (guaranteed) or when we know that the data has been received exactly once (unique). unreliable by default. When using reliable data, consider you might incur in additional resource usage on both the transport and the device's end. Type : string Required : No, default: "unreliable" Allowed values : "unreliable" "guaranteed" "unique" astarte.mapping.schema.explicit_timestamp Allow to set a custom timestamp, otherwise a timestamp is added when the message is received. If true explicit timestamp will also be used for sorting. This feature is only supported on datastreams. Type : boolean Required : No, default: false astarte.mapping.schema.retention Useful only with datastream. Defines whether the sent data should be discarded if the transport is temporarily uncapable of delivering it (discard) or should be kept in a cache in memory (volatile) or on disk (stored), and guaranteed to be delivered in the timeframe defined by the expiry. discard by default. Type : string Required : No, default: "discard" Allowed values : "discard" "volatile" "stored" astarte.mapping.schema.expiry Useful when retention is stored. Defines after how many seconds a specific data entry should be kept before giving up and erasing it from the persistent cache. A value <= 0 means the persistent cache never expires, and is the default. Type : integer Required : No, default: 0 astarte.mapping.schema.database_retention_policy Useful only with datastream. Defines whether data is expired from the database after a given time to live interval. When "no_ttl" is used data are not expired. Type : string Required : No astarte.mapping.schema.database_retention_ttl Useful when database_retention_policy is "use_ttl" . Defines how many seconds a specific data entry should be kept before erasing it from the database. Type : integer Required : No astarte.mapping.schema.allow_unset Used only with properties. Used with producers, it generates a method to unset the property. Used with consumers, it generates code to call an unset method when an empty payload is received. Type : boolean Required : No, default: false astarte.mapping.schema.description An optional description of the mapping. Type : string Required : No astarte.mapping.schema.doc A string containing documentation that will be injected in the generated client code. Type : string Required : No","ref":"040-interface_schema.html#mapping"},{"type":"extras","title":"Pairing Mechanism","doc":"Astarte's Pairing is a unified mechanism for Registering Devices and obtaining Transport Credentials . Even though in Astarte each Transport is free to choose its own Authentication mechanisms and Credentials autonomously, Pairing defines a well-known mechanism for Registering Devices and for orchestrating the exchange of Transport Credentials . Pairing is the main endpoint which orchestrates Device Authentication in Astarte, abstracting all details.","ref":"050-pairing_mechanism.html"},{"type":"extras","title":"Pairing Mechanism - Authentication flow","doc":"","ref":"050-pairing_mechanism.html#authentication-flow"},{"type":"extras","title":"Pairing Mechanism - Credentials Secret vs. Transport Credentials","doc":"Each device is identified by a Device ID and, on top of that, it has two different credentials directly associated to its ID: Credentials Secret and Transport Credentials . Credentials Secret is a shared secret between Astarte and a Device, which are used only to authenticate against Pairing API. Each device has a single Credentials Secret which remains valid throughout its whole lifecycle, and cannot be changed (unless operating manually). Transport Credentials are Transport-specific credentials usually orchestrated by Pairing. Pairing emits these Credentials through a policy which is usually imposed by the Authority emitting the Credentials or by Pairing itself. They are designed to be transient, revokable and reasonably short-lived - however, the actual behavior and their lifecycle is entirely orchestrated by the Authority emitting them. The emission of Transport Credentials can be inhibited for a specific Device, you can read how to do that in the User Guide Transports, by design, have no knowledge nor access to Credentials Secret , but have full authority over the authentication mechanism for devices. In fact, each Transport is free to choose the authentication mechanism which fits it best. Credentials Secret storage recommendations As losing or disclosing a Credentials Secret might mean a device is compromised or requires manual intervention to be fixed and secured, storing it appropriately is critical. Usually, when it comes to embedded devices, it is advised to store the Credentials Secret into an OTP, if available. Otherwise, storing it into the bootloader's variables is a viable and safe alternative. Other options might be having a separate, isolated storage containing Credentials Secret . In general, Astarte SDK does not provide a streamlined mechanism for retrieving Credentials Secret as the storage detail is strongly dependent on the target hardware - device developers should implement the safest strategy which better complies with their policies. Tuning devices for security is out of the scope of this guide, however it is advised to make sure only Astarte SDK has access to Credentials Secret .","ref":"050-pairing_mechanism.html#credentials-secret-vs-transport-credentials"},{"type":"extras","title":"Pairing Mechanism - Using SSL Certificates as Transport Credentials","doc":"Whenever possible, Transports are advised to implement their Authentication through the use of SSL certificates and a certificate authority by using Mutual Authentication , to ensure identities of the endpoint and the client are well-known to each other - this is especially the case with Astarte's MQTT Protocol on top of VerneMQ Transport. In this case, Transport Credentials are a SSL Certificate, and Pairing will interact with a Certificate Authority. The certificate rotates depending on the emission policy of the CA and can be renewed and invalidated countless times over the device lifecycle. The Certificate is a transient, asymmetric, device-specific, non-critical Transport Credential which can be in turn used to authenticate against the chosen Transport. In this case, Transports should have no knowledge nor access to secrets or Authorization details: they rather have to comply with the configured CA and the certificate parsing, as the Certificate contains all needed information for Authorization as well. Mutual SSL Authentication Flow Side note: the Transport usually bears the public certificate of the CA, and actually interacts with the CA itself only if it exposes an OCSP endpoint and the Transport is capable of understanding it. In case the CA exposes a CRL, the Transport just makes sure to update its CRL from the CA every once in a while. In both cases, Transport's only interaction with the CA is the configuration of its SSL endpoint. Certificate Authority Pairing is designed to interact with an abstract certificate authority, given this authority is capable of: Emitting SSL Certificates with a custom CN (this is important in the Transport authentication flow) Revoking emitted certificates and exposing CRL/OCSP revocation information and is accessible from a 3rd party (e.g. from a REST API). By default, Astarte supports Cloudflare's CFSSL , and also provides a minimal installation in its default deploy scripts. For bigger installations, especially in terms of number of connected devices, it is strongly advised to use a dedicated CFSSL installation. Also, Astarte Enterprise provides a number of additional features including support for other external CAs. Certificate flow During the Pairing flow, the device must generate autonomously a Certificate Signing Request (CSR) which will be in turn relayed by Pairing to the configured Certificate Authority. Pairing will also provide the Certificate Authority with a custom CN, which maps to <realm>/<device id> . The CA must ensure the signed certificate carries this information, as it will be used by the Transport to authenticate the caller inside Astarte. Pairing, in fact, will also perform sanity checks over the signed certificate and reject it in case the CA fails to comply.","ref":"050-pairing_mechanism.html#using-ssl-certificates-as-transport-credentials"},{"type":"extras","title":"Pairing Mechanism - Agents","doc":"Agents are realm-level entities capable of registering a device into Astarte. Agents are a core concept in the Pairing mechanism, as no Device can request its Transport Credentials nor be authenticated against any Transport unless an Agent previously gave its consent and delivered its Credentials Secret . The recommended configuration includes an authenticated Agent in a trusted physical environment (e.g.: the distribution facility of the device) which guarantees an isolated and safe routine for generating Credentials Secret . However, such a setup might not always be possible, and Astarte's SDK has an On Board Agent concept to allow a simpler registration procedure. On Board Agent In the On Board Agent use case, the device is preloaded with an Agent Key , a shared secret which is not tied to a specific Device in the realm . In fact, this secret is usually the same for all Devices in the same realm. This secret will be used only once, upon the device's first interaction with Astarte (Registration), and can be safely discarded afterwards. This approach largely simplifies the deploy procedure, but leaves every device with a secret which, if retrieved, can allow an entity to register an arbitrary Device in the realm. If following the On Board Agent approach, it is advised to store the Agent Key in a safe area inside the device and delete it after retrieving a Credentials Secret (some OTPs allow this configuration).","ref":"050-pairing_mechanism.html#agents"},{"type":"extras","title":"Pairing Mechanism - Transport responsibility","doc":"Once a device obtains its Transport Credentials , it is then capable of connecting to the Transport the credentials were forged for. Transports have full responsibility in terms of authenticating the client, reporting and relaying its connection state to Astarte via its internal AMQP API. As such, it is fundamental that 3rd parties implementing new Transports not only adhere to protocol specifications, but also make sure to implement the authentication procedure meticolously, as a vulnerable Transport acts as a single point of failure of the whole system, and is capable of bypassing the Pairing workflow entirely. For this very reason, we encourage users to be extremely cautious when using 3rd party Transports which have not been verified and hardly tested, especially when it comes to the Client Authentication stage. Even though there are valid use cases where Mutual Authentication is not usable, Transports are advised to stick to Mutual SSL Authentication where possible. This, among other benefits, allows to use Pairing's core features for handling SSL Certificates.","ref":"050-pairing_mechanism.html#transport-responsibility"},{"type":"extras","title":"Pairing Mechanism - Pairing facilities","doc":"Pairing's Device API exposes two additional facilities: first and foremost an endpoint which bears a set of information about both Pairing itself and Transports the device should use or choose from. This endpoint is Device and Realm specific and can be found at /{realm_name}/devices/{hw_id} . This allows granting each Device a specific Transport configuration, which can be useful in installations with more than a single Transport, and automates the configuration on the Device's end, which knows in advance what is supported and how to access its Transport(s). Moreover, each Transport implementation has a /verify endpoint where a client, authenticating with its Credentials Secret , can verify whether its Transport Credentials are valid or not. This, in case SSL is used, is especially useful for checking against revocation lists.","ref":"050-pairing_mechanism.html#pairing-facilities"},{"type":"extras","title":"Triggers","doc":"Triggers in Astarte are the go-to mechanism for generating push events. In contrast with AppEngine's REST APIs, Triggers allow users to specify conditions upon which a custom payload is delivered to a recipient, using a specific action , which usually maps to a specific transport/protocol, such as HTTP. Given this kind of flexibility, triggers are the most powerful way to push data to an external service, potentially without any additional customization. Triggers can be managed from Realm Management API , astartectl with the astartectl realm-management triggers subcommand, or Astarte Dashboard in the Triggers page.","ref":"060-triggers.html"},{"type":"extras","title":"Triggers - Building Triggers","doc":"Triggers can be either built manually or using Astarte Dashboard's Trigger Editor. Trigger Editor dynamically loads installed Interfaces in the Realm and eases trigger creation by providing not only linting and validation, but also dynamic resolution of Interface names. Trigger Editor works in a very similar fashion to Interfaces Editor, and shares the same User Interface. Format A trigger is described using a JSON document. Each trigger is defined by two main parts: condition and action . This is a JSON representation of an example trigger: { "name": "example_trigger", "action": { "http_url": "https://example.com/my_hook", "http_method": "post" }, "simple_triggers": [ { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.4 } ] } The condition is represented by the simple_triggers array. In this release, Astarte supports only a single entry in the simple_triggers array, but support for multiple simple triggers (and different ways to combine them) is planned for future releases. The condition in the example specifies that when data is received on the org.astarte-platform.genericsensors.Values interface on /streamTest/value path, if the value of said data is > 0.4 , then the trigger is activated. For more information about all the possible conditions, check out the Conditions section The action object describes what the result of the trigger will be. In this specific case, an HTTP POST request will be sent to https://example.com/my_hook , with the payload: { "timestamp": "<event_timestamp>", "device_id": "<device_id>", "event": { "type": "incoming_data", "interface": "org.astarte-platform.genericsensors.Values", "path": "/streamTest/value", "value": <some_value> } } To know more about all possible actions, check the Actions section","ref":"060-triggers.html#building-triggers"},{"type":"extras","title":"Triggers - Conditions","doc":"A condition defines the event upon which an action is triggered. Conditions are expressed through simple triggers. Astarte monitors incoming events and triggers a corresponding action whenever there is a match. Simple triggers are divided into two types: Device Triggers and Data Triggers . Device Triggers Device triggers express conditions matching the state of a device. This is the generic representation of a Device Trigger: { "type": "device_trigger", "on": "<device_trigger_type>", "device_id": "<device_id>", "group_name": "<group_name>" } Parameters device_trigger_type can be one of the following: device_connected : triggered when a device connects to its transport. device_disconnected : triggered when a device disconnects from its transport. device_error : triggered when data from a device causes an error. device_id can be used to pass a specific Device ID to restrict the trigger to a single device. * is also accepted as device_id to maintain backwards compatibility and it is considered equivalent to no device_id specified. group_name can be used to restrict the trigger to all devices that are member of the group. device_id and group_name are mutually exclusive and if neither of them is specified in the simple trigger, the simple trigger will be installed for all devices in a realm. Data Triggers Data triggers express conditions matching data coming from a device. This is the generic representation of a Data Trigger: { "type": "data_trigger", "device_id": "<device_id>", "group_name": "<group_name>", "on": "<data_trigger_type>", "interface_name": "<interface_name>", "interface_major": "<interface_major>", "match_path": "<match_path>", "value_match_operator": "<value_match_operator>", "known_value": <known_value> } Data triggers are installed for all devices in a Realm. Data Triggers Parameters device_id can be used to pass a specific Device ID to restrict the trigger to a single device. * is also accepted as device_id to maintain backwards compatibility and it is considered equivalent to no device_id specified. group_name can be used to restrict the trigger to all devices that are member of the group. device_id and group_name are mutually exclusive and if neither of them is specified in the simple trigger, the simple trigger will be installed for all devices in a realm. data_trigger_type can be one of the following: incoming_data : verifies the condition whenever new data arrives. value_stored : verifies the condition whenever new data arrives, after it is saved to the database. value_change : works only with properties interface; verifies the condition whenever the received value is different from the previous one. value_change_applied : works only with properties interface; verifies the condition whenever the last value received is different from the last one previously received, after it is saved to the database. path_created : verifies the condition whenever a new path in a property interface is set or the first value is streamed on a datastream interface. path_removed : works only with properties interface; verifies the condition whenever a property path is unset. interface_name and interface_major represent, respectively, the Interface name and major version that uniquely identify an Astarte Interface. interface_name can be * to match all interfaces; in that case interface_major is ignored and all major numbers are matched. match_path is the path that will be used to match the condition. It can be /* to match all the paths of an interface. value_match_operator is the operator used to match the incoming data against a known value. It can be * to indicate that all values should be matched ( known_value is ignored in that case), otherwise it can be one of these operators: == , != , > , >= , < , <= , contains , not_contains . The match is always performed with <incoming_value> <operator> <known_value> . contains and not_contains can be used only with type string , binaryblob and with array types. known_value is the value used with value_match_operator to perform the comparison on the incoming value. It must have the same type as the incoming value, except in the contains and not_contains case.","ref":"060-triggers.html#conditions"},{"type":"extras","title":"Triggers - Actions","doc":"Actions are triggered by a matching condition. An Action defines how the event should be sent to the outer world (e.g. an http POST on a certain URL). In addition, most actions have a Payload, which carries the body of the event. HTTP Actions Payloads are most of the time represented as text, and Astarte provides several ways to generate them. By default Astarte generates a JSON payload with all the relevant information of the event. This is also the format used when delivering payloads in Astarte Channels. The format of the payload can be found in the Default action section. Astarte also provides a powerful templating mechanism for plain-text payloads based on top of Mustache . This is especially useful for integrating with third-party actors which require custom plain-text payloads. Keep in mind that Mustache templates are only able to produce text/plain payloads, not valid JSON. Default action This is the configuration object representing a minimal default action: { "http_url": "<http_url>", "http_method": "<method>" } The default action sends an HTTP request to the specified http_url using http_method method (e.g. POST ). Further options might be used, such as "http_static_headers", enabling auth to remote services: { "http_url": "<http_url>", "http_method": "<method>", "http_static_headers": { "Authorization": "Bearer <token>" }, "ignore_ssl_errors": <true|false> } The ignore_ssl_errors key is optional and defaults to false . If set to true , any SSL error encountered while doing the HTTP request will be ignored. This can be useful if the trigger must ignore self-signed or expired certificates. Please, beware that some http headers might be not allowed or reserved for http connection signaling. SimpleEvent payloads The payload delivered in a default HTTP action or in Astarte Channels is a JSON document with this format: { "timestamp": "<timestamp>", "device_id": "<device_id>", "trigger_name": "<trigger_name>", "event": <event> } timestamp is an UTC ISO 8601 timestamp (e.g. "2019-10-16T08:56:08.534377Z" ) representing when the event happened. device_id identifies the device that triggered the event. trigger_name identifies the trigger that fired the event. event is a JSON object that has a specific structure depending on the type of the simple_trigger that generated it. Event objects are detailed below. Additionally, the realm that originated the trigger is available in the request in the Astarte-Realm header. Event objects DeviceConnectedEvent { "type": "device_connected", "device_ip_address": "<device_ip_address>" } device_ip_address is the IP address of the device. DeviceDisconnectedEvent { "type": "device_disconnected" } DeviceErrorEvent { "type": "device_error", "error_name": "<error_name>", "metadata": { "<key>": "<value>" } } error_name is a string identifying the error. More details can be found in the device errors documentation metadata is a map with string key and string values that may contain additional information about the error. Some metadata ( e.g. binary payloads) might be encoded in base64 if they cannot be represented as string. In that case, the key is prepended with the base64_ prefix. IncomingDataEvent { "type": "incoming_data", "interface": "<interface>", "path": "<path>", "value": <value> } interface is the interface on which data was received. path is the path on which data was received. value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. ValueStoredEvent { "type": "value_stored", "interface": "<interface>", "path": "<path>", "value": <value> } interface is the interface on which data was received. path is the path on which data was received. value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. ValueChangeEvent { "type": "value_change", "interface": "<interface>", "path": "<path>", "old_value": <old_value>, "new_value": <new_value> } interface is the interface on which data was received. path is the path on which data was received. old_value is the previous value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. new_value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. ValueChangeAppliedEvent { "type": "value_change_applied", "interface": "<interface>", "path": "<path>", "old_value": <old_value>, "new_value": <new_value> } interface is the interface on which data was received. path is the path on which data was received. old_value is the previous value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. new_value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. PathCreatedEvent { "type": "path_created", "interface": "<interface>", "path": "<path>", "value": <value> } interface is the interface on which data was received. path is the path that has been created. value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. PathRemovedEvent { "type": "path_removed", "interface": "<interface>", "path": "<path>" } interface is the interface on which data was received. path is the path that has been removed. Mustache action This is the configuration object representing a Mustache action: { "http_url": "<http_url>", "http_method": "<http_method>", "template_type": "mustache", "template": "<template>" "ignore_ssl_errors": <true|false> } The Mustache action sends an HTTP request to the specified http_url , with the payload built with the Mustache template specified in template . If the template contains a key inside a double curly bracket (like so: {{ key }} ), it will be substituted with the actual value of that key in the event. The basic keys that can be use to populate the template are: {{ realm }} : the realm the trigger belongs to. {{ device_id }} : the device that originated the trigger. {{ trigger_name }} : the trigger name. {{ event_type }} : the type of the received event. The ignore_ssl_errors key is optional and defaults to false . If set to true , any SSL error encountered while doing the HTTP request will be ignored. This can be useful if the trigger must ignore self-signed or expired certificates. Moreover, depending on the event type, all keys that are contained in the events described in the previous section are available, always by wrapping them in {{ }} . The realm is also sent in the Astarte-Realm header. Example This is an example of a trigger that uses Mustache action. { "name": "example_mustache_trigger", "action": { "template_type": "mustache", "template": "Device {{ device_id }} just connected from IP {{ device_ip_address }}", "http_url": "https://example.com/my_mustache_hook", "http_method": "post" }, "simple_triggers": [ { "type": "device_trigger", "on": "device_connected", "device_id": "*" } ] } When a device is connected, the following request will be received by https://example.com/my_mustache_hook : POST /my_mustache_hook HTTP/1.1 Astarte-Realm: test Content-Length: 63 Content-Type: text/plain Host: example.com User-Agent: hackney/1.13.0 Device ydqBlFsGQ--xZ-_efQxuLw just connected from IP 172.18.0.1 Trigger Delivery Policies When an HTTP action is triggered, an event is sent to a specific URL. However, it is possible that the request is not successfully completed, e.g. the required resource is momentarily not available. Trigger Delivery Policies specify what to do in case of delivery errors and how to handle events which have not been successfully delivered. A Trigger can be linked to one (at most) Trigger Delivery Policy by specifying the name of the policy in the "policy" field. If no Trigger Delivery Policies are specified, Astarte will resort to the default (pre v1.1) behaviour, i.e. ignoring delivery errors. Refer to the relevant documentation for more information on Trigger Delivery Policies. AMQP 0-9-1 Actions AMQP 0-9-1 actions might be configured as an alternative to HTTP actions for advanced use cases. AMQP 0-9-1 is the right choice for a number of scenarios, including Astarte Flow integration, high performance ingestion, integration with an existing AMQP infrastructure, etc... Payloads are always encoded using protobuf , therefore if any other format is required Astarte Flow should be employed as a format converter. This is a minimal configuration object representing an AMQP 0-9-1 action: { "amqp_exchange": "astarte_events_<realm-name>_<exchange-suffix>", "amqp_routing_key": "my_routing_key", "amqp_message_expiration_ms": <expiration in milliseconds>, "amqp_message_persistent": <true when disk persistency is used> } It is possible to configure more advanced AMQP 0-9-1 actions: { "amqp_exchange": "astarte_events_myrealm_myexchange", "amqp_routing_key": "my routing key", "amqp_static_headers": { "key": "value" }, "amqp_message_expiration_ms": 10000, "amqp_message_priority": 0, "amqp_message_persistent": true, } Some Astarte specific restrictions apply: amqp_exchange must have astarte_events_<realm-name>_<any-allowed-string> format. amqp_routing_key must not contain { and } , which are reserved for future uses. For further details RabbitMQ documentation is suggested.","ref":"060-triggers.html#actions"},{"type":"extras","title":"Triggers - Relationship with Channels","doc":"Channels are part of AppEngine, and allow users to monitor device events through WebSockets, on top of Phoenix Channels . Under the hood, Channels use transient triggers to define which kind of events will flow through a specific room.","ref":"060-triggers.html#relationship-with-channels"},{"type":"extras","title":"Trigger Delivery Policies","doc":"When an HTTP action is triggered, an event is sent to a specific URL. However, it is possible that the request is not successfully completed, e.g. the required resource is momentarily not available. Trigger Delivery Policies (also referred to as policies or delivery policies ) specify what to do in case of delivery errors and how to handle events which have not been successfully delivered. In the same fashion as triggers and interfaces, policies are realm-scoped objects, too. While a trigger may specify at most one delivery policy, there is no upper bound on the number of triggers handled by the same policy. Under the hood, a policy is mapped to a queue on which the events to be delivered are stored. This queue is referred to as event queue and has a 1-to-1 relationship to the policy, i.e. there is one and only one event queue for each delivery policy. The size of the event queue is given in the policy specification. This provides an upper bound on the amount of space the event queue can fill. In the context of a policy, the strategy to handle delivery errors is describe by a list of error handlers , which specify how to react to an error (e.g. resend or discard the event) and what kind of error is to be handled. Each policy specifies at least one error handler. Every event in the event queue can be resent up to a number of times given in the policy specification.","ref":"062-trigger_delivery_policies.html"},{"type":"extras","title":"Trigger Delivery Policies - Trigger Delivery Policy Components","doc":"A Trigger Delivery Policy is composed of: Name: a string of at most 127 characters, unique for each realm. Names starting with " ? ", " ! ", or " @ " are reserved. This is a required component and uniquely identifies the policy in the realm. Error handlers: a non-empty list of handlers. Each handler acts on groups of delivery errors and describes the strategy Astarte should take when they occur. In pseudo-BNF syntax, an handler is described by the following grammar: handler :: = "{" "on" ":" "client_error" | "server_error" | "any_error" | [ < int > ] "," "strategy" ":" "discard" | "retry" "}" There are two possible strategies: either discard the event or retry. If the strategy is retry , events will be requeued in the event queue. The default retry strategy is discarding events. The on field specifies the group of HTTP errors the handler refers to: client errors (400-499), server errors (500-599), all errors (400-599), or a custom range of error codes (e.g. [418, 419, 420] ). Handlers can be at most 200 and must be not overlapping (i.e. there can not be two handlers which refer to at least one shared error code). Maximum capacity: the maximum size of the event queue which refers to the policy. If the number of messages to be retried exceeds the event queue size, older events will be discarded. Retry times: the maximum number of times an event in the event queue can be resent. A single policy does not allow to retry sending events for different amount of times, but different policies may have different numbers. This is optional, but required if the policy specifies at least one handler with retry strategy. Event TTL: in order to further lower the space requirement of the event queue, events may be equipped with a TTL which specifies the amount of seconds an event is retained in the event queue. When an event expires, it is discarded from the event queue, even if it has not been delivered. This is optional.","ref":"062-trigger_delivery_policies.html#trigger-delivery-policy-components"},{"type":"extras","title":"Trigger Delivery Policies - Known issues","doc":"At the moment, Trigger Delivery Policies in general do not provide a guarantee of in-order delivery of events. Note that, since previous Astarte versions (i.e. < 1.1) did not provide a retry mechanism for events, this change does not impact the expected behaviour if Trigger Delivery Policies are not used. If the Astarte Trigger Engine service is replicated, events could be delivered out of order, as data from event queues are delivered to consumers in a round-robin fashion. If the retry strategy is specified, in-order delivery cannot be guaranteed because a > 1 consumer prefetch count is being used. This allows for higher throughput at the cost of consistency. In the future, the user will be allowed to choose between having an higher number of events handled, but out of order, or ordered event handling at a lower rate.","ref":"062-trigger_delivery_policies.html#known-issues"},{"type":"extras","title":"Groups","doc":"Astarte supports creating groups of devices in a realm. Groups are currently useful mainly to provide access control, combining Astarte's path based authorization with the fact that devices can be queried with a group URL. This makes it possible to emit tokens allowing a user to operate only on devices that belong to a specific group. Groups can be managed using astartectl or using AppEngine API . See the Managing Groups page in the User Guide for some usage examples. Keep in mind a group is existing as long as there's at least one device in it. Once the last device is removed from the group, the group does not exist anymore, since groups are a tag (or label) of devices.","ref":"065-groups.html"},{"type":"extras","title":"Authentication and Authorization","doc":"Authentication and authorization are crucial, as Astarte likely holds sensitive resources and is capable to send mass commands to a device fleet. First of all: when talking about auth in Astarte, we are talking about anything which isn't a Device - those are Authenticated through Pairing and Authorized by their Transport (which uses Pairing for the Authentication policies). Astarte's authentication/authorization stage identifies the principal through a token (with JWT as the first class citizen), which is the only currency the platform supports.","ref":"070-auth.html"},{"type":"extras","title":"Authentication and Authorization - Authentication Realms","doc":"In Astarte, realms are logically separated and have completely different data partitions. This is also true in terms of authentication, as caller is always authenticated on a per-realm basis. As such, an authentication realm matches 1:1 an Astarte realm. Superadmin APIs, such as housekeeping, are part of a different authentication realm which is defined upon cluster setup.","ref":"070-auth.html#authentication-realms"},{"type":"extras","title":"Authentication and Authorization - Authentication in Astarte","doc":"Astarte, by design, does not have a concept of per-user authentication built in. The definition of an authentication realm is a mean to verify a token's validity, that is most likely a public key. This makes integrating Astarte with 3rd party authentication/authorization frameworks and SSOs extremely easy, as the whole logic for addressing user management is managed out of the cluster by a dedicated party. Depending on one's use case, it is possible to use either a very simple, dedicated OAuth server for each realm, or a full fledged SSO such as Keycloak which matches its authentication realms to Astarte's realms. Especially if you are aiming at the latter, make sure to read the advised best practices for authentication afterwards.","ref":"070-auth.html#authentication-in-astarte"},{"type":"extras","title":"Authentication and Authorization - Authorization","doc":"Currently, Astarte supports a URL-based authorization for the API. Given that Astarte's data access APIs match the devices' topology like a tree, declaring the authorization in terms of path allow-listing gives enough flexibility to give each user the correct permissions without limitations. As said, Astarte does not have the concept of user, and neither has a durable storage which tracks permissions. As such, it expects the authorization information to be inside the token, which is the only entity Astarte can trust - given it has been verified and authenticated through its signature. Paths are given in form of a set of Perl-like Regular Expressions , and on a per-API basis. This means that each API endpoint (app, realm, etc...) has its own regular expression which defines what the user can do. Moreover, each HTTP verb in an API endpoint (e.g.: GET, POST, PUT, DELETE) can have its own regular expression, to fine-grain permissions on each path. Note: given Astarte's interface are either read only or write only, HTTP verb fine-graining in AppEngine API is mostly useful for preventing a user from deleting a consumer Datastream even though it has write access to it. Most of the time, using only a single regular expression with no verb fine-graining works. Examples of valid regular expressions on AppEngine API are: POST::devices/.*/interfaces/com\\\\.my\\\\.interface/.* : Allows to set individual values on the com.my.interface interface on any individual device in the realm. .*::.*/interfaces/com\\\\.my\\\\.monitoring\\\\.interface.* : Allows to get/set/delete either the aggregate or the individual values of the com.my.monitoring.interface interface on any device or device aggregation in the realm. .*::devices/j0zbvbQp9ZNnanwvh4uOCw.* : Allows every operation on device j0zbvbQp9ZNnanwvh4uOCw GET::devices/[a-zA-Z0-9-_]* : Allows to get every individual device's status, but denies access to any additional information/operation on them. Examples of valid regular expressions on Realm Management API are: POST::interfaces\\/.* : Allows installing new interfaces in the realm. GET::interfaces\\/.* : Allows inspecting every interface in the realm. PUT::interfaces\\/.*\\/0 : Allows updating all draft interfaces in the realm. Other valid examples are: .*::.* : Allows any operation on the given API. Both verb and path regular expressions are implicitly delimited by adding ^ before and $ after the regular expression string. For example, if you use GET::interfaces as regular expression in Realm Management API, the verb will be matched against ^GET$ and the path will be matched against ^interfaces$ . This way the only operation allowed will be listing all the interfaces, while all operation on interfaces/ subpaths will be denied. Token claims and formats Authorization regular expressions have to be contained in the token's claims. Only the JWT case will be considered given it is the primary currency Astarte supports. Every claim is an array of regular expressions, which act as a logical OR. A similar behavior could be of course achieved (and might be more efficient) with a singular regular expression, but for the sake of readability and simplicity it is allowed nonetheless. Of course, keeping the authorization claims simple and pragmatic helps in terms of performance. Supported token claims are: a_aea : Defines the regular expressions for AppEngine API a_rma : Defines the regular expressions for Realm Management API a_ha : Defines the regular expressions for Housekeeping API a_pa : Defines the regular expressions for Pairing API a_ch : Defines the regular expressions for Channels Of course, claims are considered only after a successful token verification. This means that the claim will be processed only if the caller is authenticated against the correct authentication realm - this is especially the case for what concerns Housekeeping, which has a dedicated Authentication realm not tied to any Astarte realms. An example of a valid token claim is: { "a_aea": ["GET::devices/[a-zA-Z0-9-_]*", ".*::.*/interfaces/com\\\\.my\\\\.monitoring\\\\.interface.*", ".*::devices/j0zbvbQp9ZNnanwvh4uOCw.*"], "a_rma": ["GET::.*"] } Which allows very specific permissions on AppEngine API, and a "read all" on Realm Management API. The client by default has no permission to do anything: as such, if a token is missing a claim it is simply assumed that the client isn't authorized to access that specific API. However, keeping in mind that Astarte has no concept of User, it is also true that your authentication backend might choose to emit a different token with only a subset of its real permissions to keep claims and regular expressions as pragmatic as possible. See Granular Claims in Best Practices for more details on this. Natively supported tokens Astarte supports only JWT natively, which has to be signed using one of the following algorithms: ES256 ES384 ES512 PS256 PS384 PS512 RS256 RS384 RS512","ref":"070-auth.html#authorization"},{"type":"extras","title":"Authentication and Authorization - Authorization for REST APIs","doc":"Valid tokens can be used for calling into Astarte's public APIs. Depending on which token mechanism is used, the HTTP call must adhere to some requirements. JWT Every API call must have an Authorization: Bearer <token> header. Not providing the token or providing a token which can't be validated for the authentication realm of the context results in a 401 reply.","ref":"070-auth.html#authorization-for-rest-apis"},{"type":"extras","title":"Authentication and Authorization - Authorization for Channels","doc":"A valid token should be supplied when opening the WebSocket, in the very same fashion to what happens with REST APIs. However, the claims in this token will support different verbs compared to the REST APIs, namely JOIN and WATCH . These have very specific meanings and are well explained in Channels' User Guide . The behavior and supported tokens are equivalent to REST APIs.","ref":"070-auth.html#authorization-for-channels"},{"type":"extras","title":"Authentication and Authorization - Supported integrations","doc":"Astarte, by default, is extremely easy to configure assuming your chosen SSO is capable of issuing JWT, as it is currently the only natively supported authentication currency. However, virtually any token-based system can be used as an auth framework for Astarte. The main purpose of Astarte's design, however, is to keep things simple for everyone. Putting up a full-fledged SSO dedicated to Astarte is beyond the scope of this documentation, and we favor the use case where an existing SSO infrastructure is integrated with Astarte, rather than built ad-hoc. For simple use cases and instant satisfaction, it is strongly advised to use a simpler solution, such as a dedicated OAuth server. Almost all popular languages and frameworks provide great projects which can spin up an OAuth2 server + user management in a matter of hours, from Elixir/Phoenix to Java/Spring to Go . Astarte's Enterprise Distribution includes other add-ons, such as automation and configuration for popular SSOs.","ref":"070-auth.html#supported-integrations"},{"type":"extras","title":"Authentication and Authorization - Best practices","doc":"Due to the nature of tokens, applications and SSOs must take care of emission and storage of the token themselves. In most production cases, Astarte will be part of a larger SSO infrastructure being one of the clients (this is especially true for OAuth). Among best practices, emitting short-lived tokens should always be considered, but depending on the use case, the authentication pipeline can be further tuned to address a number of potential issues. Token exchange OAuth, like other protocols supports the concept of a Token Exchange . Consider a web dashboard with a logged in user. The user will, most likely, have a token which is used by its frontend to call upon the backend/APIs of the web dashboard. For the sake of simplicity, one might include in this token the adequate claims to give the user access to Astarte, but this might not be desirable for a number of reasons outlined above. Token exchange, if supported by your SSO, provides a great way to work around this: whenever the backend or the frontend requires access to Astarte, it can invoke the token exchange mechanism of the SSO to generate a short lived token for the API call from the original authentication, which can then be used even as a single shot access mechanism. Granular claims The token exchange approach can be efficiently paired with a mechanism of granular claims. Consider the use case above, and let's assume the frontend needs direct, frequent access to Astarte's APIs. Exchanging tokens too many times might put a burden on the SSO and might become impractical. However, Astarte decouples entirely authentication and authorization - that means, if two subsequent (valid) tokens which represent the same identity have substantially different claims, it doesn't care. This is intentional, as it allows for a much more efficient pattern: the token used by an hypotetical frontend can have a subset of the user's claims - for example, allowing him to read data from its devices, whereas token exchange can be used whenever more specific operations should be performed - for example, sending some commands or data to devices. This also addresses the objection that regular expressions can grow big or quite complicated in case users need a large number of very granular permissions. In such complex cases, the SSO can be tuned to give out only a subset of claims depending on the user's operation. Token revocation Token revocation isn't natively supported in Astarte for two reason: the first one is performance, as keeping a revocation list is expensive in many regards. The second is the fact that the revocation list is, most of the time, SSO specific, and a dedicated SSO integration would be required. Rather than token revocation, a better practice is to make sure every emitted token has a short enough lifetime. However, it is possible to extend Astarte's authorization stage to support revocation, even though there are no plans to provide upstream support for that. Changing a Realm's validation mean Over the lifetime of a cluster, it might be necessary to change a realm's validation mean for the most diverse reasons. By design, validation means are meant to be long lived, and changing them is supposed to be an extraordinary operation. Astarte supports only one validation mean at a time. When the validation mean is changed, all tokens emitted which could be validated with the previous mean become invalid. It is also possible that there might be a delay between the request of a validation mean change and its actuation. This means during this grace period tokens will be validated against the previously configured mean. As such, it is advised to treat a validation mean change as a maintenance operation for the realm. More details can be found in the Administrator Guide.","ref":"070-auth.html#best-practices"},{"type":"extras","title":"Astarte MQTT v1 Protocol","doc":"Astarte MQTT v1 Protocol allows communication between Astarte and devices. It is the first protocol that has been implemented in Astarte, and it exploits every feature provided by Astarte itself. Astarte MQTT v1 doesn't mandate a specific Transport Credentials format: the broker must handle Authentication, Authorization and Pairing integration the way it sees fit. Astarte MQTT v1 is implemented by Astarte's Reference Transport, Astarte/VerneMQ - a client wishing to interact with it must implement MQTT v3.1.1 and all needed features for Pairing to work. MQTT doesn't mandate the data serialization format, so any application might implement its own format. Data serialization might be a tricky task and protocols might be hard to design, Astarte MQTT takes care of this and provides a higher level protocol which abstracts this detail from the end user. Astarte MQTT v1 Protocol builds upon MQTT v3.1.1 itself, BSON (Binary JSON, version 1.1) serialized payloads and on optional zlib deflate. All communications are ordered and asynchronous. A protocol reference implementation is provided with an Astarte SDK, however developers might implement it from scratch using 3rd party libraries with their favourite languages: all formats and protocols described here are open and well documented. Last but not least Astarte doesn't mandate this protocol, and a different one can be used with a different transport.","ref":"080-mqtt-v1-protocol.html"},{"type":"extras","title":"Astarte MQTT v1 Protocol - MQTT Topics Overview","doc":"Astarte MQTT v1 Protocol relies on few well known reserved topics. Topic Purpose Published By QoS Payload Format <realm name>/<device id> Introspection Device 2 ASCII plain text, ':' and ';' delimited <realm name>/<device id>/control/emptyCache Empty Cache Device 2 ASCII plain text (always "1") <realm name>/<device id>/control/consumer/properties Purge Properties Astarte 2 deflated plain text <realm name>/<device id>/control/producer/properties Purge Properties Device 2 deflated plain text <realm name>/<device id>/<interface name>/<path> Publish Data Both 0, 1, 2 BSON (or empty) For clarity reasons all <realm name>/<device id> prefixes will be omitted on the following paragraphs, those topics will be called device topics. Topics are not bidirectional, devices must not publish data for server owned topics and viceversa, onwership is explicitly stated in interfaces files.","ref":"080-mqtt-v1-protocol.html#mqtt-topics-overview"},{"type":"extras","title":"Astarte MQTT v1 Protocol - BSON","doc":"BSON allows saving precious bytes compared to JSON, while offering the advantages of a schema-less protocol. Consider, for example, a simple value and timestamp payload. The encoded JSON version, {"v":25.367812,"t":1537346756844} counts 33 bytes. The hexdump of the same message encoded with BSON is: 0000000 1 b 00 00 00 09 74 00 ec e0 01 f1 65 01 00 00 01 0000020 76 00 8 c 13 5 f ed 28 5 e 39 40 00 that fits just in 27 bytes. BSON format BSON is a really simple binary format, breaking down the previous example is very easy thanks to BSON simplicity: the first 4 bytes ( 1b 00 00 00 ) are the document size header, follows the timestamp marker ( 09 ), the timestamp key name ( 74 00 , that is "t"), the timestamp value ( 5f 48 06 f1 65 01 00 00 as int64), the double value marker ( 01 ), the value key name ( 76 00 , that is "v"), the actual value ( cd cc cc cc cc 4c 39 40 as 64-bit IEEE 754-2008 floating point) and the end of document marker ( 00 ). Astarte payload standard fields Key Type Mandatory Description v Any Astarte type Yes The value being sent (both properties and datastream) t UTC datetime No Explicit timestamp, if present (optional, datastream only) Astarte data types to BSON types Astarte Data Type BSON Type Size in Bytes double double (0x01) 8 integer int32 (0x10) 4 boolean boolean (0x08) 1 longinteger int64 (0x12) 8 string UTF-8 string (0x02) >= length (encoding dependent) binaryblob binary (0x05) length datetime UTC datetime (0x09) 8 doublearray Array (0x04) (8 + keysize) * count integerarray Array (0x04) (4 + keysize) * count booleanarray Array (0x04) (1 + keysize) * count longintegerarray Array (0x04) (1 + keysize) * count stringarray Array (0x04) depends on count, length, keys length and encoding binaryblobarray Array (0x04) depends on count, keys length and length integer and long integer are signed integer values, double must be a valid number ( +inf , NaN , etc... are not supported), variable data types might be subject to size limitations and object aggregations are encoded as embedded documents.","ref":"080-mqtt-v1-protocol.html#bson"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Connection and Disconnection","doc":"A device is not required to publish any additional connection or disconnection messages, the MQTT broker will automatically keep track of these events and relay them to Astarte. When connecting, before publishing any data message, a device should check MQTT session present flag. When the MQTT session present flag is true no further actions are required, when false the device should take following actions: Publish its introspection Publish an empty cache message Publish all of its existing and set properties on all its property interfaces If a device is unable to inspect session present all previous actions must be taken at every reconnection.","ref":"080-mqtt-v1-protocol.html#connection-and-disconnection"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Introspection","doc":"Each device must declare the set of supported interfaces and their version. Astarte needs to know which interfaces the device advertises before processing any further data publish. This message in Astarte jargon is called introspection and it's performed by publishing on the device root topic the list of interfaces that are installed on the device. Introspection payload is a simple plain text string, and it has the following format (in BNF like syntax): introspection :: = introspection_list introspection_list :: = introspection_entry ";" introspection_list | introspection_entry introspection_entry :: = interface_name ":" interface_major_version ":" interface_minor_version The following example is a valid introspection payload: com . example . MyInterface : 1 : 0 ; org . example . DraftInterface : 0 : 3","ref":"080-mqtt-v1-protocol.html#introspection"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Empty Cache","doc":"Astarte MQTT v1 strives to save bandwidth upon reconnections, to make sure even frequent reconnections don't affect bandwidth consumption. As such, upon connecting and if MQTT advertises a session present, both sides assume that data flow is ordered and consistent. However, there might be cases where this guarantee isn't respected by the device for a number of reasons (e.g.: new device, factory reset, cache lost...). In this case, a device might declare that it has no confidence about its status and its known properties, and can request to resynchronise entirely with Astarte. In Astarte jargon this message is called empty cache and it is performed by publising "1" on the device /control/emptyCache topic. After an empty cache message properties might be purged and Astarte might publish all the server owned properties again.","ref":"080-mqtt-v1-protocol.html#empty-cache"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Session Present","doc":"In the very same fashion as the device, Astarte (or the broker) might be inconsistent with a Device's known status and its known properties. Although unlikely, as Astarte should always keep knowledge about remote device status, this might happen, for example, after an internal error. Astarte performs this task by telling the broker to disconnect the device and clear its session. After this, when the device will attempt reconnection, session present will be false. After a clean session properties might be purged.","ref":"080-mqtt-v1-protocol.html#session-present"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Purge Properties","doc":"Either a Device or Astarte may tell the remote host the set properties list. Any property that is not part of the list will be deleted from any cache or database. This task is called purge properties in Astarte jargon, and it is performed by publishing a the list of known set properties to /control/consumer/properties or /control/producer/properties . Purge Properties payload is a zlib deflated plain text, with an additional 4 bytes header. The additional 4 bytes header is the size of the uncompressed payload, encoded as big endian uint32. The following example is a payload compressed using zlib default compression, with the additional 4 bytes header: 0000000 00 00 00 46 78 9 c 4 b ce cf d5 4 b ad 48 cc 2 d c8 0000020 49 d5 f3 ad f4 cc 2 b 49 2 d 4 a 4 b 4 c 4 e d5 2 f ce 0000040 cf 4 d d5 2 f 48 2 c c9 b0 ce 2 f 4 a 87 ab 70 29 4 a 0000060 4 c 2 b 41 28 ca 2 f c9 48 2 d 0 a 00 2 a 02 00 b2 0 c 0000100 1 a c9 The uncompressed plain text payload has the following format (in BNF like syntax): properties :: = properties_list properties_list :: = properties_entry ";" properties_list | properties_entry properties_entry :: = interface_name path The following example is the inflated previous payload: com . example . MyInterface / some / path ; org . example . DraftInterface / otherPath This protocol feature is fundamental when a device has any interface with an allow_unset mapping, purge properties allows to correct any error due to unhandled unset messages.","ref":"080-mqtt-v1-protocol.html#purge-properties"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Publishing Data","doc":"Either Astarte or a device might publish new data on a interface/endpoint specific topic. The topic is built using /<interface name>/<path> schema, and it is used regardless of the type of interface or mapping being used. Also / path is a valid path for object aggregated interfaces. The following device topics are valid: /com.example.MyInterface/some/path /org.example.DraftInterface/otherPath /com.example.astarte.ObjectAggregatedInterface/ Data messages QoS is chosen according to mapping settings, such as reliability. Properties are always published using QoS 2. Interface Type Reliability QoS properties always unique 2 datastream unreliable 0 datastream guaranteed 1 datastream unique 2 Payload Format Payload format might change according to the message type. Payloads are always BSON encoded, except for unset messages that are empty. Property Message Property messages have a "v" key (which means value). Valid examples are: {"v": "string property value"} {"v": 10} {"v": true} Previous payloads are BSON encoded as the following hex dumps: 0000000 22 00 00 00 02 76 00 16 00 00 00 73 74 72 69 6 e 0000020 67 20 70 72 6 f 70 65 72 74 79 20 76 61 6 c 75 65 0000040 00 00 0000000 0 c 00 00 00 10 76 00 0 a 00 00 00 00 0000000 09 00 00 00 08 76 00 01 00 Property messages order must be preserved and they must be consumed in order. The same property with the same value can be sent several times, this behavior is allowed but discouraged: it's up to the device to avoid useless messages. A device must also make sure to publish all the properties that have been changed while the device was offline. Unset Property Message Properties can be unset with an unset message. An unset message is just an empty 0 bytes payload. Datastream Message (individual aggregation) Datastream messages for interfaces with individual aggregation have a "v" key and an optional "t" key (which means timestamp). Valid examples are: {"v": false} {"v": 16.73} {"v": 16.73, "t": 1537449422890} Timestamps are UTC timestamps (BSON 0x09 type), when not provided reception timestamp is used. Previous payloads are BSON encoded as the following hex dumps: 0000000 09 00 00 00 08 76 00 00 00 0000000 10 00 00 00 01 76 00 7 b 14 ae 47 e1 ba 30 40 00 0000000 1 b 00 00 00 09 74 00 2 a 70 20 f7 65 01 00 00 01 0000020 76 00 7 b 14 ae 47 e1 ba 30 40 00 Datastream Message (object aggregation) Datastream messages for interfaces with object aggregation support every Astarte payload standard field (such as "t"), but in this case value is a BSON subdocument, in which each key represent a mapping of the aggregation. Valid examples are: {"v": {"temp": 25.3123, "hum": 67.112}} {"v": {"temp": 25.3123, "hum": 67.112}, "t": 1537452514811} Timestamps are UTC timestamps (BSON 0x09 type), when not provided reception timestamp is used. Previous payloads are BSON encoded as following hex dumps: 0000000 28 00 00 00 03 76 00 20 00 00 00 01 68 75 6 d 00 0000020 ba 49 0 c 02 2 b c7 50 40 01 74 65 6 d 70 00 72 8 a 0000040 8 e e4 f2 4 f 39 40 00 00 0000000 33 00 00 00 09 74 00 fb 9 d 4 f f7 65 01 00 00 03 0000020 76 00 20 00 00 00 01 68 75 6 d 00 ba 49 0 c 02 2 b 0000040 c7 50 40 01 74 65 6 d 70 00 72 8 a 8 e e4 f2 4 f 39 0000060 40 00 00","ref":"080-mqtt-v1-protocol.html#publishing-data"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Minimal Protocol","doc":"A device might implement a subset of this protocol if needed. /control/consumer/properties , /control/producer/properties and /emptyCache might be ignored or not implemented if a device has no property interfaces. A further simplification might remove any requirement for any introspection message when previously provisioned, but this feature is not supported out of the box.","ref":"080-mqtt-v1-protocol.html#minimal-protocol"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Error Handling","doc":"A device might be forcefully disconnected due to any kind of error. Devices should wait a random amount of time before trying to connect again to the broker. session present might be also set to false to ensure a clean and consistent state (in that case messages such as introspection and empty cache should published as previously described). Malformed or unexpected messages are discarded and further actions might be taken.","ref":"080-mqtt-v1-protocol.html#error-handling"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Authentication","doc":"In Astarte, every Transport orchestrates its credentials through Pairing. Astarte/VerneMQ authenticates devices using Mutual SSL Autentication - as such, devices use SSL certificates emitted through Pairing API to authenticate against the broker. To achieve this, the device must ensure it is capable of performing http(s) calls to Pairing API to obtain its certificates, performing SSL/X509 operations and connecting to the MQTT Broker through the use of SSL certificates.","ref":"080-mqtt-v1-protocol.html#authentication"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Authorization","doc":"Device can only publish and subscribe to its device topic ( <realm name>/<device id> ) and its subtopics. The broker will deny any publish or subscribe outside that hierarchy.","ref":"080-mqtt-v1-protocol.html#authorization"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Connecting to the Broker","doc":"In the same fashion as Authentication, Pairing provides the client with information about how to connect to the MQTT broker. When invoking relevant Pairing API's method to gather information about available transports for a device, if Astarte advertises Astarte MQTT v1, a similar reply will be returned: { "data": { "version": "<version string>", "status": "<status string>", "protocols": { "astarte_mqtt_v1": { "broker_url": "mqtts://broker.astarte.example.com:8883" } } } }","ref":"080-mqtt-v1-protocol.html#connecting-to-the-broker"},{"type":"extras","title":"Astarte Database","doc":"Astarte leverages Cassandra to store all of its data, including data ingested from devices (which might scale to insane amounts). Cassandra offers scalability and high availability with good performances . Cassandra offers linear scalability and can span from really small clusters to hundreds of nodes, without compromising on reliability. ScyllaDB >= 3.3 is also supported as a drop-in replacement when a performance boost is needed. Cassandra is also the ideal storage for large-scale data processing with Apache Spark . Astarte is multi-tenant by design, with each tenant mapping to an Astarte Realm. Each Realm has its own Cassandra keyspace, which can be tuned according to Realm-specific needs (e.g.: Realms might have different replication levels). For this reason, in the scope of this section, realm and keyspace can be used as synonyms, except for the astarte keyspace.","ref":"090-database.html"},{"type":"extras","title":"Astarte Database - Schema and Keyspace Creation","doc":"Astarte automatically takes care of keyspaces, tables creation and intra-version migrations (those tasks are performed by astarte_housekeeping or astarte_realm_management , depending on the context). The following documentation is just a reference about Astarte's internal statements, and is related to the release series referenced by the documentation. Astarte Keyspace Astarte needs an astarte keyspace to store its own data. astarte keyspace and tables are created with following CQL statements: CREATE KEYSPACE astarte WITH replication = {'class': 'SimpleStrategy', 'replication_factor': <replication factor>} AND durable_writes = true; CREATE TABLE astarte.realms ( realm_name varchar, PRIMARY KEY (realm_name) ); CREATE TABLE astarte.kv_store ( group text, key text, value blob, PRIMARY KEY (group, key) ) Realm Creation Each realm needs several tables to store data for all the functionalities. Realm tables can be grouped in the following functionalities: Configuration & key-value store Interfaces schema Device management Groups management Triggers storage Data storage Some data storage tables might be created when required, whereas all other tables are created when a keyspace is created, using the following statements: CREATE KEYSPACE <realm name> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': :replication_factor} AND durable_writes = true; CREATE TABLE <realm name>.kv_store ( group varchar, key varchar, value blob, PRIMARY KEY ((group), key) ); CREATE TABLE <realm name>.names ( object_name varchar, object_type int, object_uuid uuid, PRIMARY KEY ((object_name), object_type) ); CREATE TABLE <realm_name>.devices ( device_id uuid, aliases map<ascii, varchar>, introspection map<ascii, int>, introspection_minor map<ascii, int>, old_introspection map<frozen<tuple<ascii, int>>, int>, protocol_revision int, first_registration timestamp, credentials_secret ascii, inhibit_credentials_request boolean, cert_serial ascii, cert_aki ascii, first_credentials_request timestamp, last_connection timestamp, last_disconnection timestamp, connected boolean, pending_empty_cache boolean, total_received_msgs bigint, total_received_bytes bigint, exchanged_bytes_by_interface map<frozen<tuple<ascii, int>>, bigint>, exchanged_msgs_by_interface map<frozen<tuple<ascii, int>>, bigint>, last_credentials_request_ip inet, last_seen_ip inet, attributes map<varchar, varchar>, groups map<text, timeuuid>, PRIMARY KEY (device_id) ); CREATE TABLE <realm name>.grouped_devices ( group_name varchar, insertion_uuid timeuuid, device_id uuid, PRIMARY KEY ((group_name), insertion_uuid, device_id) ); CREATE TABLE <realm name>.endpoints ( interface_id uuid, endpoint_id uuid, interface_name ascii, interface_major_version int, interface_minor_version int, interface_type int, endpoint ascii, value_type int, reliability int, retention int, expiry int, database_retention_ttl int, database_retention_policy int, allow_unset boolean, explicit_timestamp boolean, description text, doc text, PRIMARY KEY ((interface_id), endpoint_id) ); CREATE TABLE <realm name>.interfaces ( name ascii, major_version int, minor_version int, interface_id uuid, storage_type int, storage ascii, type int, ownership int, aggregation int, automaton_transitions blob, automaton_accepting_states blob, description text, doc text, PRIMARY KEY (name, major_version) ); CREATE TABLE <realm name>.simple_triggers ( object_id uuid, object_type int, parent_trigger_id uuid, simple_trigger_id uuid, trigger_data blob, trigger_target blob, PRIMARY KEY ((object_id, object_type), parent_trigger_id, simple_trigger_id) ); CREATE TABLE <realm name>.individual_datastreams ( device_id uuid, interface_id uuid, endpoint_id uuid, path text, value_timestamp timestamp, reception_timestamp timestamp, reception_timestamp_submillis smallint, binaryblob_value blob, binaryblobarray_value list<blob>, boolean_value boolean, booleanarray_value list<boolean>, datetime_value timestamp, datetimearray_value list<timestamp>, double_value double, doublearray_value list<double>, integer_value int, integerarray_value list<int>, longinteger_value bigint, longintegerarray_value list<bigint>, string_value text, stringarray_value list<text>, PRIMARY KEY ((device_id, interface_id, endpoint_id, path), value_timestamp, reception_timestamp, reception_timestamp_submillis) ) CREATE TABLE <realm name>.individual_properties ( device_id uuid, interface_id uuid, endpoint_id uuid, path text, reception_timestamp timestamp, reception_timestamp_submillis smallint, double_value double, integer_value int, boolean_value boolean, longinteger_value bigint, string_value text, binaryblob_value blob, datetime_value timestamp, doublearray_value list<double>, integerarray_value list<int>, booleanarray_value list<boolean>, longintegerarray_value list<bigint>, stringarray_value list<text>, binaryblobarray_value list<blob>, datetimearray_value list<timestamp>, PRIMARY KEY((device_id, interface_id), endpoint_id, path) ); The following table is generated upon datastream interface creation for keeping all data sent to Astarte through the interface. The table name is derived from lower case interface name where . and - have been replaced by _ and "" (empty string), then the major version is appended with a _v prefix. For example, com.Astarte.TestInterface version 1 becomes com_astarte_testinterface_v1 . If, after all the required transformations, the resulting name is too long (>45 chars), it will be encoded and truncated. CREATE TABLE <interpolated interface name>_v<major_version> ( device_id uuid, path text, reception_timestamp timestamp, reception_timestamp_submillis smallint, v_<property_mapping> <property_type> v_<property_mapping> <property_type> ... PRIMARY KEY ((device_id, path), reception_timestamp, reception_timestamp_submillis) )","ref":"090-database.html#schema-and-keyspace-creation"},{"type":"extras","title":"Astarte Database - Tables","doc":"Devices The devices table stores the list of all the devices for a certain realm and all their metadata, including the introspection, the device status and credentials information. Column Name Column Type Description device_id uuid Device unique 128 bits ID. aliases map<ascii, varchar> Alias purpose and alias map. introspection map<ascii, int> Device interface name to interface major version map based on most recent device introspection. introspection_minor map<ascii, int> Device interface name to interface minor version map based on most recent device introspection. old_introspection map<frozen<tuple<ascii, int>>, int> All previous device interfaces. This column is used to keep track of all interfaces that have been used and might still have some recorded data. The column maps interface (name, major) to minor. protocol_revision int Spoken Astarte MQTT v1 protocol revision. first_registration timestamp First registration attempt timestamp. credentials_secret ascii The bcrypt hash of the credential secret, that the device uses to obtain new credentials. inhibit_credentials_request boolean Ban device credentials renewal, device will be able to connect to the transport up to the credential expiry. cert_serial ascii Device certificate serial used by the CA. cert_aki ascii Device certificate Authority Key Identifier. first_credentials_request timestamp First credentials request timestamp. last_connection timestamp Most recent device connection event timestamp. last_disconnection timestamp Most recent device disconnection event timestamp. connected boolean True if the device is connected, otherwise is false. pending_empty_cache boolean Device is in an unclean state and an empty cache message is being waited. total_received_msgs bigint Count of received messages since the device registration. total_received_bytes bigint Amount of received messages bytes since the device registration. exchanged_msgs_by_interface bigint Count of exchanged messages since the device registration. exchanged_bytes_by_interface bigint Amount of exchanged messages bytes since the device registration. last_credentials_request_ip inet Device IP address used during the last credential request. last_seen_ip inet Most recent device IP address. attributes map<varchar, varchar> Device attributes. It can contain arbitrary string key and values associated with the device. groups map<text, timeuuid> Groups which the device belongs to, the key is the group name, and the value is its insertion timeuuid, which is used as part of the key on grouped_devices table. Endpoints The endpoints table stores the list of all endpoints of all interfaces for realm, with all the data needed to define an endpoint, such as retention, realiability, value type and so on. Column Name Column Type Description interface_id uuid Interface unique 128 bits ID. endpoint_id uuid Endpoint unique 128 bits ID. interface_name ascii Human-readable name for interface. interface_major_version int Interface major version related to the endpoint. interface_minor_version int Interface minor version related to the endpoint. interface_type int Interface type identifier related to the endpoint. endpoint ascii Human-readable endpoint string. value_type int Value type identifier related to the endpoint. reliability int Reliability identifier related to the endpoint. retention int Retention identifier related to the endpoint. expiry int Expiry identifier related to the endpoint. database_retention_ttl int Milliseconds before data deletion. database_retention_policy int Database_retention_policy identifier related to the endpoint. allow_unset boolean Enable or disable possibility of setting value to null. explicit_timestamp boolean Set or unset explicit timestamp. description text Description of endpoint. doc text Documentation for endpoint. Interfaces The interfaces table stores the list of all interfaces for realm, with all the data needed to define an endpoint, such as retention, realiability, value type and so on. Column Name Column Type Description interface_id uuid Interface unique 128 bits ID. name ascii Human-readable name for interface. major_version int Interface major version related to the endpoint. minor_version int Interface minor version related to the endpoint. storage_type int Storage type identifier related to the endpoint. storage ascii Interface storage. type int Identifies the type of this Interface. Currently two types are supported: datastream and properties. ownership int Identifies the quality of the interface. Interfaces are meant to be unidirectional, and this property defines who's sending or receiving data. aggregation int Identifies the aggregation of the mappings of the interface. automaton_transitions blob Automaton internal field. automaton_accepting_states blob Automaton internal field. description text Description of interface. doc text Documentation of interface.","ref":"090-database.html#tables"},{"type":"extras","title":"Astarte Database - Schema changes","doc":"This section describes the schema changes happening between different Astarte Versions. They are divided between Astarte Keyspace (changes that affect the Astarte Keyspace), and Realm Keyspaces (changes that affect all realm keyspaces). Every change is followed by the CQL statement that produces the change. From v0.10 to v0.11 Astarte Keyspace v0.11 Changes Remove astarte_schema table DROP TABLE astarte_schema; Remove replication_factor column from the realms table ALTER TABLE realms DROP replication_factor; Realm Keyspaces v0.11 Changes Add grouped_devices table CREATE TABLE <realm_name>.grouped_devices ( group_name varchar, insertion_uuid timeuuid, device_id uuid, PRIMARY KEY ((group_name), insertion_uuid, device_id) ); Add groups , exchanged_bytes_by_interface and exchanged_msgs_by_interface columns to the devices table ALTER TABLE <realm_name>.devices ADD (groups map<text, timeuuid>, exchanged_bytes_by_interface map<frozen<tuple<ascii, int>>, bigint>, exchanged_msgs_by_interface map<frozen<tuple<ascii, int>>, bigint>); Add database_retention_ttl and database_retention_policy columns to the endpoints table ALTER TABLE <realm_name>.endpoints ADD ( database_retention_ttl int, database_retention_policy int ); From v0.11 to v1.0.0-beta.1 Realm Keyspace v1.0.0-beta.1 Changes The connected field of the devices table is now saved with a TTL, so it automatically expires if it doesn't gets refreshed by the hearbeat sent by the broker. This behaviour was added to avoid stale connected devices if they disconnect while the broker is down. Add metadata column to the devices table ALTER TABLE devices ADD ( metadata map<varchar, varchar> ); From v1.0-beta.1 to v1.0.0 Realm Keyspace v1.0.0 Changes Rename the metadata to attributes in the devices table Warning : migrating data from the metadata column to the attributes one is possible but is out of scope of this guide since this change happened between development releases. The procedure below just creates the new column and then deletes the old one without migrating data . You're free to implement a migration procedure between the two steps. ALTER TABLE devices ADD ( attributes map<varchar, varchar> ); ALTER TABLE devices DROP metadata;","ref":"090-database.html#schema-changes"},{"type":"extras","title":"Introduction","doc":"Astarte is an Open Source IoT platform focused on Data management. It takes care of everything from collecting data from devices to delivering data to end-user applications. To achieve such a thing, it uses a mixture of mechanisms and paradigm to store organized data, perform live queries. This guide focuses on daily operations for Astarte users and integrators. It goes through fundamental operations such as setting up triggers, querying APIs, integrating 3rd party applications and more. The user guide starts from the assumption that the reader is interacting with one or more well-known realms , and throughout the manual the assumption is that we're always operating inside a test realm, unless otherwise specified. Setting up realms is out of the scope of this guide, also because it's not a task the average user has to deal with. Please refer to the dedicated chapter of the Administrator manual to learn more about this specific topic. Before you begin, make sure you are familiar with Astarte's architecture, design and concepts .","ref":"001-intro_user.html"},{"type":"extras","title":"Interacting with Astarte","doc":"Astarte's interaction is logically divided amongst two main entities. Devices are the bottom end, and represent your IoT fleet. They can access Astarte only through a Transport, they are defined by a set of Interfaces which, in turn, also define on a very granular level which kind of data they can exchange. By design, they can't access any resource which isn't their own: such a behavior can be configured using Astarte as a middleman to act as a secure Gateway. Users are actual users, applications or anything else which needs to interact directly with Astarte. They are bound to a realm, and can virtually access any resource in that realm given they're authorized to do so. Users can also manage triggers and perform maintenance activity on the Realm.","ref":"010-interacting_with_astarte.html"},{"type":"extras","title":"Interacting with Astarte - User-side Tools","doc":"When interacting with Astarte as a User, you have several options to choose from: astartectl : astartectl is the main command-line tool to interact with Astarte clusters, which packs in a number of subcommands to interact with Astarte API sets. It is a swiss army knife to perform daily operations on Astarte Clusters, and it abstracts most Astarte API interactions in a user-friendly way. Astarte Dashboard : Astarte provides a built-in UI that can be used for managing Interfaces, Devices and Triggers. It is meant to be a graphical, user-friendly tool to perform daily operations on Realms. Astarte API Clients: API Clients are provided for a variety of languages. These clients abstract API interaction with language-friendly paradigms, and provide API automations for several operations. Currently, the main API client available is astarte-go . Astarte APIs: The base APIs are the lower level interaction layer. They are accessible, in standard installations, at api.<base Astarte URL>/<apiset> , and are the main mean of interaction upon which all other clients are based upon. Grafana Datasource Plugin for Astarte : Thanks to the Astarte Datasource Plugin, data coming from Astarte may be visualised in custom dashboards provided by Grafana, the open source observability platform. Depending on the context, you might want to choose what suits you best. Over the course of the documentation, several examples will be provided with interaction means. Setting up astartectl In the documentation, it is assumed that astartectl is properly configured to interact with your Realm or your Cluster. Please refer to its documentation to make sure all needed configurations are in place.","ref":"010-interacting_with_astarte.html#user-side-tools"},{"type":"extras","title":"Interacting with Astarte - Interacting with a Device","doc":"Devices interact with Astarte through their associated Transport. In this guide, we'll assume the Transport is MQTT/VerneMQ as per Astarte's defaults. However, rather than implementing the whole Astarte protocol over MQTT, it is usually a better idea to rely on one of Astarte's SDKs . Authentication/Pairing Depending on how you plan on implementing Astarte's pairing mechanism , your devices might need an Agent for their first authentication or not. However, once they retrieve their Credentials Secret, they can implement Astarte's standard pairing routine to rotate their SSL certificate for accessing the transport. In the most likely scenario in which you are using one of Astarte's SDKs, the SDK takes care of the whole pairing routine under the hood and, depending on your agent implementation, you just need to feed the SDK with either the Credentials Secret or the Agent Key. Exchanging data As per Astarte's protocol specification, data is exchanged based on the device's introspection. The device will be able to publish data on the transport on device interfaces, and receive data on server interfaces. In the MQTT case, the device will subscribe to its server interfaces' topics, and publish on its device interfaces topics. Isolation and RBAC are guaranteed by the transport's ACL, which are usually orchestrated though a dedicated Astarte extension (as in the VerneMQ/MQTT case). Again, Astarte's SDK allows you to interact with your device interfaces directly without caring about the underlying protocol and exchange details.","ref":"010-interacting_with_astarte.html#interacting-with-a-device"},{"type":"extras","title":"Interacting with Astarte - Interacting as a User","doc":"Astarte is mainly accessed through its APIs. Astarte's APIs are exposed through dedicated microservices (see Components ) and are meant both for configuration and for accessing data. There are two main sets of APIs we'll be using frequently: AppEngine API : This API is meant for querying/pushing data from/to devices. This maps to astartectl 's astartectl appengine subcommand. Realm Management API : This API is meant for configuring a target realm, and most notably for managing triggers. This maps to astartectl 's astartectl realm-management subcommand. Authentication Authenticating against Astarte is out of the scope of this guide, especially due to the fact that Astarte does not manage authentication directly . We'll assume either the authentication isn't enabled, or that the user is always interacting with the APIs with a token with the following claims { "a_aea": ".*:.*", "a_rma": ".*:.*" } Which represents a realm administrator. In real life use cases, you should always make sure to give out more granular permissions and to obtain the token in the right way from your authentication server. When using astartectl or any other client, you can also pass a Realm Private Key as an authentication mean, and have the token be automatically generated for you. Accessing the APIs In a standard Astarte installation, AppEngine API and Realm Management API are usually accessible at api.<your astarte domain>/appengine and api.<your astarte domain>/realmmanagement . If your Astarte installation has Swagger UI enabled, you can use the /swagger endpoint to access it, and to issue API calls straight from your browser to follow this guide.","ref":"010-interacting_with_astarte.html#interacting-as-a-user"},{"type":"extras","title":"Astarte Dashboard","doc":"Astarte provides a built-in UI that can be used for managing Interfaces, Devices, and Triggers. The Dashboard simplifies the development phases of applications that make use of Astarte, as well as troubleshooting activities. You can browse the source code of the Dashboard software on its GitHub repository .","ref":"015-astarte_dashboard.html"},{"type":"extras","title":"Astarte Dashboard - Introduction","doc":"The Astarte Dashboard is a Single Page Application that provides users with an overview of their Realm and a user-friendy way of managing it on any web browser. The Dashboard is designed to be a quick and easy way to give you immediate feedback on your work and as a quick and intuitive way to configure your realm. It is not designed to be operated by end-users, rather by infrastructure maintainers, owners who need information on the system status, and those working on projects based on Astarte. It is shipped by default with the Astarte standard distribution. The Dashboard is a graphical client for Astarte APIs; it shares similar features with the CLI client astartectl , the command-line utility to manage Astarte. The Dashboard helps you manage: Triggers Interfaces Devices Groups Realm Settings In case your Astarte distribution comes packaged with the Flow framework , the Dashboard is probably configured to manage Flow resources as well. Blocks Pipelines Flows Please note that the Flow framework is not available for use in a docker-compose environment since it relies on Kubernetes APIs to operate. How to access it Depending on how you are using Astarte, here is where you can find the Dashboard: Docker-compose: if you are using a local instance of Astarte via docker-compose , you will find it by pointing your browser to the default address http://dashboard.astarte.localhost . To login, fill in the name of your realm and a valid JWT token: if you possess the realm private key, as it is the case if you followed the Astarte in 5 minutes guide, you can generate the token with the command astartectl utils gen-jwt all-realm-apis -k <private_key> . Astarte Cloud: if you are using our managed Astarte option, you can hop onto our Console , find or create your realm and click the Dashboard button. Kubernetes cluster: in this case, if the Dashboard is enabled, it can be usually found at dashboard.<base Astarte URL> . Refer to your system administrator for more details.","ref":"015-astarte_dashboard.html#introduction"},{"type":"extras","title":"Astarte Dashboard - Main overview","doc":"Upon successful login, the main screen is the home page that provides an overview of the realm status and resources. API Status The API Status gives you general information about the status of services: Realm Management is an administrator-like API for configuring a Realm. It is used for managing Interfaces and Triggers. AppEngine is Astarte's main API endpoint for end-users. AppEngine exposes a RESTful API to retrieve and send data from/to devices, according to their interfaces. Every direct device interaction can be done from here. It also exposes Channels, a WebSocket-based solution for listening to device events in real-time with the same mechanism and semantics used by Triggers. Pairing takes care of Device Authentication and Authorization. It interacts with Astarte's CA and orchestrates the way devices connect and interact with Transports. It also handles Device Registration. Agent, Device and Pairing interaction is described in detail here . Flow is the API endpoint for Astarte Flow, used for managing Blocks, Pipelines, and Flows. Possible statuses are: This service is operating normally. This service appears offline. A general status on API health is also present in the app's sidebar, thus always providing a feedback regardless of which page you are currently visiting. Realm resources Within the main overview, a brief summary is available as well for the existing resources of the realm: registered and connected devices, installed interfaces, and installed triggers. More detailed overviews of each resource are available in the dedicated Dashboard sections, accessible via the navigation links in the Dashboard's side menu.","ref":"015-astarte_dashboard.html#main-overview"},{"type":"extras","title":"Astarte Dashboard - Interfaces","doc":"Interfaces are a core concept of Astarte which defines how data is exchanged between Astarte and its peers. You can navigate to this section thanks to the side menu of the Dashboard. A list of all installed interfaces is displayed, together with their major versions. Installing interfaces From the Interface list, clicking on the Install a new interface button will load up the Interface Editor , an interactive tool that you can use to configure your interfaces. The Interface Editor provides you with two ways to define your interfaces: on the left panel, a graphical frontend, while on the right panel you may input a JSON definition to achieve the same result. Each panel updates automatically whenever the other is changed. While defining a new interface, the Interface Editor will help you in supplying the right options and filling in mandatory entries such as: Name: an arbitrary name, formatted in reversed DNS casing. Major and Minor versions: based on Semantic Versioning . Type: indicates whether data is streamed continuously ( datastream ) or is stateful and persistent ( properties ). Ownership: the write-only allowed actor. All the other actors are read-only. Mappings: a list of endpoints that represent the data structure, following REST controller semantics. You can learn more about Interface definitions in their documentation's section . Note that when creating interface drafts, or for testing purposes in general, it is recommended to use 0 as the major version: to prevent data loss, Astarte allows only interfaces where major_version equals 0 to be deleted. Managing interfaces From the Interface list, you can select an interface to load and view its details in the Interface Editor. Clicking on the name of the interface will select its latest revision; clicking on a specific major of the interface will select the latest revision for that major. Note that interface revisions follow the Semantic Versioning convention. Once the Interface Editor is loaded you can review, update or delete the definition of the interface. Note that to prevent data loss, Astarte allows only interfaces where major_version equals 0 to be deleted. For similar reasons, when updating the definition of an interface, the Interface Editor will not allow you to change core properties on a minor version update. If you need to apply substantial changes, you can define and install a new major version for the interface.","ref":"015-astarte_dashboard.html#interfaces"},{"type":"extras","title":"Astarte Dashboard - Triggers","doc":"Triggers in Astarte are the go-to mechanism for generating push events. You can navigate to this section thanks to the side menu of the Dashboard. A list of all installed triggers is displayed. Installing triggers From the Trigger list, clicking on the Install a new trigger button will load up the Trigger Editor , an interactive tool that you can use to configure your triggers. It works in a very similar fashion to Interface Editor and shares the same User Interface. The Trigger Editor provides you with two ways to define your triggers: on the left panel, a graphical frontend, while on the right panel you may input a JSON definition to achieve the same result. Each panel updates automatically whenever the other is changed. The graphical tool dynamically loads installed Interfaces in the Realm and eases trigger creation by providing not only linting and validation, but also dynamic resolution of Interface names. You can learn more about Trigger definitions in their documentation's section . Note that due to how triggers work, you should install the trigger before a device connects. Doing otherwise will cause the trigger to kick in at a later time, and as such no events will be streamed for a while. Managing triggers From the Trigger list, you can select a trigger to load and view its details in the Trigger Editor. Once the Trigger Editor is loaded you can review the definition of the trigger. You can also delete the trigger instance by clicking on the Delete trigger button.","ref":"015-astarte_dashboard.html#triggers"},{"type":"extras","title":"Astarte Dashboard - Devices","doc":"Devices are Astarte's main entities for exchanging data. You can navigate to this section thanks to the side menu of the Dashboard. A list of all registered devices is displayed. Each device in the list is displayed together with info regarding its status and the last connection event . The status is represented by a grey dot if the device never connected to Astarte, a green dot if it is currently connected, a red dot if it is currently disconnected. The last connection event reports, if available, the date of the last connection or disconnection. A filter section is present on the side of the list to aid the search for specific devices, filtering the list by device ID, name, connection status, or configured attributes. Registering a device From the Device list, new devices may be registered by clicking on the Register a new device button which will take you to the registration page. Here you can proceed with the registration of the device by: providing a device ID: either by generating a random ID or by specifying a Name and Namespace UUID to generate the ID in a deterministic fashion. optionally declaring the initial introspection of the device: this is an indication of the list of interfaces that the device will use to exchange data. You can learn more about Devices and the registration process in their documentation's section . Device status and details From the Device list, you can select a device to navigate to its dedicated page. Here you can review and manage different info about your device. Device Info Device Info : displays info such as the device ID and the device Name alias, if set. It reports whether the device is currently connected or disconnected, or if it was never connected. From this section you can also momentarily Inhibit credentials for the device, preventing it to obtain access to Astarte; or you can directly Wipe credential secret of the device, a permanent action which will require to register the device again to have a new Credential Secret. Aliases : where you can manage custom aliases for the device. Note that setting a name alias will provide a name for the device. Attributes : a dedicated section to attach arbitrary info to the device, in a key-value form. Groups : where you can review and manage the Groups the device belongs to. Interfaces : a list of all currently and previously used interfaces. Clicking on an interface's name will load a dedicated page to review data exchanged by the device through that interface. Stats : a rundown on exchanged data via different interfaces. Here you can review, in both visual and numeric form, the quantity of bytes and messages the device is exchanging over each interface. This way you can always know at a glance which interfaces are the busiest and how chatty your device is. Status Events : here is info collected by Astarte regarding the IP addresses involved in the connections of the device to Astarte, the dates of first registration and credentials request, and the dates of last connection and disconnection, if available. Live Events : a section that reports live events regarding the device. It makes use of Astarte Channels and displays the connections, exchanged data, and errors of the device, as they happen in real-time.","ref":"015-astarte_dashboard.html#devices"},{"type":"extras","title":"Astarte Dashboard - Groups","doc":"Groups are logical collections of devices to ease the management and querying of devices. You can navigate to this section thanks to the side menu of the Dashboard. A list of all existing groups is displayed, together with the number of total and connected devices for each group. Creating a group From the Group list, clicking on the Create a new group button will load a dedicated page to setup the new group. You are required to specify a name for the group and to select at least one device that will belong to it. Indeed, note that a group must contain at least one device to exist. To confirm the creation of the new group, click the Create group button. Managing groups From the Group list, you can select a group to view its configuration on a dedicated page. Here you can review the list of devices that belong to it. To remove a device from a group, click on the Delete icon next to it. To add a device to a group, you can first navigate to the device's page and then add it to a group from there. To delete a group, remove all devices that belong to it and the group will automatically cease to exist.","ref":"015-astarte_dashboard.html#groups"},{"type":"extras","title":"Astarte Dashboard - Blocks","doc":"Blocks are computation units that can be chained together to define a logical computation topology. You can navigate to this section thanks to the side menu of the Dashboard. Here is a list of all existing blocks, both custom and native ones: the former ones are those defined by you, the latter ones are those provided by Astarte and are displayed with a native label. Creating blocks From the Block list, clicking on the Create button will load a dedicated page to define a new block. Here you can define the block by specifying: A name for the block. A type , between Producer , Consumer or Producer & Consumer , depending on how the block should treat messages and connect to other blocks. A schema , reporting the JSON Schema definition of how a configuration should look like for the block. A source , containing the implementation of the block, written in the DSL format for Pipelines. Confirm the creation of the block by clicking on the Create new block button. To learn more about block definition you can read their documentation's section . Managing blocks From the Block list, you can select a block to view its definition in a dedicated page. Here you can review the details of the block such as the block type, the schema and, if it is a custom block, its source. To delete a block, click on the Delete block button. Note that you cannot delete native blocks provided by Astarte.","ref":"015-astarte_dashboard.html#blocks"},{"type":"extras","title":"Astarte Dashboard - Pipelines","doc":"A Pipeline is a computation blueprint (therefore a description) built as a chain of blocks. You can navigate to this section thanks to the side menu of the Dashboard. Here is a list of all existing pipelines from which you can review, manage and instantiate them. Creating pipelines From the Pipeline list, clicking on the Create button will load up the Pipeline Editor page, an interactive tool that you can use to configure your pipeline. The Pipeline Editor is composed of two parts: A sidebar listing all available Blocks, grouped by type A space where you can drag & drop blocks, connecting them to effectively design a pipeline You can read more about the Pipeline Editor on the dedicated documentation . Once you have designed your pipeline, you can review and specify its: name source , containing the implementation of the pipeline, written in the DSL format for Pipelines. schema , reporting the JSON Schema definition of how a configuration should look like for the pipeline. description , explaining the scope and supposed usage of the pipeline. Then hit the Create new pipeline button to confirm the definition of the pipeline. Managing pipelines From the Pipeline list, you can select a pipeline to view its definition in a dedicated page. Here you can review the details of the pipeline such as the pipeline description, its schema, and its source. To delete a pipeline, click on the Delete pipeline button.","ref":"015-astarte_dashboard.html#pipelines"},{"type":"extras","title":"Astarte Dashboard - Flows","doc":"Flows are specific instances of a pipeline, created providing concrete values to the parametric configuration of a pipeline. You can navigate to this section thanks to the side menu of the Dashboard. Here is a list of all existing flows. Each flow reports its current status and the pipeline it originated from. To review the details of a Flow, click on its name to load up the dedicated page. To delete a flow, click on the Delete icon next to it. Instantiating flows From the Pipeline list, clicking on the Instantiate button of a Pipeline will load the Flow configuration page; from there you can supply a name and a configuration for the flow before hitting the Instantiate Flow button to confirm.","ref":"015-astarte_dashboard.html#flows"},{"type":"extras","title":"Astarte Dashboard - Realm Settings","doc":"You can navigate to this section thanks to the side menu of the Dashboard. Here you can review and update settings for your realm. You can update the public key of the realm, which is useful if you mean to use a new private key to generate auth tokens. Please note that it is a permanent action and Astarte will prevent interactions that use auth tokens generated with the previous key.","ref":"015-astarte_dashboard.html#realm-settings"},{"type":"extras","title":"Accessing and Exploring a Realm","doc":"In Astarte, a Realm is a logical partition which holds a number of devices and an Authentication Realm. The Astarte Dashboard allows you to explore all resources of a given Realm, such as e.g. Interfaces, Triggers, Devices, Groups etc...","ref":"020-accessing_and_exploring_a_realm.html"},{"type":"extras","title":"Accessing and Exploring a Realm - Device limit in a Realm","doc":"While there is no limit on the number of Devices registered in a Realm, it is possible that an upper bound has been set in the Realm configuration using the Housekeeping API . When it is set, trying to register more Devices past the limit will result in an error. The limit can be retrieved from Realm Management with GET <astarte base API URL>/realmmanagement/v1/<realm name>/config/device_registration_limit The HTTP payload of the response will have the following format: { "data": <value> } If such a limit is set, the value will be a non negative integer. If not, the value will be null .","ref":"020-accessing_and_exploring_a_realm.html#device-limit-in-a-realm"},{"type":"extras","title":"Interface Design Guide","doc":"Before we begin, let's get this straight: The way you design your interfaces will determine the overall performance and efficiency of your cluster This is because interfaces define not only the way data is exchanged between Astarte and Devices/Applications, but also how it will be stored, managed and queried . As such, it is fundamental to spend enough time on finding the most correct Interface design for your use case, keeping in mind how your users will consume your data, what might change in the future, what is fundamental and what is optional, and more.","ref":"029-interface_design_guide.html"},{"type":"extras","title":"Interface Design Guide - Use the right tools","doc":"Before you begin, you might want to take a look at Astarte Interfaces Editor , which is also available in any Astarte Dashboard installation. Astarte Interfaces Editor gives you automated validation and linting for Astarte Interfaces, and also gives you a declarative editor with automatic JSON generation. It is well maintained and used as a reference for Interface design. Consider using it for building your interfaces.","ref":"029-interface_design_guide.html#use-the-right-tools"},{"type":"extras","title":"Interface Design Guide - Rationale","doc":"Without going into deeper details on what concerns Astarte's DB internals, there are some considerations one should always keep in mind when designing interfaces. Querying an Interface is fast, querying across Interfaces is painful Astarte's data modeling is designed to optimize queries within a single interface. Querying across interfaces is supported, but might affect performances significantly, especially if done frequently and with complex queries. This is especially true for triggers, as they could be evaluated very frequently. In general, if you plan on having different mappings which are frequently queried altogether, or dependent on each other for several triggers, you might be better off in having them all in the same Interface. Aggregation makes a difference Aggregation is a powerful feature, which comes with price and benefits. Even though each series has only one timestamp for all values, it is also true that losing granularity for endpoints might cause storage of redundant data if only one of the aggregated mappings change value. Moreover, in terms of data modeling, Aggregated interfaces imply the creation of a dedicated Cassandra table. Having a lot of aggregated interfaces might end up putting additional pressure on the Cassandra Cluster in terms of memory and overall performance. Your Cluster administrator might (rightfully) choose to limit the amount of installed aggregate interfaces in a Realm, or in the overall Cluster.","ref":"029-interface_design_guide.html#rationale"},{"type":"extras","title":"Interface Design Guide - Interface Atomicity","doc":"Rule of thumb: Favor extreme atomicity in case you expect your interfaces to change often, be as atomic as reasonably possible in case you want to favor performance and flexibility in querying data.","ref":"029-interface_design_guide.html#interface-atomicity"},{"type":"extras","title":"Managing Interfaces","doc":"Interfaces define how data is exchanged over Astarte. For a Device to be capable of exchanging data into its Realm, its interfaces have to be registered into the Realm first. Let's walk over the whole process. It is assumed that you have read the Interface design guide before, to avoid bad surprises once your fleet starts rolling.","ref":"030-manage_interfaces.html"},{"type":"extras","title":"Managing Interfaces - Querying Interfaces","doc":"Listing Interfaces You can list all installed interfaces in a given Realm. This will return all the valid installed Interface names, without any versioning. List Interfaces using astartectl $ astartectl realm-management interfaces list [com.my.Interface1 com.my.Interface2 com.my.Interface3] List Interfaces using Astarte Dashboard From your Dashboard, after logging in, click on "Interfaces" in the left menu. List Interfaces using Realm Management API GET <astarte base API URL>/realmmanagement/v1/test/interfaces {"data": ["com.my.Interface1","com.my.Interface2","com.my.Interface3"]} Listing Major Versions for an Interface For each installed Interface, there can be any number of Major versions installed. This information can be retrieved by listing the available Major versions for a specific interface. In a realm, only the latest minor version of each major version of an Interface is kept. This can be done due to the fact that Semantic Versioning implies a new minor version doesn't introduce any breaking change (e.g.: deleting or renaming a mapping), and as such querying an older version of an interface using a newer one as a model is always compatible - some mappings might be empty, as expected, and will be disregarded. Astarte ensures upon Interface installation for this constraint, and as such you can always query the latest minor version of an Interface safely. List Versions using astartectl $ astartectl realm-management interfaces versions com.my.Interface1 [0 1 2] List Versions using Astarte Dashboard In the Dashboard's Interface page, click on any Interface name. A drop-down will appear, showing installed major versions for that Interface name. List Versions using Realm Management API GET <astarte base API URL>/realmmanagement/v1/test/interfaces/com.my.Interface1 {"data": [0,1,2]} Getting an Interface Definition Astarte allows you to retrieve the Interface Definition for a given Name and Major Version pair. The definition is in the standard Interface JSON format. Get Interface Definition using astartectl $ astartectl realm-management interfaces show com.my.Interface1 0 { "data": { "version_minor": 2, "version_major": 0, "type": "properties", "ownership": "device", "mappings": [ { "type": "integer", "path": "/myValue", "description": "This is quite an important value." }, { "type": "integer", "path": "/myBetterValue", "description": "A better revision, introduced in minor 2, supported only by some devices" }, { "type": "boolean", "path": "/awesome", "allow_unset": true, "description": "Introduced in minor 1, tells you if the device is awesome. Optional." } ], "interface_name": "com.my.Interface1" } } Get Interface Definition using Astarte Dashboard From the Interfaces page, click on an Interface name, and click on the Major version for which you'd like to see the definition. The Interfaces Editor window will open, with the Interface definition in the text box on the right. From the Editor page, it is also possible to add new mappings to the Interface and bump it to a new Minor. Get Interface Definition using Realm Management API GET <astarte base API URL>/realmmanagement/v1/test/interfaces/com.my.Interface1/0 { "data": { "version_minor": 2, "version_major": 0, "type": "properties", "ownership": "device", "mappings": [ { "type": "integer", "path": "/myValue", "description": "This is quite an important value." }, { "type": "integer", "path": "/myBetterValue", "description": "A better revision, introduced in minor 2, supported only by some devices" }, { "type": "boolean", "path": "/awesome", "allow_unset": true, "description": "Introduced in minor 1, tells you if the device is awesome. Optional." } ], "interface_name": "com.my.Interface1" } }","ref":"030-manage_interfaces.html#querying-interfaces"},{"type":"extras","title":"Managing Interfaces - Installing/Updating an interface","doc":"Interfaces are supposed to change over time, and are dynamic. As such, they can be installed and updated. Interface installation means adding either a whole new Interface (as in: an Interface with a new name), or a new major version of an already known Interface. Interface update means updating a specific, existing interface name/major version with a new minor version. When designing interfaces, it is strongly advised to use Astarte Interface Editor. The Editor is embedded into any Astarte Dashboard installation but, in case your Astarte installation does not provide you with a Dashboard, you can use Astarte Interface Editor public online instance . Use it to write and validate your definitions, and install the resulting JSON file through either astartectl or Realm Management APIs. Synchronizing interfaces using astartectl astartectl provides a handy sync command that, given a list of Interface files, will synchronize the state of the Astarte Realm with your local interfaces. It is handy in those cases where your Realm has several interfaces, and you're storing Interfaces in a common place, such as a Git Repository - this is the average case for Astarte-based applications/clouds. Assuming you have a set of Interface files in your folder all with the .json extension, invoking astartectl realm-management interfaces sync will result in something like this: $ astartectl realm-management interfaces sync *.json Will install interface com.my.Interface1 version 0.2 Will install interface com.my.Interface2 version 1.1 Will update interface com.my.Interface3 to version 1.4 Do you want to continue? [y/n] y Interface com.my.Interface1 installed successfully Interface com.my.Interface2 installed successfully Interface com.my.Interface3 updated successfully to version 1.4 After invocation, your Astarte Realm will be up to date with all Interfaces in your local directory. Note: astartectl realm-management interfaces sync currently synchronizes Interfaces only from your local machine to the Realm, and not the other way round. In case the Realm has a more recent version of an interface compared to your local files, or it has some interfaces which are not referenced by your local files, no action will be taken. Install an Interface using Astarte Dashboard Access the Editor by going to the Interfaces page, and clicking on "Install a New Interface..." in the top-right corner. The Editor will open. From there, you can either paste in an existing JSON definition, which will be validated and will update the left-screen declarative Editor, or you can build a whole new Interface from scratch. Once you're done, hit the "Install Interface" button at the bottom of the declarative Editor (left side) to install the Interface in the Realm. Install an Interface using astartectl First of all, ensure that you have the Interface you'd like to install saved in a file on your local machine. We will assume the interface is available as interface1.json . $ astartectl realm-management interfaces install interface1.json ok Install an Interface using Realm Management API Realm Management currently implements a completely asynchronous API for Interface installation - as such, the only feedback received by the API is that the Interface is valid and the request was accepted by the backend. However, this is no guarantee that the Interface will be installed successfully. As a best practice, it is advised to either wait a few seconds in between Realm Management API invocations, or verify through a GET operation whether the Interface has been installed or not. POST <astarte base API URL>/realmmanagement/v1/test/interfaces The POST request must have the following request body, with content type application/json { "data": { "version_minor": 2, "version_major": 0, "type": "properties", "ownership": "device", "mappings": [ { "type": "integer", "path": "/myValue", "description": "This is quite an important value." }, { "type": "integer", "path": "/myBetterValue", "description": "A better revision, introduced in minor 2, supported only by some devices" }, { "type": "boolean", "path": "/awesome", "allow_unset": true, "description": "Introduced in minor 1, tells you if the device is awesome. Optional." } ], "interface_name": "com.my.Interface1" } } The call will return either 201 Created or an error. Most common failure cases are: The interface/major combination already exists in the Realm The interface schema fails validation In any case, the API returns details on what caused the error and how to solve it through Astarte's standard error reply schema. Update an Interface using astartectl First of all, ensure that you have the Interface you'd like to update saved in a file on your local machine. We will assume the interface is available as interface1_3.json . $ astartectl realm-management interfaces update interface1_3.json ok Update an Interface using Astarte Dashboard Go to the Interfaces page, click on the Interface Name you'd like to update, and click on the Major version which is referred by your upgrade (e.g.: if you're updating from 1.2 to 1.3, you want to click on Major Version 1). The Editor will appear, populated with the currently installed Interface definition. Paste in your updated JSON file, or use the declarative editor to make your changes. The editor will be limited to Semantic Version-compatible operations (as in - adding new mappings). Once you're done, hit the "Apply Changes" button at the bottom of the declarative Editor (left side) to update the Interface in the Realm. Update an Interface using Realm Management API To update an existing interface, issue a PUT /interfaces/<name>/<major> endpoint of the realm with the very same semantics as the Installation procedure. The call will return either 201 Created or an error. Apart from the very same errors that could be triggered upon installation, Update will also fail if the interface doesn't provide a compatible upgrade path from the previously installed minor. Interface update limitations Major version updates Major version updates have no intrinsic limitations as they are not meant to ensure compatibility with older versions of the same interface. Therefore, if you plan to bump your interface major you are allowed to update your interface at your preference. Please, refer to the Interface Design Guide to follow the best practices while developing your new updated interface. Minor version updates Minor version updates are conceived to guarantee retro-compatibility and, as such, they allows only for a limited subset of update operations. Currently, based on the interface type and aggregation, different update capabilities are provided: properties : at interface root level, doc and description updates are allowed; at mapping level, doc and description updates are allowed. Moreover, an arbitrary number of new mappings can be added; individual datastream : at interface root level, doc and description updates are allowed; at mapping level, doc , description and explicit_timestamp updates are allowed. Moreover, an arbitrary number of new mappings can be added; object aggregated datastream : currently, due to a limitation in how data are stored within Cassandra, the doc , descriprion and explicit-timestamp fields can not be updated; at mapping level, an arbitrary number of mappings can be added. Where not explicitly stated, all the other values are to be considered as not updatable. In case you need to update one of those fields, please consider updating your interface major version.","ref":"030-manage_interfaces.html#installing-updating-an-interface"},{"type":"extras","title":"Managing Interfaces - Interfaces lifecycle","doc":"Interfaces are versioned through a semantic versioning-like mechanism. A Realm can hold any number of interfaces and any number of major versions of a single interface. It holds, however, only the latest installed minor version of each major version, due to the inherent compatibility of Semantic Versioning. There is no significant cost in adding a non-aggregated interface to a Realm or updating a non-aggregated interface frequently - keep in mind, however, that you might incur in dangling data in your devices if you don't plan your interface update strategy accurately. For what concerns Aggregated interfaces, instead, there is an inherent cost which might end up in putting pressure on your Cluster . Once an interface has been installed in a Realm, it can't be uninstalled without performing manual operations on Astarte's DB, unless its major version number is 0 . This is a safety measure to prevent dangling data from appearing in the cluster. For this reason, when developing an Astarte-based interface, it is strongly advised to keep its major number to 0 to allow quick changes at the expense of data loss. Please note, however, that deleting a major 0 interface is possible if the Realm has no devices left declaring that specific interface in their introspection. This is done to avoid forever dangling data and potential consistency errors. This limitation might be lifted in the future through a mass-deletion mechanism, but there is no guarantee this will ever be done. It is advised to test new interfaces on a limited number of devices to ease operations.","ref":"030-manage_interfaces.html#interfaces-lifecycle"},{"type":"extras","title":"Managing Interfaces - Realm vs. Device Interface relationship","doc":"There is a clear difference between how Interfaces are managed in a Realm and its Devices (e.g.: the device Introspection). Whereas a Realm can have any number of versions of a single interface, a Device is allowed to expose in its introspection only a single, specific version of an Interface. In general, Realm interfaces are kept as a shared agreement between its entities, but when it comes to interacting with a Device, the Realm honors its introspection (as long as the Device declares interfaces the Realm is knowledgeable about). As such, installing an interface in a Realm is a completely safe and non-disruptive operation: by design, Devices aren't aware of which interfaces a Realm supports, and Realms don't impose any interface versioning on a Device.","ref":"030-manage_interfaces.html#realm-vs-device-interface-relationship"},{"type":"extras","title":"Managing Interfaces - Caveats","doc":"Due to how minor versions work, it is responsibility of the end user to prevent accidental data loss due to missing data. Every mapping declared in a new minor release must be assumed as optional, as there is no guarantee that a Device will be able to publish (or receive) data on that specific mapping. Minor version bumps work great in case they represents additional, optional features which might be available on an arbitrarly large subset of Devices implementing that interface's major version, and are not necessary or fundamental for normal operations. If that is not the case, consider a major version update or a whole new interface instead. Also, please keep in mind that designing interfaces in the right way, especially being as atomic as reasonably possible, helps a lot in preventing situations where a minor interface update can't be done without disrupting operations. Again, the Interface design guide covers this topic extensively.","ref":"030-manage_interfaces.html#caveats"},{"type":"extras","title":"Managing Interfaces - Dangling data","doc":"In several situations, it is possible to have dangling data inside Astarte. This happens by design, as the liquid nature of a Device makes it possible for data to be stored in interfaces no longer present in its introspection. Astarte does not delete data unless requested explicitly: as such, data remains available inside its database, but potentially inaccessible through the cluster's APIs and standard mechanism. As of the current version, Astarte has no mechanism for retrieving and acting upon a device's dangling data - this is a limitation that will be lifted in future releases with additions to the current API. Interface major version change If a device upgrades one of its interfaces to a new major version, the previous interface is parked and its data remains dangling. Every API call, trigger, or reference to the interface will always target the major version declared in the introspection, regardless of the fact that a more recent version might have been installed in the realm. Interface deletion from device A device might arbitrarly decide to remove an interface from its introspection. In such a case, Astarte won't return any data and will consider all data previously pushed to said interface inaccessible. In case the interface comes back again in the introspection, previously pushed data will be available as if nothing happened.","ref":"030-manage_interfaces.html#dangling-data"},{"type":"extras","title":"Registering a Device","doc":"Devices are Astarte's main entities for exchanging data. Even though a Device usually represents the physical Device communicating with Astarte, they might as well be mapped to other entities, such as individual sensors or aggregated gateways. A Device always belongs to a Realm and is identified by a Device ID , which has to be unique at least within its Realm. Devices communicate with Astarte through Transports - in most installations, this means through an MQTT Broker (VerneMQ with Astarte's plugin). Before this happens, though, Devices must obtain credentials for accessing their Transport and, most of all, make themselves known to Astarte. This happens through the Registration process. In Astarte, Registering a device means obtaining an unique Credentials Secret (Registration Credentials), univocally associated to a Device ID, through a well-known workflow and pipeline. If you are not familiar with these concepts, please refer to Pairing Architecture to learn more about Pairing's workflow basics. The Credentials Secret can then be used by the Device for accessing Pairing API and getting information and Credentials for its Transport. As such, registration happens only once during a Device's lifecycle, and is a security-sensitive process. As such, this process is usually carried over (in production scenarios) through an Agent .","ref":"035-register_device.html"},{"type":"extras","title":"Registering a Device - Registration Agent","doc":"An Agent 's purpose is to perform Registration on behalf of a Device. Agents should be the only components in your infrastructure with enough credentials to access Pairing's Agent APIs (as a rule of thumb, it is a bad idea to give access to Pairing API to anything which isn't an Agent). When setting up an Astarte project, it is fundamental to define beforehand how your Devices will be registered and hence where your Agent(s) will belong. There's two main ways for implementing an Agent, even though in production scenarios On Board Agents are strongly discouraged as they expose a single point of failure in terms of a Realm's whole fleet security. On Board Agent Please keep in mind that On Board Agents are not advised in production, as a single compromised device/token might compromise the Registration routine for your entire fleet. They should be used only in non-critical use cases or during testing and development. On Board Agents are provided as a feature by Astarte's SDK, and hide the detail of Device registration by integrating an Agent into the SDK itself. This allows to deliver the same credentials to each device belonging to a Realm. Of course, this also opens up a single point of failure in the whole fleet's security, as Credentials aren't tied to a specific device - as such, if compromised, they might allow an attacker to register an arbitrary device into a Realm, unless other policies prevent him from doing so. To create a On Board Agent, you simply need to emit a long-enough lived token from your Realm's private key with access to Pairing's Agent APIs . This token should then be delivered to your devices and provided to the SDK in order to carry over the Registration. The SDK will do this automatically and without any need for additional code, as long as you set the agentKey configuration key to a meaningful value, and no Credentials Secret has been set. 3rd Party Agent A more secure approach to the Registration process is having a 3rd Party agent. In such a case, an external component is in charge of requesting a Credentials Secret to Pairing and delivering it to the target Device. This approach has a number of benefits: in terms of Security, the Agent uses a short-lived token and can follow the Realm's authentication workflow just like any other application. For what concerns daily operations, the Agent can implement any arbitrary logic to make a decision on whether a Device should be registered or not. In such cases, Devices have an out-of-band communication mechanism with the Agent in which the Credentials are exchanged. Usually, these cases fall under two main categories: "Local" or "Plant" Agents In this scenario, devices are imprinted with their Credentials Secret in the production plant. The Device might not even be connected to the Internet, whereas the machine running the Agent has access to the target Astarte Cluster and adequate Credentials for Registration. Once the Agent acquires the Device ID of the Device which should be registered, it issues the request to Astarte's Pairing API and obtains the Device's Credentials Secret . At this stage, the Agent is in charge of delivering the Credentials Secret to the Device the way it sees fit. As a best practice, the Credentials Secret should then be saved to an OTP area or a dedicated secure storage in the device to prevent tampering or accidental loss. Even though this is arguably the most secure mechanism available for Registering a Device, it might not fit every use case as the Device will be irrevocabily assigned to a specific Astarte Cluster and a specific Realm in that Cluster before it even connects. "Remote" Agents If your use case demands more flexibility, Registering a Device in a plant might not fit your Device's lifecycle. This could be likely if, for example, Realm or Cluster assignment should be done dynamically once the Device reaches its final user. In this case, this role is usually delegated to an external web application acting as an Agent. In this case, it's up to the user setting up all mechanisms for delivering the Credentials Secret to the Device, which includes securing the communication channel. On the other hand, this allows an extremely flexible approach to Registration, which can be implemented through an entirely custom logic.","ref":"035-register_device.html#registration-agent"},{"type":"extras","title":"Registering a Device - Credentials Secret Lifecycle","doc":"Credentials Secrets are meant to be immutable - as such, they should be handled with extreme care. Credentials Secrets are used only for interacting with Pairing, hence to obtain Credentials for a Transport which, on the other hand, are meant to be volatile. A Device can be Registered an arbitrary number of times before its Credentials Secret is used for the first time for interacting with Pairing. This is done to ensure the entire Registration process, including any kind of external custom logic of the Agents, has been carried over successfully, allowing a de-facto "retry" until there's certainty the Device has access to its Credentials Secret . Please note that when Registering a Device, a new Credentials Secret is generated every time.","ref":"035-register_device.html#credentials-secret-lifecycle"},{"type":"extras","title":"Registering a Device - Unregistering a device","doc":"Once the Credentials Secret is used for retrieving Credentials for a Transport for the first time, Astarte prevents further registration of the same Device again. If there's the need of registering the device again (e.g.: a Device has been tampered and got back to its plant with its previous Credentials Secret compromised), it is possible to explicitly unregister the device to obtain a new Credentials Secret using Pairing's Agent APIs or with astartectl (see the output of astartectl pairing agent unregister -h for more documentation).","ref":"035-register_device.html#unregistering-a-device"},{"type":"extras","title":"Connecting a Device","doc":"Once a Device has been Registered in Astarte, it is capable of connecting to it. Devices connect to Astarte through the use of Transports . A Transport is an arbitrary protocol implementation which maps Astarte's concepts (mainly Interfaces) to a communication channel. Astarte's main supported Transport is Astarte/MQTT, implemented on top of VerneMQ through an additional plugin , and it is used by Astarte's SDKs for communication. However, virtually any protocol can be integrated in Astarte by creating a corresponding Transport. Transports also define the authentication/authorization mechanism of their Devices. For instance, Astarte/MQTT uses mutual SSL Authentication with Certificate Rotation for securing its Ingress and identifying its clients. To manage their Transport(s) and Credentials, Devices have to interact with Pairing.","ref":"040-connect_device.html"},{"type":"extras","title":"Connecting a Device - Credentials Secret, Pairing and Transports","doc":"Once a Device has performed its first registration through an Agent, it holds its Credentials Secret . This Credentials Secret is the token the device uses for performing the actual Pairing routine, which results in the device obtaining its Credentials for accessing its designated Transport. A Device's Credentials Secret allows access to Pairing API's Device REST API , which is then used for obtaining information about which Transports the Device can use for communicating, and for obtaining Credentials for its assigned Transports. The ability to request Credentials of a Device can be inhibited with AppEngine API or using astartectl with this command: astartectl appengine devices credentials inhibit <device_id_or_alias> true \\ -k <appengine-key> -r <realm-name> -u <astarte-api-url> Once its credentials_inhibited field is set to true , a Device is not able to request new Credentials. Note that Credentials that were already emitted will still be valid until their expiration. As, from a user's standpoint, the way a Device communicates with Astarte is entirely Transport-specific, this guide will cover using Astarte/MQTT through one of Astarte's SDKs. If you are using a different Transport, please refer to its User Guide, or if you wish to implement your own, head over to Transport Developer Documentation .","ref":"040-connect_device.html#credentials-secret-pairing-and-transports"},{"type":"extras","title":"Connecting a Device - Using Astarte/MQTT through Astarte SDK","doc":"If you are using one of Astarte's SDK, the Pairing routine is entirely managed, and you won't need to do any of the aforementioned steps. Just make sure your Credentials Secret is passed as the apiKey configuration key, to allow the SDK to perform automatically the Pairing routine when needed. The SDK does a number of automated things under the hood. Its flow is: The SDK verifies if a SSL certificate for connecting to the broker is present. If it is, it attempts connecting to the Transport. If the Transport doesn't accept the connection due to an SSL error, it queries Pairing API about its certificate status. If Pairing API returns a problem with the certificate or, in general, the certificate isn't valid, the certificate is erased and the Pairing procedure begins. The SDK invokes Pairing API until it manages to obtain a valid Certificate for the Transport. The SDK considers a Device successfully paired when it has a valid certificate and manages to connect to the Transport. Once in this state, the Device can start exchanging data. Note: the Pairing procedure is secure as long as Pairing API is queried using HTTPS. Plain HTTP installations are vulnerable to a number of different attacks and should NEVER be used in production. Interfaces and Introspection A Device must have some installed interfaces to be capable of exchanging data. These interfaces must be made known to the SDK and installed in the Device's Realm, as previously explained . The SDK expects the user to provide a directory containing a set of valid interfaces. It then takes care of making Astarte aware of its registered interfaces through a process called Introspection. Introspection is a special control message in Astarte's protocol which makes Astarte aware of a list of Interfaces and relative versions which are installed on the Device. Again, Astarte's SDK, given a directory, is capable of performing the correct procedures for keeping Introspecting in sync correctly without any kind of user intervention. Astarte's SDK also takes care of updating a Device's Introspection if its interfaces change. Exchanging data When a Device connects successfully, it must then subscribe to its server Interfaces. The SDK takes care of this detail and exposes a higher level interface. For example, using the Qt5 SDK: { m_sdk = new AstarteDeviceSDK(QStringLiteral("/path/to/transport-astarte.conf"), QStringLiteral("/path/to/interfaces"), deviceId); connect(m_sdk->init(), &Hemera::Operation::finished, this, &AstarteStreamQt5Test::checkInitResult); connect(m_sdk, &AstarteDeviceSDK::dataReceived, this, &AstarteStreamQt5Test::handleIncomingData); } void AstarteStreamQt5Test::handleIncomingData(const QByteArray &interface, const QByteArray &path, const QVariant &value) { qDebug() << "Received data, interface: " << interface << "path: " << path << ", value: " << value << ", Qt type name: " << value.typeName(); } Applications can simply connect to the handleIncomingData signal and have data correctly formatted and delivered as it runs through the transport. On the other hand, for sending data: m_sdk->sendData(interface, path, value); The SDK will check if data is coherent with its introspection, and send data onto the transport in the correct way. Reliability, retention and persistency in the SDK Astarte's SDK has an internal concept of persistency, depending on the behaviour defined in its installed Interfaces. The retention parameter, specifically, tells Astarte's SDK how hard it should try to send a specific message. In case the Transport is unreachable, the SDK might try to persist, either in memory or on disk, and send the message when the connection is available again. Please note that these parameters declared in Interfaces are to be considered on a best effort basis. In case your SDK does not support persistency or has persistency disabled, a number of warranties requested by an Interface might not be satisfied. Make sure your SDK is configured correctly before moving to production.","ref":"040-connect_device.html#using-astarte-mqtt-through-astarte-sdk"},{"type":"extras","title":"Device errors","doc":"This page details the errors that can affect a device while it's sending data. The user can monitor device errors by installing a device trigger on device_error or checking the Devices tab in the Astarte Dashboard. The same errors are also provided as log messages on Data Updater Plant.","ref":"045-device_errors.html"},{"type":"extras","title":"Device errors - write_on_server_owned_interface","doc":"The device is trying to write on a server owned interface. The device can only push data on device owned interfaces.","ref":"045-device_errors.html#write_on_server_owned_interface"},{"type":"extras","title":"Device errors - invalid_interface","doc":"The interface name received in the message is invalid.","ref":"045-device_errors.html#invalid_interface"},{"type":"extras","title":"Device errors - invalid_path","doc":"The path received in the message is invalid. This might happen when a path does not have a valid path format or it's not a valid UTF-8 string.","ref":"045-device_errors.html#invalid_path"},{"type":"extras","title":"Device errors - mapping_not_found","doc":"The path received in the message can't be found in the interface mappings. This could be the result of the device having a more recent version of the inteface than the one installed in the realm or an interface with the same name and version but different contents.","ref":"045-device_errors.html#mapping_not_found"},{"type":"extras","title":"Device errors - interface_loading_failed","doc":"The target interface was not found in the database. Usually this means the interface is not installed in the realm, but the error can also derive from the database being temporarily unavailable.","ref":"045-device_errors.html#interface_loading_failed"},{"type":"extras","title":"Device errors - ambiguous_path","doc":"The path received in the message can't be mapped univocally on a mapping. This is often the result of an incomplete path.","ref":"045-device_errors.html#ambiguous_path"},{"type":"extras","title":"Device errors - undecodable_bson_payload","doc":"The payload of the message can't be decoded as BSON.","ref":"045-device_errors.html#undecodable_bson_payload"},{"type":"extras","title":"Device errors - unexpected_value_type","doc":"The value of the message does not have the expected type ( e.g. the mapping expects a string value but an integer value was received instead).","ref":"045-device_errors.html#unexpected_value_type"},{"type":"extras","title":"Device errors - value_size_exceeded","doc":"The value of the message exceeds the maximum size of its type. The size limitations of the types are documented here .","ref":"045-device_errors.html#value_size_exceeded"},{"type":"extras","title":"Device errors - unexpected_object_key","doc":"An object aggregated value with an unexpected key was received.","ref":"045-device_errors.html#unexpected_object_key"},{"type":"extras","title":"Device errors - invalid_introspection","doc":"The introspection sent from the device can't be parsed correctly. The introspection format is documented here .","ref":"045-device_errors.html#invalid_introspection"},{"type":"extras","title":"Device errors - unexpected_control_message","doc":"The device sent a message on an unhandled control path. The supported control paths are detailed in the protocol documentation .","ref":"045-device_errors.html#unexpected_control_message"},{"type":"extras","title":"Device errors - device_session_not_found","doc":"Data Updater Plant failed to push data towards the device. This could result from the device being currently offline and not having a persistent session on the MQTT broker or from the device not having all the MQTT subscriptions required by the Astarte protocol","ref":"045-device_errors.html#device_session_not_found"},{"type":"extras","title":"Device errors - resend_interface_properties_failed","doc":"Data Updater Plant failed to resend the properties of an interface. This could result from the device declaring a uninstalled properties interface in its introspection right before an emptyCache.","ref":"045-device_errors.html#resend_interface_properties_failed"},{"type":"extras","title":"Device errors - empty_cache_error","doc":"The empty cache operation for a device failed. This could result from a temporary database failure.","ref":"045-device_errors.html#empty_cache_error"},{"type":"extras","title":"Querying a Device","doc":"Once you have your devices connected, up and running in Astarte, you can start interacting with them.","ref":"050-query_device.html"},{"type":"extras","title":"Querying a Device - Device status","doc":"First things first, you can check if your device is correctly registered in Astarte, and its current status. Let's assume our Device has f0VMRgIBAQAAAAAAAAAAAA as its id. A Device's status includes a number of useful information, among which whether it is connected or not to its Transport, its introspection, the amount of exchanged data and more. Query Device status using astartectl $ astartectl appengine devices show f0VMRgIBAQAAAAAAAAAAAA Device ID: f0VMRgIBAQAAAAAAAAAAAA Connected: false Last Connection: 2018-02-07 18:38:57.266 +0000 UTC Last Disconnection: 2018-02-08 09:49:26.566 +0000 UTC Introspection: com.example.ExampleInterface v1.0 exchanged messages: 20 exchanged bytes: 200B org.example.TestInterface v0.2 exchanged messages: 8 exchanged bytes: 147B Received Messages: 221 Data Received: 11.7K Last Seen IP: 203.0.113.89 Last Credentials Request IP: 203.0.113.201 First Registration: 2018-01-31 17:10:59.270 +0000 UTC First Credentials Request: 2018-01-31 17:10:59.270 +0000 UTC Query Device status using Astarte Dashboard After logging in to Astarte dashboard, go to the "Devices" page clicking on the menu on your left. A list of available Device IDs will appear. If you do not see your device at a glance, use the search bar on the top right to find it. Clicking on the Device ID will take you to its details page. Query Device status using AppEngine API GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA { "data": { "total_received_msgs": 221, "total_received_bytes": 11660, "last_seen_ip": "203.0.113.89", "last_credentials_request_ip": "203.0.113.201", "last_disconnection": "2018-02-07T18:38:57.266Z", "last_connection": "2018-02-08T09:49:26.556Z", "id": "f0VMRgIBAQAAAAAAAAAAAA", "first_registration": "2018-01-31T17:10:59.270Z", "connected": true, "introspection": { "com.example.ExampleInterface" : { "major" : 1, "minor" : 0, "exchanged_msgs": 20, "exchanged_bytes": 200 }, "org.example.TestInterface" : { "major" : 0, "minor" : 2, "exchanged_msgs": 8, "exchanged_bytes": 147 } }, "aliases": { "name": "device_a" }, "attributes": { "attributes_key": "attributes_value" }, "groups": [ "my_group", ], "previous_interfaces": [ { "name": "com.example.ExampleInterface", "major" : 0, "minor" : 2, "exchanged_msgs": 3, "exchanged_bytes": 120 } ] } } Through the API, it is also possible to get the Introspection of the device only: GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces { "data": [ "com.example.ExampleInterface", "com.example.TestInterface" ] } This returns the Interfaces which the device reported in its Introspection and which are known to the Realm. Arbitrary information can be added to the device by means of attributes : they allow to store any number of string values associated to a corresponding string key. To set, modify and delete attributes , a PATCH on the device endpoint is required: PATCH api . < your astarte domain > / appengine / v1 / test / devices / f0VMRgIBAQAAAAAAAAAAAA In the request body, the data JSON object should have a attributes key which bears a dictionary of strings. A valid request body which changes only device attributes, for example, is {"data":{"attributes": {"<key>": "<value>"}}} . To delete an attribute entry, set the value of the corresponding key to null . For example, POSTing {"data":{"attributes": {"my_key": null}}} will remove the my_key attribute entry from the device. Depending on the aggregation and ownership of the Interface, you can GET / PUT / POST on the interface itself or one of its mappings, or use astartectl to perform the same operation on the command line. Some examples are: Get data from an aggregate device properties interface astartectl invocation: astartectl appengine devices data-snapshot f0VMRgIBAQAAAAAAAAAAAA com.example.ExampleInterface AppEngine API invocation: GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces/com.example.ExampleInterface Get last sent value from an individual device datastream interface astartectl invocation: astartectl appengine devices data-snapshot f0VMRgIBAQAAAAAAAAAAAA com.example.TestInterface AppEngine API invocation: GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces/com.example.TestInterface/myValue?limit=1 Set values in an individual server datastream interface astartectl invocation: astartectl appengine devices send-data f0VMRgIBAQAAAAAAAAAAAA com.example.OtherTestInterface /myOtherValue <value> AppEngine API invocation: POST api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces/com.example.OtherTestInterface/myOtherValue Request body: {"data": <value>} API Query semantics In general, to query AppEngine, the following things must be kept in mind When sending data, use PUT if dealing with properties , POST if dealing with datastream . When GET ting, if you are querying an aggregate interface, make sure to query the interface itself rather than its mappings. When GET ting datastream , keep in mind that AppEngine's default behavior is to return a large as possible timeseries.","ref":"050-query_device.html#device-status"},{"type":"extras","title":"Querying a Device - Navigating and retrieving Datastream results through APIs","doc":"The Datastream case is significant, as it might be common to have a lot of values for each endpoint/interface. As such, returning all of them in a single API call is most of the times not desirable nor recommended. To avoid putting the cluster under excessive pressure, AppEngine API is configured with a hard cap on the maximum number of returned results for each single call, with a sane default of 10000 . Although this hard cap is entirely configurable, please be aware that AppEngine API is designed to process a lot of reasonably small requests in the shortest possible time, and hence is not optimised nor strongly tested against big requests . Make sure that AppEngine API has enough resources available to cope with the maximum dataset size. AppEngine API provides you with a variety of mechanisms to make retrieval and navigation of large data sets as smooth and efficient as possible. Limit Adding a limit=n to the URL query tells AppEngine to return no more than n results. This acts similarly to a LIMIT SQL statement, but, as it stands, it does not impose a hard limit on the whole retrieved dataset but on the amount of the results displayed by the API call - see Pagination and Time Windows for more details on this topic and the performance implications of different limits in queries. If the specified limit is beyond the hard cap, the query won't fail, but will return at most the amount set by the hard cap, without further warnings. Since/To/Since After Results can be limited to a specific time window. since and to can be set to a ISO 8601 valid timestamp to limit on an upper and lower bound the result set. This can also be combined with limit to make sure that no more than n results are returned. Also, since and to can as well be set independently to provide only an upper or lower bound. In case you're dealing with a very large dataset and you want to dump it, it is likely that you need to go beyond what a reasonable default limit looks like. In those cases, you can use the since_after query parameter to retrieve parameters within a time window. since_after slices the time window just like since does, but it does not include values matching the specified timestamp, if any. This is especially useful when paginating, to start right after a returned result. Pagination and time windows AppEngine API provides you automatically with a time window-based pagination. When GET ting a datastream , if more results are available beyond the chosen time window/limit, a links map will be provided, in JSON-API style, to allow the user to paginate the results accordingly using since_after . You can use limit to determine each page's size. When specifying a valid limit , the links will keep the page size consistent over the next calls. However, limit should be used wisely to lower the pressure on the cluster. Each API call maps to a query that, no matter how efficient, has a computational cost. A few mid-sized queries should always be preferred over a large amount of smaller queries. Given your cluster is configured correctly, limit should be omitted in most cases when paginating, and you should rather trust your cluster's hard cap to be the sweet spot in efficiency and cluster pressure. Downsampling Especially when plotting graphs, retrieving all points in a time series isn't desirable. Astarte provides you with an implementation of the LTTB Downsampling Algorithm , which is used to return only a fixed number of samples from a time series. When setting downsample_to=n , AppEngine will return a maximum of n results, which are the most significant over the considered time series according to the algorithm. Due to how LTTB works, downsample_to must be >2 , as the algorithm will return the two ends of the considered value bucket, and n-2 values which are the picked samples. Please refer to the LTTB implementation used by Astarte to learn more about how this algorithm affects samples and its limitations. downsample_to=x can be used in conjunction with other query parameters, including limit=y . When doing so, Astarte will downsample to x samples the dataset composed of the last y values. Every feature previously outlined is in fact available with downsampling, including pagination - bear in mind, though, that for how the algorithm works, some options have drastically different semantic effects. Also, the hard cap has a very different meaning in downsampling. In this case, the hard cap applies to downsample_to instead of limit . limit can be an arbitrarly large amount of samples taken out of the DB, and can be used mainly to alleviate pressure in case of extremely large datasets which would require a lot of time for being processed by LTTB - even though, most of the time, you might want to define a time window to downsample instead. Astarte is also capable of downsampling aggregated interfaces, as long as a downsample_key is specified, which has to match the last token of an endpoint of the queried interface (i.e. in case the interface has a /%{id}/myValue mapping which should be used as the downsample_key , you should specify downsample_key=myValue in the query). When doing so, the aggregate will be downsampled using the chosen endpoint value as the y axis value, whereas its other endpoints will be disregarded when applying the algorithm. Please note that, no matter what downsample_key is used, a sample will be composed by the whole aggregation. If there is no way an interface can be downsampled (this is true, for example, if no downsample_key has been specified for aggregations , or for types such as strings ), AppEngine API will return a 4xx error. In general, downsampling is a powerful mechanism with a lot of limitations which really shines when plotting. Once again, this is a fundamental factor to consider when designing your interfaces .","ref":"050-query_device.html#navigating-and-retrieving-datastream-results-through-apis"},{"type":"extras","title":"Querying a Device - Real-Time Updates","doc":"The http REST API returns a static result, therefore API clients should either poll the REST API when displaying real-time changes or use a WebSocket. WebSockets are called Astarte Channels in Astarte jargon, and they should be considered as a more efficient alternative to polling. When using Astarte Channels the REST API should be used to retrieve the initial status.","ref":"050-query_device.html#real-time-updates"},{"type":"extras","title":"Querying a Device - astartectl-specific features","doc":"astartectl implements some convenience methods that make navigation easier. In particular, astartectl allows for any of the AppEngine API query parameters/mechanisms, but also implements automated pagination, snapshots and more. Data Snapshot astartectl has a unique feature that allows to retrieve a "Data Snapshot" of a device, namely the last known value for every interface available in the Device's introspection. This is extremely useful to have an at-a-glance view of the Device status with regards to data. Simply invoke astartectl appengine devices data-snapshot <device ID> , or astartectl appengine devices data-snapshot <device ID> <interface name> to get a snapshot for a single interface. Advanced querying astartectl appengine devices get-samples is astartectl 's frontend to advanced query. Refer to the command line documentation to learn about all available parameters, which match all of the parameters found in AppEngine API. The main difference is that, in case a query would break the boundaries of the page limit, astartectl will automatically paginate the request, and return all of the samples. Exporting Devices Data with astartectl The previous feature makes astartectl extremely useful when it comes to export or dump data. Moreover, get-samples features a --output option, which allows to print the results in different formats, such as json or CSV . This way, exporting values becomes extremely easy, as get-samples can easily tap into an Interface's entire data set and print it into a CSV file.","ref":"050-query_device.html#astartectl-specific-features"},{"type":"extras","title":"Querying a Device - Deleting a Device","doc":"If you want to entirely remove a device from your realm along with its data, it is possible to do so with the device deletion API. Note that the Astarte API also allows for the unregistration and the inhibition of a specific device, which should be enough in most use cases and do not cause any data loss. [!IMPORTANT] Once the deletion of a device is started, all data related to the device will be lost. Device deletion using Realm Management API Let's assume our Device has f0VMRgIBAQAAAAAAAAAAAA as its id. To delete it, use the following API. The deletion operation is asynchronous and can take up to 15 minutes. During the process, you device status will appear as "In deletion". DELETE api.<your astarte domain>/realmmanagement/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA","ref":"050-query_device.html#deleting-a-device"},{"type":"extras","title":"Using Astarte Channels (WebSockets)","doc":"Especially when building Frontend applications, it is useful to receive real-time updates about data sent from Devices. Astarte leverages Phoenix Channels to provide such a thing over WebSockets in AppEngine API. WebSockets can be used natively from a Web Browser and follow the same authentication pattern as a standard HTTP call. The AppEngine API for Astarte Channels is located at the URL: ws://<astarte-base-url>/appengine/v1/socket/websocket . Astarte Channels define a semantic on top of Phoenix Channels which allows read-only monitoring of device Interfaces. Authentication and Authorization over Channels happens in the very same way as AppEngine , and the a_ch claim in the token is respected when joining rooms and installing triggers. See Authentication and Authorization for more details on Auth semantics in Astarte.","ref":"052-using_channels.html"},{"type":"extras","title":"Using Astarte Channels (WebSockets) - Rooms","doc":"Rooms in Astarte Channels map 1:1 to Topics in Phoenix Channels, and can be joined in the very same way. Once a connection is established, the user can join any number of rooms, given he is authorized to do so . A Room is identified by a topic with the following semantics: rooms:<realm>:<name> . For example, rooms:test:myroom will join the Room myroom in the Realm test . A room can be joined by any number of concurrent users. Rooms serve as containers for Transient Triggers, which can be installed by any authorized user. Transient Triggers are actual Triggers , with the difference that they exist within a Channels Room rather than within a Realm - this mostly affects their timespan - and that the action can't be configured - every time a Condition is triggered an event is delivered to users in the Room. Events Everytime a Condition of an installed Trigger is triggered, an event is sent to the Phoenix Channel, with a similar payload: { "device_id": "f0VMRgIBAQAAAAAAAAAAAA", "event": { "type": "device_connected", "device_ip_address": "1.2.3.4" } } device_id is always present (as long as the trigger matches a device) and identifies the device emitting the event. event , instead, depends on the kind of installed trigger. It always carries a type string, which identifies the content of the object. The complete list of possible event payloads can be found here . Lifecycle Once a room is created, it remains valid and active with all of its subscriptions. There's little overhead in having a large number of rooms, as the only components leeching resources are Transient Triggers. As of today, Transient Triggers never expire - it is responsibility of the user to clean them up once the room becomes empty, if needed. In future versions, Transient Triggers will likely expire after some time, if left in an empty room.","ref":"052-using_channels.html#rooms"},{"type":"extras","title":"Using Astarte Channels (WebSockets) - Managing Transient Triggers","doc":"To install a Transient Trigger, one should issue a watch event in the Channel, given he is authorized to do so. The payload of such an event is identical to a Trigger definition, hence it looks like this: { "name": "datatrigger", "device_id": "f0VMRgIBAQAAAAAAAAAAAA", "simple_trigger": { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.6 } } This installs in the Room a Transient Trigger which will trigger an event everytime a value higher than 0.6 is sent on the path /streamTest/value of the datastream interface org.astarte-platform.genericsensors.Values by the device f0VMRgIBAQAAAAAAAAAAAA , and will be received by every user currently in the room. If a user isn't in the room at the time of the event, he will not get it, and there's no way he can retrieve it if he joined at a later time. Triggers can be uninstalled by issuing an unwatch event in the Channel. The payload of the event should be the name of the trigger which should be uninstalled. Group Triggers Transient triggers can also target an Astarte group instead of a single device. To install a group volatile trigger, pass the group_name key in the JSON payload instead of the device_id key. For example, the trigger below is equivalent to the one in the previous section, but it targets all devices that are in the group mygroup . { "name": "groupdevicetrigger", "group_name": "mygroup", "simple_trigger": { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.6 } } Note that the devices belonging to the group are evaluated when the trigger is installed, i.e. if a device is added to the group when the trigger is already installed, the trigger will not target the newly added device. The same goes for devices removed from the group, that will still be targeted by the trigger until it is removed.","ref":"052-using_channels.html#managing-transient-triggers"},{"type":"extras","title":"Using Astarte Channels (WebSockets) - Authorization","doc":"Just like any other Astarte component, Authorization is encapsulated in a token claim, in particular the a_ch claim. However, the mechanism is rather different compared to a REST API, and uses different verbs. JOIN The JOIN verb implies that a user can join a room. This only allows him to receive events and to interact in a read-only fashion with the room itself. There is no restriction to which events a user sees - if he is authorized to enter in a room, he will be capable of seeing all events flowing in. More granular permissions can be done simply by creating more rooms in which different triggers will be installed. The JOIN verb has the following semantic: JOIN::<regex> , where regex matches a room name (the room name is what follows rooms:<realm>: - the realm is implicit in the context of the authorization token). For example, a user authorized with the JOIN::test.* claim in the test realm will be able to join, for example, rooms:test:testthis , rooms:test:testme , rooms:test:test . The realm is always implicit in the regex, as the token is authenticated in the context of a Realm. WATCH The WATCH verb allows a user to install a Trigger within a room. Its semantics define which kind of trigger, and upon which entities the user is allowed to act. Watch semantics are WATCH::<regex> , where regex is a regular expression which matches a device, path or interface (or a mixture of them) in almost very same fashion as the a_aea claim (which is used in AppEngine). Given different kind of triggers impact different Astarte entities, the Authorization claim implicitly defines which kind of triggers a user will be able to install. For example, f0VMRgIBAQAAAAAAAAAAAA/org.astarte-platform.genericsensors.Values.* will allow installing data triggers such as the one shown in the previous example, but won't let the user install device-wide triggers (such as connect/disconnect events). A claim such as f0VMRgIBAQAAAAAAAAAAAA or f0VMRgIBAQAAAAAAAAAAAA.* , instead, will allow device-level triggers to be installed.","ref":"052-using_channels.html#authorization"},{"type":"extras","title":"Using Triggers","doc":"Triggers allow receiving notifications when a device connects, disconnects or publishes specific data. More details on Triggers can be found in the Architecture Documentation . Astarte allows you to install and delete Triggers dynamically through its clients. Upon installation or deletion, changes to the Trigger infrastructure might take some time to propagate, and some devices might pick up changes at a later time. If a Trigger shows as installed, it will eventually be loaded. This propagation can take up to 10 minutes.","ref":"060-using_triggers.html"},{"type":"extras","title":"Using Triggers - Listing Triggers","doc":"At any time, you can list existing Triggers in a Realm and fetch their details and definitions. Listing and querying Triggers using Astarte Dashboard After logging in, navigate to the Triggers page using the menu on the left. The list of Triggers installed in the Realm will be shown in the page. Clicking on a Trigger will open the Trigger editor in view-only mode, showing its definition on the right panel. Listing and querying Triggers using astartectl To list all existing Triggers in a Realm: $ astartectl realm-management triggers list [my_trigger other_trigger my_connection_trigger my_data_trigger] To get a Trigger definition: $ astartectl realm-management triggers show my_connection_trigger { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "<method>" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } Listing and querying Triggers using Realm Management API To list all existing Triggers in a Realm: GET api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers { "data": [ "my_trigger", "other_trigger", "my_connection_trigger", "my_data_trigger" ] } To get a Trigger definition: GET api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers/my_connection_trigger { "data": { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "<method>" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } }","ref":"060-using_triggers.html#listing-triggers"},{"type":"extras","title":"Using Triggers - Installing a Trigger","doc":"To install a Trigger, you need its JSON definition. If you have access to the Astarte Dashboard, you can use its Trigger Editor to build your JSON definition and install the Trigger directly. If you already have a JSON definition instead, you can either use astartectl or Realm Management APIs. The name of the Trigger must be unique within the Realm, or an error will be returned. Installing a Trigger using Astarte Dashboard After logging in, navigate to the Triggers page using the menu on the left. Click on "Install a new Trigger..." in the top-right corner. The Trigger Editor will open, and you can either paste/write a JSON definition, or use the declarative editor. When you are done, click on the "Install Trigger" button at the bottom of the declarative editor to install the Trigger in the Realm. Installing a Trigger using astartectl Assuming the Trigger definition is contained in my_connection_trigger.json , $ astartectl realm-management triggers install my_connection_trigger.json ok Installing a Trigger using Realm Management APIs POST api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers The POST request must have the following request body, with content type application/json { "data": { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "<method>" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } }","ref":"060-using_triggers.html#installing-a-trigger"},{"type":"extras","title":"Using Triggers - Deleting a Trigger","doc":"To delete a Trigger, you need to know its name. Just like when installing a Trigger, deleting a Trigger might not stop the data flow out of the Trigger immediately, which will eventually terminate at some point. Deleting a Trigger using Astarte Dashboard After logging in, navigate to the Triggers page using the menu on the left. Click on "Install a new Trigger..." in the top-right corner. Click on the Trigger name you want to delete. The Trigger Editor will open, and a "Delete" button will become available next to the Trigger name. Click on it to delete the Trigger. Deleting a Trigger using astartectl $ astartectl realm-management triggers delete my_connection_trigger ok Deleting a Trigger using Realm Management APIs DELETE api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers/my_connection_trigger","ref":"060-using_triggers.html#deleting-a-trigger"},{"type":"extras","title":"Using Triggers - Trigger examples","doc":"This section outlines two examples for the two main Trigger types (connection and data), and a sample payload for its HTTP Post URL action. Connection Trigger This trigger will send a POST request to <url> every time any device connects to its transport. This is the JSON representation of the trigger: { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "post" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } If the Trigger is installed, when a device connects, <url> will receive the following JSON payload: { "timestamp": "<timestamp>", "device_id": "<device_id>", "event": { "type": "device_connected", "device_ip_address": "<device_ip_address>" } } Data Trigger This trigger will send a POST request to http://www.example.com/hook every time a device sends data to the org.astarte-platform.genericsensors.Values major version 0 interface on the /streamTest/value path. This is the JSON representation of the trigger { "name": "my_data_trigger", "action": { "http_url": "http://www.example.com/hook", "http_method": "post" }, "simple_triggers": [ { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": "*" } ] } If the Trigger is installed, when a device sends data to the interface/path defined above, <url> will receive the following JSON payload: { "timestamp": "<timestamp>", "device_id": "<device_id>", "event": { "type": "incoming_data", "interface": "org.astarte-platform.genericsensors.Values", "path": "/streamTest/value", "value": <value> } }","ref":"060-using_triggers.html#trigger-examples"},{"type":"extras","title":"Using Triggers - Restricting triggers to a single device or group","doc":"Both device and data triggers accept the device_id and group_name keys to restrict a trigger to a single device or a single group. Triggers containing the device_id key will be triggered only for the specified device, while triggers containing the group_name key will be triggered only if the device is member of the group that is indicated in the group_name key. Note that when devices in a group are added or removed, the changes are not reflected immediately in group triggers. It can take up to 10 minutes to see the propagation of said changes.","ref":"060-using_triggers.html#restricting-triggers-to-a-single-device-or-group"},{"type":"extras","title":"Using Triggers - Trigger Delivery Policies","doc":"Trigger Delivery Policies allow customizing what Astarte is supposed to do in case of delivery errors on HTTP events and how to handle events that have not been delivered. A Trigger can be linked to one (at most) Trigger Delivery Policy by specifying the name of the policy in the "policy" field. If no Trigger Delivery Policies are specified, Astarte will resort to the default (pre v1.1) behaviour, i.e. ignoring delivery errors. Note that the Trigger Delivery Policy must be installed before installing a Trigger that is linked to it, or an error will be returned. The following is an example of a Connection Trigger linked to the simple_trigger_delivery_policy Trigger Delivery Policy: { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "post" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ], "policy" : "simple_trigger_delivery_policy" } Refer to the relevant documentation for more information on Trigger Delivery Policies.","ref":"060-using_triggers.html#trigger-delivery-policies"},{"type":"extras","title":"Using Trigger Delivery Policies","doc":"Note: Trigger Delivery Policies are an experimental feature, see Known Issues for more information about their current status. Trigger Delivery Policies allow customizing what Astarte is supposed to do in case of delivery errors on HTTP events and how to handle events that have not been delivered. More details on Trigger Delivery Policies can be found in the Architecture Documentation . Astarte allows you to install and delete Trigger Delivery Policies dynamically through its clients. A Trigger Delivery Policy is linked to an HTTP Trigger by specifying its name in the Trigger definition. A Trigger can be linked to at most one Trigger Delivery Policy, but a single Trigger Delivery Policy may serve any number of Triggers.","ref":"062-using_trigger_delivery_policies.html"},{"type":"extras","title":"Using Trigger Delivery Policies - Listing Trigger Delivery Policies","doc":"At any time, you can list existing Trigger Delivery Policies in a Realm and fetch their details and definitions. Listing and querying Trigger Delivery Policies using Realm Management API To list all existing Trigger Delivery Policies in a Realm: GET api.<your astarte domain>/realmmanagement/v1/<realm name>/policies { "data": [ "my_trigger_delivery_policy", "simple_trigger_delivery_policy", "other_trigger_delivery_policy", "retry_upon_all_errors_delivery_policy" ] } To get a Trigger Delivery Policy definition: GET api.<your astarte domain>/realmmanagement/v1/<realm name>/policies/simple_trigger_delivery_policy { "data": { "name" : "simple_trigger_delivery_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "any_error", "strategy" : "discard" } ] } }","ref":"062-using_trigger_delivery_policies.html#listing-trigger-delivery-policies"},{"type":"extras","title":"Using Trigger Delivery Policies - Installing a Trigger Delivery Policy","doc":"To install a Trigger Delivery Policy, you need its JSON definition. Then you can install the Trigger Delivery Policy via Realm Management APIs. The name of the Trigger Delivery Policy must be unique within the Realm, or an error will be returned. Installing a Trigger Delivery Policy using Realm Management APIs POST api.<your astarte domain>/realmmanagement/v1/<realm name>/policies The POST request must have the following request body, with content type application/json { "data": { "name" : "simple_trigger_delivery_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "any_error", "strategy" : "discard" } ] } }","ref":"062-using_trigger_delivery_policies.html#installing-a-trigger-delivery-policy"},{"type":"extras","title":"Using Trigger Delivery Policies - Deleting a Trigger Delivery Policy","doc":"To delete a Trigger Delivery Policy, you need to know its name. A Trigger Delivery Policy can be deleted only if no Triggers linked to it are present in the Realm. Deleting a Trigger Delivery Policy using Realm Management APIs DELETE api.<your astarte domain>/realmmanagement/v1/<realm name>/policies/simple_trigger_delivery_policy","ref":"062-using_trigger_delivery_policies.html#deleting-a-trigger-delivery-policy"},{"type":"extras","title":"Using Trigger Delivery Policies - Trigger Delivery Policy examples","doc":"This section outlines two examples of Trigger Delivery Policy. A Simple Trigger Delivery Policy The following Trigger Delivery Policy discards events on any error. This is the only behaviour previous Astarte versions (i.e. < 1.1) allowed. { "name" : "simple_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "any_error", "strategy" : "discard" } ] } A More Complex Trigger Delivery Policy The following policy has a different behaviour depending on whether the HTTP delivery error is a client or a server one. { "name" : "complex_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "server_error", "strategy" : "retry" }, { "on" : "client_error", "strategy" : "discard" } ], "retry_times" : 5, "event_ttl" : 10 } If an HTTP client error occurs, then Astarte will try to resend the event up to 5 times. If there occurs an HTTP server error, then Astarte will do nothing. At most 100 events can be in the queue at any time; if more than 100 events are present in the queue, the oldest ones will be deleted (even if they were resent less than 5 times in the case of HTTP client errors). If any event lasts for longer than 10 second in the queue, it will be discarded.","ref":"062-using_trigger_delivery_policies.html#trigger-delivery-policy-examples"},{"type":"extras","title":"Using Trigger Delivery Policies - Known issues","doc":"At the moment, trigger delivery policies in general do not provide a guarantee of in-order delivery of events. Refer to the Architecture Documentation for more information on this.","ref":"062-using_trigger_delivery_policies.html#known-issues"},{"type":"extras","title":"Managing Groups","doc":"Devices can be divided in groups to provide group-specific access to the APIs. The examples below will use astartectl but you can achieve the same results using AppEngine API.","ref":"065-managing-groups.html"},{"type":"extras","title":"Managing Groups - Reserved group prefixes","doc":"Some prefixes are reserved for internal use. It's not possible to create groups with a name starting with the ~ and @ characters.","ref":"065-managing-groups.html#reserved-group-prefixes"},{"type":"extras","title":"Managing Groups - Creating a group","doc":"You can create a group with astartectl with this command astartectl appengine groups create mygroup <device_identifier>,<device_identifier> device_identifier can be a Device ID or an Alias, and you can put multiple devices by separating them with a comma. You can check the group was created by listing groups in your realm astartectl appengine groups list","ref":"065-managing-groups.html#creating-a-group"},{"type":"extras","title":"Managing Groups - Adding or removing a device to/from a group","doc":"Once you created a group, you can add or remove devices from it. To add a device, use: astartectl appengine groups devices add <device_identifier> To remove a device, use: astartectl appengine groups devices remove <device_identifier> Keep in mind that a group exists as long as it has at least one device in it, so if you remove the last device from a group, the group will cease to exist. You can always check which devices are in a group with: astartectl appengine groups devices list","ref":"065-managing-groups.html#adding-or-removing-a-device-to-from-a-group"},{"type":"extras","title":"Managing Groups - Accessing Devices in a group with Astarte AppEngine API","doc":"Once a device is in a group, you can access to its data on this URL: https://<astarte-api>/appengine/v1/groups/<group_name>/devices/<device_id> The hierarchy is exactly the same that is found under https://<astarte-api>/appengine/v1/devices/<device-id> which is documented here , but this makes it possible to emit a JWT that only allows access to devices belonging to a specific group.","ref":"065-managing-groups.html#accessing-devices-in-a-group-with-astarte-appengine-api"},{"type":"extras","title":"Connect 3rd party applications","doc":"","ref":"070-connect_application.html"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana","doc":"Astarte Datasource Plugin conveys data from Astarte to Grafana , the open-source platform for monitoring and observability, developed by Grafana Labs . Actual data visualisation is provided by Grafana via its plugins . You can browse the source code of this plugin on its GitHub repository .","ref":"080-grafana_datasource.html"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Try it!","doc":"When deploying locally using docker-compose as mentioned in the Astarte in 5 mins tutorial , Astarte Datasource Plugin will be automatically installed. You may then access Grafana by visiting http://grafana.astarte.localhost . When first logging into Grafana, you will be prompted to change default credentials user admin , password: admin .","ref":"080-grafana_datasource.html#try-it"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Configure the datasource","doc":"In order to get data from Astarte, you will need to create a new datasource referring to your own Astarte instance. You will need to provide Astarte API URLs, the realm name and Astarte AppEngine token. Save the configuration by clicking on Save & Test","ref":"080-grafana_datasource.html#configure-the-datasource"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Setting up a graph","doc":"After successfully configuring your datasources, you will be able to select them as a source for your graph as depicted below. You will then be able to choose which device and interface data should be retrieved from.","ref":"080-grafana_datasource.html#setting-up-a-graph"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Data manipulation","doc":"Grafana offers data-aggregation features, for more information check the official Grafana documentation","ref":"080-grafana_datasource.html#data-manipulation"},{"type":"extras","title":"Troubleshooting","doc":"Be sure to check known issues to see if your problem is already covered there.","ref":"090-troubleshooting.html"},{"type":"extras","title":"Troubleshooting - Devices","doc":"Devices cannot connect to Astarte There might be some network issues or network misconfiguration Devices need a working network connection in order to communicate with Astarte. There might be some temporary network issues, or any network setting or appliance might not be properly configured. Make sure that devices are allowed to make outbound connections on ports 443 (https) and any port the transport needs for accepting connections from devices. For Astarte/VerneMQ, this defaults to 8883 (MQTT over SSL), but might also be configured otherwise. SSL issues Devices need to be able to connect to Astarte using SSL. Make sure that the clock has been synched to avoid certificate issue/expiry date errors, make also sure to have all the root CAs up to date. Device gets disconnected from Astarte Some interfaces might be missing When a device reports an interface that Astarte doesn't have, it gets disconnected when the introspection is published. Make sure that all device interfaces have been previously installed on Astarte. Make also sure that interface name and major exactly matches installed version. Device is publishing unexpected or malformed data When a device sends invalid, malformed or unexpected data it gets disconnected, make sure that the device is sending valid data. An interface mismatch might be the most common reason for this kind of issues. e.g. the interface has been declared device owned on the device, and astarte owned on astarte. Make sure to use exactly the same JSON file on both ends.","ref":"090-troubleshooting.html#devices"},{"type":"extras","title":"Troubleshooting - Triggers","doc":"Triggers are not executed Triggers have not been loaded yet Triggers might take some time before being loaded for devices that have been recently connected, make sure to wait some time before the triggers cache is populated again. If you are on a hurry make sure to test a trigger on a device that has not been recently connected yet.","ref":"090-troubleshooting.html#triggers"},{"type":"extras","title":"Known issues","doc":"This page collects some notable issues which affect Astarte v1.0 . This is by no means an exhaustive list and you should also check Github issues to see if your problem is already covered there.","ref":"095-known_issues.html"},{"type":"extras","title":"Known issues - Realm deletion","doc":"Astarte v1.0 introduces the possibility of deleting a Realm, but currently devices which have a valid certificate are not disconnected from the Realm. The issue is tracked here . Since publishing with devices on unexisting realms can cause problems (namely: RabbitMQ data queues filling up) which can also impact devices on other realms, a realm should be deleted only after ensuring that no devices are connected to it. Due to the problems that realm deletion can cause, currently the feature must be explicitly enabled with a feature gate, i.e. by adding features: realmDeletion: true components: ... to the Astarte Custom Resource (which maps to setting the HOUSEKEEPING_ENABLE_REALM_DELETION environment variable to true in the astarte_housekeeping container). We also encourage avoiding to recreate realms with the same name to avoid having devices from the old realm reconnecting back to the new one.","ref":"095-known_issues.html#realm-deletion"},{"type":"extras","title":"Known issues - Group permissions","doc":"Currently Astarte authorization mechanism doesn't allow permissions on groups with a device granularity. Specifically there's no way to authorize a user to add only specific devices to a group. The issue is tracked here . This means that right now the best way to allow users to add or remove devices from a group they have access to is to provide a backend which is able to perform the necessary authorization checks and then performs the necessary additions/removals, while the user only has a read access to the group resource. In the long term a minor semantic change is going to employed, therefore currently we discourage emitting long living tokens which allow a non-root user to manage groups (i.e. create and modify them) since the current tokens could become incompatible with future changes.","ref":"095-known_issues.html#group-permissions"},{"type":"extras","title":"Known issues - Ghost connected devices","doc":"In some circumstances, prior to Astarte v1.0 , a device might be mistakenly reported as connected. This bug has been fixed in v1.0 , however this bug may still affect devices that have connected last time while using v0.11 (prior to the upgrade to v1.0 ). This issue is likely to happen when upgrading to v1.0 since it might be caused by VerneMQ shutdown. A device reconnection fixes this issue, and the connection state will always be reliably reported.","ref":"095-known_issues.html#ghost-connected-devices"},{"type":"extras","title":"Managing Realms","doc":"Once the Cluster is set up, you can start managing it by creating Realms.","ref":"070-manage_realms.html"},{"type":"extras","title":"Managing Realms - Accessing Housekeeping key","doc":"When creating a new Cluster, Astarte Operator also creates a brand new keypair and stores it in the cluster. To retrieve it (assuming you deployed an instance named astarte in namespace astarte ): kubectl get secret -n astarte astarte-housekeeping-private-key -o=jsonpath={.data.private-key} | base64 -d > housekeeping.key You can then use housekeeping.key to authenticate against Housekeeping API.","ref":"070-manage_realms.html#accessing-housekeeping-key"},{"type":"extras","title":"Managing Realms - Device limit in a Realm","doc":"While there is no limit on the number of Devices registered in a Realm, it is possible to impose an upper bound using the <astarte base API URL>/housekeeping/v1/realms/<realm name> API. By default, such an upper bound does not exist. The limit can be retrieved, updated or removed (meaning that any number of registered Devices is allowed). Setting, updating or removing the limit on the number of registered Devices in a Realm PATCH <astarte base API URL>/housekeeping/v1/realms/<realm name> The HTTP payload of the request must have the following format: { "data": { "device_registration_limit": <value> } } To set or update the limit, the value must be a non negative integer. If the value is less than the number of currently registered devices, no devices will be removed, but it will not be possible to register new devices. When it is set, trying to register more Devices past the limit will result in an error. To remove the limit, the value must be null . Fetching the limit on the number of registered Devices in a Realm GET <astarte base API URL>/housekeeping/v1/realms/<realm name> The HTTP payload of the response will have the following format: { "data": { "realm_name": <realm name>, "jwt_public_key_pem": <realm public key>, "replication_factor": <realm replication factor>, "device_registration_limit": <realm device registration limit> } } If no limit is currently set, the value of the "device_registration_limit" field will be null .","ref":"070-manage_realms.html#device-limit-in-a-realm"},{"type":"extras","title":"Managing Realms - Work in progress","doc":"This guide is not yet complete, as this part is a moving target within astartectl . Please refer to the API Documentation to manage Realms manually once here.","ref":"070-manage_realms.html#work-in-progress"},{"type":"extras","title":"Monitoring","doc":"Astarte is a complex, distributed system that may pose several challenges when deployed in production. Individual services report health and metrics to ensure production clusters can be properly monitored and proactive actions can be taken in case of faults or unexpected behavior.","ref":"090-monitoring.html"},{"type":"extras","title":"Monitoring - Health checks","doc":"Every Astarte service, whether it's an API service or not, exposes an HTTP endpoint /health , without versioning, on its HTTP port. By default, services use port 4000 . /health is meant to be called frequently and reports the individual health state of a service. It will return 200 in case the service is healthy, or other errors in case the service is having issues. Among those issues, there might be failure in accessing RabbitMQ/RPC communications or failure in accessing the Database. Health checks and Kubernetes The aforementioned health checks are integrated in Kubernetes, when using Astarte Operator, as LivenessProbe and ReadinessProbe . As such, health monitoring and forced restarts are automatically handled without the need for the administrator to integrate any additional logic.","ref":"090-monitoring.html#health-checks"},{"type":"extras","title":"Monitoring - Service metrics","doc":"Just like /health , every service exposes a /metrics endpoint. This endpoint exposes a series of metrics in Prometheus format, which can be easily integrated and queried from any Prometheus-compatible monitoring solution. Each service, besides exposing stats on its Erlang VM, resource consumption and HTTP stats (where applicable), also exposes a number of service-specific metric, which can be queried to obtain information about Astarte's usage and behavior. Authentication and access to metrics /metrics , being Prometheus-compatible, does not implement any kind of authentication or access control. Ideally, only your scraper should have access to /metrics , as it can leak sensitive information and should not be exposed to the outer world. Astarte Operator, by default, forbids access to /metrics through its ingress, as it assumes your scraper lives within the Kubernetes cluster or has means to access the cluster on its own. However, this behavior can be overridden through by setting serveMetrics: true in the api section. An additional parameter, serveMetricsToSubnet , can be specified to restrict access to /metrics only to source IPs in a specific subnet. It is strongly recommended to set this up in case an external scraper needs to have access to /metrics , to ensure access is restricted.","ref":"090-monitoring.html#service-metrics"},{"type":"extras","title":"Advanced operations","doc":"This section provides guidance on some delicate operations that must be performed manually as they could potentially result in data loss or other types of irrecoverable damage. Always be careful while performing these operations!","ref":"095-advanced-operations.html"},{"type":"extras","title":"Advanced operations - Manual deletion of interfaces","doc":"Right now, Astarte only allows deleting draft interfaces, i.e. interfaces with major version 0 and not used by any device. If you want to delete an interface that has already published data, you must proceed manually with the steps described below. This guide assumes the aim of the operation is deleting the org.astarte-platform.genericsensors.Values interface in the test realm. The guide requires that cqlsh is connected to the Cassandra/ScyllaDB instance that your Astarte instance is using. Switch to the target keyspace The keyspace has the same name of the realm, in this case it is test : cqlsh > use test ; Find out the interface id cqlsh :test > SELECT interface_id FROM interfaces WHERE name = 'org.astarte-platform.genericsensors.Values' AND major_version = 1 ; cqlsh will reply with the interface id: interface_id -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac Delete the interface WARNING: This is a destructive step that will erase the correlation between the Interface name and internal ID. Before proceeding, ensure you saved the interface ID, or you will end up with dangling data. Further steps in this guide will require the interface ID. To delete the interface, cqlsh :test > DELETE FROM interfaces WHERE name = 'org.astarte-platform.genericsensors.Values' AND major_version = 1 ; Keep in mind that after this step, all existing devices that attempt to publish on this interface will be disconnected as soon as they try to do so. Delete interface data The interface data is stored in a different place depending on the interface type ( datastream or properties ) and aggregation. Individual datastream interfaces store their data in the individual_datastreams table. Individual properties interfaces store their data in the individual_properties table. Object datastream interfaces store their data in a dedicated table which is created starting from the interface (e.g. an interface called com.test.Sensors with major version 1 creates a com_test_sensors_v1 table in the realm keyspace). To delete data from object datastreams, a simple DROP of the table where the data is stored is needed. Deleting data from individual interfaces requires more steps. In this example the interface is an individual datastream, but the procedure for individual properties is the same, but concerns the individual_properties table instead. To delete the interface data, first all relevant primary keys must be found: cqlsh :test > SELECT DISTINCT device_id , interface_id , endpoint_id , path FROM individual_datastreams WHERE interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac ALLOW FILTERING ; This will return a set of primary keys of data belonging to that interface: device_id | interface_id | endpoint_id | path -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- - 41 c1c072 - d416 - 4686 - ba23 - 673 fe4ad926f | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / test / value 81 c60277 - 4645 - 441 f - a49b - 66 a71ce54b83 | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / foo / value ... After that, all data belonging to those primary keys must be deleted: cqlsh :test > DELETE FROM individual_datastreams WHERE device_id = 41 c1c072 - d416 - 4686 - ba23 - 673 fe4ad926f AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac AND endpoint_id = 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 AND path = '/test/value' ; cqlsh :test > DELETE FROM individual_datastreams WHERE device_id = 81 c60277 - 4645 - 441 f - a49b - 66 a71ce54b83 AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac AND endpoint_id = 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 AND path = '/foo/value' ; ... devices-by-interface cleanup If this guide is being used so as to remove a draft interface (i.e. with major version 0 ) that cannot be deleted since it has data on it, an additional step is required for a complete cleanup. The information about which devices are using draft interfaces is kept in the kv_store table. You can inspect the groups with: cqlsh :test > SELECT group FROM kv_store ; The group that has to be deleted may be easily identified by inspecting the returned group s, since it is the one with its name derived from the interface name. For example, if the purpose of the operation is removing all data from the org.astarte-platform.genericevents.DeviceEvents v0.1 interface, the corresponding group in kv_store will be devices-by-interface-org.astarte-platform.genericevents.DeviceEvents-v0 . As the target group is identified, just remove all its entries with: cqlsh :test > DELETE FROM kv_store WHERE group = 'devices-by-interface-org.astarte-platform.genericevents.DeviceEvents-v0' ; Conclusions After performing all the steps above, the interface will be completely removed from Astarte. You can then proceed to install a new interface with the same name and major version without any conflict. Remember to remove the interface also on the device side, otherwise devices will keep getting disconnected if they try to publish on the deleted interface.","ref":"095-advanced-operations.html#manual-deletion-of-interfaces"},{"type":"extras","title":"Advanced operations - Manual deletion of a device","doc":"Starting from Astarte 1.2, a device can be deleted using Realm Management API (see here ). The following manual procedure remains for historical reasons. This section assumes: that cqlsh is connected to the Cassandra/ScyllaDB instance that your Astarte is using; that astartectl is installed on your machine. Please keep in mind that this is a destructive procedure. Before moving on, ensure you saved your device ID or you might end up with dangling data and possibly a damaged Astarte deployment. Retrieve the device uuid To interact with the device and its data, the device uuid must be retrieved. Assuming that the id of the device to be deleted is k3oPTXaGGGGGGGGGGGGGGG , its uuid can be obtained with the following: $ astartectl utils device-id to-uuid k3oPTXaGGGGGGGGGGGGGGG 937a0f4d-7686-1861-8618-618618618618 Please, make sure not to lose the device uuid as it will be employed in all the following steps of this section. Switch to the target keyspace The keyspace takes its name from the realm, in this case it is test . cqlsh > use test ; Delete device data on a specific interface Depending on the interface type and aggregation, data published by the device is stored into different tables: data published over an individual datastream interface are available within the individual_datastreams table; data published over an individual property interface are available within the individual_properties table; data published over an object datastream interfaces are stored in a dedicated table named after the interface name: e.g. an interface called com.test.Sensors with major version 1 creates a com_test_sensors_v1 table in the realm keyspace. Delete device data from an individual_datastreams interface The first step consists in finding all the relevant primary keys for the device. To do so, simply run: cqlsh :test > SELECT DISTINCT device_id , interface_id , endpoint_id , path FROM individual_datastreams WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; The output will show a set of primary keys of data belonging to your device: device_id | interface_id | endpoint_id | path -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- - 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / test / value 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | 1 e6fb841 - 9 ee3 - 0 e60 - 72 ed - 1 f55b334b832 | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / foo / value ... It is now time to perform the actual data deletion: to do so, repeat the following instruction iterating over every combination of primary keys as obtained from the output of the previous command: cqlsh :test > DELETE FROM individual_datastreams WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac AND endpoint_id = 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 AND path = '/test/value' ; Delete device data from an individual_properties interface The first step consists in retrieving the primary keys for the device. Just run: cqlsh :test > SELECT DISTINCT device_id , interface_id FROM individual_properties WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; The output will be similar to the following one: device_id | interface_id -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | 8 ed086db - 0 bcc - 5 a9f - 2 fc2 - ddf49c35e87d 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | c61879ce - c60c - adaf - c6b4 - d04b1e1b14c4 To perform the actual data deletion, run the following query for each pair of device_id and interface_id obtained from the previous query: cqlsh :test > DELETE FROM individual_properties WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac ; Delete device data for object datastreams The first step consists in retrieving the primary keys for the device. For this particular example the sample interface named com.test.Sensors with major version v1 is employed. Please note that the upcoming steps must be repeated for each object datastream interface installed in your realm. cqlsh :test > SELECT DISTINCT device_id , path FROM com_test_sensors_v1 WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; The output will show something like: device_id | path -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + -- -- -- 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | / foo ... It is now time to perform the actual data deletion: cqlsh :test > DELETE FROM com_test_sensors_v1 WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 AND path = '/foo' ; Delete device aliases If your device has one or more aliases you will find them in the names table. First, you have to find the primary key for the device: cqlsh :test > SELECT object_name FROM names WHERE object_uuid = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; If your device has any aliases, the output will show object_name -- -- -- -- -- -- -- -- my - device - alias ... Thus, you can delete the alias simply executing: cqlsh :test > DELETE FROM names WHERE object_name = 'my-device-alias' ; Delete the device from groups To delete the device from a device group let's find the needed keys: SELECT group_name , insertion_uuid , device_id FROM grouped_devices WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; If the device is contained in one or more groups, the output will be: group_name | insertion_uuid | device_id -- -- -- -- -- -- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- my - group | c1a0dade - 43 bc - 11 ec - 95 be - 41 f7663270b3 | 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ... The actual deletion can be performed with: cqlsh :test > DELETE FROM grouped_devices WHERE group_name = 'my-group' AND insertion_uuid = c1a0dade - 43 bc - 11 ec - 95 be - 41 f7663270b3 AND device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ; Delete entries from kv_store If your device is publishing over one or more interfaces with version v0 , you will need to handle also the kv_store table. Retrieve all the entries that must be handled: cqlsh :test > SELECT group , key FROM kv_store WHERE key = 'k3oPTXaGGGGGGGGGGGGGGG' ALLOW FILTERING ; The output of the query will show something similar to group | key -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- devices - by - interface - com . test . Sensor - v0 | k3oPTXaGGGGGGGGGGGGGGG devices - with - data - on - interface - com . test . Sensor - v0 | k3oPTXaGGGGGGGGGGGGGGG ... To remove the entries, simply execute the following queries to remove the proper rows from the table. Please, make sure to remove all the entries referencing your device ID. cqlsh :test > DELETE FROM kv_store WHERE group = 'devices-by-interface-com.test.Sensor-v0' AND key = 'k3oPTXaGGGGGGGGGGGGGGG' ; cqlsh :test > DELETE FROM kv_store WHERE group = 'devices-with-data-on-interface-com.test.Sensor-v0' AND key = 'k3oPTXaGGGGGGGGGGGGGGG' ; Eventually delete your device Deleting your device from the devices table is as simple as cqlsh :test > DELETE FROM devices WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ; Conclusions If you managed to remove all the device-related entries as described in the previous sections, then your device and its data have been properly deleted from Astarte. Before trying to reconnect your device you must make sure that the SSL certificate and all the credentials onboard your device are deleted. This is crucial for ensuring that new data published by the device can be properly ingested and processed by Astarte.","ref":"095-advanced-operations.html#manual-deletion-of-a-device"},{"type":"extras","title":"Astarte in 5 minutes","doc":"This documentation page describes a development version, for production systems please use the stable version instead. This tutorial will guide you through bringing up your Astarte instance, creating a realm and streaming your first data from a device simulator (or a real device) before your cup of tea is ready.","ref":"010-astarte_in_5_minutes.html"},{"type":"extras","title":"Astarte in 5 minutes - Before you begin","doc":"First of all, please keep in mind that this setup is not meant to be used in production : by default, no persistence is involved, the installation does not have any recovery mechanism, and you will have to restart services manually in case something goes awry. This guide is great if you want to take Astarte for a spin, or if you want to use an isolated instance for development. You will need a machine with at least 4GB of RAM, a recent 64-bit operating system with Docker , Docker Compose and astartectl installed. If you don't have astartectl installed on your machine yet, you should install it by following the instructions in astartectl's README Also, on the machine(s) or device(s) you will use as a client, you will need either Docker, or a Qt5 installation with development components if you wish to build and run components locally. Due to ScyllaDB requirements, if you're working on a Linux machine you should make sure that aio-max-nr is at least 1048576 : cat /proc/sys/fs/aio-max-nr 1048576 If it's less than that, you'll need to edit your /etc/sysctl.conf file fs . aio - max - nr = 1048576 and to persist this configuration sudo sysctl -p","ref":"010-astarte_in_5_minutes.html#before-you-begin"},{"type":"extras","title":"Astarte in 5 minutes - Checking prerequistes","doc":"Docker version >= 19 is recommended: $ docker -v Docker version 19.03.8 Docker compose version >= 2.21 is recommended: $ docker compose version Docker Compose version v2.21.0 astartectl >= 22.11 is recommended: $ astartectl version astartectl 22.11.02 This procedure has been tested on several systems, and is validated and maintained against Ubuntu 22.04 and macOS 10.15 Catalina, but any other modern operating system should work.","ref":"010-astarte_in_5_minutes.html#checking-prerequistes"},{"type":"extras","title":"Astarte in 5 minutes - Install Astarte","doc":"To get our Astarte instance running as fast as possible, we will install Astarte's standalone distribution. It includes a tunable Docker Compose which brings up Astarte and every companion service needed for it to work. To do so, simply clone Astarte's main repository and use its scripts to bring it up: $ git clone https://github.com/astarte-platform/astarte.git -b v1.2.0 && cd astarte $ docker run -v $(pwd)/compose:/compose astarte/docker-compose-initializer:1.2.0 $ docker compose pull $ docker compose up -d docker-compose-initializer will generate a root CA for devices, a key pair for Housekeeping, and a self-signed certificate for the broker (note: this is a really bad idea in production). You can tune the compose file further to use legitimate certificates and custom keys, but this is out of the scope of this tutorial. Compose might take some time to bring everything up, but usually within a minute from the containers creation Astarte will be ready. You can reach Astarte at the following addresses: api.astarte.localhost : Astarte API, in detail: api.astarte.localhost/appengine : AppEngine api.astarte.localhost/housekeeping : Housekeeping api.astarte.localhost/pairing : Pairing api.astarte.localhost/realmmanagement : Realm Management broker.astarte.localhost : VerneMQ broker dashboard.astarte.localhost : Astarte Dashboard Moreover, Compose will forward the following ports to your machine: 80 : HTTP 8883 : MQTTS To check everything went fine, use docker ps to verify relevant containers are up: Astarte itself, VerneMQ, CFSSL, RabbitMQ and ScyllaDB should be now running on your system. If any of them isn't up and running, docker ps -a should show it stopped or failed. In those cases, it is advised to issue docker compose up -d again to fix potential temporary failures.","ref":"010-astarte_in_5_minutes.html#install-astarte"},{"type":"extras","title":"Astarte in 5 minutes - Create a Realm","doc":"Now that we have our instance up and running, we can start setting up a Realm for our device. We'll call our Realm test . Given we have no SSO or Authentication mechanism set up, we're just going to generate a public key to sign our JWTs with. You can create one with astartectl : $ astartectl utils gen-keypair test Also, we will need a JWT token to authenticate against Housekeeping. generate-compose-files.sh created a keypair automatically, which is in compose/astarte-keys/housekeeping_{private,public}.pem . To perform all of our Astarte interactions, we will use astartectl . Use astartectl to create a new Realm: $ astartectl housekeeping realms create test --astarte-url http://api.astarte.localhost --realm-public-key test_public.pem -k compose/astarte-keys/housekeeping_private.pem This creates a test realm, which should be ready to be used almost immediately. To ensure your realm is available and ready, check if it exists in Astarte by issuing: $ astartectl housekeeping realms ls --astarte-url http://api.astarte.localhost -k compose/astarte-keys/housekeeping_private.pem","ref":"010-astarte_in_5_minutes.html#create-a-realm"},{"type":"extras","title":"Astarte in 5 minutes - Install interfaces","doc":"We will use Astarte's Qt5 Stream Generator to feed data into Astarte. However before starting, we will have to install the org.astarte- platform.genericsensors.Values and the org.astarte-platform.genericcommands.ServerCommands interfaces into our new realm. To do that, we can use astartectl again: $ astartectl realm-management interfaces sync standard-interfaces/org.astarte-platform.genericsensors.Values.json standard-interfaces/org.astarte-platform.genericcommands.ServerCommands.json --astarte-url http://api.astarte.localhost -r test -k test_private.pem -y Now org.astarte-platform.genericsensors.Values and org.astarte- platform.genericcommands.ServerCommands should show up among our available interfaces: $ astartectl realm-management interfaces ls --astarte-url http://api.astarte.localhost -r test -k test_private.pem Our Astarte instance is now ready for our devices.","ref":"010-astarte_in_5_minutes.html#install-interfaces"},{"type":"extras","title":"Astarte in 5 minutes - Install a trigger","doc":"We will also test Astarte's push capabilities with a trigger. This will send a POST to a URL of our choice every time the value generated by stream_test is above 0.6. Due to how triggers work, it is fundamental to install the trigger before a device connects. Doing otherwise will cause the trigger to kick in at a later time, and as such no events will be streamed for a while. Replace $TRIGGER_TARGET_URL with your target URL in the example below, you can use a Postbin service like Mailgun Postbin to generate a URL and see the POST requests. The resulting trigger would be: { "name": "my_trigger", "action": { "http_url": "$TRIGGER_TARGET_URL", "http_method": "post" }, "simple_triggers": [ { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 1, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.6 } ] } Replace $TRIGGER_TARGET_URL with the URL your Trigger will target. Assuming you saved this as my_trigger.json , you can now install it through astartectl : $ astartectl realm-management triggers install my_trigger.json --astarte-url http://api.astarte.localhost -r test -k test_private.pem You can now check that your trigger is correctly installed: $ astartectl realm-management triggers ls --astarte-url http://api.astarte.localhost -r test -k test_private.pem","ref":"010-astarte_in_5_minutes.html#install-a-trigger"},{"type":"extras","title":"Astarte in 5 minutes - Stream data","doc":"If you already have an Astarte compliant device, you can configure it and connect it straight away, and it will just work with your new installation - provided you skip SSL checks on the broker's certificate. If you don't, you can use Astarte's stream-qt5-test to emulate an Astarte device and generate a datastream . You can do this either on the same machine where you are running Astarte, or from another machine or device on the same network. Using a container for stream-qt5-test Astarte's stream-qt5-test can be pulled from Docker Hub with: $ docker pull astarte/astarte-stream-qt5-test:1.0.4 Its most basic invocation (from your astarte repository tree) is: $ docker run --net="host" -e "DEVICE_ID=$(astartectl utils device-id generate-random)" -e "PAIRING_URL=http://api.astarte.localhost/pairing" -e "REALM=test" -e "PAIRING_JWT=$(astartectl utils gen-jwt pairing -k test_private.pem)" -e "IGNORE_SSL_ERRORS=true" astarte/astarte-stream-qt5-test:1.0.4 This will generate a random datastream from a brand new, random Device ID. You can tweak those parameters to whatever suits you better by having a look at the Dockerfile. You can spawn any number of instances you like, or you can have the same Device ID send longer streams of data by saving the container's persistency through a Docker Volume. If you wish to do so, simply add -v /persistency:<your persistency path> to your docker run invocation. Refer to stream-qt5-test README for more details on which variables can be passed to the container. Also, please note that the --net="host" parameter is required to make localhost work. If this is not desirable, you can change PAIRING_URL to an host reachable from within the container network. Obviously, that parameter isn't required if you're running the container on a different machine and PAIRING_URL is pointing to a different URL.","ref":"010-astarte_in_5_minutes.html#stream-data"},{"type":"extras","title":"Astarte in 5 minutes - Grab your tea","doc":"Congratulations! Your devices or fake devices are now communicating with Astarte, and your tea should be ready by now. You can check if everything is working out by invoking AppEngine APIs to get some values. In case you are using stream-qt5-test , you can get the last sent value with astartectl : $ astartectl appengine devices get-samples <your device id> org.astarte-platform.genericsensors.Values /streamTest/value --count 1 --astarte-url http://api.astarte.localhost -r test -k test_private.pem If you get a meaningful value, congratulations - you have a working Astarte installation with your first datastream coming in! Moreover, Astarte's Docker Compose also installs Astarte Dashboard , from which you can manage your Realms and install Triggers, Interfaces and more from a Web UI. It is accessible by default at http://dashboard.astarte.localhost - remember that if you are not exposing Astarte from localhost , you have to change Realm Management API's URL in Dashboard's configuration file, to be found in compose/astarte-dashboard/config.json in Astarte's repository. You can generate a token for Astarte Dashboard, as usual, through astartectl utils gen-jwt all-realm-apis -k test_private.pem . By default, astartectl will generate a token valid for 8 hours, but you can set a specific expiration by using the -e <seconds> parameter. From here on, you can use all of Astarte's APIs and features from your own installation. You can add devices, experiment with interfaces, or develop your own applications on top of Astarte's triggers or AppEngine's APIs. And have a lot of fun!","ref":"010-astarte_in_5_minutes.html#grab-your-tea"},{"type":"extras","title":"Astarte in 5 minutes - Cleaning up","doc":"When you're done with your tests and developments, you can use docker compose again to tear down your Astarte instance simply by issuing: $ docker compose down Unless you add the -v option, persistencies will be kept and next time you will docker compose up the cluster will come back in the very same state you left it last time. docker compose down -v is extremely useful during development, especially if you want a clean slate for testing your applications or your routines every time.","ref":"010-astarte_in_5_minutes.html#cleaning-up"},{"type":"extras","title":"Astarte in 5 minutes - Final notes","doc":"Running Astarte through docker compose is the fastest way for going from zero to hero. However, please keep in mind this setup is unlikely to hold for long in production, and is by design broken for large installations . We can't stop you from running such a thing in production, but do so as long as you know you voided your warranty by doing so. This method is great for development and for trying out the system. If you wish to deploy Astarte in a more robust environment, have a look at Astarte Enterprise or, if you want to go the DIY way, make sure that at least every service which requires persistency has reliable storage and adequate redundancy beneath it.","ref":"010-astarte_in_5_minutes.html#final-notes"},{"type":"extras","title":"Introduction","doc":"","ref":"001-intro_api.html"},{"type":"extras","title":"Introduction - REST API","doc":"Astarte's APIs are documented through Swagger . Your Astarte installation probably already has Swagger UI support, which serves as the reference for your installed APIs. To browse API documentation online, follow this link .","ref":"001-intro_api.html#rest-api"},{"type":"extras","title":"Introduction - WebSocket Astarte Channels","doc":"The AppEngine API for Astarte Channels is located at the URL: ws://<astarte-base-url>/appengine/v1/socket/websocket . To learn how to use the Channels API, you can read the documentation at: Using Astarte Channels (WebSockets)","ref":"001-intro_api.html#websocket-astarte-channels"}]
\ No newline at end of file
+searchNodes=[{"type":"extras","title":"Introduction","doc":"This documentation page describes a development version, for production systems please use the stable version instead. Astarte is a collection of components written in Elixir meant to orchestrate and pilot a number of 3rd party components. These components include: One or more ingresses (the most popular implementation being an MQTT broker) An AMQP broker for handling messages and queues between Astarte's services A Cassandra-like Database for ingesting and retrieving data (currently Cassandra and ScyllaDB are both supported) These components are never directly exposed to Astarte's end user, who requires no knowledge whatsoever of the mentioned frameworks - they are rather orchestrated and managed directly by Astarte's services. It is, however, responsability of Astarte's administrators to make sure these services are made available the way they are meant to. For more details on this topic and, in general, on how to deal with Astarte's installation and maintenance, please refer to the Administrator Guide .","ref":"001-intro_architecture.html"},{"type":"extras","title":"Design Principles","doc":"Astarte has a strongly opinionated design aimed at the generic IoT / data-driven use case. As such, and unlike other platforms, it strives to streamline a very simple user workflow for ingesting, distributing and retrieving data, built on a set of concepts and principles.","ref":"010-design_principles.html"},{"type":"extras","title":"Design Principles - Declarative vs. Explicit Data Management","doc":"Astarte does not allow exchanging raw data - it rather forces the user to describe data before it is sent into the platform. Data is described with a mechanism named Interfaces, explained in detail in the user guide . Through Interfaces, Astarte creates and maintains a data model autonomously, sparing the user from the complexity of dealing with Databases and Data Management in general.","ref":"010-design_principles.html#declarative-vs-explicit-data-management"},{"type":"extras","title":"Design Principles - AMQP as internal API mechanism","doc":"Astarte services use a Protobuf-based API to exchange data over AMQP in a gRPC like fashion. As such, as long as a service conforms with the policies defined by the queues, it is possible to extend Astarte in virtually any language that can deliver a compliant AMQP client.","ref":"010-design_principles.html#amqp-as-internal-api-mechanism"},{"type":"extras","title":"Design Principles - Device ID","doc":"Astarte identifies each device with a 128 bit Device ID which has to be unique within its Realm. As a best practice, it is advised to generate such an ID from hardware unique IDs or using dedicated hardware modules, to make it consistent across device reflashes. It is advised to use a cryptographic hash function (such as sha256) when generating it using a software module. Astarte will use URL encoded base64 (without padding) strings like V_zv6ThCCtXWveQ8mPjsKg in its representation. Although not required, it is strongly advised to use UUIDs as Astarte Device IDs. In fact, Astarte Device ID's specification is 100% compatible with UUIDs Base64 encoded adhering to RFC 7515. In the same fashion, UUIDv5 can be used to generate a deterministic Device ID from any kind of input data. Astarte Clients which generate Astarte Device IDs (such as astartectl or Astarte Dashboard) will always generate a Device ID out of UUIDv4 (random ID) or UUIDv5 (deterministic ID). This detail is relevant not only for identifying and querying the device, but also for the Pairing mechanism , as a device's credentials are associated to its Device ID. Note: currently, Astarte accepts Device IDs longer than 128 bit, which are then truncated to 128 bit internally. This behaviour exists for compatibility reasons but it's not supported and will likely change in future releases - hence, refrain from using anything which is not a 128-bit Device ID. Note: As much as Device IDs should effectively be unique per-realm and this configuration will always be supported, some future optional optimizations might be available on top of the assumption that Device IDs are globally unique to an Astarte installation. Given the Device ID format has a 2<sup>-128</sup> chance of collision, it is safe to assume that as long as best practices for Device ID generation are followed, Device IDs will always be globally unique.","ref":"010-design_principles.html#device-id"},{"type":"extras","title":"Design Principles - Device interaction","doc":"Astarte assumes devices are capable of exchanging data over a transport/protocol supporting SSL/TLS (e.g.: MQTT). This is a strong requirement, as Astarte identifies devices through client SSL certificates when it comes to data exchange. Each transport implementation must be capable of mapping interfaces and out-of-band messages on top of it. Astarte itself does not care about the implementation detail of the transport itself, as the transport is in charge of converting its input to an AMQP message following Astarte's internal API specification. Astarte's official reference and recommended design is MQTT using VerneMQ and its Astarte plugin. Device SDK and code generation Device SDKs can take advantage of the interface design to dynamically generate code for exchanging data with Astarte. This way, developers using Device SDKs are spared from knowing details about the underlying transports and protocols, and can use a data-driven API. However, there are some limitations and requirements: The SDK requires SSL support - Astarte does not allow exchanging data over unencrypted channels and its design builds on the assumption that everything runs on top of SSL. If your device isn't capable of SSL, you are probably looking for Gateway support in Astarte. As much as the SDK can implement virtually any transport protocol, it is required that the SDK supports at least HTTP(s) for Pairing.","ref":"010-design_principles.html#device-interaction"},{"type":"extras","title":"Design Principles - Realms and multitenancy","doc":"Astarte is natively multitenant through the concept of Realms. Each Realm is a logical portion of Astarte, and usually represents an organization or, in general, a set of devices physically/logically isolated. Realms build upon the concept of keyspaces in Cassandra. Each Realm has its very own keyspace and has no shared data with other Realms. In fact, it is even possible to have a dedicated Cassandra cluster for a single realm in complex installations.","ref":"010-design_principles.html#realms-and-multitenancy"},{"type":"extras","title":"Design Principles - Message Ordering","doc":"In Astarte, transports are given the task to deliver messages in a well-known AMQP structure. The ordering of such messages is then preserved on a set of criterias: There is no such thing as "in-order" among devices. A message X sent to device A can be processed after a message Y sent to device B even if Y was ingested in the AMQP queue before X. This is intentional and by design. All messages to a specific device A are always guaranteed to be processed in the very same order of the transport ingestion. Ordering is not dependent on the message timestamp, which can be set by different sources (depending on the interface's definition of timestamp). For example, interface A has explicit timestamping while interface B doesn't. Message X from A has an earlier timestamp than message Y from B, but if message Y has been ingested before X, Y will be processed before X regardless. Responsibility of message ordering before entering AMQP is entirely up to the transport, and different transports might have different behaviors when it comes to message ordering. Astarte provides this guarantee right after the transport itself. Message ordering concerns only pipelines in the DUP , including but not limited to data ingestion in the Database and Simple Triggers.","ref":"010-design_principles.html#message-ordering"},{"type":"extras","title":"Design Principles - Triggers","doc":"Triggers are rules which are "triggered" whenever one or more conditions are satisfied. Every satisfied condition generates an ordered event for the Trigger Engine to be processed. They are one of the core concepts in Astarte and are the preferred way to handle push interactions between Astarte and connected applications. More details about triggers can be found in the dedicated section .","ref":"010-design_principles.html#triggers"},{"type":"extras","title":"Components","doc":"Astarte is a distributed system interacting over AMQP, as explained in Design Principles . This is an overview of its main internal services.","ref":"020-components.html"},{"type":"extras","title":"Components - Pairing","doc":"Pairing takes care of Device Authentication and Authorization. It interacts with Astarte's CA and orchestrates the way devices connect and interact with Transports. It also handles Device Registration. Agent, Device and Pairing interaction is described in detail here .","ref":"020-components.html#pairing"},{"type":"extras","title":"Components - Data Updater Plant (DUP)","doc":"Data Updater Plant is a replicable, scalable component which takes care of the ingestion pipeline. It gathers data from devices and orchestrates data flow amongst other components. It is, arguably, the most critical component of the system and the most resource hungry - the way DUP is deployed, replicated and configured has a tremendous impact on Astarte's performances, especially when dealing with massive data flows.","ref":"020-components.html#data-updater-plant-dup"},{"type":"extras","title":"Components - Trigger Engine","doc":"Trigger Engine takes care of processing Triggers. It is a purely computational component which handles every Trigger's pipeline and triggers actions accordingly.","ref":"020-components.html#trigger-engine"},{"type":"extras","title":"Components - AppEngine","doc":"AppEngine is Astarte's main API endpoint for end users. AppEngine exposes a RESTful API to retrieve and send data from/to devices, according to their interfaces. Every direct device interaction can be done from here. It also exposes Channels, a WebSocket-based solution for listening to device events in real-time with Triggers' same mechanism and semantics.","ref":"020-components.html#appengine"},{"type":"extras","title":"Components - Realm Management","doc":"Realm Management is an administrator-like API for configuring a Realm. It is used for managing Interfaces and Triggers.","ref":"020-components.html#realm-management"},{"type":"extras","title":"Components - Housekeeping","doc":"Housekeeping is the equivalent of a superadmin API. It is usually not accessible to the end user but rather to Astarte's administrator who, in most cases, might deny overall outside access. It allows to manage and create Realms, and perform cluster-wide maintenance actions.","ref":"020-components.html#housekeeping"},{"type":"extras","title":"Interfaces","doc":"Interfaces are a core concept of Astarte which defines how data is exchanged between Astarte and its peers. They are not to be intended as OOP interfaces, but rather as the following definition: In computing, an interface is a shared boundary across which two or more separate components of a computer system exchange information. In Astarte each interface has an owner, can represent either a continuous data stream or a snapshot of a set of properties, and can be either aggregated into an object or be an independent set of individual members. If you are already familiar with interface's basic concepts, you might want to jump directly to the Interface Schema .","ref":"030-interface.html"},{"type":"extras","title":"Interfaces - Versioning","doc":"Interfaces are versioned, each interface having both a major version and a minor version number. The concept behind these two version numbers mimics Semantic Versioning : arbitrary changes can happen exclusively between different major versions (e.g. removing members, changing types, etc...), whereas minor versions allow incremental additive changes only (e.g. adding members). Several different major versions of the same interface can coexist at the same time in Astarte, although a Device can hold only a single version of an interface at a time (even though interfaces can be updated over time). Interfaces, internally, are univocally identified by their name and their major version.","ref":"030-interface.html#versioning"},{"type":"extras","title":"Interfaces - Format","doc":"Interfaces are described using a JSON document. Each interface is identified by an unique interface name of maximum 128 characters, which must be a Reverse Domain Name . As a convention, the interface name usually contains its author's URI Reverse Internet Domain Name. An example skeleton looks like this: { "interface_name": "com.test.MyInterfaceName", "version_major": 1, "version_minor": 0, [...] } Valid values and variables are listed in the Interface Schema . Name limitations A valid interface name consists of a Reverse Domain Name containing alphanumeric characters, hyphens and dots. By design, both the top level domain and last domain component can not contain hyphens, although hypens are allowed in other parts of the interface name (e.g.: org.astarte-platform.Values is a valid interface name). Interface names have to be fully-defined Reverse Domain Names. Values will not be accepted as an Astarte interface name, whereas org.astarte-platform.Values is a valid one. Interface's uniqueness is case insensitive - this means you cannot install two interfaces with the same name and different casing (e.g.: org.astarte-platform.MyValues and org.astarte-platform.Myvalues ). This also applies to Major versioning: interfaces sharing the same name with a different major version cannot have different casing. Although not enforced, naming conventions for Astarte Interfaces require lowercasing for anything but the last part of the Interface name, which should be CamelCase. Valid examples are: org.astarte-platform.conventions.ValidInterfaceName org.astarte-platform.ValidInterfaceName org.astarte-platform.conventions.satisfied.ValidInterfaceName Non-valid examples are: org.astarte-platform.Conventions.ValidInterfaceName org.astarte-platform.validInterfaceName org.astarte-platform.Conventions.satisfied.ValidInterfaceName","ref":"030-interface.html#format"},{"type":"extras","title":"Interfaces - Interface Type","doc":"Interfaces have a well-known, predefined type, which can be either property or datastream . Every Device in Astarte can have any number of interfaces of any different types. Datastream datastream represents a mutable, ordered stream of data, with no concept of persistent state or synchronization. As a rule of thumb, datastream interfaces should be used when dealing with values such as sensor samples, commands and events. datastream are stored as time series in the database, making them suitable for time span filtering and any other common time series operation, and they are not idempotent in the REST API semantics. Due to their nature, datastream interfaces have a number of additional properties which fine tune their behavior. Properties properties represent a persistent, stateful, synchronized state with no concept of history or timestamping. properties are useful, for example, when dealing with settings, states or policies/rules. properties are stored in a key-value fashion, and grouped according to their interface, and they are idempotent in the REST API semantics. Rather than being able to act on a stream like in the datastream case, properties can be retrieved, or can be used as a trigger whenever they change. Values in a properties interface can be unset (or deleted according to the http jargon): to allow such a thing, the interface must have its allow_unset property set to true . Please refer to the JSON Schema for further details.","ref":"030-interface.html#interface-type"},{"type":"extras","title":"Interfaces - Ownership","doc":"Astarte's design mandates that each interface has an owner. The owner of an interface has a write-only access to it, whereas other actors have read-only access. Interface ownership can be either device or server : the owner is the actor producing the data, whereas the other actor consumes data.","ref":"030-interface.html#ownership"},{"type":"extras","title":"Interfaces - Mappings","doc":"Every interface must have an array of mappings. Mappings are designed around REST controller semantics: each mapping describes an endpoint which is resolved to a path, it is strongly typed, and can have additional options. Just like in REST controllers, Endpoints can be parametrized to build REST-like collection and trees. Parameters are identified by %{parameterName} , with each endpoint supporting any number of parameters (see Limitations ). This is how a parametrized mapping looks like: [...] "mappings": [ { "endpoint": "/%{itemIndex}/value", "type": "integer", "reliability": "unique", "retention": "discard" }, [...] In this example, /0/value , /1/value or /test/value all map to a valid endpoint, while /te/st/value can't be resolved by any endpoint. Supported data types The following types are supported: double : A double-precision floating-point number as specified by binary64, by the IEEE 754 standard (NaNs and other non numerical values are not supported). integer : A signed 32 bit integer. boolean : Either true or false , adhering to JSON boolean type. longinteger : A signed 64 bit integer (please note that longinteger is represented as a string by default in JSON-based APIs.). string : An UTF-8 string, at most 65536 bytes long. binaryblob : An arbitrary sequence of any byte that should be shorter than 64 KiB. ( binaryblob is represented as a base64 string by default in JSON-based APIs.). datetime : A UTC timestamp, internally represented as milliseconds since 1st Jan 1970 using a signed 64 bits integer. ( datetime is represented as an ISO 8601 string by default in JSON based APIs.) doublearray , integerarray , booleanarray , longintegerarray , stringarray , binaryblobarray , datetimearray : A list of values, represented as a JSON Array. Arrays can have up to 1024 items and each item must respect the limits of its scalar type ( i.e. each string in a stringarray must be at most 65535 bytes long, each binary blob in a binaryblobarray must be shorter than 64 KiB. Make sure that the differences between two distinct interface names are not limited to the casing or the presence of hyphens. This situation leads to a collision in the interface names which brings to an error in the interface installation process. Limitations A valid interface must resolve a path univocally to a single endpoint. Take the following example: [...] "mappings": [ { "endpoint": "/%{itemIndex}/value", "type": "integer" }, { "endpoint": "/myPath/value", "type": "integer" }, [...] In such a case, the interface isn't valid and is rejected, due to the fact that path /myPath/value is ambiguous and could be resolved to two different endpoints. Any endpoint configuration must not generate paths that are prefix of other paths, for this reason the following example is also invalid: [...] "mappings": [ { "endpoint": "/some/thing", "type":"integer" }, { "endpoint": "/some/%{param}/value", "type": "integer" }, [...] In case the interface's aggregation is object , additional restrictions apply. Endpoints in the same interface must all have the same depth, and the same number of parameters. If the interface is parametrized, every endpoint must have the same parameter name at the same level. This is an example of a valid aggregated interface mapping: [...] "mappings": [ { "endpoint": "/%{itemIndex}/value", "type": "integer" }, { "endpoint": "/%{itemIndex}/otherValue", "type": "string" }, [...] Additional limitations (which stem from the MQTT protocol specification) can be outlined. When using parametric endpoints, the actual values used in place of parameter placeholders must fulfill the following requirements: endpoint parameters must be non-empty UTF-8 encoded strings; endpoint parameters must not contain the following characters: + and # . Those characters are treated as wildcards for MQTT topics and therefore must be avoided; endpoint parameters must not contain the / character.","ref":"030-interface.html#mappings"},{"type":"extras","title":"Interfaces - Aggregation","doc":"In a real world scenario, such as an array of sensors, there are usually two main cases. A sensor might have one or more independent values which are sampled individually and sent whenever they become available independently. Or a sensor might sample at the same time a number of values, which might as well have some form of correlation. In Astarte, this concept is mapped to interface aggregation . In case aggregation is individual , each mapping is treated as an independent value and is managed individually. In case aggregation is object , Astarte expects the owner to send all of the interface's mappings at the same time, packed in a single message. In this case, all of the mappings share some core properties such as the timestamp. Aggregation is a powerful mechanism that can be used to map interfaces to real world "objects" . Moreover, aggregated interfaces can also be parametrized, although with some limitations . Endpoints and aggregation Since Astarte 0.11, Aggregations cannot have endpoints with depth 1. This was an erroneously allowed behavior in Astarte 0.10 which is kept for retrocompatibility - however, new interfaces should ensure each endpoint in an aggreate has at least depth 2, as support for depth 1 will be removed in a future release. This change has been done to be consistent with AppEngine API design, and to ensure that path / is not ambiguous. This is the correct way to set up a valid endpoint structure for an aggregate: [...] "mappings": [ { "endpoint": "/objects/value", "type": "integer" }, { "endpoint": "/objects/otherValue", "type": "string" }, [...] The following structure, instead, is deprecated: [...] "mappings": [ { "endpoint": "/value", "type": "integer" }, { "endpoint": "/otherValue", "type": "string" }, [...]","ref":"030-interface.html#aggregation"},{"type":"extras","title":"Interfaces - Datastream-specific features","doc":"datastream interfaces are highly tunable, depending on the kind of data they are representing: it is possible to fine tune several aspects of how data is stored, transferred and indexed. The following properties can be set at mapping level. NOTE: In case the interface is aggregated, additional properties must be the same for each mapping. explicit_timestamp : By default, Astarte associates a timestamp to data whenever it is collected (or - when the message hits the data collection stage). However, when setting this property to true , Astarte expects the owner to attach a valid timestamp each time it produces data. In that case, the provided timestamp is used for indexing. reliability : Each mapping can be unreliable (default), guaranteed , unique . This defines whether data should be considered delivered when the transport successfully sends the data regardless of the outcome ( unreliable ), when data has been received at least once by the recipient ( guaranteed ) or when data has been received exactly once by the recipient ( unique ). When using reliable data, consider you might incur in additional resource usage on both the transport and the device's end. retention : Each mapping can have a discard (default), volatile , stored retention. This defines whether data should be discarded if the transport is temporarily uncapable of delivering it ( discard ), should be kept in a cache in memory ( volatile ) or on disk ( stored ), and guaranteed to be delivered in the timeframe defined by the expiry . expiry : Meaningful only when retention is stored . Defines how many seconds a specific data entry should be kept before giving up and erasing it from the persistent cache. A value <= 0 means the persistent cache never expires, and is the default. database_retention_policy : Useful only with datastream. Defines whether data should expire from the database after a given interval. Valid values are: no_ttl and use_ttl. database_retention_ttl : Useful when database_retention_policy is "use_ttl" . Defines how many seconds a specific data entry should be kept before erasing it from the database.","ref":"030-interface.html#datastream-specific-features"},{"type":"extras","title":"Interfaces - Best practices","doc":"When creating interface drafts, or for testing purposes in general, it is recommended to use 0 as the major version, to make maintenance and testing easier. Currently, Astarte allows only interfaces with major_version == 0 to be deleted, and this limitation will probably be never lifted to prevent data loss. When sending real time commands in datastream interfaces, discard is usually the best option. Even though it does not guarantee delivery, it prevents users from unwillingly sending the same command over and over if the recipient isn't available, causing a queue of commands to be sent to the recipient when it gets back online. In general, retention should be used to keep track of low traffic/important events","ref":"030-interface.html#best-practices"},{"type":"extras","title":"Interface Schema","doc":"The schema contains the following objects: Interface (root object) Mapping","ref":"040-interface_schema.html"},{"type":"extras","title":"Interface Schema - Interface","doc":"This schema describes how an Astarte interface should be declared Properties Type Description Required interface_name string The name of the interface. This has to be an unique, alphanumeric reverse internet domain name, shorther than 128 characters. ✔ Yes version_major integer A Major version qualifier for this interface. Interfaces with the same id and different version_major number are deemed incompatible. It is then acceptable to redefine any property of the interface when changing the major version number. ✔ Yes version_minor integer A Minor version qualifier for this interface. Interfaces with the same id and major version number and different version_minor number are deemed compatible between each other. When changing the minor number, it is then only possible to insert further mappings. Any other modification might lead to incompatibilities and undefined behavior. ✔ Yes type string Identifies the type of this Interface. Currently two types are supported: datastream and properties. datastream should be used when dealing with streams of non-persistent data, where a single path receives updates and there's no concept of state. properties, instead, are meant to be an actual state and as such they have only a change history, and are retained. ✔ Yes ownership string Identifies the quality of the interface. Interfaces are meant to be unidirectional, and this property defines who's sending or receiving data. device means the device/gateway is sending data to Astarte, consumer means the device/gateway is receiving data from Astarte. Bidirectional mode is not supported, you should instantiate another interface for that. ✔ Yes aggregation string Identifies the aggregation of the mappings of the interface. Individual means every mapping changes state or streams data independently, whereas an object aggregation treats the interface as an object, making all the mappings changes interdependent. Choosing the right aggregation might drastically improve performances. No, default: "individual" description string An optional description of the interface. No doc string A string containing documentation that will be injected in the generated client code. No mappings Astarte Mapping Schema [1-1024] Mappings define the endpoint of the interface, where actual data is stored/streamed. They are defined as relative URLs (e.g. /my/path) and can be parametrized (e.g.: /%{myparam}/path). A valid interface must have no mappings clash, which means that every mapping must resolve to a unique path or collection of paths (including parametrization). Every mapping acquires type, quality and aggregation of the interface. ✔ Yes Additional properties are allowed. astarte.interface.schema.interface_name ✔ The name of the interface. This has to be an unique, alphanumeric reverse internet domain name, shorther than 128 characters. Type : string Required : Yes Minimum Length : >= 1 Allowed Pattern : ^([a-zA-Z][a-zA-Z0-9]*.([a-zA-Z0-9][a-zA-Z0-9-]*.)*)?[a-zA-Z][a-zA-Z0-9]*$ astarte.interface.schema.version_major ✔ A Major version qualifier for this interface. Interfaces with the same id and different version_major number are deemed incompatible. It is then acceptable to redefine any property of the interface when changing the major version number. Type : integer Required : Yes astarte.interface.schema.version_minor ✔ A Minor version qualifier for this interface. Interfaces with the same id and major version number and different version_minor number are deemed compatible between each other. When changing the minor number, it is then only possible to insert further mappings. Any other modification might lead to incompatibilities and undefined behavior. Type : integer Required : Yes astarte.interface.schema.type ✔ Identifies the type of this Interface. Currently two types are supported: datastream and properties. datastream should be used when dealing with streams of non-persistent data, where a single path receives updates and there's no concept of state. properties, instead, are meant to be an actual state and as such they have only a change history, and are retained. Type : string Required : Yes Allowed values : "datastream" "properties" astarte.interface.schema.ownership ✔ Identifies the quality of the interface. Interfaces are meant to be unidirectional, and this property defines who's sending or receiving data. device means the device/gateway is sending data to Astarte, consumer means the device/gateway is receiving data from Astarte. Bidirectional mode is not supported, you should instantiate another interface for that. Type : string Required : Yes Allowed values : "device" "server" astarte.interface.schema.aggregation Identifies the aggregation of the mappings of the interface. Individual means every mapping changes state or streams data independently, whereas an object aggregation treats the interface as an object, making all the mappings changes interdependent. Choosing the right aggregation might drastically improve performances. Type : string Required : No, default: "individual" Allowed values : "individual" "object" astarte.interface.schema.description An optional description of the interface. Type : string Required : No astarte.interface.schema.doc A string containing documentation that will be injected in the generated client code. Type : string Required : No astarte.interface.schema.mappings ✔ Mappings define the endpoint of the interface, where actual data is stored/streamed. They are defined as relative URLs (e.g. /my/path) and can be parametrized (e.g.: /%{myparam}/path). A valid interface must have no mappings clash, which means that every mapping must resolve to a unique path or collection of paths (including parametrization). Every mapping acquires type, quality and aggregation of the interface. Type : Astarte Mapping Schema [1-1024] Each element in the array must be unique. Required : Yes","ref":"040-interface_schema.html#interface"},{"type":"extras","title":"Interface Schema - Mapping","doc":"Identifies a mapping for an interface. A mapping must consist at least of an endpoint and a type. Properties Type Description Required endpoint string The template of the path. This is a UNIX-like path (e.g. /my/path) and can be parametrized. Parameters are in the %{name} form, and can be used to create interfaces which represent dictionaries of mappings. When the interface aggregation is object, an object is composed by all the mappings for one specific parameter combination. ✔ Yes type string Defines the type of the mapping. ✔ Yes reliability string Useful only with datastream. Defines whether the sent data should be considered delivered when the transport successfully sends the data (unreliable), when we know that the data has been received at least once (guaranteed) or when we know that the data has been received exactly once (unique). unreliable by default. When using reliable data, consider you might incur in additional resource usage on both the transport and the device's end. No, default: "unreliable" explicit_timestamp boolean Allow to set a custom timestamp, otherwise a timestamp is added when the message is received. If true explicit timestamp will also be used for sorting. This feature is only supported on datastreams. No, default: false retention string Useful only with datastream. Defines whether the sent data should be discarded if the transport is temporarily uncapable of delivering it (discard) or should be kept in a cache in memory (volatile) or on disk (stored), and guaranteed to be delivered in the timeframe defined by the expiry. discard by default. No, default: "discard" expiry integer Useful when retention is stored. Defines after how many seconds a specific data entry should be kept before giving up and erasing it from the persistent cache. A value <= 0 means the persistent cache never expires, and is the default. No, default: 0 database_retention_policy string Useful only with datastream. Defines whether data should expire from the database after a given interval. Valid values are: no_ttl and use_ttl. No, default: "no_ttl" database_retention_ttl integer Useful when database_retention_policy is "use_ttl" . Defines how many seconds a specific data entry should be kept before erasing it from the database. No allow_unset boolean Used only with properties. Used with producers, it generates a method to unset the property. Used with consumers, it generates code to call an unset method when an empty payload is received. No, default: false description string An optional description of the mapping. No doc string A string containing documentation that will be injected in the generated client code. No Additional properties are allowed. astarte.mapping.schema.endpoint ✔ The template of the path. This is a UNIX-like path (e.g. /my/path) and can be parametrized. Parameters are in the %{name} form, and can be used to create interfaces which represent dictionaries of mappings. When the interface aggregation is object, an object is composed by all the mappings for one specific parameter combination. Refer to the mapping section of the interface page for further details on allowed parameter substitution values. Type : string Required : Yes Minimum Length : >= 2 Allowed Pattern : ^(/(%{([a-zA-Z_][a-zA-Z0-9_]*)}|[a-zA-Z_][a-zA-Z0-9_]*)){1,64}$ astarte.mapping.schema.type ✔ Defines the type of the mapping. Type : string Required : Yes Allowed values : "double" "integer" "boolean" "longinteger" "string" "binaryblob" "datetime" "doublearray" "integerarray" "booleanarray" "longintegerarray" "stringarray" "binaryblobarray" "datetimearray" astarte.mapping.schema.reliability Useful only with datastream. Defines whether the sent data should be considered delivered when the transport successfully sends the data (unreliable), when we know that the data has been received at least once (guaranteed) or when we know that the data has been received exactly once (unique). unreliable by default. When using reliable data, consider you might incur in additional resource usage on both the transport and the device's end. Type : string Required : No, default: "unreliable" Allowed values : "unreliable" "guaranteed" "unique" astarte.mapping.schema.explicit_timestamp Allow to set a custom timestamp, otherwise a timestamp is added when the message is received. If true explicit timestamp will also be used for sorting. This feature is only supported on datastreams. Type : boolean Required : No, default: false astarte.mapping.schema.retention Useful only with datastream. Defines whether the sent data should be discarded if the transport is temporarily uncapable of delivering it (discard) or should be kept in a cache in memory (volatile) or on disk (stored), and guaranteed to be delivered in the timeframe defined by the expiry. discard by default. Type : string Required : No, default: "discard" Allowed values : "discard" "volatile" "stored" astarte.mapping.schema.expiry Useful when retention is stored. Defines after how many seconds a specific data entry should be kept before giving up and erasing it from the persistent cache. A value <= 0 means the persistent cache never expires, and is the default. Type : integer Required : No, default: 0 astarte.mapping.schema.database_retention_policy Useful only with datastream. Defines whether data is expired from the database after a given time to live interval. When "no_ttl" is used data are not expired. Type : string Required : No astarte.mapping.schema.database_retention_ttl Useful when database_retention_policy is "use_ttl" . Defines how many seconds a specific data entry should be kept before erasing it from the database. Type : integer Required : No astarte.mapping.schema.allow_unset Used only with properties. Used with producers, it generates a method to unset the property. Used with consumers, it generates code to call an unset method when an empty payload is received. Type : boolean Required : No, default: false astarte.mapping.schema.description An optional description of the mapping. Type : string Required : No astarte.mapping.schema.doc A string containing documentation that will be injected in the generated client code. Type : string Required : No","ref":"040-interface_schema.html#mapping"},{"type":"extras","title":"Pairing Mechanism","doc":"Astarte's Pairing is a unified mechanism for Registering Devices and obtaining Transport Credentials . Even though in Astarte each Transport is free to choose its own Authentication mechanisms and Credentials autonomously, Pairing defines a well-known mechanism for Registering Devices and for orchestrating the exchange of Transport Credentials . Pairing is the main endpoint which orchestrates Device Authentication in Astarte, abstracting all details.","ref":"050-pairing_mechanism.html"},{"type":"extras","title":"Pairing Mechanism - Authentication flow","doc":"","ref":"050-pairing_mechanism.html#authentication-flow"},{"type":"extras","title":"Pairing Mechanism - Credentials Secret vs. Transport Credentials","doc":"Each device is identified by a Device ID and, on top of that, it has two different credentials directly associated to its ID: Credentials Secret and Transport Credentials . Credentials Secret is a shared secret between Astarte and a Device, which are used only to authenticate against Pairing API. Each device has a single Credentials Secret which remains valid throughout its whole lifecycle, and cannot be changed (unless operating manually). Transport Credentials are Transport-specific credentials usually orchestrated by Pairing. Pairing emits these Credentials through a policy which is usually imposed by the Authority emitting the Credentials or by Pairing itself. They are designed to be transient, revokable and reasonably short-lived - however, the actual behavior and their lifecycle is entirely orchestrated by the Authority emitting them. The emission of Transport Credentials can be inhibited for a specific Device, you can read how to do that in the User Guide Transports, by design, have no knowledge nor access to Credentials Secret , but have full authority over the authentication mechanism for devices. In fact, each Transport is free to choose the authentication mechanism which fits it best. Credentials Secret storage recommendations As losing or disclosing a Credentials Secret might mean a device is compromised or requires manual intervention to be fixed and secured, storing it appropriately is critical. Usually, when it comes to embedded devices, it is advised to store the Credentials Secret into an OTP, if available. Otherwise, storing it into the bootloader's variables is a viable and safe alternative. Other options might be having a separate, isolated storage containing Credentials Secret . In general, Astarte SDK does not provide a streamlined mechanism for retrieving Credentials Secret as the storage detail is strongly dependent on the target hardware - device developers should implement the safest strategy which better complies with their policies. Tuning devices for security is out of the scope of this guide, however it is advised to make sure only Astarte SDK has access to Credentials Secret .","ref":"050-pairing_mechanism.html#credentials-secret-vs-transport-credentials"},{"type":"extras","title":"Pairing Mechanism - Using SSL Certificates as Transport Credentials","doc":"Whenever possible, Transports are advised to implement their Authentication through the use of SSL certificates and a certificate authority by using Mutual Authentication , to ensure identities of the endpoint and the client are well-known to each other - this is especially the case with Astarte's MQTT Protocol on top of VerneMQ Transport. In this case, Transport Credentials are a SSL Certificate, and Pairing will interact with a Certificate Authority. The certificate rotates depending on the emission policy of the CA and can be renewed and invalidated countless times over the device lifecycle. The Certificate is a transient, asymmetric, device-specific, non-critical Transport Credential which can be in turn used to authenticate against the chosen Transport. In this case, Transports should have no knowledge nor access to secrets or Authorization details: they rather have to comply with the configured CA and the certificate parsing, as the Certificate contains all needed information for Authorization as well. Mutual SSL Authentication Flow Side note: the Transport usually bears the public certificate of the CA, and actually interacts with the CA itself only if it exposes an OCSP endpoint and the Transport is capable of understanding it. In case the CA exposes a CRL, the Transport just makes sure to update its CRL from the CA every once in a while. In both cases, Transport's only interaction with the CA is the configuration of its SSL endpoint. Certificate Authority Pairing is designed to interact with an abstract certificate authority, given this authority is capable of: Emitting SSL Certificates with a custom CN (this is important in the Transport authentication flow) Revoking emitted certificates and exposing CRL/OCSP revocation information and is accessible from a 3rd party (e.g. from a REST API). By default, Astarte supports Cloudflare's CFSSL , and also provides a minimal installation in its default deploy scripts. For bigger installations, especially in terms of number of connected devices, it is strongly advised to use a dedicated CFSSL installation. Also, Astarte Enterprise provides a number of additional features including support for other external CAs. Certificate flow During the Pairing flow, the device must generate autonomously a Certificate Signing Request (CSR) which will be in turn relayed by Pairing to the configured Certificate Authority. Pairing will also provide the Certificate Authority with a custom CN, which maps to <realm>/<device id> . The CA must ensure the signed certificate carries this information, as it will be used by the Transport to authenticate the caller inside Astarte. Pairing, in fact, will also perform sanity checks over the signed certificate and reject it in case the CA fails to comply.","ref":"050-pairing_mechanism.html#using-ssl-certificates-as-transport-credentials"},{"type":"extras","title":"Pairing Mechanism - Agents","doc":"Agents are realm-level entities capable of registering a device into Astarte. Agents are a core concept in the Pairing mechanism, as no Device can request its Transport Credentials nor be authenticated against any Transport unless an Agent previously gave its consent and delivered its Credentials Secret . The recommended configuration includes an authenticated Agent in a trusted physical environment (e.g.: the distribution facility of the device) which guarantees an isolated and safe routine for generating Credentials Secret . However, such a setup might not always be possible, and Astarte's SDK has an On Board Agent concept to allow a simpler registration procedure. On Board Agent In the On Board Agent use case, the device is preloaded with an Agent Key , a shared secret which is not tied to a specific Device in the realm . In fact, this secret is usually the same for all Devices in the same realm. This secret will be used only once, upon the device's first interaction with Astarte (Registration), and can be safely discarded afterwards. This approach largely simplifies the deploy procedure, but leaves every device with a secret which, if retrieved, can allow an entity to register an arbitrary Device in the realm. If following the On Board Agent approach, it is advised to store the Agent Key in a safe area inside the device and delete it after retrieving a Credentials Secret (some OTPs allow this configuration).","ref":"050-pairing_mechanism.html#agents"},{"type":"extras","title":"Pairing Mechanism - Transport responsibility","doc":"Once a device obtains its Transport Credentials , it is then capable of connecting to the Transport the credentials were forged for. Transports have full responsibility in terms of authenticating the client, reporting and relaying its connection state to Astarte via its internal AMQP API. As such, it is fundamental that 3rd parties implementing new Transports not only adhere to protocol specifications, but also make sure to implement the authentication procedure meticolously, as a vulnerable Transport acts as a single point of failure of the whole system, and is capable of bypassing the Pairing workflow entirely. For this very reason, we encourage users to be extremely cautious when using 3rd party Transports which have not been verified and hardly tested, especially when it comes to the Client Authentication stage. Even though there are valid use cases where Mutual Authentication is not usable, Transports are advised to stick to Mutual SSL Authentication where possible. This, among other benefits, allows to use Pairing's core features for handling SSL Certificates.","ref":"050-pairing_mechanism.html#transport-responsibility"},{"type":"extras","title":"Pairing Mechanism - Pairing facilities","doc":"Pairing's Device API exposes two additional facilities: first and foremost an endpoint which bears a set of information about both Pairing itself and Transports the device should use or choose from. This endpoint is Device and Realm specific and can be found at /{realm_name}/devices/{hw_id} . This allows granting each Device a specific Transport configuration, which can be useful in installations with more than a single Transport, and automates the configuration on the Device's end, which knows in advance what is supported and how to access its Transport(s). Moreover, each Transport implementation has a /verify endpoint where a client, authenticating with its Credentials Secret , can verify whether its Transport Credentials are valid or not. This, in case SSL is used, is especially useful for checking against revocation lists.","ref":"050-pairing_mechanism.html#pairing-facilities"},{"type":"extras","title":"Triggers","doc":"Triggers in Astarte are the go-to mechanism for generating push events. In contrast with AppEngine's REST APIs, Triggers allow users to specify conditions upon which a custom payload is delivered to a recipient, using a specific action , which usually maps to a specific transport/protocol, such as HTTP. Given this kind of flexibility, triggers are the most powerful way to push data to an external service, potentially without any additional customization. Triggers can be managed from Realm Management API , astartectl with the astartectl realm-management triggers subcommand, or Astarte Dashboard in the Triggers page.","ref":"060-triggers.html"},{"type":"extras","title":"Triggers - Building Triggers","doc":"Triggers can be either built manually or using Astarte Dashboard's Trigger Editor. Trigger Editor dynamically loads installed Interfaces in the Realm and eases trigger creation by providing not only linting and validation, but also dynamic resolution of Interface names. Trigger Editor works in a very similar fashion to Interfaces Editor, and shares the same User Interface. Format A trigger is described using a JSON document. Each trigger is defined by two main parts: condition and action . This is a JSON representation of an example trigger: { "name": "example_trigger", "action": { "http_url": "https://example.com/my_hook", "http_method": "post" }, "simple_triggers": [ { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.4 } ] } The condition is represented by the simple_triggers array. In this release, Astarte supports only a single entry in the simple_triggers array, but support for multiple simple triggers (and different ways to combine them) is planned for future releases. The condition in the example specifies that when data is received on the org.astarte-platform.genericsensors.Values interface on /streamTest/value path, if the value of said data is > 0.4 , then the trigger is activated. For more information about all the possible conditions, check out the Conditions section The action object describes what the result of the trigger will be. In this specific case, an HTTP POST request will be sent to https://example.com/my_hook , with the payload: { "timestamp": "<event_timestamp>", "device_id": "<device_id>", "event": { "type": "incoming_data", "interface": "org.astarte-platform.genericsensors.Values", "path": "/streamTest/value", "value": <some_value> } } To know more about all possible actions, check the Actions section","ref":"060-triggers.html#building-triggers"},{"type":"extras","title":"Triggers - Conditions","doc":"A condition defines the event upon which an action is triggered. Conditions are expressed through simple triggers. Astarte monitors incoming events and triggers a corresponding action whenever there is a match. Simple triggers are divided into two types: Device Triggers and Data Triggers . Device Triggers Device triggers express conditions matching the state of a device. This is the generic representation of a Device Trigger: { "type": "device_trigger", "on": "<device_trigger_type>", "device_id": "<device_id>", "group_name": "<group_name>" } Parameters device_trigger_type can be one of the following: device_connected : triggered when a device connects to its transport. device_disconnected : triggered when a device disconnects from its transport. device_error : triggered when data from a device causes an error. device_id can be used to pass a specific Device ID to restrict the trigger to a single device. * is also accepted as device_id to maintain backwards compatibility and it is considered equivalent to no device_id specified. group_name can be used to restrict the trigger to all devices that are member of the group. device_id and group_name are mutually exclusive and if neither of them is specified in the simple trigger, the simple trigger will be installed for all devices in a realm. Data Triggers Data triggers express conditions matching data coming from a device. This is the generic representation of a Data Trigger: { "type": "data_trigger", "device_id": "<device_id>", "group_name": "<group_name>", "on": "<data_trigger_type>", "interface_name": "<interface_name>", "interface_major": "<interface_major>", "match_path": "<match_path>", "value_match_operator": "<value_match_operator>", "known_value": <known_value> } Data triggers are installed for all devices in a Realm. Data Triggers Parameters device_id can be used to pass a specific Device ID to restrict the trigger to a single device. * is also accepted as device_id to maintain backwards compatibility and it is considered equivalent to no device_id specified. group_name can be used to restrict the trigger to all devices that are member of the group. device_id and group_name are mutually exclusive and if neither of them is specified in the simple trigger, the simple trigger will be installed for all devices in a realm. data_trigger_type can be one of the following: incoming_data : verifies the condition whenever new data arrives. value_stored : verifies the condition whenever new data arrives, after it is saved to the database. value_change : works only with properties interface; verifies the condition whenever the received value is different from the previous one. value_change_applied : works only with properties interface; verifies the condition whenever the last value received is different from the last one previously received, after it is saved to the database. path_created : verifies the condition whenever a new path in a property interface is set or the first value is streamed on a datastream interface. path_removed : works only with properties interface; verifies the condition whenever a property path is unset. interface_name and interface_major represent, respectively, the Interface name and major version that uniquely identify an Astarte Interface. interface_name can be * to match all interfaces; in that case interface_major is ignored and all major numbers are matched. match_path is the path that will be used to match the condition. It can be /* to match all the paths of an interface. value_match_operator is the operator used to match the incoming data against a known value. It can be * to indicate that all values should be matched ( known_value is ignored in that case), otherwise it can be one of these operators: == , != , > , >= , < , <= , contains , not_contains . The match is always performed with <incoming_value> <operator> <known_value> . contains and not_contains can be used only with type string , binaryblob and with array types. known_value is the value used with value_match_operator to perform the comparison on the incoming value. It must have the same type as the incoming value, except in the contains and not_contains case.","ref":"060-triggers.html#conditions"},{"type":"extras","title":"Triggers - Actions","doc":"Actions are triggered by a matching condition. An Action defines how the event should be sent to the outer world (e.g. an http POST on a certain URL). In addition, most actions have a Payload, which carries the body of the event. HTTP Actions Payloads are most of the time represented as text, and Astarte provides several ways to generate them. By default Astarte generates a JSON payload with all the relevant information of the event. This is also the format used when delivering payloads in Astarte Channels. The format of the payload can be found in the Default action section. Astarte also provides a powerful templating mechanism for plain-text payloads based on top of Mustache . This is especially useful for integrating with third-party actors which require custom plain-text payloads. Keep in mind that Mustache templates are only able to produce text/plain payloads, not valid JSON. Default action This is the configuration object representing a minimal default action: { "http_url": "<http_url>", "http_method": "<method>" } The default action sends an HTTP request to the specified http_url using http_method method (e.g. POST ). Further options might be used, such as "http_static_headers", enabling auth to remote services: { "http_url": "<http_url>", "http_method": "<method>", "http_static_headers": { "Authorization": "Bearer <token>" }, "ignore_ssl_errors": <true|false> } The ignore_ssl_errors key is optional and defaults to false . If set to true , any SSL error encountered while doing the HTTP request will be ignored. This can be useful if the trigger must ignore self-signed or expired certificates. Please, beware that some http headers might be not allowed or reserved for http connection signaling. SimpleEvent payloads The payload delivered in a default HTTP action or in Astarte Channels is a JSON document with this format: { "timestamp": "<timestamp>", "device_id": "<device_id>", "trigger_name": "<trigger_name>", "event": <event> } timestamp is an UTC ISO 8601 timestamp (e.g. "2019-10-16T08:56:08.534377Z" ) representing when the event happened. device_id identifies the device that triggered the event. trigger_name identifies the trigger that fired the event. event is a JSON object that has a specific structure depending on the type of the simple_trigger that generated it. Event objects are detailed below. Additionally, the realm that originated the trigger is available in the request in the Astarte-Realm header. Event objects DeviceConnectedEvent { "type": "device_connected", "device_ip_address": "<device_ip_address>" } device_ip_address is the IP address of the device. DeviceDisconnectedEvent { "type": "device_disconnected" } DeviceErrorEvent { "type": "device_error", "error_name": "<error_name>", "metadata": { "<key>": "<value>" } } error_name is a string identifying the error. More details can be found in the device errors documentation metadata is a map with string key and string values that may contain additional information about the error. Some metadata ( e.g. binary payloads) might be encoded in base64 if they cannot be represented as string. In that case, the key is prepended with the base64_ prefix. IncomingDataEvent { "type": "incoming_data", "interface": "<interface>", "path": "<path>", "value": <value> } interface is the interface on which data was received. path is the path on which data was received. value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. ValueStoredEvent { "type": "value_stored", "interface": "<interface>", "path": "<path>", "value": <value> } interface is the interface on which data was received. path is the path on which data was received. value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. ValueChangeEvent { "type": "value_change", "interface": "<interface>", "path": "<path>", "old_value": <old_value>, "new_value": <new_value> } interface is the interface on which data was received. path is the path on which data was received. old_value is the previous value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. new_value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. ValueChangeAppliedEvent { "type": "value_change_applied", "interface": "<interface>", "path": "<path>", "old_value": <old_value>, "new_value": <new_value> } interface is the interface on which data was received. path is the path on which data was received. old_value is the previous value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. new_value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. PathCreatedEvent { "type": "path_created", "interface": "<interface>", "path": "<path>", "value": <value> } interface is the interface on which data was received. path is the path that has been created. value is the received value. Its type depends on the type of the mapping it's coming from. binaryblob and binaryblobarray type values are encoded with Base64. PathRemovedEvent { "type": "path_removed", "interface": "<interface>", "path": "<path>" } interface is the interface on which data was received. path is the path that has been removed. Mustache action This is the configuration object representing a Mustache action: { "http_url": "<http_url>", "http_method": "<http_method>", "template_type": "mustache", "template": "<template>" "ignore_ssl_errors": <true|false> } The Mustache action sends an HTTP request to the specified http_url , with the payload built with the Mustache template specified in template . If the template contains a key inside a double curly bracket (like so: {{ key }} ), it will be substituted with the actual value of that key in the event. The basic keys that can be use to populate the template are: {{ realm }} : the realm the trigger belongs to. {{ device_id }} : the device that originated the trigger. {{ trigger_name }} : the trigger name. {{ event_type }} : the type of the received event. The ignore_ssl_errors key is optional and defaults to false . If set to true , any SSL error encountered while doing the HTTP request will be ignored. This can be useful if the trigger must ignore self-signed or expired certificates. Moreover, depending on the event type, all keys that are contained in the events described in the previous section are available, always by wrapping them in {{ }} . The realm is also sent in the Astarte-Realm header. Example This is an example of a trigger that uses Mustache action. { "name": "example_mustache_trigger", "action": { "template_type": "mustache", "template": "Device {{ device_id }} just connected from IP {{ device_ip_address }}", "http_url": "https://example.com/my_mustache_hook", "http_method": "post" }, "simple_triggers": [ { "type": "device_trigger", "on": "device_connected", "device_id": "*" } ] } When a device is connected, the following request will be received by https://example.com/my_mustache_hook : POST /my_mustache_hook HTTP/1.1 Astarte-Realm: test Content-Length: 63 Content-Type: text/plain Host: example.com User-Agent: hackney/1.13.0 Device ydqBlFsGQ--xZ-_efQxuLw just connected from IP 172.18.0.1 Trigger Delivery Policies When an HTTP action is triggered, an event is sent to a specific URL. However, it is possible that the request is not successfully completed, e.g. the required resource is momentarily not available. Trigger Delivery Policies specify what to do in case of delivery errors and how to handle events which have not been successfully delivered. A Trigger can be linked to one (at most) Trigger Delivery Policy by specifying the name of the policy in the "policy" field. If no Trigger Delivery Policies are specified, Astarte will resort to the default (pre v1.1) behaviour, i.e. ignoring delivery errors. Refer to the relevant documentation for more information on Trigger Delivery Policies. AMQP 0-9-1 Actions AMQP 0-9-1 actions might be configured as an alternative to HTTP actions for advanced use cases. AMQP 0-9-1 is the right choice for a number of scenarios, including Astarte Flow integration, high performance ingestion, integration with an existing AMQP infrastructure, etc... Payloads are always encoded using protobuf , therefore if any other format is required Astarte Flow should be employed as a format converter. This is a minimal configuration object representing an AMQP 0-9-1 action: { "amqp_exchange": "astarte_events_<realm-name>_<exchange-suffix>", "amqp_routing_key": "my_routing_key", "amqp_message_expiration_ms": <expiration in milliseconds>, "amqp_message_persistent": <true when disk persistency is used> } It is possible to configure more advanced AMQP 0-9-1 actions: { "amqp_exchange": "astarte_events_myrealm_myexchange", "amqp_routing_key": "my routing key", "amqp_static_headers": { "key": "value" }, "amqp_message_expiration_ms": 10000, "amqp_message_priority": 0, "amqp_message_persistent": true, } Some Astarte specific restrictions apply: amqp_exchange must have astarte_events_<realm-name>_<any-allowed-string> format. amqp_routing_key must not contain { and } , which are reserved for future uses. For further details RabbitMQ documentation is suggested.","ref":"060-triggers.html#actions"},{"type":"extras","title":"Triggers - Relationship with Channels","doc":"Channels are part of AppEngine, and allow users to monitor device events through WebSockets, on top of Phoenix Channels . Under the hood, Channels use transient triggers to define which kind of events will flow through a specific room.","ref":"060-triggers.html#relationship-with-channels"},{"type":"extras","title":"Trigger Delivery Policies","doc":"When an HTTP action is triggered, an event is sent to a specific URL. However, it is possible that the request is not successfully completed, e.g. the required resource is momentarily not available. Trigger Delivery Policies (also referred to as policies or delivery policies ) specify what to do in case of delivery errors and how to handle events which have not been successfully delivered. In the same fashion as triggers and interfaces, policies are realm-scoped objects, too. While a trigger may specify at most one delivery policy, there is no upper bound on the number of triggers handled by the same policy. Under the hood, a policy is mapped to a queue on which the events to be delivered are stored. This queue is referred to as event queue and has a 1-to-1 relationship to the policy, i.e. there is one and only one event queue for each delivery policy. The size of the event queue is given in the policy specification. This provides an upper bound on the amount of space the event queue can fill. In the context of a policy, the strategy to handle delivery errors is describe by a list of error handlers , which specify how to react to an error (e.g. resend or discard the event) and what kind of error is to be handled. Each policy specifies at least one error handler. Every event in the event queue can be resent up to a number of times given in the policy specification.","ref":"062-trigger_delivery_policies.html"},{"type":"extras","title":"Trigger Delivery Policies - Trigger Delivery Policy Components","doc":"A Trigger Delivery Policy is composed of: Name: a string of at most 127 characters, unique for each realm. Names starting with " ? ", " ! ", or " @ " are reserved. This is a required component and uniquely identifies the policy in the realm. Error handlers: a non-empty list of handlers. Each handler acts on groups of delivery errors and describes the strategy Astarte should take when they occur. In pseudo-BNF syntax, an handler is described by the following grammar: handler :: = "{" "on" ":" "client_error" | "server_error" | "any_error" | [ < int > ] "," "strategy" ":" "discard" | "retry" "}" There are two possible strategies: either discard the event or retry. If the strategy is retry , events will be requeued in the event queue. The default retry strategy is discarding events. The on field specifies the group of HTTP errors the handler refers to: client errors (400-499), server errors (500-599), all errors (400-599), or a custom range of error codes (e.g. [418, 419, 420] ). Handlers can be at most 200 and must be not overlapping (i.e. there can not be two handlers which refer to at least one shared error code). Maximum capacity: the maximum size of the event queue which refers to the policy. If the number of messages to be retried exceeds the event queue size, older events will be discarded. Retry times: the maximum number of times an event in the event queue can be resent. A single policy does not allow to retry sending events for different amount of times, but different policies may have different numbers. This is optional, but required if the policy specifies at least one handler with retry strategy. Event TTL: in order to further lower the space requirement of the event queue, events may be equipped with a TTL which specifies the amount of seconds an event is retained in the event queue. When an event expires, it is discarded from the event queue, even if it has not been delivered. This is optional.","ref":"062-trigger_delivery_policies.html#trigger-delivery-policy-components"},{"type":"extras","title":"Trigger Delivery Policies - Known issues","doc":"At the moment, Trigger Delivery Policies in general do not provide a guarantee of in-order delivery of events. Note that, since previous Astarte versions (i.e. < 1.1) did not provide a retry mechanism for events, this change does not impact the expected behaviour if Trigger Delivery Policies are not used. If the Astarte Trigger Engine service is replicated, events could be delivered out of order, as data from event queues are delivered to consumers in a round-robin fashion. If the retry strategy is specified, in-order delivery cannot be guaranteed because a > 1 consumer prefetch count is being used. This allows for higher throughput at the cost of consistency. In the future, the user will be allowed to choose between having an higher number of events handled, but out of order, or ordered event handling at a lower rate.","ref":"062-trigger_delivery_policies.html#known-issues"},{"type":"extras","title":"Groups","doc":"Astarte supports creating groups of devices in a realm. Groups are currently useful mainly to provide access control, combining Astarte's path based authorization with the fact that devices can be queried with a group URL. This makes it possible to emit tokens allowing a user to operate only on devices that belong to a specific group. Groups can be managed using astartectl or using AppEngine API . See the Managing Groups page in the User Guide for some usage examples. Keep in mind a group is existing as long as there's at least one device in it. Once the last device is removed from the group, the group does not exist anymore, since groups are a tag (or label) of devices.","ref":"065-groups.html"},{"type":"extras","title":"Authentication and Authorization","doc":"Authentication and authorization are crucial, as Astarte likely holds sensitive resources and is capable to send mass commands to a device fleet. First of all: when talking about auth in Astarte, we are talking about anything which isn't a Device - those are Authenticated through Pairing and Authorized by their Transport (which uses Pairing for the Authentication policies). Astarte's authentication/authorization stage identifies the principal through a token (with JWT as the first class citizen), which is the only currency the platform supports.","ref":"070-auth.html"},{"type":"extras","title":"Authentication and Authorization - Authentication Realms","doc":"In Astarte, realms are logically separated and have completely different data partitions. This is also true in terms of authentication, as caller is always authenticated on a per-realm basis. As such, an authentication realm matches 1:1 an Astarte realm. Superadmin APIs, such as housekeeping, are part of a different authentication realm which is defined upon cluster setup.","ref":"070-auth.html#authentication-realms"},{"type":"extras","title":"Authentication and Authorization - Authentication in Astarte","doc":"Astarte, by design, does not have a concept of per-user authentication built in. The definition of an authentication realm is a mean to verify a token's validity, that is most likely a public key. This makes integrating Astarte with 3rd party authentication/authorization frameworks and SSOs extremely easy, as the whole logic for addressing user management is managed out of the cluster by a dedicated party. Depending on one's use case, it is possible to use either a very simple, dedicated OAuth server for each realm, or a full fledged SSO such as Keycloak which matches its authentication realms to Astarte's realms. Especially if you are aiming at the latter, make sure to read the advised best practices for authentication afterwards.","ref":"070-auth.html#authentication-in-astarte"},{"type":"extras","title":"Authentication and Authorization - Authorization","doc":"Currently, Astarte supports a URL-based authorization for the API. Given that Astarte's data access APIs match the devices' topology like a tree, declaring the authorization in terms of path allow-listing gives enough flexibility to give each user the correct permissions without limitations. As said, Astarte does not have the concept of user, and neither has a durable storage which tracks permissions. As such, it expects the authorization information to be inside the token, which is the only entity Astarte can trust - given it has been verified and authenticated through its signature. Paths are given in form of a set of Perl-like Regular Expressions , and on a per-API basis. This means that each API endpoint (app, realm, etc...) has its own regular expression which defines what the user can do. Moreover, each HTTP verb in an API endpoint (e.g.: GET, POST, PUT, DELETE) can have its own regular expression, to fine-grain permissions on each path. Note: given Astarte's interface are either read only or write only, HTTP verb fine-graining in AppEngine API is mostly useful for preventing a user from deleting a consumer Datastream even though it has write access to it. Most of the time, using only a single regular expression with no verb fine-graining works. Examples of valid regular expressions on AppEngine API are: POST::devices/.*/interfaces/com\\\\.my\\\\.interface/.* : Allows to set individual values on the com.my.interface interface on any individual device in the realm. .*::.*/interfaces/com\\\\.my\\\\.monitoring\\\\.interface.* : Allows to get/set/delete either the aggregate or the individual values of the com.my.monitoring.interface interface on any device or device aggregation in the realm. .*::devices/j0zbvbQp9ZNnanwvh4uOCw.* : Allows every operation on device j0zbvbQp9ZNnanwvh4uOCw GET::devices/[a-zA-Z0-9-_]* : Allows to get every individual device's status, but denies access to any additional information/operation on them. Examples of valid regular expressions on Realm Management API are: POST::interfaces\\/.* : Allows installing new interfaces in the realm. GET::interfaces\\/.* : Allows inspecting every interface in the realm. PUT::interfaces\\/.*\\/0 : Allows updating all draft interfaces in the realm. Other valid examples are: .*::.* : Allows any operation on the given API. Both verb and path regular expressions are implicitly delimited by adding ^ before and $ after the regular expression string. For example, if you use GET::interfaces as regular expression in Realm Management API, the verb will be matched against ^GET$ and the path will be matched against ^interfaces$ . This way the only operation allowed will be listing all the interfaces, while all operation on interfaces/ subpaths will be denied. Token claims and formats Authorization regular expressions have to be contained in the token's claims. Only the JWT case will be considered given it is the primary currency Astarte supports. Every claim is an array of regular expressions, which act as a logical OR. A similar behavior could be of course achieved (and might be more efficient) with a singular regular expression, but for the sake of readability and simplicity it is allowed nonetheless. Of course, keeping the authorization claims simple and pragmatic helps in terms of performance. Supported token claims are: a_aea : Defines the regular expressions for AppEngine API a_rma : Defines the regular expressions for Realm Management API a_ha : Defines the regular expressions for Housekeeping API a_pa : Defines the regular expressions for Pairing API a_ch : Defines the regular expressions for Channels Of course, claims are considered only after a successful token verification. This means that the claim will be processed only if the caller is authenticated against the correct authentication realm - this is especially the case for what concerns Housekeeping, which has a dedicated Authentication realm not tied to any Astarte realms. An example of a valid token claim is: { "a_aea": ["GET::devices/[a-zA-Z0-9-_]*", ".*::.*/interfaces/com\\\\.my\\\\.monitoring\\\\.interface.*", ".*::devices/j0zbvbQp9ZNnanwvh4uOCw.*"], "a_rma": ["GET::.*"] } Which allows very specific permissions on AppEngine API, and a "read all" on Realm Management API. The client by default has no permission to do anything: as such, if a token is missing a claim it is simply assumed that the client isn't authorized to access that specific API. However, keeping in mind that Astarte has no concept of User, it is also true that your authentication backend might choose to emit a different token with only a subset of its real permissions to keep claims and regular expressions as pragmatic as possible. See Granular Claims in Best Practices for more details on this. Natively supported tokens Astarte supports only JWT natively, which has to be signed using one of the following algorithms: ES256 ES384 ES512 PS256 PS384 PS512 RS256 RS384 RS512","ref":"070-auth.html#authorization"},{"type":"extras","title":"Authentication and Authorization - Authorization for REST APIs","doc":"Valid tokens can be used for calling into Astarte's public APIs. Depending on which token mechanism is used, the HTTP call must adhere to some requirements. JWT Every API call must have an Authorization: Bearer <token> header. Not providing the token or providing a token which can't be validated for the authentication realm of the context results in a 401 reply.","ref":"070-auth.html#authorization-for-rest-apis"},{"type":"extras","title":"Authentication and Authorization - Authorization for Channels","doc":"A valid token should be supplied when opening the WebSocket, in the very same fashion to what happens with REST APIs. However, the claims in this token will support different verbs compared to the REST APIs, namely JOIN and WATCH . These have very specific meanings and are well explained in Channels' User Guide . The behavior and supported tokens are equivalent to REST APIs.","ref":"070-auth.html#authorization-for-channels"},{"type":"extras","title":"Authentication and Authorization - Supported integrations","doc":"Astarte, by default, is extremely easy to configure assuming your chosen SSO is capable of issuing JWT, as it is currently the only natively supported authentication currency. However, virtually any token-based system can be used as an auth framework for Astarte. The main purpose of Astarte's design, however, is to keep things simple for everyone. Putting up a full-fledged SSO dedicated to Astarte is beyond the scope of this documentation, and we favor the use case where an existing SSO infrastructure is integrated with Astarte, rather than built ad-hoc. For simple use cases and instant satisfaction, it is strongly advised to use a simpler solution, such as a dedicated OAuth server. Almost all popular languages and frameworks provide great projects which can spin up an OAuth2 server + user management in a matter of hours, from Elixir/Phoenix to Java/Spring to Go . Astarte's Enterprise Distribution includes other add-ons, such as automation and configuration for popular SSOs.","ref":"070-auth.html#supported-integrations"},{"type":"extras","title":"Authentication and Authorization - Best practices","doc":"Due to the nature of tokens, applications and SSOs must take care of emission and storage of the token themselves. In most production cases, Astarte will be part of a larger SSO infrastructure being one of the clients (this is especially true for OAuth). Among best practices, emitting short-lived tokens should always be considered, but depending on the use case, the authentication pipeline can be further tuned to address a number of potential issues. Token exchange OAuth, like other protocols supports the concept of a Token Exchange . Consider a web dashboard with a logged in user. The user will, most likely, have a token which is used by its frontend to call upon the backend/APIs of the web dashboard. For the sake of simplicity, one might include in this token the adequate claims to give the user access to Astarte, but this might not be desirable for a number of reasons outlined above. Token exchange, if supported by your SSO, provides a great way to work around this: whenever the backend or the frontend requires access to Astarte, it can invoke the token exchange mechanism of the SSO to generate a short lived token for the API call from the original authentication, which can then be used even as a single shot access mechanism. Granular claims The token exchange approach can be efficiently paired with a mechanism of granular claims. Consider the use case above, and let's assume the frontend needs direct, frequent access to Astarte's APIs. Exchanging tokens too many times might put a burden on the SSO and might become impractical. However, Astarte decouples entirely authentication and authorization - that means, if two subsequent (valid) tokens which represent the same identity have substantially different claims, it doesn't care. This is intentional, as it allows for a much more efficient pattern: the token used by an hypotetical frontend can have a subset of the user's claims - for example, allowing him to read data from its devices, whereas token exchange can be used whenever more specific operations should be performed - for example, sending some commands or data to devices. This also addresses the objection that regular expressions can grow big or quite complicated in case users need a large number of very granular permissions. In such complex cases, the SSO can be tuned to give out only a subset of claims depending on the user's operation. Token revocation Token revocation isn't natively supported in Astarte for two reason: the first one is performance, as keeping a revocation list is expensive in many regards. The second is the fact that the revocation list is, most of the time, SSO specific, and a dedicated SSO integration would be required. Rather than token revocation, a better practice is to make sure every emitted token has a short enough lifetime. However, it is possible to extend Astarte's authorization stage to support revocation, even though there are no plans to provide upstream support for that. Changing a Realm's validation mean Over the lifetime of a cluster, it might be necessary to change a realm's validation mean for the most diverse reasons. By design, validation means are meant to be long lived, and changing them is supposed to be an extraordinary operation. Astarte supports only one validation mean at a time. When the validation mean is changed, all tokens emitted which could be validated with the previous mean become invalid. It is also possible that there might be a delay between the request of a validation mean change and its actuation. This means during this grace period tokens will be validated against the previously configured mean. As such, it is advised to treat a validation mean change as a maintenance operation for the realm. More details can be found in the Administrator Guide.","ref":"070-auth.html#best-practices"},{"type":"extras","title":"Astarte MQTT v1 Protocol","doc":"Astarte MQTT v1 Protocol allows communication between Astarte and devices. It is the first protocol that has been implemented in Astarte, and it exploits every feature provided by Astarte itself. Astarte MQTT v1 doesn't mandate a specific Transport Credentials format: the broker must handle Authentication, Authorization and Pairing integration the way it sees fit. Astarte MQTT v1 is implemented by Astarte's Reference Transport, Astarte/VerneMQ - a client wishing to interact with it must implement MQTT v3.1.1 and all needed features for Pairing to work. MQTT doesn't mandate the data serialization format, so any application might implement its own format. Data serialization might be a tricky task and protocols might be hard to design, Astarte MQTT takes care of this and provides a higher level protocol which abstracts this detail from the end user. Astarte MQTT v1 Protocol builds upon MQTT v3.1.1 itself, BSON (Binary JSON, version 1.1) serialized payloads and on optional zlib deflate. All communications are ordered and asynchronous. A protocol reference implementation is provided with an Astarte SDK, however developers might implement it from scratch using 3rd party libraries with their favourite languages: all formats and protocols described here are open and well documented. Last but not least Astarte doesn't mandate this protocol, and a different one can be used with a different transport.","ref":"080-mqtt-v1-protocol.html"},{"type":"extras","title":"Astarte MQTT v1 Protocol - MQTT Topics Overview","doc":"Astarte MQTT v1 Protocol relies on few well known reserved topics. Topic Purpose Published By QoS Payload Format <realm name>/<device id> Introspection Device 2 ASCII plain text, ':' and ';' delimited <realm name>/<device id>/control/emptyCache Empty Cache Device 2 ASCII plain text (always "1") <realm name>/<device id>/control/consumer/properties Purge Properties Astarte 2 deflated plain text <realm name>/<device id>/control/producer/properties Purge Properties Device 2 deflated plain text <realm name>/<device id>/<interface name>/<path> Publish Data Both 0, 1, 2 BSON (or empty) For clarity reasons all <realm name>/<device id> prefixes will be omitted on the following paragraphs, those topics will be called device topics. Topics are not bidirectional, devices must not publish data for server owned topics and viceversa, onwership is explicitly stated in interfaces files.","ref":"080-mqtt-v1-protocol.html#mqtt-topics-overview"},{"type":"extras","title":"Astarte MQTT v1 Protocol - BSON","doc":"BSON allows saving precious bytes compared to JSON, while offering the advantages of a schema-less protocol. Consider, for example, a simple value and timestamp payload. The encoded JSON version, {"v":25.367812,"t":1537346756844} counts 33 bytes. The hexdump of the same message encoded with BSON is: 0000000 1 b 00 00 00 09 74 00 ec e0 01 f1 65 01 00 00 01 0000020 76 00 8 c 13 5 f ed 28 5 e 39 40 00 that fits just in 27 bytes. BSON format BSON is a really simple binary format, breaking down the previous example is very easy thanks to BSON simplicity: the first 4 bytes ( 1b 00 00 00 ) are the document size header, follows the timestamp marker ( 09 ), the timestamp key name ( 74 00 , that is "t"), the timestamp value ( 5f 48 06 f1 65 01 00 00 as int64), the double value marker ( 01 ), the value key name ( 76 00 , that is "v"), the actual value ( cd cc cc cc cc 4c 39 40 as 64-bit IEEE 754-2008 floating point) and the end of document marker ( 00 ). Astarte payload standard fields Key Type Mandatory Description v Any Astarte type Yes The value being sent (both properties and datastream) t UTC datetime No Explicit timestamp, if present (optional, datastream only) Astarte data types to BSON types Astarte Data Type BSON Type Size in Bytes double double (0x01) 8 integer int32 (0x10) 4 boolean boolean (0x08) 1 longinteger int64 (0x12) 8 string UTF-8 string (0x02) >= length (encoding dependent) binaryblob binary (0x05) length datetime UTC datetime (0x09) 8 doublearray Array (0x04) (8 + keysize) * count integerarray Array (0x04) (4 + keysize) * count booleanarray Array (0x04) (1 + keysize) * count longintegerarray Array (0x04) (1 + keysize) * count stringarray Array (0x04) depends on count, length, keys length and encoding binaryblobarray Array (0x04) depends on count, keys length and length integer and long integer are signed integer values, double must be a valid number ( +inf , NaN , etc... are not supported), variable data types might be subject to size limitations and object aggregations are encoded as embedded documents.","ref":"080-mqtt-v1-protocol.html#bson"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Connection and Disconnection","doc":"A device is not required to publish any additional connection or disconnection messages, the MQTT broker will automatically keep track of these events and relay them to Astarte. When connecting, before publishing any data message, a device should check MQTT session present flag. When the MQTT session present flag is true no further actions are required, when false the device should take following actions: Publish its introspection Publish an empty cache message Publish all of its existing and set properties on all its property interfaces If a device is unable to inspect session present all previous actions must be taken at every reconnection.","ref":"080-mqtt-v1-protocol.html#connection-and-disconnection"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Introspection","doc":"Each device must declare the set of supported interfaces and their version. Astarte needs to know which interfaces the device advertises before processing any further data publish. This message in Astarte jargon is called introspection and it's performed by publishing on the device root topic the list of interfaces that are installed on the device. Introspection payload is a simple plain text string, and it has the following format (in BNF like syntax): introspection :: = introspection_list introspection_list :: = introspection_entry ";" introspection_list | introspection_entry introspection_entry :: = interface_name ":" interface_major_version ":" interface_minor_version The following example is a valid introspection payload: com . example . MyInterface : 1 : 0 ; org . example . DraftInterface : 0 : 3","ref":"080-mqtt-v1-protocol.html#introspection"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Empty Cache","doc":"Astarte MQTT v1 strives to save bandwidth upon reconnections, to make sure even frequent reconnections don't affect bandwidth consumption. As such, upon connecting and if MQTT advertises a session present, both sides assume that data flow is ordered and consistent. However, there might be cases where this guarantee isn't respected by the device for a number of reasons (e.g.: new device, factory reset, cache lost...). In this case, a device might declare that it has no confidence about its status and its known properties, and can request to resynchronise entirely with Astarte. In Astarte jargon this message is called empty cache and it is performed by publising "1" on the device /control/emptyCache topic. After an empty cache message properties might be purged and Astarte might publish all the server owned properties again.","ref":"080-mqtt-v1-protocol.html#empty-cache"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Session Present","doc":"In the very same fashion as the device, Astarte (or the broker) might be inconsistent with a Device's known status and its known properties. Although unlikely, as Astarte should always keep knowledge about remote device status, this might happen, for example, after an internal error. Astarte performs this task by telling the broker to disconnect the device and clear its session. After this, when the device will attempt reconnection, session present will be false. After a clean session properties might be purged.","ref":"080-mqtt-v1-protocol.html#session-present"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Purge Properties","doc":"Either a Device or Astarte may tell the remote host the set properties list. Any property that is not part of the list will be deleted from any cache or database. This task is called purge properties in Astarte jargon, and it is performed by publishing a the list of known set properties to /control/consumer/properties or /control/producer/properties . Purge Properties payload is a zlib deflated plain text, with an additional 4 bytes header. The additional 4 bytes header is the size of the uncompressed payload, encoded as big endian uint32. The following example is a payload compressed using zlib default compression, with the additional 4 bytes header: 0000000 00 00 00 46 78 9 c 4 b ce cf d5 4 b ad 48 cc 2 d c8 0000020 49 d5 f3 ad f4 cc 2 b 49 2 d 4 a 4 b 4 c 4 e d5 2 f ce 0000040 cf 4 d d5 2 f 48 2 c c9 b0 ce 2 f 4 a 87 ab 70 29 4 a 0000060 4 c 2 b 41 28 ca 2 f c9 48 2 d 0 a 00 2 a 02 00 b2 0 c 0000100 1 a c9 The uncompressed plain text payload has the following format (in BNF like syntax): properties :: = properties_list properties_list :: = properties_entry ";" properties_list | properties_entry properties_entry :: = interface_name path The following example is the inflated previous payload: com . example . MyInterface / some / path ; org . example . DraftInterface / otherPath This protocol feature is fundamental when a device has any interface with an allow_unset mapping, purge properties allows to correct any error due to unhandled unset messages.","ref":"080-mqtt-v1-protocol.html#purge-properties"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Publishing Data","doc":"Either Astarte or a device might publish new data on a interface/endpoint specific topic. The topic is built using /<interface name>/<path> schema, and it is used regardless of the type of interface or mapping being used. Also / path is a valid path for object aggregated interfaces. The following device topics are valid: /com.example.MyInterface/some/path /org.example.DraftInterface/otherPath /com.example.astarte.ObjectAggregatedInterface/ Data messages QoS is chosen according to mapping settings, such as reliability. Properties are always published using QoS 2. Interface Type Reliability QoS properties always unique 2 datastream unreliable 0 datastream guaranteed 1 datastream unique 2 Payload Format Payload format might change according to the message type. Payloads are always BSON encoded, except for unset messages that are empty. Property Message Property messages have a "v" key (which means value). Valid examples are: {"v": "string property value"} {"v": 10} {"v": true} Previous payloads are BSON encoded as the following hex dumps: 0000000 22 00 00 00 02 76 00 16 00 00 00 73 74 72 69 6 e 0000020 67 20 70 72 6 f 70 65 72 74 79 20 76 61 6 c 75 65 0000040 00 00 0000000 0 c 00 00 00 10 76 00 0 a 00 00 00 00 0000000 09 00 00 00 08 76 00 01 00 Property messages order must be preserved and they must be consumed in order. The same property with the same value can be sent several times, this behavior is allowed but discouraged: it's up to the device to avoid useless messages. A device must also make sure to publish all the properties that have been changed while the device was offline. Unset Property Message Properties can be unset with an unset message. An unset message is just an empty 0 bytes payload. Datastream Message (individual aggregation) Datastream messages for interfaces with individual aggregation have a "v" key and an optional "t" key (which means timestamp). Valid examples are: {"v": false} {"v": 16.73} {"v": 16.73, "t": 1537449422890} Timestamps are UTC timestamps (BSON 0x09 type), when not provided reception timestamp is used. Previous payloads are BSON encoded as the following hex dumps: 0000000 09 00 00 00 08 76 00 00 00 0000000 10 00 00 00 01 76 00 7 b 14 ae 47 e1 ba 30 40 00 0000000 1 b 00 00 00 09 74 00 2 a 70 20 f7 65 01 00 00 01 0000020 76 00 7 b 14 ae 47 e1 ba 30 40 00 Datastream Message (object aggregation) Datastream messages for interfaces with object aggregation support every Astarte payload standard field (such as "t"), but in this case value is a BSON subdocument, in which each key represent a mapping of the aggregation. Valid examples are: {"v": {"temp": 25.3123, "hum": 67.112}} {"v": {"temp": 25.3123, "hum": 67.112}, "t": 1537452514811} Timestamps are UTC timestamps (BSON 0x09 type), when not provided reception timestamp is used. Previous payloads are BSON encoded as following hex dumps: 0000000 28 00 00 00 03 76 00 20 00 00 00 01 68 75 6 d 00 0000020 ba 49 0 c 02 2 b c7 50 40 01 74 65 6 d 70 00 72 8 a 0000040 8 e e4 f2 4 f 39 40 00 00 0000000 33 00 00 00 09 74 00 fb 9 d 4 f f7 65 01 00 00 03 0000020 76 00 20 00 00 00 01 68 75 6 d 00 ba 49 0 c 02 2 b 0000040 c7 50 40 01 74 65 6 d 70 00 72 8 a 8 e e4 f2 4 f 39 0000060 40 00 00","ref":"080-mqtt-v1-protocol.html#publishing-data"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Minimal Protocol","doc":"A device might implement a subset of this protocol if needed. /control/consumer/properties , /control/producer/properties and /emptyCache might be ignored or not implemented if a device has no property interfaces. A further simplification might remove any requirement for any introspection message when previously provisioned, but this feature is not supported out of the box.","ref":"080-mqtt-v1-protocol.html#minimal-protocol"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Error Handling","doc":"A device might be forcefully disconnected due to any kind of error. Devices should wait a random amount of time before trying to connect again to the broker. session present might be also set to false to ensure a clean and consistent state (in that case messages such as introspection and empty cache should published as previously described). Malformed or unexpected messages are discarded and further actions might be taken.","ref":"080-mqtt-v1-protocol.html#error-handling"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Authentication","doc":"In Astarte, every Transport orchestrates its credentials through Pairing. Astarte/VerneMQ authenticates devices using Mutual SSL Autentication - as such, devices use SSL certificates emitted through Pairing API to authenticate against the broker. To achieve this, the device must ensure it is capable of performing http(s) calls to Pairing API to obtain its certificates, performing SSL/X509 operations and connecting to the MQTT Broker through the use of SSL certificates.","ref":"080-mqtt-v1-protocol.html#authentication"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Authorization","doc":"Device can only publish and subscribe to its device topic ( <realm name>/<device id> ) and its subtopics. The broker will deny any publish or subscribe outside that hierarchy.","ref":"080-mqtt-v1-protocol.html#authorization"},{"type":"extras","title":"Astarte MQTT v1 Protocol - Connecting to the Broker","doc":"In the same fashion as Authentication, Pairing provides the client with information about how to connect to the MQTT broker. When invoking relevant Pairing API's method to gather information about available transports for a device, if Astarte advertises Astarte MQTT v1, a similar reply will be returned: { "data": { "version": "<version string>", "status": "<status string>", "protocols": { "astarte_mqtt_v1": { "broker_url": "mqtts://broker.astarte.example.com:8883" } } } }","ref":"080-mqtt-v1-protocol.html#connecting-to-the-broker"},{"type":"extras","title":"Astarte Database","doc":"Astarte leverages Cassandra to store all of its data, including data ingested from devices (which might scale to insane amounts). Cassandra offers scalability and high availability with good performances . Cassandra offers linear scalability and can span from really small clusters to hundreds of nodes, without compromising on reliability. ScyllaDB >= 3.3 is also supported as a drop-in replacement when a performance boost is needed. Cassandra is also the ideal storage for large-scale data processing with Apache Spark . Astarte is multi-tenant by design, with each tenant mapping to an Astarte Realm. Each Realm has its own Cassandra keyspace, which can be tuned according to Realm-specific needs (e.g.: Realms might have different replication levels). For this reason, in the scope of this section, realm and keyspace can be used as synonyms, except for the astarte keyspace.","ref":"090-database.html"},{"type":"extras","title":"Astarte Database - Schema and Keyspace Creation","doc":"Astarte automatically takes care of keyspaces, tables creation and intra-version migrations (those tasks are performed by astarte_housekeeping or astarte_realm_management , depending on the context). The following documentation is just a reference about Astarte's internal statements, and is related to the release series referenced by the documentation. Astarte Keyspace Astarte needs an astarte keyspace to store its own data. astarte keyspace and tables are created with following CQL statements: CREATE KEYSPACE astarte WITH replication = {'class': 'SimpleStrategy', 'replication_factor': <replication factor>} AND durable_writes = true; CREATE TABLE astarte.realms ( realm_name varchar, PRIMARY KEY (realm_name) ); CREATE TABLE astarte.kv_store ( group text, key text, value blob, PRIMARY KEY (group, key) ) Realm Creation Each realm needs several tables to store data for all the functionalities. Realm tables can be grouped in the following functionalities: Configuration & key-value store Interfaces schema Device management Groups management Triggers storage Data storage Some data storage tables might be created when required, whereas all other tables are created when a keyspace is created, using the following statements: CREATE KEYSPACE <realm name> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': :replication_factor} AND durable_writes = true; CREATE TABLE <realm name>.kv_store ( group varchar, key varchar, value blob, PRIMARY KEY ((group), key) ); CREATE TABLE <realm name>.names ( object_name varchar, object_type int, object_uuid uuid, PRIMARY KEY ((object_name), object_type) ); CREATE TABLE <realm_name>.devices ( device_id uuid, aliases map<ascii, varchar>, introspection map<ascii, int>, introspection_minor map<ascii, int>, old_introspection map<frozen<tuple<ascii, int>>, int>, protocol_revision int, first_registration timestamp, credentials_secret ascii, inhibit_credentials_request boolean, cert_serial ascii, cert_aki ascii, first_credentials_request timestamp, last_connection timestamp, last_disconnection timestamp, connected boolean, pending_empty_cache boolean, total_received_msgs bigint, total_received_bytes bigint, exchanged_bytes_by_interface map<frozen<tuple<ascii, int>>, bigint>, exchanged_msgs_by_interface map<frozen<tuple<ascii, int>>, bigint>, last_credentials_request_ip inet, last_seen_ip inet, attributes map<varchar, varchar>, groups map<text, timeuuid>, PRIMARY KEY (device_id) ); CREATE TABLE <realm name>.grouped_devices ( group_name varchar, insertion_uuid timeuuid, device_id uuid, PRIMARY KEY ((group_name), insertion_uuid, device_id) ); CREATE TABLE <realm name>.endpoints ( interface_id uuid, endpoint_id uuid, interface_name ascii, interface_major_version int, interface_minor_version int, interface_type int, endpoint ascii, value_type int, reliability int, retention int, expiry int, database_retention_ttl int, database_retention_policy int, allow_unset boolean, explicit_timestamp boolean, description text, doc text, PRIMARY KEY ((interface_id), endpoint_id) ); CREATE TABLE <realm name>.interfaces ( name ascii, major_version int, minor_version int, interface_id uuid, storage_type int, storage ascii, type int, ownership int, aggregation int, automaton_transitions blob, automaton_accepting_states blob, description text, doc text, PRIMARY KEY (name, major_version) ); CREATE TABLE <realm name>.simple_triggers ( object_id uuid, object_type int, parent_trigger_id uuid, simple_trigger_id uuid, trigger_data blob, trigger_target blob, PRIMARY KEY ((object_id, object_type), parent_trigger_id, simple_trigger_id) ); CREATE TABLE <realm name>.individual_datastreams ( device_id uuid, interface_id uuid, endpoint_id uuid, path text, value_timestamp timestamp, reception_timestamp timestamp, reception_timestamp_submillis smallint, binaryblob_value blob, binaryblobarray_value list<blob>, boolean_value boolean, booleanarray_value list<boolean>, datetime_value timestamp, datetimearray_value list<timestamp>, double_value double, doublearray_value list<double>, integer_value int, integerarray_value list<int>, longinteger_value bigint, longintegerarray_value list<bigint>, string_value text, stringarray_value list<text>, PRIMARY KEY ((device_id, interface_id, endpoint_id, path), value_timestamp, reception_timestamp, reception_timestamp_submillis) ) CREATE TABLE <realm name>.individual_properties ( device_id uuid, interface_id uuid, endpoint_id uuid, path text, reception_timestamp timestamp, reception_timestamp_submillis smallint, double_value double, integer_value int, boolean_value boolean, longinteger_value bigint, string_value text, binaryblob_value blob, datetime_value timestamp, doublearray_value list<double>, integerarray_value list<int>, booleanarray_value list<boolean>, longintegerarray_value list<bigint>, stringarray_value list<text>, binaryblobarray_value list<blob>, datetimearray_value list<timestamp>, PRIMARY KEY((device_id, interface_id), endpoint_id, path) ); The following table is generated upon datastream interface creation for keeping all data sent to Astarte through the interface. The table name is derived from lower case interface name where . and - have been replaced by _ and "" (empty string), then the major version is appended with a _v prefix. For example, com.Astarte.TestInterface version 1 becomes com_astarte_testinterface_v1 . If, after all the required transformations, the resulting name is too long (>45 chars), it will be encoded and truncated. CREATE TABLE <interpolated interface name>_v<major_version> ( device_id uuid, path text, reception_timestamp timestamp, reception_timestamp_submillis smallint, v_<property_mapping> <property_type> v_<property_mapping> <property_type> ... PRIMARY KEY ((device_id, path), reception_timestamp, reception_timestamp_submillis) )","ref":"090-database.html#schema-and-keyspace-creation"},{"type":"extras","title":"Astarte Database - Tables","doc":"Devices The devices table stores the list of all the devices for a certain realm and all their metadata, including the introspection, the device status and credentials information. Column Name Column Type Description device_id uuid Device unique 128 bits ID. aliases map<ascii, varchar> Alias purpose and alias map. introspection map<ascii, int> Device interface name to interface major version map based on most recent device introspection. introspection_minor map<ascii, int> Device interface name to interface minor version map based on most recent device introspection. old_introspection map<frozen<tuple<ascii, int>>, int> All previous device interfaces. This column is used to keep track of all interfaces that have been used and might still have some recorded data. The column maps interface (name, major) to minor. protocol_revision int Spoken Astarte MQTT v1 protocol revision. first_registration timestamp First registration attempt timestamp. credentials_secret ascii The bcrypt hash of the credential secret, that the device uses to obtain new credentials. inhibit_credentials_request boolean Ban device credentials renewal, device will be able to connect to the transport up to the credential expiry. cert_serial ascii Device certificate serial used by the CA. cert_aki ascii Device certificate Authority Key Identifier. first_credentials_request timestamp First credentials request timestamp. last_connection timestamp Most recent device connection event timestamp. last_disconnection timestamp Most recent device disconnection event timestamp. connected boolean True if the device is connected, otherwise is false. pending_empty_cache boolean Device is in an unclean state and an empty cache message is being waited. total_received_msgs bigint Count of received messages since the device registration. total_received_bytes bigint Amount of received messages bytes since the device registration. exchanged_msgs_by_interface bigint Count of exchanged messages since the device registration. exchanged_bytes_by_interface bigint Amount of exchanged messages bytes since the device registration. last_credentials_request_ip inet Device IP address used during the last credential request. last_seen_ip inet Most recent device IP address. attributes map<varchar, varchar> Device attributes. It can contain arbitrary string key and values associated with the device. groups map<text, timeuuid> Groups which the device belongs to, the key is the group name, and the value is its insertion timeuuid, which is used as part of the key on grouped_devices table. Endpoints The endpoints table stores the list of all endpoints of all interfaces for realm, with all the data needed to define an endpoint, such as retention, realiability, value type and so on. Column Name Column Type Description interface_id uuid Interface unique 128 bits ID. endpoint_id uuid Endpoint unique 128 bits ID. interface_name ascii Human-readable name for interface. interface_major_version int Interface major version related to the endpoint. interface_minor_version int Interface minor version related to the endpoint. interface_type int Interface type identifier related to the endpoint. endpoint ascii Human-readable endpoint string. value_type int Value type identifier related to the endpoint. reliability int Reliability identifier related to the endpoint. retention int Retention identifier related to the endpoint. expiry int Expiry identifier related to the endpoint. database_retention_ttl int Milliseconds before data deletion. database_retention_policy int Database_retention_policy identifier related to the endpoint. allow_unset boolean Enable or disable possibility of setting value to null. explicit_timestamp boolean Set or unset explicit timestamp. description text Description of endpoint. doc text Documentation for endpoint. Interfaces The interfaces table stores the list of all interfaces for realm, with all the data needed to define an endpoint, such as retention, realiability, value type and so on. Column Name Column Type Description interface_id uuid Interface unique 128 bits ID. name ascii Human-readable name for interface. major_version int Interface major version related to the endpoint. minor_version int Interface minor version related to the endpoint. storage_type int Storage type identifier related to the endpoint. storage ascii Interface storage. type int Identifies the type of this Interface. Currently two types are supported: datastream and properties. ownership int Identifies the quality of the interface. Interfaces are meant to be unidirectional, and this property defines who's sending or receiving data. aggregation int Identifies the aggregation of the mappings of the interface. automaton_transitions blob Automaton internal field. automaton_accepting_states blob Automaton internal field. description text Description of interface. doc text Documentation of interface.","ref":"090-database.html#tables"},{"type":"extras","title":"Astarte Database - Schema changes","doc":"This section describes the schema changes happening between different Astarte Versions. They are divided between Astarte Keyspace (changes that affect the Astarte Keyspace), and Realm Keyspaces (changes that affect all realm keyspaces). Every change is followed by the CQL statement that produces the change. From v0.10 to v0.11 Astarte Keyspace v0.11 Changes Remove astarte_schema table DROP TABLE astarte_schema; Remove replication_factor column from the realms table ALTER TABLE realms DROP replication_factor; Realm Keyspaces v0.11 Changes Add grouped_devices table CREATE TABLE <realm_name>.grouped_devices ( group_name varchar, insertion_uuid timeuuid, device_id uuid, PRIMARY KEY ((group_name), insertion_uuid, device_id) ); Add groups , exchanged_bytes_by_interface and exchanged_msgs_by_interface columns to the devices table ALTER TABLE <realm_name>.devices ADD (groups map<text, timeuuid>, exchanged_bytes_by_interface map<frozen<tuple<ascii, int>>, bigint>, exchanged_msgs_by_interface map<frozen<tuple<ascii, int>>, bigint>); Add database_retention_ttl and database_retention_policy columns to the endpoints table ALTER TABLE <realm_name>.endpoints ADD ( database_retention_ttl int, database_retention_policy int ); From v0.11 to v1.0.0-beta.1 Realm Keyspace v1.0.0-beta.1 Changes The connected field of the devices table is now saved with a TTL, so it automatically expires if it doesn't gets refreshed by the hearbeat sent by the broker. This behaviour was added to avoid stale connected devices if they disconnect while the broker is down. Add metadata column to the devices table ALTER TABLE devices ADD ( metadata map<varchar, varchar> ); From v1.0-beta.1 to v1.0.0 Realm Keyspace v1.0.0 Changes Rename the metadata to attributes in the devices table Warning : migrating data from the metadata column to the attributes one is possible but is out of scope of this guide since this change happened between development releases. The procedure below just creates the new column and then deletes the old one without migrating data . You're free to implement a migration procedure between the two steps. ALTER TABLE devices ADD ( attributes map<varchar, varchar> ); ALTER TABLE devices DROP metadata;","ref":"090-database.html#schema-changes"},{"type":"extras","title":"Introduction","doc":"Astarte is an Open Source IoT platform focused on Data management. It takes care of everything from collecting data from devices to delivering data to end-user applications. To achieve such a thing, it uses a mixture of mechanisms and paradigm to store organized data, perform live queries. This guide focuses on daily operations for Astarte users and integrators. It goes through fundamental operations such as setting up triggers, querying APIs, integrating 3rd party applications and more. The user guide starts from the assumption that the reader is interacting with one or more well-known realms , and throughout the manual the assumption is that we're always operating inside a test realm, unless otherwise specified. Setting up realms is out of the scope of this guide, also because it's not a task the average user has to deal with. Please refer to the dedicated chapter of the Administrator manual to learn more about this specific topic. Before you begin, make sure you are familiar with Astarte's architecture, design and concepts .","ref":"001-intro_user.html"},{"type":"extras","title":"Interacting with Astarte","doc":"Astarte's interaction is logically divided amongst two main entities. Devices are the bottom end, and represent your IoT fleet. They can access Astarte only through a Transport, they are defined by a set of Interfaces which, in turn, also define on a very granular level which kind of data they can exchange. By design, they can't access any resource which isn't their own: such a behavior can be configured using Astarte as a middleman to act as a secure Gateway. Users are actual users, applications or anything else which needs to interact directly with Astarte. They are bound to a realm, and can virtually access any resource in that realm given they're authorized to do so. Users can also manage triggers and perform maintenance activity on the Realm.","ref":"010-interacting_with_astarte.html"},{"type":"extras","title":"Interacting with Astarte - User-side Tools","doc":"When interacting with Astarte as a User, you have several options to choose from: astartectl : astartectl is the main command-line tool to interact with Astarte clusters, which packs in a number of subcommands to interact with Astarte API sets. It is a swiss army knife to perform daily operations on Astarte Clusters, and it abstracts most Astarte API interactions in a user-friendly way. Astarte Dashboard : Astarte provides a built-in UI that can be used for managing Interfaces, Devices and Triggers. It is meant to be a graphical, user-friendly tool to perform daily operations on Realms. Astarte API Clients: API Clients are provided for a variety of languages. These clients abstract API interaction with language-friendly paradigms, and provide API automations for several operations. Currently, the main API client available is astarte-go . Astarte APIs: The base APIs are the lower level interaction layer. They are accessible, in standard installations, at api.<base Astarte URL>/<apiset> , and are the main mean of interaction upon which all other clients are based upon. Grafana Datasource Plugin for Astarte : Thanks to the Astarte Datasource Plugin, data coming from Astarte may be visualised in custom dashboards provided by Grafana, the open source observability platform. Depending on the context, you might want to choose what suits you best. Over the course of the documentation, several examples will be provided with interaction means. Setting up astartectl In the documentation, it is assumed that astartectl is properly configured to interact with your Realm or your Cluster. Please refer to its documentation to make sure all needed configurations are in place.","ref":"010-interacting_with_astarte.html#user-side-tools"},{"type":"extras","title":"Interacting with Astarte - Interacting with a Device","doc":"Devices interact with Astarte through their associated Transport. In this guide, we'll assume the Transport is MQTT/VerneMQ as per Astarte's defaults. However, rather than implementing the whole Astarte protocol over MQTT, it is usually a better idea to rely on one of Astarte's SDKs . Authentication/Pairing Depending on how you plan on implementing Astarte's pairing mechanism , your devices might need an Agent for their first authentication or not. However, once they retrieve their Credentials Secret, they can implement Astarte's standard pairing routine to rotate their SSL certificate for accessing the transport. In the most likely scenario in which you are using one of Astarte's SDKs, the SDK takes care of the whole pairing routine under the hood and, depending on your agent implementation, you just need to feed the SDK with either the Credentials Secret or the Agent Key. Exchanging data As per Astarte's protocol specification, data is exchanged based on the device's introspection. The device will be able to publish data on the transport on device interfaces, and receive data on server interfaces. In the MQTT case, the device will subscribe to its server interfaces' topics, and publish on its device interfaces topics. Isolation and RBAC are guaranteed by the transport's ACL, which are usually orchestrated though a dedicated Astarte extension (as in the VerneMQ/MQTT case). Again, Astarte's SDK allows you to interact with your device interfaces directly without caring about the underlying protocol and exchange details.","ref":"010-interacting_with_astarte.html#interacting-with-a-device"},{"type":"extras","title":"Interacting with Astarte - Interacting as a User","doc":"Astarte is mainly accessed through its APIs. Astarte's APIs are exposed through dedicated microservices (see Components ) and are meant both for configuration and for accessing data. There are two main sets of APIs we'll be using frequently: AppEngine API : This API is meant for querying/pushing data from/to devices. This maps to astartectl 's astartectl appengine subcommand. Realm Management API : This API is meant for configuring a target realm, and most notably for managing triggers. This maps to astartectl 's astartectl realm-management subcommand. Authentication Authenticating against Astarte is out of the scope of this guide, especially due to the fact that Astarte does not manage authentication directly . We'll assume either the authentication isn't enabled, or that the user is always interacting with the APIs with a token with the following claims { "a_aea": ".*:.*", "a_rma": ".*:.*" } Which represents a realm administrator. In real life use cases, you should always make sure to give out more granular permissions and to obtain the token in the right way from your authentication server. When using astartectl or any other client, you can also pass a Realm Private Key as an authentication mean, and have the token be automatically generated for you. Accessing the APIs In a standard Astarte installation, AppEngine API and Realm Management API are usually accessible at api.<your astarte domain>/appengine and api.<your astarte domain>/realmmanagement . If your Astarte installation has Swagger UI enabled, you can use the /swagger endpoint to access it, and to issue API calls straight from your browser to follow this guide.","ref":"010-interacting_with_astarte.html#interacting-as-a-user"},{"type":"extras","title":"Astarte Dashboard","doc":"Astarte provides a built-in UI that can be used for managing Interfaces, Devices, and Triggers. The Dashboard simplifies the development phases of applications that make use of Astarte, as well as troubleshooting activities. You can browse the source code of the Dashboard software on its GitHub repository .","ref":"015-astarte_dashboard.html"},{"type":"extras","title":"Astarte Dashboard - Introduction","doc":"The Astarte Dashboard is a Single Page Application that provides users with an overview of their Realm and a user-friendy way of managing it on any web browser. The Dashboard is designed to be a quick and easy way to give you immediate feedback on your work and as a quick and intuitive way to configure your realm. It is not designed to be operated by end-users, rather by infrastructure maintainers, owners who need information on the system status, and those working on projects based on Astarte. It is shipped by default with the Astarte standard distribution. The Dashboard is a graphical client for Astarte APIs; it shares similar features with the CLI client astartectl , the command-line utility to manage Astarte. The Dashboard helps you manage: Triggers Interfaces Devices Groups Realm Settings In case your Astarte distribution comes packaged with the Flow framework , the Dashboard is probably configured to manage Flow resources as well. Blocks Pipelines Flows Please note that the Flow framework is not available for use in a docker-compose environment since it relies on Kubernetes APIs to operate. How to access it Depending on how you are using Astarte, here is where you can find the Dashboard: Docker-compose: if you are using a local instance of Astarte via docker-compose , you will find it by pointing your browser to the default address http://dashboard.astarte.localhost . To login, fill in the name of your realm and a valid JWT token: if you possess the realm private key, as it is the case if you followed the Astarte in 5 minutes guide, you can generate the token with the command astartectl utils gen-jwt all-realm-apis -k <private_key> . Astarte Cloud: if you are using our managed Astarte option, you can hop onto our Console , find or create your realm and click the Dashboard button. Kubernetes cluster: in this case, if the Dashboard is enabled, it can be usually found at dashboard.<base Astarte URL> . Refer to your system administrator for more details.","ref":"015-astarte_dashboard.html#introduction"},{"type":"extras","title":"Astarte Dashboard - Main overview","doc":"Upon successful login, the main screen is the home page that provides an overview of the realm status and resources. API Status The API Status gives you general information about the status of services: Realm Management is an administrator-like API for configuring a Realm. It is used for managing Interfaces and Triggers. AppEngine is Astarte's main API endpoint for end-users. AppEngine exposes a RESTful API to retrieve and send data from/to devices, according to their interfaces. Every direct device interaction can be done from here. It also exposes Channels, a WebSocket-based solution for listening to device events in real-time with the same mechanism and semantics used by Triggers. Pairing takes care of Device Authentication and Authorization. It interacts with Astarte's CA and orchestrates the way devices connect and interact with Transports. It also handles Device Registration. Agent, Device and Pairing interaction is described in detail here . Flow is the API endpoint for Astarte Flow, used for managing Blocks, Pipelines, and Flows. Possible statuses are: This service is operating normally. This service appears offline. A general status on API health is also present in the app's sidebar, thus always providing a feedback regardless of which page you are currently visiting. Realm resources Within the main overview, a brief summary is available as well for the existing resources of the realm: registered and connected devices, installed interfaces, and installed triggers. More detailed overviews of each resource are available in the dedicated Dashboard sections, accessible via the navigation links in the Dashboard's side menu.","ref":"015-astarte_dashboard.html#main-overview"},{"type":"extras","title":"Astarte Dashboard - Interfaces","doc":"Interfaces are a core concept of Astarte which defines how data is exchanged between Astarte and its peers. You can navigate to this section thanks to the side menu of the Dashboard. A list of all installed interfaces is displayed, together with their major versions. Installing interfaces From the Interface list, clicking on the Install a new interface button will load up the Interface Editor , an interactive tool that you can use to configure your interfaces. The Interface Editor provides you with two ways to define your interfaces: on the left panel, a graphical frontend, while on the right panel you may input a JSON definition to achieve the same result. Each panel updates automatically whenever the other is changed. While defining a new interface, the Interface Editor will help you in supplying the right options and filling in mandatory entries such as: Name: an arbitrary name, formatted in reversed DNS casing. Major and Minor versions: based on Semantic Versioning . Type: indicates whether data is streamed continuously ( datastream ) or is stateful and persistent ( properties ). Ownership: the write-only allowed actor. All the other actors are read-only. Mappings: a list of endpoints that represent the data structure, following REST controller semantics. You can learn more about Interface definitions in their documentation's section . Note that when creating interface drafts, or for testing purposes in general, it is recommended to use 0 as the major version: to prevent data loss, Astarte allows only interfaces where major_version equals 0 to be deleted. Managing interfaces From the Interface list, you can select an interface to load and view its details in the Interface Editor. Clicking on the name of the interface will select its latest revision; clicking on a specific major of the interface will select the latest revision for that major. Note that interface revisions follow the Semantic Versioning convention. Once the Interface Editor is loaded you can review, update or delete the definition of the interface. Note that to prevent data loss, Astarte allows only interfaces where major_version equals 0 to be deleted. For similar reasons, when updating the definition of an interface, the Interface Editor will not allow you to change core properties on a minor version update. If you need to apply substantial changes, you can define and install a new major version for the interface.","ref":"015-astarte_dashboard.html#interfaces"},{"type":"extras","title":"Astarte Dashboard - Triggers","doc":"Triggers in Astarte are the go-to mechanism for generating push events. You can navigate to this section thanks to the side menu of the Dashboard. A list of all installed triggers is displayed. Installing triggers From the Trigger list, clicking on the Install a new trigger button will load up the Trigger Editor , an interactive tool that you can use to configure your triggers. It works in a very similar fashion to Interface Editor and shares the same User Interface. The Trigger Editor provides you with two ways to define your triggers: on the left panel, a graphical frontend, while on the right panel you may input a JSON definition to achieve the same result. Each panel updates automatically whenever the other is changed. The graphical tool dynamically loads installed Interfaces in the Realm and eases trigger creation by providing not only linting and validation, but also dynamic resolution of Interface names. You can learn more about Trigger definitions in their documentation's section . Note that due to how triggers work, you should install the trigger before a device connects. Doing otherwise will cause the trigger to kick in at a later time, and as such no events will be streamed for a while. Managing triggers From the Trigger list, you can select a trigger to load and view its details in the Trigger Editor. Once the Trigger Editor is loaded you can review the definition of the trigger. You can also delete the trigger instance by clicking on the Delete trigger button.","ref":"015-astarte_dashboard.html#triggers"},{"type":"extras","title":"Astarte Dashboard - Devices","doc":"Devices are Astarte's main entities for exchanging data. You can navigate to this section thanks to the side menu of the Dashboard. A list of all registered devices is displayed. Each device in the list is displayed together with info regarding its status and the last connection event . The status is represented by a grey dot if the device never connected to Astarte, a green dot if it is currently connected, a red dot if it is currently disconnected. The last connection event reports, if available, the date of the last connection or disconnection. A filter section is present on the side of the list to aid the search for specific devices, filtering the list by device ID, name, connection status, or configured attributes. Registering a device From the Device list, new devices may be registered by clicking on the Register a new device button which will take you to the registration page. Here you can proceed with the registration of the device by: providing a device ID: either by generating a random ID or by specifying a Name and Namespace UUID to generate the ID in a deterministic fashion. optionally declaring the initial introspection of the device: this is an indication of the list of interfaces that the device will use to exchange data. You can learn more about Devices and the registration process in their documentation's section . Device status and details From the Device list, you can select a device to navigate to its dedicated page. Here you can review and manage different info about your device. Device Info Device Info : displays info such as the device ID and the device Name alias, if set. It reports whether the device is currently connected or disconnected, or if it was never connected. From this section you can also momentarily Inhibit credentials for the device, preventing it to obtain access to Astarte; or you can directly Wipe credential secret of the device, a permanent action which will require to register the device again to have a new Credential Secret. Aliases : where you can manage custom aliases for the device. Note that setting a name alias will provide a name for the device. Attributes : a dedicated section to attach arbitrary info to the device, in a key-value form. Groups : where you can review and manage the Groups the device belongs to. Interfaces : a list of all currently and previously used interfaces. Clicking on an interface's name will load a dedicated page to review data exchanged by the device through that interface. Stats : a rundown on exchanged data via different interfaces. Here you can review, in both visual and numeric form, the quantity of bytes and messages the device is exchanging over each interface. This way you can always know at a glance which interfaces are the busiest and how chatty your device is. Status Events : here is info collected by Astarte regarding the IP addresses involved in the connections of the device to Astarte, the dates of first registration and credentials request, and the dates of last connection and disconnection, if available. Live Events : a section that reports live events regarding the device. It makes use of Astarte Channels and displays the connections, exchanged data, and errors of the device, as they happen in real-time.","ref":"015-astarte_dashboard.html#devices"},{"type":"extras","title":"Astarte Dashboard - Groups","doc":"Groups are logical collections of devices to ease the management and querying of devices. You can navigate to this section thanks to the side menu of the Dashboard. A list of all existing groups is displayed, together with the number of total and connected devices for each group. Creating a group From the Group list, clicking on the Create a new group button will load a dedicated page to setup the new group. You are required to specify a name for the group and to select at least one device that will belong to it. Indeed, note that a group must contain at least one device to exist. To confirm the creation of the new group, click the Create group button. Managing groups From the Group list, you can select a group to view its configuration on a dedicated page. Here you can review the list of devices that belong to it. To remove a device from a group, click on the Delete icon next to it. To add a device to a group, you can first navigate to the device's page and then add it to a group from there. To delete a group, remove all devices that belong to it and the group will automatically cease to exist.","ref":"015-astarte_dashboard.html#groups"},{"type":"extras","title":"Astarte Dashboard - Blocks","doc":"Blocks are computation units that can be chained together to define a logical computation topology. You can navigate to this section thanks to the side menu of the Dashboard. Here is a list of all existing blocks, both custom and native ones: the former ones are those defined by you, the latter ones are those provided by Astarte and are displayed with a native label. Creating blocks From the Block list, clicking on the Create button will load a dedicated page to define a new block. Here you can define the block by specifying: A name for the block. A type , between Producer , Consumer or Producer & Consumer , depending on how the block should treat messages and connect to other blocks. A schema , reporting the JSON Schema definition of how a configuration should look like for the block. A source , containing the implementation of the block, written in the DSL format for Pipelines. Confirm the creation of the block by clicking on the Create new block button. To learn more about block definition you can read their documentation's section . Managing blocks From the Block list, you can select a block to view its definition in a dedicated page. Here you can review the details of the block such as the block type, the schema and, if it is a custom block, its source. To delete a block, click on the Delete block button. Note that you cannot delete native blocks provided by Astarte.","ref":"015-astarte_dashboard.html#blocks"},{"type":"extras","title":"Astarte Dashboard - Pipelines","doc":"A Pipeline is a computation blueprint (therefore a description) built as a chain of blocks. You can navigate to this section thanks to the side menu of the Dashboard. Here is a list of all existing pipelines from which you can review, manage and instantiate them. Creating pipelines From the Pipeline list, clicking on the Create button will load up the Pipeline Editor page, an interactive tool that you can use to configure your pipeline. The Pipeline Editor is composed of two parts: A sidebar listing all available Blocks, grouped by type A space where you can drag & drop blocks, connecting them to effectively design a pipeline You can read more about the Pipeline Editor on the dedicated documentation . Once you have designed your pipeline, you can review and specify its: name source , containing the implementation of the pipeline, written in the DSL format for Pipelines. schema , reporting the JSON Schema definition of how a configuration should look like for the pipeline. description , explaining the scope and supposed usage of the pipeline. Then hit the Create new pipeline button to confirm the definition of the pipeline. Managing pipelines From the Pipeline list, you can select a pipeline to view its definition in a dedicated page. Here you can review the details of the pipeline such as the pipeline description, its schema, and its source. To delete a pipeline, click on the Delete pipeline button.","ref":"015-astarte_dashboard.html#pipelines"},{"type":"extras","title":"Astarte Dashboard - Flows","doc":"Flows are specific instances of a pipeline, created providing concrete values to the parametric configuration of a pipeline. You can navigate to this section thanks to the side menu of the Dashboard. Here is a list of all existing flows. Each flow reports its current status and the pipeline it originated from. To review the details of a Flow, click on its name to load up the dedicated page. To delete a flow, click on the Delete icon next to it. Instantiating flows From the Pipeline list, clicking on the Instantiate button of a Pipeline will load the Flow configuration page; from there you can supply a name and a configuration for the flow before hitting the Instantiate Flow button to confirm.","ref":"015-astarte_dashboard.html#flows"},{"type":"extras","title":"Astarte Dashboard - Realm Settings","doc":"You can navigate to this section thanks to the side menu of the Dashboard. Here you can review and update settings for your realm. You can update the public key of the realm, which is useful if you mean to use a new private key to generate auth tokens. Please note that it is a permanent action and Astarte will prevent interactions that use auth tokens generated with the previous key.","ref":"015-astarte_dashboard.html#realm-settings"},{"type":"extras","title":"Accessing and Exploring a Realm","doc":"In Astarte, a Realm is a logical partition which holds a number of devices and an Authentication Realm. The Astarte Dashboard allows you to explore all resources of a given Realm, such as e.g. Interfaces, Triggers, Devices, Groups etc...","ref":"020-accessing_and_exploring_a_realm.html"},{"type":"extras","title":"Accessing and Exploring a Realm - Device limit in a Realm","doc":"While there is no limit on the number of Devices registered in a Realm, it is possible that an upper bound has been set in the Realm configuration using the Housekeeping API . When it is set, trying to register more Devices past the limit will result in an error. The limit can be retrieved from Realm Management with GET <astarte base API URL>/realmmanagement/v1/<realm name>/config/device_registration_limit The HTTP payload of the response will have the following format: { "data": <value> } If such a limit is set, the value will be a non negative integer. If not, the value will be null .","ref":"020-accessing_and_exploring_a_realm.html#device-limit-in-a-realm"},{"type":"extras","title":"Interface Design Guide","doc":"Before we begin, let's get this straight: The way you design your interfaces will determine the overall performance and efficiency of your cluster This is because interfaces define not only the way data is exchanged between Astarte and Devices/Applications, but also how it will be stored, managed and queried . As such, it is fundamental to spend enough time on finding the most correct Interface design for your use case, keeping in mind how your users will consume your data, what might change in the future, what is fundamental and what is optional, and more.","ref":"029-interface_design_guide.html"},{"type":"extras","title":"Interface Design Guide - Use the right tools","doc":"Before you begin, you might want to take a look at Astarte Interfaces Editor , which is also available in any Astarte Dashboard installation. Astarte Interfaces Editor gives you automated validation and linting for Astarte Interfaces, and also gives you a declarative editor with automatic JSON generation. It is well maintained and used as a reference for Interface design. Consider using it for building your interfaces.","ref":"029-interface_design_guide.html#use-the-right-tools"},{"type":"extras","title":"Interface Design Guide - Rationale","doc":"Without going into deeper details on what concerns Astarte's DB internals, there are some considerations one should always keep in mind when designing interfaces. Querying an Interface is fast, querying across Interfaces is painful Astarte's data modeling is designed to optimize queries within a single interface. Querying across interfaces is supported, but might affect performances significantly, especially if done frequently and with complex queries. This is especially true for triggers, as they could be evaluated very frequently. In general, if you plan on having different mappings which are frequently queried altogether, or dependent on each other for several triggers, you might be better off in having them all in the same Interface. Aggregation makes a difference Aggregation is a powerful feature, which comes with price and benefits. Even though each series has only one timestamp for all values, it is also true that losing granularity for endpoints might cause storage of redundant data if only one of the aggregated mappings change value. Moreover, in terms of data modeling, Aggregated interfaces imply the creation of a dedicated Cassandra table. Having a lot of aggregated interfaces might end up putting additional pressure on the Cassandra Cluster in terms of memory and overall performance. Your Cluster administrator might (rightfully) choose to limit the amount of installed aggregate interfaces in a Realm, or in the overall Cluster.","ref":"029-interface_design_guide.html#rationale"},{"type":"extras","title":"Interface Design Guide - Interface Atomicity","doc":"Rule of thumb: Favor extreme atomicity in case you expect your interfaces to change often, be as atomic as reasonably possible in case you want to favor performance and flexibility in querying data.","ref":"029-interface_design_guide.html#interface-atomicity"},{"type":"extras","title":"Managing Interfaces","doc":"Interfaces define how data is exchanged over Astarte. For a Device to be capable of exchanging data into its Realm, its interfaces have to be registered into the Realm first. Let's walk over the whole process. It is assumed that you have read the Interface design guide before, to avoid bad surprises once your fleet starts rolling.","ref":"030-manage_interfaces.html"},{"type":"extras","title":"Managing Interfaces - Querying Interfaces","doc":"Listing Interfaces You can list all installed interfaces in a given Realm. This will return all the valid installed Interface names, without any versioning. List Interfaces using astartectl $ astartectl realm-management interfaces list [com.my.Interface1 com.my.Interface2 com.my.Interface3] List Interfaces using Astarte Dashboard From your Dashboard, after logging in, click on "Interfaces" in the left menu. List Interfaces using Realm Management API GET <astarte base API URL>/realmmanagement/v1/test/interfaces {"data": ["com.my.Interface1","com.my.Interface2","com.my.Interface3"]} Listing Major Versions for an Interface For each installed Interface, there can be any number of Major versions installed. This information can be retrieved by listing the available Major versions for a specific interface. In a realm, only the latest minor version of each major version of an Interface is kept. This can be done due to the fact that Semantic Versioning implies a new minor version doesn't introduce any breaking change (e.g.: deleting or renaming a mapping), and as such querying an older version of an interface using a newer one as a model is always compatible - some mappings might be empty, as expected, and will be disregarded. Astarte ensures upon Interface installation for this constraint, and as such you can always query the latest minor version of an Interface safely. List Versions using astartectl $ astartectl realm-management interfaces versions com.my.Interface1 [0 1 2] List Versions using Astarte Dashboard In the Dashboard's Interface page, click on any Interface name. A drop-down will appear, showing installed major versions for that Interface name. List Versions using Realm Management API GET <astarte base API URL>/realmmanagement/v1/test/interfaces/com.my.Interface1 {"data": [0,1,2]} Getting an Interface Definition Astarte allows you to retrieve the Interface Definition for a given Name and Major Version pair. The definition is in the standard Interface JSON format. Get Interface Definition using astartectl $ astartectl realm-management interfaces show com.my.Interface1 0 { "data": { "version_minor": 2, "version_major": 0, "type": "properties", "ownership": "device", "mappings": [ { "type": "integer", "path": "/myValue", "description": "This is quite an important value." }, { "type": "integer", "path": "/myBetterValue", "description": "A better revision, introduced in minor 2, supported only by some devices" }, { "type": "boolean", "path": "/awesome", "allow_unset": true, "description": "Introduced in minor 1, tells you if the device is awesome. Optional." } ], "interface_name": "com.my.Interface1" } } Get Interface Definition using Astarte Dashboard From the Interfaces page, click on an Interface name, and click on the Major version for which you'd like to see the definition. The Interfaces Editor window will open, with the Interface definition in the text box on the right. From the Editor page, it is also possible to add new mappings to the Interface and bump it to a new Minor. Get Interface Definition using Realm Management API GET <astarte base API URL>/realmmanagement/v1/test/interfaces/com.my.Interface1/0 { "data": { "version_minor": 2, "version_major": 0, "type": "properties", "ownership": "device", "mappings": [ { "type": "integer", "path": "/myValue", "description": "This is quite an important value." }, { "type": "integer", "path": "/myBetterValue", "description": "A better revision, introduced in minor 2, supported only by some devices" }, { "type": "boolean", "path": "/awesome", "allow_unset": true, "description": "Introduced in minor 1, tells you if the device is awesome. Optional." } ], "interface_name": "com.my.Interface1" } }","ref":"030-manage_interfaces.html#querying-interfaces"},{"type":"extras","title":"Managing Interfaces - Installing/Updating an interface","doc":"Interfaces are supposed to change over time, and are dynamic. As such, they can be installed and updated. Interface installation means adding either a whole new Interface (as in: an Interface with a new name), or a new major version of an already known Interface. Interface update means updating a specific, existing interface name/major version with a new minor version. When designing interfaces, it is strongly advised to use Astarte Interface Editor. The Editor is embedded into any Astarte Dashboard installation but, in case your Astarte installation does not provide you with a Dashboard, you can use Astarte Interface Editor public online instance . Use it to write and validate your definitions, and install the resulting JSON file through either astartectl or Realm Management APIs. Synchronizing interfaces using astartectl astartectl provides a handy sync command that, given a list of Interface files, will synchronize the state of the Astarte Realm with your local interfaces. It is handy in those cases where your Realm has several interfaces, and you're storing Interfaces in a common place, such as a Git Repository - this is the average case for Astarte-based applications/clouds. Assuming you have a set of Interface files in your folder all with the .json extension, invoking astartectl realm-management interfaces sync will result in something like this: $ astartectl realm-management interfaces sync *.json Will install interface com.my.Interface1 version 0.2 Will install interface com.my.Interface2 version 1.1 Will update interface com.my.Interface3 to version 1.4 Do you want to continue? [y/n] y Interface com.my.Interface1 installed successfully Interface com.my.Interface2 installed successfully Interface com.my.Interface3 updated successfully to version 1.4 After invocation, your Astarte Realm will be up to date with all Interfaces in your local directory. Note: astartectl realm-management interfaces sync currently synchronizes Interfaces only from your local machine to the Realm, and not the other way round. In case the Realm has a more recent version of an interface compared to your local files, or it has some interfaces which are not referenced by your local files, no action will be taken. Install an Interface using Astarte Dashboard Access the Editor by going to the Interfaces page, and clicking on "Install a New Interface..." in the top-right corner. The Editor will open. From there, you can either paste in an existing JSON definition, which will be validated and will update the left-screen declarative Editor, or you can build a whole new Interface from scratch. Once you're done, hit the "Install Interface" button at the bottom of the declarative Editor (left side) to install the Interface in the Realm. Install an Interface using astartectl First of all, ensure that you have the Interface you'd like to install saved in a file on your local machine. We will assume the interface is available as interface1.json . $ astartectl realm-management interfaces install interface1.json ok Install an Interface using Realm Management API Realm Management currently implements a completely asynchronous API for Interface installation - as such, the only feedback received by the API is that the Interface is valid and the request was accepted by the backend. However, this is no guarantee that the Interface will be installed successfully. As a best practice, it is advised to either wait a few seconds in between Realm Management API invocations, or verify through a GET operation whether the Interface has been installed or not. POST <astarte base API URL>/realmmanagement/v1/test/interfaces The POST request must have the following request body, with content type application/json { "data": { "version_minor": 2, "version_major": 0, "type": "properties", "ownership": "device", "mappings": [ { "type": "integer", "path": "/myValue", "description": "This is quite an important value." }, { "type": "integer", "path": "/myBetterValue", "description": "A better revision, introduced in minor 2, supported only by some devices" }, { "type": "boolean", "path": "/awesome", "allow_unset": true, "description": "Introduced in minor 1, tells you if the device is awesome. Optional." } ], "interface_name": "com.my.Interface1" } } The call will return either 201 Created or an error. Most common failure cases are: The interface/major combination already exists in the Realm The interface schema fails validation In any case, the API returns details on what caused the error and how to solve it through Astarte's standard error reply schema. Update an Interface using astartectl First of all, ensure that you have the Interface you'd like to update saved in a file on your local machine. We will assume the interface is available as interface1_3.json . $ astartectl realm-management interfaces update interface1_3.json ok Update an Interface using Astarte Dashboard Go to the Interfaces page, click on the Interface Name you'd like to update, and click on the Major version which is referred by your upgrade (e.g.: if you're updating from 1.2 to 1.3, you want to click on Major Version 1). The Editor will appear, populated with the currently installed Interface definition. Paste in your updated JSON file, or use the declarative editor to make your changes. The editor will be limited to Semantic Version-compatible operations (as in - adding new mappings). Once you're done, hit the "Apply Changes" button at the bottom of the declarative Editor (left side) to update the Interface in the Realm. Update an Interface using Realm Management API To update an existing interface, issue a PUT /interfaces/<name>/<major> endpoint of the realm with the very same semantics as the Installation procedure. The call will return either 201 Created or an error. Apart from the very same errors that could be triggered upon installation, Update will also fail if the interface doesn't provide a compatible upgrade path from the previously installed minor. Interface update limitations Major version updates Major version updates have no intrinsic limitations as they are not meant to ensure compatibility with older versions of the same interface. Therefore, if you plan to bump your interface major you are allowed to update your interface at your preference. Please, refer to the Interface Design Guide to follow the best practices while developing your new updated interface. Minor version updates Minor version updates are conceived to guarantee retro-compatibility and, as such, they allows only for a limited subset of update operations. Currently, based on the interface type and aggregation, different update capabilities are provided: properties : at interface root level, doc and description updates are allowed; at mapping level, doc and description updates are allowed. Moreover, an arbitrary number of new mappings can be added; individual datastream : at interface root level, doc and description updates are allowed; at mapping level, doc , description and explicit_timestamp updates are allowed. Moreover, an arbitrary number of new mappings can be added; object aggregated datastream : currently, due to a limitation in how data are stored within Cassandra, the doc , descriprion and explicit-timestamp fields can not be updated; at mapping level, an arbitrary number of mappings can be added. Where not explicitly stated, all the other values are to be considered as not updatable. In case you need to update one of those fields, please consider updating your interface major version.","ref":"030-manage_interfaces.html#installing-updating-an-interface"},{"type":"extras","title":"Managing Interfaces - Interfaces lifecycle","doc":"Interfaces are versioned through a semantic versioning-like mechanism. A Realm can hold any number of interfaces and any number of major versions of a single interface. It holds, however, only the latest installed minor version of each major version, due to the inherent compatibility of Semantic Versioning. There is no significant cost in adding a non-aggregated interface to a Realm or updating a non-aggregated interface frequently - keep in mind, however, that you might incur in dangling data in your devices if you don't plan your interface update strategy accurately. For what concerns Aggregated interfaces, instead, there is an inherent cost which might end up in putting pressure on your Cluster . Once an interface has been installed in a Realm, it can't be uninstalled without performing manual operations on Astarte's DB, unless its major version number is 0 . This is a safety measure to prevent dangling data from appearing in the cluster. For this reason, when developing an Astarte-based interface, it is strongly advised to keep its major number to 0 to allow quick changes at the expense of data loss. Please note, however, that deleting a major 0 interface is possible if the Realm has no devices left declaring that specific interface in their introspection. This is done to avoid forever dangling data and potential consistency errors. This limitation might be lifted in the future through a mass-deletion mechanism, but there is no guarantee this will ever be done. It is advised to test new interfaces on a limited number of devices to ease operations.","ref":"030-manage_interfaces.html#interfaces-lifecycle"},{"type":"extras","title":"Managing Interfaces - Realm vs. Device Interface relationship","doc":"There is a clear difference between how Interfaces are managed in a Realm and its Devices (e.g.: the device Introspection). Whereas a Realm can have any number of versions of a single interface, a Device is allowed to expose in its introspection only a single, specific version of an Interface. In general, Realm interfaces are kept as a shared agreement between its entities, but when it comes to interacting with a Device, the Realm honors its introspection (as long as the Device declares interfaces the Realm is knowledgeable about). As such, installing an interface in a Realm is a completely safe and non-disruptive operation: by design, Devices aren't aware of which interfaces a Realm supports, and Realms don't impose any interface versioning on a Device.","ref":"030-manage_interfaces.html#realm-vs-device-interface-relationship"},{"type":"extras","title":"Managing Interfaces - Caveats","doc":"Due to how minor versions work, it is responsibility of the end user to prevent accidental data loss due to missing data. Every mapping declared in a new minor release must be assumed as optional, as there is no guarantee that a Device will be able to publish (or receive) data on that specific mapping. Minor version bumps work great in case they represents additional, optional features which might be available on an arbitrarly large subset of Devices implementing that interface's major version, and are not necessary or fundamental for normal operations. If that is not the case, consider a major version update or a whole new interface instead. Also, please keep in mind that designing interfaces in the right way, especially being as atomic as reasonably possible, helps a lot in preventing situations where a minor interface update can't be done without disrupting operations. Again, the Interface design guide covers this topic extensively.","ref":"030-manage_interfaces.html#caveats"},{"type":"extras","title":"Managing Interfaces - Dangling data","doc":"In several situations, it is possible to have dangling data inside Astarte. This happens by design, as the liquid nature of a Device makes it possible for data to be stored in interfaces no longer present in its introspection. Astarte does not delete data unless requested explicitly: as such, data remains available inside its database, but potentially inaccessible through the cluster's APIs and standard mechanism. As of the current version, Astarte has no mechanism for retrieving and acting upon a device's dangling data - this is a limitation that will be lifted in future releases with additions to the current API. Interface major version change If a device upgrades one of its interfaces to a new major version, the previous interface is parked and its data remains dangling. Every API call, trigger, or reference to the interface will always target the major version declared in the introspection, regardless of the fact that a more recent version might have been installed in the realm. Interface deletion from device A device might arbitrarly decide to remove an interface from its introspection. In such a case, Astarte won't return any data and will consider all data previously pushed to said interface inaccessible. In case the interface comes back again in the introspection, previously pushed data will be available as if nothing happened.","ref":"030-manage_interfaces.html#dangling-data"},{"type":"extras","title":"Registering a Device","doc":"Devices are Astarte's main entities for exchanging data. Even though a Device usually represents the physical Device communicating with Astarte, they might as well be mapped to other entities, such as individual sensors or aggregated gateways. A Device always belongs to a Realm and is identified by a Device ID , which has to be unique at least within its Realm. Devices communicate with Astarte through Transports - in most installations, this means through an MQTT Broker (VerneMQ with Astarte's plugin). Before this happens, though, Devices must obtain credentials for accessing their Transport and, most of all, make themselves known to Astarte. This happens through the Registration process. In Astarte, Registering a device means obtaining an unique Credentials Secret (Registration Credentials), univocally associated to a Device ID, through a well-known workflow and pipeline. If you are not familiar with these concepts, please refer to Pairing Architecture to learn more about Pairing's workflow basics. The Credentials Secret can then be used by the Device for accessing Pairing API and getting information and Credentials for its Transport. As such, registration happens only once during a Device's lifecycle, and is a security-sensitive process. As such, this process is usually carried over (in production scenarios) through an Agent .","ref":"035-register_device.html"},{"type":"extras","title":"Registering a Device - Registration Agent","doc":"An Agent 's purpose is to perform Registration on behalf of a Device. Agents should be the only components in your infrastructure with enough credentials to access Pairing's Agent APIs (as a rule of thumb, it is a bad idea to give access to Pairing API to anything which isn't an Agent). When setting up an Astarte project, it is fundamental to define beforehand how your Devices will be registered and hence where your Agent(s) will belong. There's two main ways for implementing an Agent, even though in production scenarios On Board Agents are strongly discouraged as they expose a single point of failure in terms of a Realm's whole fleet security. On Board Agent Please keep in mind that On Board Agents are not advised in production, as a single compromised device/token might compromise the Registration routine for your entire fleet. They should be used only in non-critical use cases or during testing and development. On Board Agents are provided as a feature by Astarte's SDK, and hide the detail of Device registration by integrating an Agent into the SDK itself. This allows to deliver the same credentials to each device belonging to a Realm. Of course, this also opens up a single point of failure in the whole fleet's security, as Credentials aren't tied to a specific device - as such, if compromised, they might allow an attacker to register an arbitrary device into a Realm, unless other policies prevent him from doing so. To create a On Board Agent, you simply need to emit a long-enough lived token from your Realm's private key with access to Pairing's Agent APIs . This token should then be delivered to your devices and provided to the SDK in order to carry over the Registration. The SDK will do this automatically and without any need for additional code, as long as you set the agentKey configuration key to a meaningful value, and no Credentials Secret has been set. 3rd Party Agent A more secure approach to the Registration process is having a 3rd Party agent. In such a case, an external component is in charge of requesting a Credentials Secret to Pairing and delivering it to the target Device. This approach has a number of benefits: in terms of Security, the Agent uses a short-lived token and can follow the Realm's authentication workflow just like any other application. For what concerns daily operations, the Agent can implement any arbitrary logic to make a decision on whether a Device should be registered or not. In such cases, Devices have an out-of-band communication mechanism with the Agent in which the Credentials are exchanged. Usually, these cases fall under two main categories: "Local" or "Plant" Agents In this scenario, devices are imprinted with their Credentials Secret in the production plant. The Device might not even be connected to the Internet, whereas the machine running the Agent has access to the target Astarte Cluster and adequate Credentials for Registration. Once the Agent acquires the Device ID of the Device which should be registered, it issues the request to Astarte's Pairing API and obtains the Device's Credentials Secret . At this stage, the Agent is in charge of delivering the Credentials Secret to the Device the way it sees fit. As a best practice, the Credentials Secret should then be saved to an OTP area or a dedicated secure storage in the device to prevent tampering or accidental loss. Even though this is arguably the most secure mechanism available for Registering a Device, it might not fit every use case as the Device will be irrevocabily assigned to a specific Astarte Cluster and a specific Realm in that Cluster before it even connects. "Remote" Agents If your use case demands more flexibility, Registering a Device in a plant might not fit your Device's lifecycle. This could be likely if, for example, Realm or Cluster assignment should be done dynamically once the Device reaches its final user. In this case, this role is usually delegated to an external web application acting as an Agent. In this case, it's up to the user setting up all mechanisms for delivering the Credentials Secret to the Device, which includes securing the communication channel. On the other hand, this allows an extremely flexible approach to Registration, which can be implemented through an entirely custom logic.","ref":"035-register_device.html#registration-agent"},{"type":"extras","title":"Registering a Device - Credentials Secret Lifecycle","doc":"Credentials Secrets are meant to be immutable - as such, they should be handled with extreme care. Credentials Secrets are used only for interacting with Pairing, hence to obtain Credentials for a Transport which, on the other hand, are meant to be volatile. A Device can be Registered an arbitrary number of times before its Credentials Secret is used for the first time for interacting with Pairing. This is done to ensure the entire Registration process, including any kind of external custom logic of the Agents, has been carried over successfully, allowing a de-facto "retry" until there's certainty the Device has access to its Credentials Secret . Please note that when Registering a Device, a new Credentials Secret is generated every time.","ref":"035-register_device.html#credentials-secret-lifecycle"},{"type":"extras","title":"Registering a Device - Unregistering a device","doc":"Once the Credentials Secret is used for retrieving Credentials for a Transport for the first time, Astarte prevents further registration of the same Device again. If there's the need of registering the device again (e.g.: a Device has been tampered and got back to its plant with its previous Credentials Secret compromised), it is possible to explicitly unregister the device to obtain a new Credentials Secret using Pairing's Agent APIs or with astartectl (see the output of astartectl pairing agent unregister -h for more documentation).","ref":"035-register_device.html#unregistering-a-device"},{"type":"extras","title":"Connecting a Device","doc":"Once a Device has been Registered in Astarte, it is capable of connecting to it. Devices connect to Astarte through the use of Transports . A Transport is an arbitrary protocol implementation which maps Astarte's concepts (mainly Interfaces) to a communication channel. Astarte's main supported Transport is Astarte/MQTT, implemented on top of VerneMQ through an additional plugin , and it is used by Astarte's SDKs for communication. However, virtually any protocol can be integrated in Astarte by creating a corresponding Transport. Transports also define the authentication/authorization mechanism of their Devices. For instance, Astarte/MQTT uses mutual SSL Authentication with Certificate Rotation for securing its Ingress and identifying its clients. To manage their Transport(s) and Credentials, Devices have to interact with Pairing.","ref":"040-connect_device.html"},{"type":"extras","title":"Connecting a Device - Credentials Secret, Pairing and Transports","doc":"Once a Device has performed its first registration through an Agent, it holds its Credentials Secret . This Credentials Secret is the token the device uses for performing the actual Pairing routine, which results in the device obtaining its Credentials for accessing its designated Transport. A Device's Credentials Secret allows access to Pairing API's Device REST API , which is then used for obtaining information about which Transports the Device can use for communicating, and for obtaining Credentials for its assigned Transports. The ability to request Credentials of a Device can be inhibited with AppEngine API or using astartectl with this command: astartectl appengine devices credentials inhibit <device_id_or_alias> true \\ -k <appengine-key> -r <realm-name> -u <astarte-api-url> Once its credentials_inhibited field is set to true , a Device is not able to request new Credentials. Note that Credentials that were already emitted will still be valid until their expiration. As, from a user's standpoint, the way a Device communicates with Astarte is entirely Transport-specific, this guide will cover using Astarte/MQTT through one of Astarte's SDKs. If you are using a different Transport, please refer to its User Guide, or if you wish to implement your own, head over to Transport Developer Documentation .","ref":"040-connect_device.html#credentials-secret-pairing-and-transports"},{"type":"extras","title":"Connecting a Device - Using Astarte/MQTT through Astarte SDK","doc":"If you are using one of Astarte's SDK, the Pairing routine is entirely managed, and you won't need to do any of the aforementioned steps. Just make sure your Credentials Secret is passed as the apiKey configuration key, to allow the SDK to perform automatically the Pairing routine when needed. The SDK does a number of automated things under the hood. Its flow is: The SDK verifies if a SSL certificate for connecting to the broker is present. If it is, it attempts connecting to the Transport. If the Transport doesn't accept the connection due to an SSL error, it queries Pairing API about its certificate status. If Pairing API returns a problem with the certificate or, in general, the certificate isn't valid, the certificate is erased and the Pairing procedure begins. The SDK invokes Pairing API until it manages to obtain a valid Certificate for the Transport. The SDK considers a Device successfully paired when it has a valid certificate and manages to connect to the Transport. Once in this state, the Device can start exchanging data. Note: the Pairing procedure is secure as long as Pairing API is queried using HTTPS. Plain HTTP installations are vulnerable to a number of different attacks and should NEVER be used in production. Interfaces and Introspection A Device must have some installed interfaces to be capable of exchanging data. These interfaces must be made known to the SDK and installed in the Device's Realm, as previously explained . The SDK expects the user to provide a directory containing a set of valid interfaces. It then takes care of making Astarte aware of its registered interfaces through a process called Introspection. Introspection is a special control message in Astarte's protocol which makes Astarte aware of a list of Interfaces and relative versions which are installed on the Device. Again, Astarte's SDK, given a directory, is capable of performing the correct procedures for keeping Introspecting in sync correctly without any kind of user intervention. Astarte's SDK also takes care of updating a Device's Introspection if its interfaces change. Exchanging data When a Device connects successfully, it must then subscribe to its server Interfaces. The SDK takes care of this detail and exposes a higher level interface. For example, using the Qt5 SDK: { m_sdk = new AstarteDeviceSDK(QStringLiteral("/path/to/transport-astarte.conf"), QStringLiteral("/path/to/interfaces"), deviceId); connect(m_sdk->init(), &Hemera::Operation::finished, this, &AstarteStreamQt5Test::checkInitResult); connect(m_sdk, &AstarteDeviceSDK::dataReceived, this, &AstarteStreamQt5Test::handleIncomingData); } void AstarteStreamQt5Test::handleIncomingData(const QByteArray &interface, const QByteArray &path, const QVariant &value) { qDebug() << "Received data, interface: " << interface << "path: " << path << ", value: " << value << ", Qt type name: " << value.typeName(); } Applications can simply connect to the handleIncomingData signal and have data correctly formatted and delivered as it runs through the transport. On the other hand, for sending data: m_sdk->sendData(interface, path, value); The SDK will check if data is coherent with its introspection, and send data onto the transport in the correct way. Reliability, retention and persistency in the SDK Astarte's SDK has an internal concept of persistency, depending on the behaviour defined in its installed Interfaces. The retention parameter, specifically, tells Astarte's SDK how hard it should try to send a specific message. In case the Transport is unreachable, the SDK might try to persist, either in memory or on disk, and send the message when the connection is available again. Please note that these parameters declared in Interfaces are to be considered on a best effort basis. In case your SDK does not support persistency or has persistency disabled, a number of warranties requested by an Interface might not be satisfied. Make sure your SDK is configured correctly before moving to production.","ref":"040-connect_device.html#using-astarte-mqtt-through-astarte-sdk"},{"type":"extras","title":"Device errors","doc":"This page details the errors that can affect a device while it's sending data. The user can monitor device errors by installing a device trigger on device_error or checking the Devices tab in the Astarte Dashboard. The same errors are also provided as log messages on Data Updater Plant.","ref":"045-device_errors.html"},{"type":"extras","title":"Device errors - write_on_server_owned_interface","doc":"The device is trying to write on a server owned interface. The device can only push data on device owned interfaces.","ref":"045-device_errors.html#write_on_server_owned_interface"},{"type":"extras","title":"Device errors - invalid_interface","doc":"The interface name received in the message is invalid.","ref":"045-device_errors.html#invalid_interface"},{"type":"extras","title":"Device errors - invalid_path","doc":"The path received in the message is invalid. This might happen when a path does not have a valid path format or it's not a valid UTF-8 string.","ref":"045-device_errors.html#invalid_path"},{"type":"extras","title":"Device errors - mapping_not_found","doc":"The path received in the message can't be found in the interface mappings. This could be the result of the device having a more recent version of the inteface than the one installed in the realm or an interface with the same name and version but different contents.","ref":"045-device_errors.html#mapping_not_found"},{"type":"extras","title":"Device errors - interface_loading_failed","doc":"The target interface was not found in the database. Usually this means the interface is not installed in the realm, but the error can also derive from the database being temporarily unavailable.","ref":"045-device_errors.html#interface_loading_failed"},{"type":"extras","title":"Device errors - ambiguous_path","doc":"The path received in the message can't be mapped univocally on a mapping. This is often the result of an incomplete path.","ref":"045-device_errors.html#ambiguous_path"},{"type":"extras","title":"Device errors - undecodable_bson_payload","doc":"The payload of the message can't be decoded as BSON.","ref":"045-device_errors.html#undecodable_bson_payload"},{"type":"extras","title":"Device errors - unexpected_value_type","doc":"The value of the message does not have the expected type ( e.g. the mapping expects a string value but an integer value was received instead).","ref":"045-device_errors.html#unexpected_value_type"},{"type":"extras","title":"Device errors - value_size_exceeded","doc":"The value of the message exceeds the maximum size of its type. The size limitations of the types are documented here .","ref":"045-device_errors.html#value_size_exceeded"},{"type":"extras","title":"Device errors - unexpected_object_key","doc":"An object aggregated value with an unexpected key was received.","ref":"045-device_errors.html#unexpected_object_key"},{"type":"extras","title":"Device errors - invalid_introspection","doc":"The introspection sent from the device can't be parsed correctly. The introspection format is documented here .","ref":"045-device_errors.html#invalid_introspection"},{"type":"extras","title":"Device errors - unexpected_control_message","doc":"The device sent a message on an unhandled control path. The supported control paths are detailed in the protocol documentation .","ref":"045-device_errors.html#unexpected_control_message"},{"type":"extras","title":"Device errors - device_session_not_found","doc":"Data Updater Plant failed to push data towards the device. This could result from the device being currently offline and not having a persistent session on the MQTT broker or from the device not having all the MQTT subscriptions required by the Astarte protocol","ref":"045-device_errors.html#device_session_not_found"},{"type":"extras","title":"Device errors - resend_interface_properties_failed","doc":"Data Updater Plant failed to resend the properties of an interface. This could result from the device declaring a uninstalled properties interface in its introspection right before an emptyCache.","ref":"045-device_errors.html#resend_interface_properties_failed"},{"type":"extras","title":"Device errors - empty_cache_error","doc":"The empty cache operation for a device failed. This could result from a temporary database failure.","ref":"045-device_errors.html#empty_cache_error"},{"type":"extras","title":"Querying a Device","doc":"Once you have your devices connected, up and running in Astarte, you can start interacting with them.","ref":"050-query_device.html"},{"type":"extras","title":"Querying a Device - Device status","doc":"First things first, you can check if your device is correctly registered in Astarte, and its current status. Let's assume our Device has f0VMRgIBAQAAAAAAAAAAAA as its id. A Device's status includes a number of useful information, among which whether it is connected or not to its Transport, its introspection, the amount of exchanged data and more. Query Device status using astartectl $ astartectl appengine devices show f0VMRgIBAQAAAAAAAAAAAA Device ID: f0VMRgIBAQAAAAAAAAAAAA Connected: false Last Connection: 2018-02-07 18:38:57.266 +0000 UTC Last Disconnection: 2018-02-08 09:49:26.566 +0000 UTC Introspection: com.example.ExampleInterface v1.0 exchanged messages: 20 exchanged bytes: 200B org.example.TestInterface v0.2 exchanged messages: 8 exchanged bytes: 147B Received Messages: 221 Data Received: 11.7K Last Seen IP: 203.0.113.89 Last Credentials Request IP: 203.0.113.201 First Registration: 2018-01-31 17:10:59.270 +0000 UTC First Credentials Request: 2018-01-31 17:10:59.270 +0000 UTC Query Device status using Astarte Dashboard After logging in to Astarte dashboard, go to the "Devices" page clicking on the menu on your left. A list of available Device IDs will appear. If you do not see your device at a glance, use the search bar on the top right to find it. Clicking on the Device ID will take you to its details page. Query Device status using AppEngine API GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA { "data": { "total_received_msgs": 221, "total_received_bytes": 11660, "last_seen_ip": "203.0.113.89", "last_credentials_request_ip": "203.0.113.201", "last_disconnection": "2018-02-07T18:38:57.266Z", "last_connection": "2018-02-08T09:49:26.556Z", "id": "f0VMRgIBAQAAAAAAAAAAAA", "first_registration": "2018-01-31T17:10:59.270Z", "connected": true, "introspection": { "com.example.ExampleInterface" : { "major" : 1, "minor" : 0, "exchanged_msgs": 20, "exchanged_bytes": 200 }, "org.example.TestInterface" : { "major" : 0, "minor" : 2, "exchanged_msgs": 8, "exchanged_bytes": 147 } }, "aliases": { "name": "device_a" }, "attributes": { "attributes_key": "attributes_value" }, "groups": [ "my_group", ], "previous_interfaces": [ { "name": "com.example.ExampleInterface", "major" : 0, "minor" : 2, "exchanged_msgs": 3, "exchanged_bytes": 120 } ] } } Through the API, it is also possible to get the Introspection of the device only: GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces { "data": [ "com.example.ExampleInterface", "com.example.TestInterface" ] } This returns the Interfaces which the device reported in its Introspection and which are known to the Realm. Arbitrary information can be added to the device by means of attributes : they allow to store any number of string values associated to a corresponding string key. To set, modify and delete attributes , a PATCH on the device endpoint is required: PATCH api . < your astarte domain > / appengine / v1 / test / devices / f0VMRgIBAQAAAAAAAAAAAA In the request body, the data JSON object should have a attributes key which bears a dictionary of strings. A valid request body which changes only device attributes, for example, is {"data":{"attributes": {"<key>": "<value>"}}} . To delete an attribute entry, set the value of the corresponding key to null . For example, POSTing {"data":{"attributes": {"my_key": null}}} will remove the my_key attribute entry from the device. Depending on the aggregation and ownership of the Interface, you can GET / PUT / POST on the interface itself or one of its mappings, or use astartectl to perform the same operation on the command line. Some examples are: Get data from an aggregate device properties interface astartectl invocation: astartectl appengine devices data-snapshot f0VMRgIBAQAAAAAAAAAAAA com.example.ExampleInterface AppEngine API invocation: GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces/com.example.ExampleInterface Get last sent value from an individual device datastream interface astartectl invocation: astartectl appengine devices data-snapshot f0VMRgIBAQAAAAAAAAAAAA com.example.TestInterface AppEngine API invocation: GET api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces/com.example.TestInterface/myValue?limit=1 Set values in an individual server datastream interface astartectl invocation: astartectl appengine devices send-data f0VMRgIBAQAAAAAAAAAAAA com.example.OtherTestInterface /myOtherValue <value> AppEngine API invocation: POST api.<your astarte domain>/appengine/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA/interfaces/com.example.OtherTestInterface/myOtherValue Request body: {"data": <value>} API Query semantics In general, to query AppEngine, the following things must be kept in mind When sending data, use PUT if dealing with properties , POST if dealing with datastream . When GET ting, if you are querying an aggregate interface, make sure to query the interface itself rather than its mappings. When GET ting datastream , keep in mind that AppEngine's default behavior is to return a large as possible timeseries.","ref":"050-query_device.html#device-status"},{"type":"extras","title":"Querying a Device - Navigating and retrieving Datastream results through APIs","doc":"The Datastream case is significant, as it might be common to have a lot of values for each endpoint/interface. As such, returning all of them in a single API call is most of the times not desirable nor recommended. To avoid putting the cluster under excessive pressure, AppEngine API is configured with a hard cap on the maximum number of returned results for each single call, with a sane default of 10000 . Although this hard cap is entirely configurable, please be aware that AppEngine API is designed to process a lot of reasonably small requests in the shortest possible time, and hence is not optimised nor strongly tested against big requests . Make sure that AppEngine API has enough resources available to cope with the maximum dataset size. AppEngine API provides you with a variety of mechanisms to make retrieval and navigation of large data sets as smooth and efficient as possible. Limit Adding a limit=n to the URL query tells AppEngine to return no more than n results. This acts similarly to a LIMIT SQL statement, but, as it stands, it does not impose a hard limit on the whole retrieved dataset but on the amount of the results displayed by the API call - see Pagination and Time Windows for more details on this topic and the performance implications of different limits in queries. If the specified limit is beyond the hard cap, the query won't fail, but will return at most the amount set by the hard cap, without further warnings. Since/To/Since After Results can be limited to a specific time window. since and to can be set to a ISO 8601 valid timestamp to limit on an upper and lower bound the result set. This can also be combined with limit to make sure that no more than n results are returned. Also, since and to can as well be set independently to provide only an upper or lower bound. In case you're dealing with a very large dataset and you want to dump it, it is likely that you need to go beyond what a reasonable default limit looks like. In those cases, you can use the since_after query parameter to retrieve parameters within a time window. since_after slices the time window just like since does, but it does not include values matching the specified timestamp, if any. This is especially useful when paginating, to start right after a returned result. Pagination and time windows AppEngine API provides you automatically with a time window-based pagination. When GET ting a datastream , if more results are available beyond the chosen time window/limit, a links map will be provided, in JSON-API style, to allow the user to paginate the results accordingly using since_after . You can use limit to determine each page's size. When specifying a valid limit , the links will keep the page size consistent over the next calls. However, limit should be used wisely to lower the pressure on the cluster. Each API call maps to a query that, no matter how efficient, has a computational cost. A few mid-sized queries should always be preferred over a large amount of smaller queries. Given your cluster is configured correctly, limit should be omitted in most cases when paginating, and you should rather trust your cluster's hard cap to be the sweet spot in efficiency and cluster pressure. Downsampling Especially when plotting graphs, retrieving all points in a time series isn't desirable. Astarte provides you with an implementation of the LTTB Downsampling Algorithm , which is used to return only a fixed number of samples from a time series. When setting downsample_to=n , AppEngine will return a maximum of n results, which are the most significant over the considered time series according to the algorithm. Due to how LTTB works, downsample_to must be >2 , as the algorithm will return the two ends of the considered value bucket, and n-2 values which are the picked samples. Please refer to the LTTB implementation used by Astarte to learn more about how this algorithm affects samples and its limitations. downsample_to=x can be used in conjunction with other query parameters, including limit=y . When doing so, Astarte will downsample to x samples the dataset composed of the last y values. Every feature previously outlined is in fact available with downsampling, including pagination - bear in mind, though, that for how the algorithm works, some options have drastically different semantic effects. Also, the hard cap has a very different meaning in downsampling. In this case, the hard cap applies to downsample_to instead of limit . limit can be an arbitrarly large amount of samples taken out of the DB, and can be used mainly to alleviate pressure in case of extremely large datasets which would require a lot of time for being processed by LTTB - even though, most of the time, you might want to define a time window to downsample instead. Astarte is also capable of downsampling aggregated interfaces, as long as a downsample_key is specified, which has to match the last token of an endpoint of the queried interface (i.e. in case the interface has a /%{id}/myValue mapping which should be used as the downsample_key , you should specify downsample_key=myValue in the query). When doing so, the aggregate will be downsampled using the chosen endpoint value as the y axis value, whereas its other endpoints will be disregarded when applying the algorithm. Please note that, no matter what downsample_key is used, a sample will be composed by the whole aggregation. If there is no way an interface can be downsampled (this is true, for example, if no downsample_key has been specified for aggregations , or for types such as strings ), AppEngine API will return a 4xx error. In general, downsampling is a powerful mechanism with a lot of limitations which really shines when plotting. Once again, this is a fundamental factor to consider when designing your interfaces .","ref":"050-query_device.html#navigating-and-retrieving-datastream-results-through-apis"},{"type":"extras","title":"Querying a Device - Real-Time Updates","doc":"The http REST API returns a static result, therefore API clients should either poll the REST API when displaying real-time changes or use a WebSocket. WebSockets are called Astarte Channels in Astarte jargon, and they should be considered as a more efficient alternative to polling. When using Astarte Channels the REST API should be used to retrieve the initial status.","ref":"050-query_device.html#real-time-updates"},{"type":"extras","title":"Querying a Device - astartectl-specific features","doc":"astartectl implements some convenience methods that make navigation easier. In particular, astartectl allows for any of the AppEngine API query parameters/mechanisms, but also implements automated pagination, snapshots and more. Data Snapshot astartectl has a unique feature that allows to retrieve a "Data Snapshot" of a device, namely the last known value for every interface available in the Device's introspection. This is extremely useful to have an at-a-glance view of the Device status with regards to data. Simply invoke astartectl appengine devices data-snapshot <device ID> , or astartectl appengine devices data-snapshot <device ID> <interface name> to get a snapshot for a single interface. Advanced querying astartectl appengine devices get-samples is astartectl 's frontend to advanced query. Refer to the command line documentation to learn about all available parameters, which match all of the parameters found in AppEngine API. The main difference is that, in case a query would break the boundaries of the page limit, astartectl will automatically paginate the request, and return all of the samples. Exporting Devices Data with astartectl The previous feature makes astartectl extremely useful when it comes to export or dump data. Moreover, get-samples features a --output option, which allows to print the results in different formats, such as json or CSV . This way, exporting values becomes extremely easy, as get-samples can easily tap into an Interface's entire data set and print it into a CSV file.","ref":"050-query_device.html#astartectl-specific-features"},{"type":"extras","title":"Querying a Device - Deleting a Device","doc":"If you want to entirely remove a device from your realm along with its data, it is possible to do so with the device deletion API. Note that the Astarte API also allows for the unregistration and the inhibition of a specific device, which should be enough in most use cases and do not cause any data loss. [!IMPORTANT] Once the deletion of a device is started, all data related to the device will be lost. Device deletion using Realm Management API Let's assume our Device has f0VMRgIBAQAAAAAAAAAAAA as its id. To delete it, use the following API. The deletion operation is asynchronous and can take up to 15 minutes. During the process, you device status will appear as "In deletion". DELETE api.<your astarte domain>/realmmanagement/v1/test/devices/f0VMRgIBAQAAAAAAAAAAAA","ref":"050-query_device.html#deleting-a-device"},{"type":"extras","title":"Using Astarte Channels (WebSockets)","doc":"Especially when building Frontend applications, it is useful to receive real-time updates about data sent from Devices. Astarte leverages Phoenix Channels to provide such a thing over WebSockets in AppEngine API. WebSockets can be used natively from a Web Browser and follow the same authentication pattern as a standard HTTP call. The AppEngine API for Astarte Channels is located at the URL: ws://<astarte-base-url>/appengine/v1/socket/websocket . Astarte Channels define a semantic on top of Phoenix Channels which allows read-only monitoring of device Interfaces. Authentication and Authorization over Channels happens in the very same way as AppEngine , and the a_ch claim in the token is respected when joining rooms and installing triggers. See Authentication and Authorization for more details on Auth semantics in Astarte.","ref":"052-using_channels.html"},{"type":"extras","title":"Using Astarte Channels (WebSockets) - Rooms","doc":"Rooms in Astarte Channels map 1:1 to Topics in Phoenix Channels, and can be joined in the very same way. Once a connection is established, the user can join any number of rooms, given he is authorized to do so . A Room is identified by a topic with the following semantics: rooms:<realm>:<name> . For example, rooms:test:myroom will join the Room myroom in the Realm test . A room can be joined by any number of concurrent users. Rooms serve as containers for Transient Triggers, which can be installed by any authorized user. Transient Triggers are actual Triggers , with the difference that they exist within a Channels Room rather than within a Realm - this mostly affects their timespan - and that the action can't be configured - every time a Condition is triggered an event is delivered to users in the Room. Events Everytime a Condition of an installed Trigger is triggered, an event is sent to the Phoenix Channel, with a similar payload: { "device_id": "f0VMRgIBAQAAAAAAAAAAAA", "event": { "type": "device_connected", "device_ip_address": "1.2.3.4" } } device_id is always present (as long as the trigger matches a device) and identifies the device emitting the event. event , instead, depends on the kind of installed trigger. It always carries a type string, which identifies the content of the object. The complete list of possible event payloads can be found here . Lifecycle Once a room is created, it remains valid and active with all of its subscriptions. There's little overhead in having a large number of rooms, as the only components leeching resources are Transient Triggers. As of today, Transient Triggers never expire - it is responsibility of the user to clean them up once the room becomes empty, if needed. In future versions, Transient Triggers will likely expire after some time, if left in an empty room.","ref":"052-using_channels.html#rooms"},{"type":"extras","title":"Using Astarte Channels (WebSockets) - Managing Transient Triggers","doc":"To install a Transient Trigger, one should issue a watch event in the Channel, given he is authorized to do so. The payload of such an event is identical to a Trigger definition, hence it looks like this: { "name": "datatrigger", "device_id": "f0VMRgIBAQAAAAAAAAAAAA", "simple_trigger": { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.6 } } This installs in the Room a Transient Trigger which will trigger an event everytime a value higher than 0.6 is sent on the path /streamTest/value of the datastream interface org.astarte-platform.genericsensors.Values by the device f0VMRgIBAQAAAAAAAAAAAA , and will be received by every user currently in the room. If a user isn't in the room at the time of the event, he will not get it, and there's no way he can retrieve it if he joined at a later time. Triggers can be uninstalled by issuing an unwatch event in the Channel. The payload of the event should be the name of the trigger which should be uninstalled. Group Triggers Transient triggers can also target an Astarte group instead of a single device. To install a group volatile trigger, pass the group_name key in the JSON payload instead of the device_id key. For example, the trigger below is equivalent to the one in the previous section, but it targets all devices that are in the group mygroup . { "name": "groupdevicetrigger", "group_name": "mygroup", "simple_trigger": { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.6 } } Note that the devices belonging to the group are evaluated when the trigger is installed, i.e. if a device is added to the group when the trigger is already installed, the trigger will not target the newly added device. The same goes for devices removed from the group, that will still be targeted by the trigger until it is removed.","ref":"052-using_channels.html#managing-transient-triggers"},{"type":"extras","title":"Using Astarte Channels (WebSockets) - Authorization","doc":"Just like any other Astarte component, Authorization is encapsulated in a token claim, in particular the a_ch claim. However, the mechanism is rather different compared to a REST API, and uses different verbs. JOIN The JOIN verb implies that a user can join a room. This only allows him to receive events and to interact in a read-only fashion with the room itself. There is no restriction to which events a user sees - if he is authorized to enter in a room, he will be capable of seeing all events flowing in. More granular permissions can be done simply by creating more rooms in which different triggers will be installed. The JOIN verb has the following semantic: JOIN::<regex> , where regex matches a room name (the room name is what follows rooms:<realm>: - the realm is implicit in the context of the authorization token). For example, a user authorized with the JOIN::test.* claim in the test realm will be able to join, for example, rooms:test:testthis , rooms:test:testme , rooms:test:test . The realm is always implicit in the regex, as the token is authenticated in the context of a Realm. WATCH The WATCH verb allows a user to install a Trigger within a room. Its semantics define which kind of trigger, and upon which entities the user is allowed to act. Watch semantics are WATCH::<regex> , where regex is a regular expression which matches a device, path or interface (or a mixture of them) in almost very same fashion as the a_aea claim (which is used in AppEngine). Given different kind of triggers impact different Astarte entities, the Authorization claim implicitly defines which kind of triggers a user will be able to install. For example, f0VMRgIBAQAAAAAAAAAAAA/org.astarte-platform.genericsensors.Values.* will allow installing data triggers such as the one shown in the previous example, but won't let the user install device-wide triggers (such as connect/disconnect events). A claim such as f0VMRgIBAQAAAAAAAAAAAA or f0VMRgIBAQAAAAAAAAAAAA.* , instead, will allow device-level triggers to be installed.","ref":"052-using_channels.html#authorization"},{"type":"extras","title":"Using Triggers","doc":"Triggers allow receiving notifications when a device connects, disconnects or publishes specific data. More details on Triggers can be found in the Architecture Documentation . Astarte allows you to install and delete Triggers dynamically through its clients. Upon installation or deletion, changes to the Trigger infrastructure might take some time to propagate, and some devices might pick up changes at a later time. If a Trigger shows as installed, it will eventually be loaded. This propagation can take up to 10 minutes.","ref":"060-using_triggers.html"},{"type":"extras","title":"Using Triggers - Listing Triggers","doc":"At any time, you can list existing Triggers in a Realm and fetch their details and definitions. Listing and querying Triggers using Astarte Dashboard After logging in, navigate to the Triggers page using the menu on the left. The list of Triggers installed in the Realm will be shown in the page. Clicking on a Trigger will open the Trigger editor in view-only mode, showing its definition on the right panel. Listing and querying Triggers using astartectl To list all existing Triggers in a Realm: $ astartectl realm-management triggers list [my_trigger other_trigger my_connection_trigger my_data_trigger] To get a Trigger definition: $ astartectl realm-management triggers show my_connection_trigger { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "<method>" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } Listing and querying Triggers using Realm Management API To list all existing Triggers in a Realm: GET api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers { "data": [ "my_trigger", "other_trigger", "my_connection_trigger", "my_data_trigger" ] } To get a Trigger definition: GET api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers/my_connection_trigger { "data": { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "<method>" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } }","ref":"060-using_triggers.html#listing-triggers"},{"type":"extras","title":"Using Triggers - Installing a Trigger","doc":"To install a Trigger, you need its JSON definition. If you have access to the Astarte Dashboard, you can use its Trigger Editor to build your JSON definition and install the Trigger directly. If you already have a JSON definition instead, you can either use astartectl or Realm Management APIs. The name of the Trigger must be unique within the Realm, or an error will be returned. Installing a Trigger using Astarte Dashboard After logging in, navigate to the Triggers page using the menu on the left. Click on "Install a new Trigger..." in the top-right corner. The Trigger Editor will open, and you can either paste/write a JSON definition, or use the declarative editor. When you are done, click on the "Install Trigger" button at the bottom of the declarative editor to install the Trigger in the Realm. Installing a Trigger using astartectl Assuming the Trigger definition is contained in my_connection_trigger.json , $ astartectl realm-management triggers install my_connection_trigger.json ok Installing a Trigger using Realm Management APIs POST api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers The POST request must have the following request body, with content type application/json { "data": { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "<method>" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } }","ref":"060-using_triggers.html#installing-a-trigger"},{"type":"extras","title":"Using Triggers - Deleting a Trigger","doc":"To delete a Trigger, you need to know its name. Just like when installing a Trigger, deleting a Trigger might not stop the data flow out of the Trigger immediately, which will eventually terminate at some point. Deleting a Trigger using Astarte Dashboard After logging in, navigate to the Triggers page using the menu on the left. Click on "Install a new Trigger..." in the top-right corner. Click on the Trigger name you want to delete. The Trigger Editor will open, and a "Delete" button will become available next to the Trigger name. Click on it to delete the Trigger. Deleting a Trigger using astartectl $ astartectl realm-management triggers delete my_connection_trigger ok Deleting a Trigger using Realm Management APIs DELETE api.<astarte base URL>/realmmanagement/v1/<realm name>/triggers/my_connection_trigger","ref":"060-using_triggers.html#deleting-a-trigger"},{"type":"extras","title":"Using Triggers - Trigger examples","doc":"This section outlines two examples for the two main Trigger types (connection and data), and a sample payload for its HTTP Post URL action. Connection Trigger This trigger will send a POST request to <url> every time any device connects to its transport. This is the JSON representation of the trigger: { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "post" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ] } If the Trigger is installed, when a device connects, <url> will receive the following JSON payload: { "timestamp": "<timestamp>", "device_id": "<device_id>", "event": { "type": "device_connected", "device_ip_address": "<device_ip_address>" } } Data Trigger This trigger will send a POST request to http://www.example.com/hook every time a device sends data to the org.astarte-platform.genericsensors.Values major version 0 interface on the /streamTest/value path. This is the JSON representation of the trigger { "name": "my_data_trigger", "action": { "http_url": "http://www.example.com/hook", "http_method": "post" }, "simple_triggers": [ { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 0, "match_path": "/streamTest/value", "value_match_operator": "*" } ] } If the Trigger is installed, when a device sends data to the interface/path defined above, <url> will receive the following JSON payload: { "timestamp": "<timestamp>", "device_id": "<device_id>", "event": { "type": "incoming_data", "interface": "org.astarte-platform.genericsensors.Values", "path": "/streamTest/value", "value": <value> } }","ref":"060-using_triggers.html#trigger-examples"},{"type":"extras","title":"Using Triggers - Restricting triggers to a single device or group","doc":"Both device and data triggers accept the device_id and group_name keys to restrict a trigger to a single device or a single group. Triggers containing the device_id key will be triggered only for the specified device, while triggers containing the group_name key will be triggered only if the device is member of the group that is indicated in the group_name key. Note that when devices in a group are added or removed, the changes are not reflected immediately in group triggers. It can take up to 10 minutes to see the propagation of said changes.","ref":"060-using_triggers.html#restricting-triggers-to-a-single-device-or-group"},{"type":"extras","title":"Using Triggers - Trigger Delivery Policies","doc":"Trigger Delivery Policies allow customizing what Astarte is supposed to do in case of delivery errors on HTTP events and how to handle events that have not been delivered. A Trigger can be linked to one (at most) Trigger Delivery Policy by specifying the name of the policy in the "policy" field. If no Trigger Delivery Policies are specified, Astarte will resort to the default (pre v1.1) behaviour, i.e. ignoring delivery errors. Note that the Trigger Delivery Policy must be installed before installing a Trigger that is linked to it, or an error will be returned. The following is an example of a Connection Trigger linked to the simple_trigger_delivery_policy Trigger Delivery Policy: { "name": "my_connection_trigger", "action": { "http_url": "<url>", "http_method": "post" }, "simple_triggers": [ { "type": "device_trigger", "device_id": "*", "on": "device_connected" } ], "policy" : "simple_trigger_delivery_policy" } Refer to the relevant documentation for more information on Trigger Delivery Policies.","ref":"060-using_triggers.html#trigger-delivery-policies"},{"type":"extras","title":"Using Trigger Delivery Policies","doc":"Note: Trigger Delivery Policies are an experimental feature, see Known Issues for more information about their current status. Trigger Delivery Policies allow customizing what Astarte is supposed to do in case of delivery errors on HTTP events and how to handle events that have not been delivered. More details on Trigger Delivery Policies can be found in the Architecture Documentation . Astarte allows you to install and delete Trigger Delivery Policies dynamically through its clients. A Trigger Delivery Policy is linked to an HTTP Trigger by specifying its name in the Trigger definition. A Trigger can be linked to at most one Trigger Delivery Policy, but a single Trigger Delivery Policy may serve any number of Triggers.","ref":"062-using_trigger_delivery_policies.html"},{"type":"extras","title":"Using Trigger Delivery Policies - Listing Trigger Delivery Policies","doc":"At any time, you can list existing Trigger Delivery Policies in a Realm and fetch their details and definitions. Listing and querying Trigger Delivery Policies using Realm Management API To list all existing Trigger Delivery Policies in a Realm: GET api.<your astarte domain>/realmmanagement/v1/<realm name>/policies { "data": [ "my_trigger_delivery_policy", "simple_trigger_delivery_policy", "other_trigger_delivery_policy", "retry_upon_all_errors_delivery_policy" ] } To get a Trigger Delivery Policy definition: GET api.<your astarte domain>/realmmanagement/v1/<realm name>/policies/simple_trigger_delivery_policy { "data": { "name" : "simple_trigger_delivery_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "any_error", "strategy" : "discard" } ] } }","ref":"062-using_trigger_delivery_policies.html#listing-trigger-delivery-policies"},{"type":"extras","title":"Using Trigger Delivery Policies - Installing a Trigger Delivery Policy","doc":"To install a Trigger Delivery Policy, you need its JSON definition. Then you can install the Trigger Delivery Policy via Realm Management APIs. The name of the Trigger Delivery Policy must be unique within the Realm, or an error will be returned. Installing a Trigger Delivery Policy using Realm Management APIs POST api.<your astarte domain>/realmmanagement/v1/<realm name>/policies The POST request must have the following request body, with content type application/json { "data": { "name" : "simple_trigger_delivery_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "any_error", "strategy" : "discard" } ] } }","ref":"062-using_trigger_delivery_policies.html#installing-a-trigger-delivery-policy"},{"type":"extras","title":"Using Trigger Delivery Policies - Deleting a Trigger Delivery Policy","doc":"To delete a Trigger Delivery Policy, you need to know its name. A Trigger Delivery Policy can be deleted only if no Triggers linked to it are present in the Realm. Deleting a Trigger Delivery Policy using Realm Management APIs DELETE api.<your astarte domain>/realmmanagement/v1/<realm name>/policies/simple_trigger_delivery_policy","ref":"062-using_trigger_delivery_policies.html#deleting-a-trigger-delivery-policy"},{"type":"extras","title":"Using Trigger Delivery Policies - Trigger Delivery Policy examples","doc":"This section outlines two examples of Trigger Delivery Policy. A Simple Trigger Delivery Policy The following Trigger Delivery Policy discards events on any error. This is the only behaviour previous Astarte versions (i.e. < 1.1) allowed. { "name" : "simple_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "any_error", "strategy" : "discard" } ] } A More Complex Trigger Delivery Policy The following policy has a different behaviour depending on whether the HTTP delivery error is a client or a server one. { "name" : "complex_policy", "maximum_capacity" : 100, "error_handlers" : [ { "on" : "server_error", "strategy" : "retry" }, { "on" : "client_error", "strategy" : "discard" } ], "retry_times" : 5, "event_ttl" : 10 } If an HTTP client error occurs, then Astarte will try to resend the event up to 5 times. If there occurs an HTTP server error, then Astarte will do nothing. At most 100 events can be in the queue at any time; if more than 100 events are present in the queue, the oldest ones will be deleted (even if they were resent less than 5 times in the case of HTTP client errors). If any event lasts for longer than 10 second in the queue, it will be discarded.","ref":"062-using_trigger_delivery_policies.html#trigger-delivery-policy-examples"},{"type":"extras","title":"Using Trigger Delivery Policies - Known issues","doc":"At the moment, trigger delivery policies in general do not provide a guarantee of in-order delivery of events. Refer to the Architecture Documentation for more information on this.","ref":"062-using_trigger_delivery_policies.html#known-issues"},{"type":"extras","title":"Managing Groups","doc":"Devices can be divided in groups to provide group-specific access to the APIs. The examples below will use astartectl but you can achieve the same results using AppEngine API.","ref":"065-managing-groups.html"},{"type":"extras","title":"Managing Groups - Reserved group prefixes","doc":"Some prefixes are reserved for internal use. It's not possible to create groups with a name starting with the ~ and @ characters.","ref":"065-managing-groups.html#reserved-group-prefixes"},{"type":"extras","title":"Managing Groups - Creating a group","doc":"You can create a group with astartectl with this command astartectl appengine groups create mygroup <device_identifier>,<device_identifier> device_identifier can be a Device ID or an Alias, and you can put multiple devices by separating them with a comma. You can check the group was created by listing groups in your realm astartectl appengine groups list","ref":"065-managing-groups.html#creating-a-group"},{"type":"extras","title":"Managing Groups - Adding or removing a device to/from a group","doc":"Once you created a group, you can add or remove devices from it. To add a device, use: astartectl appengine groups devices add <device_identifier> To remove a device, use: astartectl appengine groups devices remove <device_identifier> Keep in mind that a group exists as long as it has at least one device in it, so if you remove the last device from a group, the group will cease to exist. You can always check which devices are in a group with: astartectl appengine groups devices list","ref":"065-managing-groups.html#adding-or-removing-a-device-to-from-a-group"},{"type":"extras","title":"Managing Groups - Accessing Devices in a group with Astarte AppEngine API","doc":"Once a device is in a group, you can access to its data on this URL: https://<astarte-api>/appengine/v1/groups/<group_name>/devices/<device_id> The hierarchy is exactly the same that is found under https://<astarte-api>/appengine/v1/devices/<device-id> which is documented here , but this makes it possible to emit a JWT that only allows access to devices belonging to a specific group.","ref":"065-managing-groups.html#accessing-devices-in-a-group-with-astarte-appengine-api"},{"type":"extras","title":"Connect 3rd party applications","doc":"","ref":"070-connect_application.html"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana","doc":"Astarte Datasource Plugin conveys data from Astarte to Grafana , the open-source platform for monitoring and observability, developed by Grafana Labs . Actual data visualisation is provided by Grafana via its plugins . You can browse the source code of this plugin on its GitHub repository .","ref":"080-grafana_datasource.html"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Try it!","doc":"When deploying locally using docker-compose as mentioned in the Astarte in 5 mins tutorial , Astarte Datasource Plugin will be automatically installed. You may then access Grafana by visiting http://grafana.astarte.localhost . When first logging into Grafana, you will be prompted to change default credentials user admin , password: admin .","ref":"080-grafana_datasource.html#try-it"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Configure the datasource","doc":"In order to get data from Astarte, you will need to create a new datasource referring to your own Astarte instance. You will need to provide Astarte API URLs, the realm name and Astarte AppEngine token. Save the configuration by clicking on Save & Test","ref":"080-grafana_datasource.html#configure-the-datasource"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Setting up a graph","doc":"After successfully configuring your datasources, you will be able to select them as a source for your graph as depicted below. You will then be able to choose which device and interface data should be retrieved from.","ref":"080-grafana_datasource.html#setting-up-a-graph"},{"type":"extras","title":"Astarte Datasource Plugin for Grafana - Data manipulation","doc":"Grafana offers data-aggregation features, for more information check the official Grafana documentation","ref":"080-grafana_datasource.html#data-manipulation"},{"type":"extras","title":"Troubleshooting","doc":"Be sure to check known issues to see if your problem is already covered there.","ref":"090-troubleshooting.html"},{"type":"extras","title":"Troubleshooting - Devices","doc":"Devices cannot connect to Astarte There might be some network issues or network misconfiguration Devices need a working network connection in order to communicate with Astarte. There might be some temporary network issues, or any network setting or appliance might not be properly configured. Make sure that devices are allowed to make outbound connections on ports 443 (https) and any port the transport needs for accepting connections from devices. For Astarte/VerneMQ, this defaults to 8883 (MQTT over SSL), but might also be configured otherwise. SSL issues Devices need to be able to connect to Astarte using SSL. Make sure that the clock has been synched to avoid certificate issue/expiry date errors, make also sure to have all the root CAs up to date. Device gets disconnected from Astarte Some interfaces might be missing When a device reports an interface that Astarte doesn't have, it gets disconnected when the introspection is published. Make sure that all device interfaces have been previously installed on Astarte. Make also sure that interface name and major exactly matches installed version. Device is publishing unexpected or malformed data When a device sends invalid, malformed or unexpected data it gets disconnected, make sure that the device is sending valid data. An interface mismatch might be the most common reason for this kind of issues. e.g. the interface has been declared device owned on the device, and astarte owned on astarte. Make sure to use exactly the same JSON file on both ends.","ref":"090-troubleshooting.html#devices"},{"type":"extras","title":"Troubleshooting - Triggers","doc":"Triggers are not executed Triggers have not been loaded yet Triggers might take some time before being loaded for devices that have been recently connected, make sure to wait some time before the triggers cache is populated again. If you are on a hurry make sure to test a trigger on a device that has not been recently connected yet.","ref":"090-troubleshooting.html#triggers"},{"type":"extras","title":"Known issues","doc":"This page collects some notable issues which affect Astarte v1.0 . This is by no means an exhaustive list and you should also check Github issues to see if your problem is already covered there.","ref":"095-known_issues.html"},{"type":"extras","title":"Known issues - Realm deletion","doc":"Astarte v1.0 introduces the possibility of deleting a Realm, but currently devices which have a valid certificate are not disconnected from the Realm. The issue is tracked here . Since publishing with devices on unexisting realms can cause problems (namely: RabbitMQ data queues filling up) which can also impact devices on other realms, a realm should be deleted only after ensuring that no devices are connected to it. Due to the problems that realm deletion can cause, currently the feature must be explicitly enabled with a feature gate, i.e. by adding features: realmDeletion: true components: ... to the Astarte Custom Resource (which maps to setting the HOUSEKEEPING_ENABLE_REALM_DELETION environment variable to true in the astarte_housekeeping container). We also encourage avoiding to recreate realms with the same name to avoid having devices from the old realm reconnecting back to the new one.","ref":"095-known_issues.html#realm-deletion"},{"type":"extras","title":"Known issues - Group permissions","doc":"Currently Astarte authorization mechanism doesn't allow permissions on groups with a device granularity. Specifically there's no way to authorize a user to add only specific devices to a group. The issue is tracked here . This means that right now the best way to allow users to add or remove devices from a group they have access to is to provide a backend which is able to perform the necessary authorization checks and then performs the necessary additions/removals, while the user only has a read access to the group resource. In the long term a minor semantic change is going to employed, therefore currently we discourage emitting long living tokens which allow a non-root user to manage groups (i.e. create and modify them) since the current tokens could become incompatible with future changes.","ref":"095-known_issues.html#group-permissions"},{"type":"extras","title":"Known issues - Ghost connected devices","doc":"In some circumstances, prior to Astarte v1.0 , a device might be mistakenly reported as connected. This bug has been fixed in v1.0 , however this bug may still affect devices that have connected last time while using v0.11 (prior to the upgrade to v1.0 ). This issue is likely to happen when upgrading to v1.0 since it might be caused by VerneMQ shutdown. A device reconnection fixes this issue, and the connection state will always be reliably reported.","ref":"095-known_issues.html#ghost-connected-devices"},{"type":"extras","title":"Managing Realms","doc":"Once the Cluster is set up, you can start managing it by creating Realms.","ref":"070-manage_realms.html"},{"type":"extras","title":"Managing Realms - Accessing Housekeeping key","doc":"When creating a new Cluster, Astarte Operator also creates a brand new keypair and stores it in the cluster. To retrieve it (assuming you deployed an instance named astarte in namespace astarte ): kubectl get secret -n astarte astarte-housekeeping-private-key -o=jsonpath={.data.private-key} | base64 -d > housekeeping.key You can then use housekeeping.key to authenticate against Housekeeping API.","ref":"070-manage_realms.html#accessing-housekeeping-key"},{"type":"extras","title":"Managing Realms - Device limit in a Realm","doc":"While there is no limit on the number of Devices registered in a Realm, it is possible to impose an upper bound using the <astarte base API URL>/housekeeping/v1/realms/<realm name> API. By default, such an upper bound does not exist. The limit can be retrieved, updated or removed (meaning that any number of registered Devices is allowed). Setting, updating or removing the limit on the number of registered Devices in a Realm PATCH <astarte base API URL>/housekeeping/v1/realms/<realm name> The HTTP payload of the request must have the following format: { "data": { "device_registration_limit": <value> } } To set or update the limit, the value must be a non negative integer. If the value is less than the number of currently registered devices, no devices will be removed, but it will not be possible to register new devices. When it is set, trying to register more Devices past the limit will result in an error. To remove the limit, the value must be null . Fetching the limit on the number of registered Devices in a Realm GET <astarte base API URL>/housekeeping/v1/realms/<realm name> The HTTP payload of the response will have the following format: { "data": { "realm_name": <realm name>, "jwt_public_key_pem": <realm public key>, "replication_factor": <realm replication factor>, "device_registration_limit": <realm device registration limit> } } If no limit is currently set, the value of the "device_registration_limit" field will be null .","ref":"070-manage_realms.html#device-limit-in-a-realm"},{"type":"extras","title":"Managing Realms - Work in progress","doc":"This guide is not yet complete, as this part is a moving target within astartectl . Please refer to the API Documentation to manage Realms manually once here.","ref":"070-manage_realms.html#work-in-progress"},{"type":"extras","title":"Monitoring","doc":"Astarte is a complex, distributed system that may pose several challenges when deployed in production. Individual services report health and metrics to ensure production clusters can be properly monitored and proactive actions can be taken in case of faults or unexpected behavior.","ref":"090-monitoring.html"},{"type":"extras","title":"Monitoring - Health checks","doc":"Every Astarte service, whether it's an API service or not, exposes an HTTP endpoint /health , without versioning, on its HTTP port. By default, services use port 4000 . /health is meant to be called frequently and reports the individual health state of a service. It will return 200 in case the service is healthy, or other errors in case the service is having issues. Among those issues, there might be failure in accessing RabbitMQ/RPC communications or failure in accessing the Database. Health checks and Kubernetes The aforementioned health checks are integrated in Kubernetes, when using Astarte Operator, as LivenessProbe and ReadinessProbe . As such, health monitoring and forced restarts are automatically handled without the need for the administrator to integrate any additional logic.","ref":"090-monitoring.html#health-checks"},{"type":"extras","title":"Monitoring - Service metrics","doc":"Just like /health , every service exposes a /metrics endpoint. This endpoint exposes a series of metrics in Prometheus format, which can be easily integrated and queried from any Prometheus-compatible monitoring solution. Each service, besides exposing stats on its Erlang VM, resource consumption and HTTP stats (where applicable), also exposes a number of service-specific metric, which can be queried to obtain information about Astarte's usage and behavior. Authentication and access to metrics /metrics , being Prometheus-compatible, does not implement any kind of authentication or access control. Ideally, only your scraper should have access to /metrics , as it can leak sensitive information and should not be exposed to the outer world. Astarte Operator, by default, forbids access to /metrics through its ingress, as it assumes your scraper lives within the Kubernetes cluster or has means to access the cluster on its own. However, this behavior can be overridden through by setting serveMetrics: true in the api section. An additional parameter, serveMetricsToSubnet , can be specified to restrict access to /metrics only to source IPs in a specific subnet. It is strongly recommended to set this up in case an external scraper needs to have access to /metrics , to ensure access is restricted.","ref":"090-monitoring.html#service-metrics"},{"type":"extras","title":"Advanced operations","doc":"This section provides guidance on some delicate operations that must be performed manually as they could potentially result in data loss or other types of irrecoverable damage. Always be careful while performing these operations!","ref":"095-advanced-operations.html"},{"type":"extras","title":"Advanced operations - Manual deletion of interfaces","doc":"Right now, Astarte only allows deleting draft interfaces, i.e. interfaces with major version 0 and not used by any device. If you want to delete an interface that has already published data, you must proceed manually with the steps described below. This guide assumes the aim of the operation is deleting the org.astarte-platform.genericsensors.Values interface in the test realm. The guide requires that cqlsh is connected to the Cassandra/ScyllaDB instance that your Astarte instance is using. Switch to the target keyspace The keyspace has the same name of the realm, in this case it is test : cqlsh > use test ; Find out the interface id cqlsh :test > SELECT interface_id FROM interfaces WHERE name = 'org.astarte-platform.genericsensors.Values' AND major_version = 1 ; cqlsh will reply with the interface id: interface_id -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac Delete the interface WARNING: This is a destructive step that will erase the correlation between the Interface name and internal ID. Before proceeding, ensure you saved the interface ID, or you will end up with dangling data. Further steps in this guide will require the interface ID. To delete the interface, cqlsh :test > DELETE FROM interfaces WHERE name = 'org.astarte-platform.genericsensors.Values' AND major_version = 1 ; Keep in mind that after this step, all existing devices that attempt to publish on this interface will be disconnected as soon as they try to do so. Delete interface data The interface data is stored in a different place depending on the interface type ( datastream or properties ) and aggregation. Individual datastream interfaces store their data in the individual_datastreams table. Individual properties interfaces store their data in the individual_properties table. Object datastream interfaces store their data in a dedicated table which is created starting from the interface (e.g. an interface called com.test.Sensors with major version 1 creates a com_test_sensors_v1 table in the realm keyspace). To delete data from object datastreams, a simple DROP of the table where the data is stored is needed. Deleting data from individual interfaces requires more steps. In this example the interface is an individual datastream, but the procedure for individual properties is the same, but concerns the individual_properties table instead. To delete the interface data, first all relevant primary keys must be found: cqlsh :test > SELECT DISTINCT device_id , interface_id , endpoint_id , path FROM individual_datastreams WHERE interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac ALLOW FILTERING ; This will return a set of primary keys of data belonging to that interface: device_id | interface_id | endpoint_id | path -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- - 41 c1c072 - d416 - 4686 - ba23 - 673 fe4ad926f | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / test / value 81 c60277 - 4645 - 441 f - a49b - 66 a71ce54b83 | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / foo / value ... After that, all data belonging to those primary keys must be deleted: cqlsh :test > DELETE FROM individual_datastreams WHERE device_id = 41 c1c072 - d416 - 4686 - ba23 - 673 fe4ad926f AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac AND endpoint_id = 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 AND path = '/test/value' ; cqlsh :test > DELETE FROM individual_datastreams WHERE device_id = 81 c60277 - 4645 - 441 f - a49b - 66 a71ce54b83 AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac AND endpoint_id = 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 AND path = '/foo/value' ; ... devices-by-interface cleanup If this guide is being used so as to remove a draft interface (i.e. with major version 0 ) that cannot be deleted since it has data on it, an additional step is required for a complete cleanup. The information about which devices are using draft interfaces is kept in the kv_store table. You can inspect the groups with: cqlsh :test > SELECT group FROM kv_store ; The group that has to be deleted may be easily identified by inspecting the returned group s, since it is the one with its name derived from the interface name. For example, if the purpose of the operation is removing all data from the org.astarte-platform.genericevents.DeviceEvents v0.1 interface, the corresponding group in kv_store will be devices-by-interface-org.astarte-platform.genericevents.DeviceEvents-v0 . As the target group is identified, just remove all its entries with: cqlsh :test > DELETE FROM kv_store WHERE group = 'devices-by-interface-org.astarte-platform.genericevents.DeviceEvents-v0' ; Conclusions After performing all the steps above, the interface will be completely removed from Astarte. You can then proceed to install a new interface with the same name and major version without any conflict. Remember to remove the interface also on the device side, otherwise devices will keep getting disconnected if they try to publish on the deleted interface.","ref":"095-advanced-operations.html#manual-deletion-of-interfaces"},{"type":"extras","title":"Advanced operations - Manual deletion of a device","doc":"Starting from Astarte 1.2, a device can be deleted using Realm Management API (see here ). The following manual procedure remains for historical reasons. This section assumes: that cqlsh is connected to the Cassandra/ScyllaDB instance that your Astarte is using; that astartectl is installed on your machine. Please keep in mind that this is a destructive procedure. Before moving on, ensure you saved your device ID or you might end up with dangling data and possibly a damaged Astarte deployment. Retrieve the device uuid To interact with the device and its data, the device uuid must be retrieved. Assuming that the id of the device to be deleted is k3oPTXaGGGGGGGGGGGGGGG , its uuid can be obtained with the following: $ astartectl utils device-id to-uuid k3oPTXaGGGGGGGGGGGGGGG 937a0f4d-7686-1861-8618-618618618618 Please, make sure not to lose the device uuid as it will be employed in all the following steps of this section. Switch to the target keyspace The keyspace takes its name from the realm, in this case it is test . cqlsh > use test ; Delete device data on a specific interface Depending on the interface type and aggregation, data published by the device is stored into different tables: data published over an individual datastream interface are available within the individual_datastreams table; data published over an individual property interface are available within the individual_properties table; data published over an object datastream interfaces are stored in a dedicated table named after the interface name: e.g. an interface called com.test.Sensors with major version 1 creates a com_test_sensors_v1 table in the realm keyspace. Delete device data from an individual_datastreams interface The first step consists in finding all the relevant primary keys for the device. To do so, simply run: cqlsh :test > SELECT DISTINCT device_id , interface_id , endpoint_id , path FROM individual_datastreams WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; The output will show a set of primary keys of data belonging to your device: device_id | interface_id | endpoint_id | path -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- | -- -- -- -- -- -- - 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / test / value 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | 1 e6fb841 - 9 ee3 - 0 e60 - 72 ed - 1 f55b334b832 | 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 | / foo / value ... It is now time to perform the actual data deletion: to do so, repeat the following instruction iterating over every combination of primary keys as obtained from the output of the previous command: cqlsh :test > DELETE FROM individual_datastreams WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac AND endpoint_id = 33751412 - 3 e77 - ad1f - ad57 - 280 cc9fad581 AND path = '/test/value' ; Delete device data from an individual_properties interface The first step consists in retrieving the primary keys for the device. Just run: cqlsh :test > SELECT DISTINCT device_id , interface_id FROM individual_properties WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; The output will be similar to the following one: device_id | interface_id -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | 8 ed086db - 0 bcc - 5 a9f - 2 fc2 - ddf49c35e87d 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | c61879ce - c60c - adaf - c6b4 - d04b1e1b14c4 To perform the actual data deletion, run the following query for each pair of device_id and interface_id obtained from the previous query: cqlsh :test > DELETE FROM individual_properties WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 AND interface_id = c238b244 - b90f - 4 c6d - f276 - 25768 bf6abac ; Delete device data for object datastreams The first step consists in retrieving the primary keys for the device. For this particular example the sample interface named com.test.Sensors with major version v1 is employed. Please note that the upcoming steps must be repeated for each object datastream interface installed in your realm. cqlsh :test > SELECT DISTINCT device_id , path FROM com_test_sensors_v1 WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; The output will show something like: device_id | path -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + -- -- -- 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 | / foo ... It is now time to perform the actual data deletion: cqlsh :test > DELETE FROM com_test_sensors_v1 WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 AND path = '/foo' ; Delete device aliases If your device has one or more aliases you will find them in the names table. First, you have to find the primary key for the device: cqlsh :test > SELECT object_name FROM names WHERE object_uuid = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; If your device has any aliases, the output will show object_name -- -- -- -- -- -- -- -- my - device - alias ... Thus, you can delete the alias simply executing: cqlsh :test > DELETE FROM names WHERE object_name = 'my-device-alias' ; Delete the device from groups To delete the device from a device group let's find the needed keys: SELECT group_name , insertion_uuid , device_id FROM grouped_devices WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ALLOW FILTERING ; If the device is contained in one or more groups, the output will be: group_name | insertion_uuid | device_id -- -- -- -- -- -- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- my - group | c1a0dade - 43 bc - 11 ec - 95 be - 41 f7663270b3 | 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ... The actual deletion can be performed with: cqlsh :test > DELETE FROM grouped_devices WHERE group_name = 'my-group' AND insertion_uuid = c1a0dade - 43 bc - 11 ec - 95 be - 41 f7663270b3 AND device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ; Delete entries from kv_store If your device is publishing over one or more interfaces with version v0 , you will need to handle also the kv_store table. Retrieve all the entries that must be handled: cqlsh :test > SELECT group , key FROM kv_store WHERE key = 'k3oPTXaGGGGGGGGGGGGGGG' ALLOW FILTERING ; The output of the query will show something similar to group | key -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- devices - by - interface - com . test . Sensor - v0 | k3oPTXaGGGGGGGGGGGGGGG devices - with - data - on - interface - com . test . Sensor - v0 | k3oPTXaGGGGGGGGGGGGGGG ... To remove the entries, simply execute the following queries to remove the proper rows from the table. Please, make sure to remove all the entries referencing your device ID. cqlsh :test > DELETE FROM kv_store WHERE group = 'devices-by-interface-com.test.Sensor-v0' AND key = 'k3oPTXaGGGGGGGGGGGGGGG' ; cqlsh :test > DELETE FROM kv_store WHERE group = 'devices-with-data-on-interface-com.test.Sensor-v0' AND key = 'k3oPTXaGGGGGGGGGGGGGGG' ; Eventually delete your device Deleting your device from the devices table is as simple as cqlsh :test > DELETE FROM devices WHERE device_id = 937 a0f4d - 7686 - 1861 - 8618 - 618618618618 ; Conclusions If you managed to remove all the device-related entries as described in the previous sections, then your device and its data have been properly deleted from Astarte. Before trying to reconnect your device you must make sure that the SSL certificate and all the credentials onboard your device are deleted. This is crucial for ensuring that new data published by the device can be properly ingested and processed by Astarte.","ref":"095-advanced-operations.html#manual-deletion-of-a-device"},{"type":"extras","title":"Astarte in 5 minutes","doc":"This tutorial will guide you through bringing up your Astarte instance, creating a realm and streaming your first data from a device simulator (or a real device) before your cup of tea is ready.","ref":"010-astarte_in_5_minutes.html"},{"type":"extras","title":"Astarte in 5 minutes - Before you begin","doc":"First of all, please keep in mind that this setup is not meant to be used in production : by default, no persistence is involved, the installation does not have any recovery mechanism, and you will have to restart services manually in case something goes awry. This guide is great if you want to take Astarte for a spin, or if you want to use an isolated instance for development. You will need a machine with at least 4GB of RAM, a recent 64-bit operating system with Docker , Docker Compose and astartectl installed. If you don't have astartectl installed on your machine yet, you should install it by following the instructions in astartectl's README Also, on the machine(s) or device(s) you will use as a client, you will need either Docker, or a Qt5 installation with development components if you wish to build and run components locally. Due to ScyllaDB requirements, if you're working on a Linux machine you should make sure that aio-max-nr is at least 1048576 : cat /proc/sys/fs/aio-max-nr 1048576 If it's less than that, you'll need to edit your /etc/sysctl.conf file fs . aio - max - nr = 1048576 and to persist this configuration sudo sysctl -p","ref":"010-astarte_in_5_minutes.html#before-you-begin"},{"type":"extras","title":"Astarte in 5 minutes - Checking prerequistes","doc":"Docker version >= 19 is recommended: $ docker -v Docker version 19.03.8 Docker compose version >= 2.21 is recommended: $ docker compose version Docker Compose version v2.21.0 astartectl >= 22.11 is recommended: $ astartectl version astartectl 22.11.02 This procedure has been tested on several systems, and is validated and maintained against Ubuntu 22.04 and macOS 10.15 Catalina, but any other modern operating system should work.","ref":"010-astarte_in_5_minutes.html#checking-prerequistes"},{"type":"extras","title":"Astarte in 5 minutes - Install Astarte","doc":"To get our Astarte instance running as fast as possible, we will install Astarte's standalone distribution. It includes a tunable Docker Compose which brings up Astarte and every companion service needed for it to work. To do so, simply clone Astarte's main repository and use its scripts to bring it up: $ git clone https://github.com/astarte-platform/astarte.git -b v1.2.0 && cd astarte $ docker run -v $(pwd)/compose:/compose astarte/docker-compose-initializer:1.2.0 $ docker compose pull $ docker compose up -d docker-compose-initializer will generate a root CA for devices, a key pair for Housekeeping, and a self-signed certificate for the broker (note: this is a really bad idea in production). You can tune the compose file further to use legitimate certificates and custom keys, but this is out of the scope of this tutorial. Compose might take some time to bring everything up, but usually within a minute from the containers creation Astarte will be ready. You can reach Astarte at the following addresses: api.astarte.localhost : Astarte API, in detail: api.astarte.localhost/appengine : AppEngine api.astarte.localhost/housekeeping : Housekeeping api.astarte.localhost/pairing : Pairing api.astarte.localhost/realmmanagement : Realm Management broker.astarte.localhost : VerneMQ broker dashboard.astarte.localhost : Astarte Dashboard Moreover, Compose will forward the following ports to your machine: 80 : HTTP 8883 : MQTTS To check everything went fine, use docker ps to verify relevant containers are up: Astarte itself, VerneMQ, CFSSL, RabbitMQ and ScyllaDB should be now running on your system. If any of them isn't up and running, docker ps -a should show it stopped or failed. In those cases, it is advised to issue docker compose up -d again to fix potential temporary failures.","ref":"010-astarte_in_5_minutes.html#install-astarte"},{"type":"extras","title":"Astarte in 5 minutes - Create a Realm","doc":"Now that we have our instance up and running, we can start setting up a Realm for our device. We'll call our Realm test . Given we have no SSO or Authentication mechanism set up, we're just going to generate a public key to sign our JWTs with. You can create one with astartectl : $ astartectl utils gen-keypair test Also, we will need a JWT token to authenticate against Housekeeping. generate-compose-files.sh created a keypair automatically, which is in compose/astarte-keys/housekeeping_{private,public}.pem . To perform all of our Astarte interactions, we will use astartectl . Use astartectl to create a new Realm: $ astartectl housekeeping realms create test --astarte-url http://api.astarte.localhost --realm-public-key test_public.pem -k compose/astarte-keys/housekeeping_private.pem This creates a test realm, which should be ready to be used almost immediately. To ensure your realm is available and ready, check if it exists in Astarte by issuing: $ astartectl housekeeping realms ls --astarte-url http://api.astarte.localhost -k compose/astarte-keys/housekeeping_private.pem","ref":"010-astarte_in_5_minutes.html#create-a-realm"},{"type":"extras","title":"Astarte in 5 minutes - Install interfaces","doc":"We will use Astarte's Qt5 Stream Generator to feed data into Astarte. However before starting, we will have to install the org.astarte- platform.genericsensors.Values and the org.astarte-platform.genericcommands.ServerCommands interfaces into our new realm. To do that, we can use astartectl again: $ astartectl realm-management interfaces sync standard-interfaces/org.astarte-platform.genericsensors.Values.json standard-interfaces/org.astarte-platform.genericcommands.ServerCommands.json --astarte-url http://api.astarte.localhost -r test -k test_private.pem -y Now org.astarte-platform.genericsensors.Values and org.astarte- platform.genericcommands.ServerCommands should show up among our available interfaces: $ astartectl realm-management interfaces ls --astarte-url http://api.astarte.localhost -r test -k test_private.pem Our Astarte instance is now ready for our devices.","ref":"010-astarte_in_5_minutes.html#install-interfaces"},{"type":"extras","title":"Astarte in 5 minutes - Install a trigger","doc":"We will also test Astarte's push capabilities with a trigger. This will send a POST to a URL of our choice every time the value generated by stream_test is above 0.6. Due to how triggers work, it is fundamental to install the trigger before a device connects. Doing otherwise will cause the trigger to kick in at a later time, and as such no events will be streamed for a while. Replace $TRIGGER_TARGET_URL with your target URL in the example below, you can use a Postbin service like Mailgun Postbin to generate a URL and see the POST requests. The resulting trigger would be: { "name": "my_trigger", "action": { "http_url": "$TRIGGER_TARGET_URL", "http_method": "post" }, "simple_triggers": [ { "type": "data_trigger", "on": "incoming_data", "interface_name": "org.astarte-platform.genericsensors.Values", "interface_major": 1, "match_path": "/streamTest/value", "value_match_operator": ">", "known_value": 0.6 } ] } Replace $TRIGGER_TARGET_URL with the URL your Trigger will target. Assuming you saved this as my_trigger.json , you can now install it through astartectl : $ astartectl realm-management triggers install my_trigger.json --astarte-url http://api.astarte.localhost -r test -k test_private.pem You can now check that your trigger is correctly installed: $ astartectl realm-management triggers ls --astarte-url http://api.astarte.localhost -r test -k test_private.pem","ref":"010-astarte_in_5_minutes.html#install-a-trigger"},{"type":"extras","title":"Astarte in 5 minutes - Stream data","doc":"If you already have an Astarte compliant device, you can configure it and connect it straight away, and it will just work with your new installation - provided you skip SSL checks on the broker's certificate. If you don't, you can use Astarte's stream-qt5-test to emulate an Astarte device and generate a datastream . You can do this either on the same machine where you are running Astarte, or from another machine or device on the same network. Using a container for stream-qt5-test Astarte's stream-qt5-test can be pulled from Docker Hub with: $ docker pull astarte/astarte-stream-qt5-test:1.0.4 Its most basic invocation (from your astarte repository tree) is: $ docker run --net="host" -e "DEVICE_ID=$(astartectl utils device-id generate-random)" -e "PAIRING_URL=http://api.astarte.localhost/pairing" -e "REALM=test" -e "PAIRING_JWT=$(astartectl utils gen-jwt pairing -k test_private.pem)" -e "IGNORE_SSL_ERRORS=true" astarte/astarte-stream-qt5-test:1.0.4 This will generate a random datastream from a brand new, random Device ID. You can tweak those parameters to whatever suits you better by having a look at the Dockerfile. You can spawn any number of instances you like, or you can have the same Device ID send longer streams of data by saving the container's persistency through a Docker Volume. If you wish to do so, simply add -v /persistency:<your persistency path> to your docker run invocation. Refer to stream-qt5-test README for more details on which variables can be passed to the container. Also, please note that the --net="host" parameter is required to make localhost work. If this is not desirable, you can change PAIRING_URL to an host reachable from within the container network. Obviously, that parameter isn't required if you're running the container on a different machine and PAIRING_URL is pointing to a different URL.","ref":"010-astarte_in_5_minutes.html#stream-data"},{"type":"extras","title":"Astarte in 5 minutes - Grab your tea","doc":"Congratulations! Your devices or fake devices are now communicating with Astarte, and your tea should be ready by now. You can check if everything is working out by invoking AppEngine APIs to get some values. In case you are using stream-qt5-test , you can get the last sent value with astartectl : $ astartectl appengine devices get-samples <your device id> org.astarte-platform.genericsensors.Values /streamTest/value --count 1 --astarte-url http://api.astarte.localhost -r test -k test_private.pem If you get a meaningful value, congratulations - you have a working Astarte installation with your first datastream coming in! Moreover, Astarte's Docker Compose also installs Astarte Dashboard , from which you can manage your Realms and install Triggers, Interfaces and more from a Web UI. It is accessible by default at http://dashboard.astarte.localhost - remember that if you are not exposing Astarte from localhost , you have to change Realm Management API's URL in Dashboard's configuration file, to be found in compose/astarte-dashboard/config.json in Astarte's repository. You can generate a token for Astarte Dashboard, as usual, through astartectl utils gen-jwt all-realm-apis -k test_private.pem . By default, astartectl will generate a token valid for 8 hours, but you can set a specific expiration by using the -e <seconds> parameter. From here on, you can use all of Astarte's APIs and features from your own installation. You can add devices, experiment with interfaces, or develop your own applications on top of Astarte's triggers or AppEngine's APIs. And have a lot of fun!","ref":"010-astarte_in_5_minutes.html#grab-your-tea"},{"type":"extras","title":"Astarte in 5 minutes - Cleaning up","doc":"When you're done with your tests and developments, you can use docker compose again to tear down your Astarte instance simply by issuing: $ docker compose down Unless you add the -v option, persistencies will be kept and next time you will docker compose up the cluster will come back in the very same state you left it last time. docker compose down -v is extremely useful during development, especially if you want a clean slate for testing your applications or your routines every time.","ref":"010-astarte_in_5_minutes.html#cleaning-up"},{"type":"extras","title":"Astarte in 5 minutes - Final notes","doc":"Running Astarte through docker compose is the fastest way for going from zero to hero. However, please keep in mind this setup is unlikely to hold for long in production, and is by design broken for large installations . We can't stop you from running such a thing in production, but do so as long as you know you voided your warranty by doing so. This method is great for development and for trying out the system. If you wish to deploy Astarte in a more robust environment, have a look at Astarte Enterprise or, if you want to go the DIY way, make sure that at least every service which requires persistency has reliable storage and adequate redundancy beneath it.","ref":"010-astarte_in_5_minutes.html#final-notes"},{"type":"extras","title":"Introduction","doc":"","ref":"001-intro_api.html"},{"type":"extras","title":"Introduction - REST API","doc":"Astarte's APIs are documented through Swagger . Your Astarte installation probably already has Swagger UI support, which serves as the reference for your installed APIs. To browse API documentation online, follow this link .","ref":"001-intro_api.html#rest-api"},{"type":"extras","title":"Introduction - WebSocket Astarte Channels","doc":"The AppEngine API for Astarte Channels is located at the URL: ws://<astarte-base-url>/appengine/v1/socket/websocket . To learn how to use the Channels API, you can read the documentation at: Using Astarte Channels (WebSockets)","ref":"001-intro_api.html#websocket-astarte-channels"}]
\ No newline at end of file
diff --git a/astarte/1.2/search.html b/astarte/1.2/search.html
index dda05198..c894450b 100644
--- a/astarte/1.2/search.html
+++ b/astarte/1.2/search.html
@@ -102,7 +102,7 @@