Synchronising properties from a supplier with Roomorama

This is supposed to be a step-by-step guide on how to perform the synchronisation of properties with a Roomorama supplier. It describes the usual flow expected by Concierge. However, the project enforces no rigid constraints. A supplier implementation may choose to do things differently if the model described below does not fit the supplier specific rules.

Database Format

Concierge maintains three important database tables, as far as synchronising properties is concerned:

suppliers: a table of suppliers that are integrated on Concierge. Only suppliers included in this table will ever have a synchronisation process triggered. Its content is not manually updated, but loaded from a config/suppliers.yml file, which declares a list of supplier names.
hosts: a host entry indicates, most of the times, an account on the supplier system. The relationship between hosts and suppliers is that one supplier has many hosts. However, for some suppliers, there is no distinction between different hosts, and all properties are grouped under the same host account. In such cases, the supplier will have only one associated host record. Important to notice is the fact that a host record also corresponds to a Roomorama user account, with a properly configured access token and permissions to publish properties.
properties: indicates a property that was published on Roomorama, and is currently active on Roomorama's database. If a property is no longer shared with Roomorama, the corresponding entry on the properties table should be removed. The relationship between properties and hosts is that one host has many properties. Records of this table include all property information known at a given time (including metadata, images and units) so that, on future synchronisation processes, it is possible to determine which attributes changed, and perform a corresponding API call to update the information on Roomorama.

Dealing with operations that can fail

Concierge handles operations that can fail by wrapping the return method of operations as Result instances. It is a simple implementation of the Result design pattern, and is pretty similar to what can be found in the Rust language (though without the types declaration since types are dynamic in Ruby.)

During the synchronisation process, the core classes rely on the fact that supplier implementations perform their operations and return a Result instance to indicate whether it was successful or not.

The source code for that class is very small, so it is worth to check it as well as its documentation and usage across the app. The class is defined at lib/concierge/result.rb.

A working example

The best way to explain how all compoenents work and fit together is by going through a simple, working example of an fake integration.

First, the code on its entirety is reproduced below. After, it is examined, piece by piece, with a corresponding explanation.

# config/suppliers.yml
- Acme

# apps/workers/suppliers/acme/metadata.rb

module Workers::Suppliers::
  class Metadata

    attr_reader :synchronisation, :host

    def initialize(host)
      @host = host
      @synchronisation = Workers::PropertySynchronisation.new(host)
    end

    def perform
      properties = synchronisation.new_context do
        importer.fetch
      end

      properties.each do |property|
        synchronisation.start(property["id"]) do
          roomorama_property = Roomorama::Property.new(property["id"])
          roomorama_property.title = property["identifier"]
          roomorama_property.instant_booking = true
          # other fields

          property["images"].each do |attr|
            identifier = attr["id"] || Digest::MD5.hexdigest(img["url"])
            image = Roomorama::Image.new(identifier)
            image.url = attr["url"]
            image.caption = attr["caption"]

            roomorama_property.add_image(image)
          end

          property["units"].each do |attr|
            unit = Roomorama::Unit.new(attr["id"])
            unit.title = attr["title"]
            # other attributes

            attr["images"].each do |img|
              url = extract_url(img)

              unless url.success?
                no_url_information(attr["id"], property["id"])
                return url
              end

              identifier = img["id"] || Digest::MD5.hexdigest(url.value)
              image = Roomorama::Image.new(identifier)
              image.url = img["url"]
              image.caption = img["caption"]

              unit.add_image(image)
            end
 
            roomorama_property.add_unit(unit)
          end

          Result.new(roomorama_property)
        end
      end

      synchronisation.finish!
    end

    private

    def importer
      @properties ||= Acme::Importer.new(credentials)
    end

    def extract_url(hash)
      if hash["url"]
        Result.new(hash["url"])
      else
        Result.error(:no_url)
      end
    end

    def credentials
      @credentials ||= Concierge::Credentials.for("acme")
    end
  end

end

Concierge::Announcer.on("metadata.Acme") do |host|
  Workers::Suppliers::Acme::Metadata.new(host).perform
end

In this example, we are adding support for an unexistent supplier called Acme. What follows is a close explanation of the steps involved to achieving the final solution.

Step 1. Letting Concierge know about the supplier

As mentioned before, Concierge keeps track of suppliers via the config/suppliers.yml file. That should include a definition of the workers to be configured for a given supplier. The only supported workers at the moment are metadata and availabilities. This document describes the metadata worker exclusively. The metadata worker is required for every supplier, whereas the availabilities worker might not be implemented due to supplier's limitations.

The content of the suppliers.yml file will cause Concierge to create the corresponding records on the suppliers table on the next deployment, through the suppliers:load Rake task. Every host associated with a given supplier will have workers that run according to the specifications in this file.

# config/supplires.yml
Acme:
  workers:
    metadata:
      every: "1d"
    availabilities:
      every: "2h"

Supplier2:
  workers:
    metadata:
      every: "12h"
    availabilities:
      absence: "Supplier2 does not provide an API for retrieving the calendar of availabilities."

As can be seen above, two suppliers are declared: our example, Acme, as well as a supporting Supplier2 supplier. This entry exemplifies the use of the absence field for the availabilities worker: in case it is not implemented, a reason for its non-existence can be included there, which will in turn show up at the suppliers page in Concierge's web app.

The every field indicates how often a given worker should run (for a given host). The format is supposed to be always a number, followed by a time identifier: d for days; h for hours; and m for minutes.

Step 2. Properly setting up credentials on Roomorama

Concierge uses the Roomorama API in order to publish, update and disable properties. Access to the Roomorama API is authenticated via access tokens. In order for the API calls Concierge is going to perform to happen successfuly, host accounts need to be created properly, both on Roomorama and on Concierge.

On Roomorama

A regular user account needs to exist on Roomorama. That is where the properties are going to be published under. The host account needs to be a valid host account, have a confirmed and valid email account, be an elite host (if the cancellation policy requires) and a valid access token needs to be created. Someone on Roomorama staff is responsible for setting up the account properly. Alternatively, the create-host API call on Roomorama is able to configure a host account on Roomorama with everything properly set up in order to support properties with instant confirmation.

On Concierge

Once the user account above is created, the host account on Concierge needs to be created. It needs:

linking to the related supplier via the supplier_id foreign key
the identifier, on the supplier system, of the host account, configured on the identifier column
the Roomorama username, only for presentation purposes
the API access_token must be set on the database record accordingly.

Step 3. Announcing an implementation

Concierge announces that a synchronisation process for a given host should be performed via its support class Concierge::Announcer. Supplier implementations should listen for events using that class and, when something is received, the implementation for a given supplier should kick-in.

Whenever a synchronisation process for a given supplier should happen, the metadata.<supplierName> event is triggered in that class. Our Acme implementation should properly listen to that message and do its job:

Concierge::Announcer.on("metadata.Acme") do |host|
  Workers::Suppliers::Acme.new(host).perform
end

The event receives a single parameter, which is the instance of the Host entity which is to be synchronised. Whatever happens inside the block is supplier-implementation specific. In the example above, it is ilustrated the way in which the Acme implementation is organised.

Step 4. Implementing the synchronisation

It is highly recommended to wrap the work to be carried out in the block mentioned above in a separate class, for obvious reasons (more modular, testable, reproducible approach.)

In our Acme implementation, the Workers::Suppliers::Acme class receives the Host instance on initialization, with that instantiates a new object of type Workers::PropertySynchronisation:

def initialize(host)
  @host = host
  @synchronisation = Workers::PropertySynchronisation.new(host)
end

More about Workers::PropertySynchronisation will be outlined later.

Analysing the metadata.Acme block passed to Concierge::Announcer, we see that the perform method is called. Let's take a look at that, line by line:

properties = synchronisation.new_context
  importer.fetch
end

# ...
 
def importer
  @properties ||= Acme::Importer.new(credentials)
end

In this example, Acme::Importer is a wrapper for the Acme API to import properties. Client libraries are often included in files under lib/concierge/suppliers/<name>. The fetch method, unimportant for this example, is responsible for hitting the Acme API and returning the list of properties, in a raw format.

Because the fetch method (or any work done in this stage) can announce external errors, we need to ensure that the error is recorded in the right context, ie. during a synchronisation, but not related to any property. Synchronisation provides a #new_context(property_id=nil) to wrap around such work.

Notice that it is recommended (or at least not necessary) to wrap the response of the fetch method in a proper object since that the final result will have to be an instance of Roomorama::Property (more on that later).

Moving on, the perform method iterates over a collection of properties in the resulting response (note that the format of the Acme response is fictitious, but representative of what is expected from a supplier API.)

synchronisation.start(property["id"]) do

In the line above, we call the start method of the synchronisation object (remember, that is an instance of Workers::PropertySynchronisation). When a synchronisation is started, it means that the process of importing a new property is going to be performed.

The start method receives a block that performs the conversion of the response payload of the Acme API into a Roomorama::Property object. Note that Workers::PropertySynchronisation#start expects the return of the invoked block to be an instance of Result which, when successful, should wrap the corresponding Roomorama::Property object. If the result indicates failure, that is logged into the external_errors table for later analysis (for more on external errors, see https://github.com/roomorama/concierge/wiki/Concierge-Service-Goals#error-handling), and no request to roomorama is performed.

roomorama_property = Roomorama::Property.new(property["id"])
roomorama_property.title = property["identifier"]
roomorama_property.instant_booking = true
# other fields

This builds the Roomorama::Property object that is to be returned by this block (wrapped as a Result). For convenience and ease of presentation, the other fields related to a property were not included, but should be added for a working integration (such as the property description, amenities, taxes and so on). Note that if the property being synchronised is supposed to have instant confirmation, the instant_booking field must be set to true.

Also worth mentioning is the fact that Roomorama::Property is not related to the toplevel Property. The former represents the property format of a property according to Roomorama's API, whereas the latter is the Concierge entity which keeps track of the properties table of previously published properties.

property["images"].each do |attr|
  identifier = attr["id"] || Digest::MD5.hexdigest(img["url"])
  image = Roomorama::Image.new(identifier)
  image.url = attr["url"]
  image.caption = attr["caption"]

  roomorama_property.add_image(image)
end

Here, we iterate and build all images related to a property. Images are wrapped as Roomorama::Image objects. Note that - as was the case with Roomorama::Property - we need to give the image an identifier.

Concierge does not keep track of Roomorama IDs for properties/units/images. It only deals with supplier-provided indentifiers. For images, it is often the case that suppliers will not share an identifier. In that case, it is possible to use a hash of the image location (URL), so that images can be uniquely identified. That is what is ilustrated in the following line:

identifier = attr["id"] || Digest::MD5.hexdigest(img["url"])

When the Roomorama::Image object is built, it can be associated with a property via the Roomorama::Property#add_image method.

property["units"].each do |attr|
  unit = Roomorama::Unit.new(attr["id"])
  unit.title = attr["title"]
  # other attributes

Here, we simulate the scenario where a property is composed of many units (a multi-unit property). In case the property being synchronised is not multi-unit, the steps to follow can be skipped.

Units are very similar to properties. They have title, description and some attributes in common with properties. Units are wrapped in Roomorama::Unit objects, receiving a required identifier on initilization.

attr["images"].each do |img|
  url = extract_url(img)

  unless url.success?
    no_url_information(attr["id"], property["id"])
    return url
  end

  identifier = img["id"] || Digest::MD5.hexdigest(url.value)
  image = Roomorama::Image.new(identifier)
  image.url = img["url"]
  image.caption = img["caption"]

  unit.add_image(image)
end

Units also have images. In a similar way to what happened for the property itself, we iterate over a collection of image attributes and create Roomorama::Image objects for them. Images are associated with a unit via the Roomorama::Unit#add_image method.

Note that we simulate a scenario where there is an invalid field on the response:

url = extract_url(img)

unless url.success?
  no_url_information(attr["id"], property["id"])
  return url
end

def extract_url(hash)
  if hash["url"]
    Result.new(hash["url"])
  else
    Result.error(:no_url)
  end
end

Here, we suppose that, for some reason, some images contain no url field for the image. If that is the case, we return an error result (Result.error(:no_url)), and augment the context of the process by reporting the mismatch.

Concierge's context is defined as a series of events that happen when a transaction is being performed (price being calculated, a property being imported, a booking operation.) If an error happens (that is, the block passed to start returns a non-successful Result), then the context of the process will be stored in the database and will be available for later analysis.

roomorama_property.add_unit(unit)

Units are associated to a property via the Roomorama::Property#add_unit method.

Result.new(roomorama_property)

In case everything went well, the block passed to Workers::Synchronisation#start returns a successful Result containing the final Roomorama::Property object. At the end of the block, the synchronisation object will:

check if the property known to Concierge (that is, was already published before). In case it is not, it will publish the property on Roomorama using the publish API. In case there is any error with this call (example, validations failed on Roomorama), the failure will be logged for later analysis.
if the property is already known, the synchronisation process will check for any changes in the attributes since the last time the property was fetched. It will then calculate the set of differences (including meta-attributes, new/updated/removed images, new/updated/removed units), and will use Roomorama's diff API to apply the changes. In case the API call fails, that is logged for later analysis.

synchronisation.finish!

At the end of the process, when all properties have been processed, we invoke the Workers::Synchronisation#finish! method. This method indicates that the synchronisation process is finished, and all properties have been processed. Concierge will then verify if there is any previously known property for that host that was not processed (meaning the property was disabled/removed on the supplier's end.) It will then proceed to disable such properties on Roomorama through the disable API. That is to ensure that properties that no longer fit into the integration or are removed from the supplier account are reflected back (that is, removed), from Roomorama.

Final Considerations

Supplier implementations need not care whether or not the property is new or to be updated. That is taken care of by the core classes of Concierge.
Context tracking is one of the main features of Concierge which will allow us to have better visibility of what went wrong when there is missing data or any kind of failure. Therefore, it is important to properly augment the context (Concierge.context) accordingly with useful information.
Using the core classes provided by Concierge to perform common tasks (Concierge::HTTPClient, Concierge::JSONRPC, Concierge::JSON), guarantees Result-aware implementations. If custom logic is required, it is highly encouraged to follow the Result approach that is taken in those classes so as to provide useful data to the external error report.
If the supplier provides all properties at once in a single network call, it makes sense to perform that call outside of Workers::Synchronisation#start, so as to avoid keeping in memory a potentially huge payload. However, if one API call is required for each different property, then it makes sense to perform the call inside the start block, so that the request and response will be properly tracked in the Concierge.context object (as long as Concierge::HTTPClient is used.)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly