Progress on automations, roadmap for 10.x and 11.x

[jstanden]

We're pretty close to a release for 10.0.

I've been pushing out beta builds at https://github.com/jstanden/cerb/tree/10.0

The release notes preview has been staying reasonably in-sync.

I've been wrapping up some of the last work that potentially changes the schema or the automations syntax. It's a lot easier to make those changes now before thousands of production workflows are running on them.

I recommend running a clean test environment for the builds, since the schema can still change. We're not guaranteeing the 9.x->10.0 migration will handle all the incremental variations that happen within 10.0-dev. Once that's finalized, we'll be able to guarantee migrations between all versions for real data.

I'm very encouraged by the design decisions in automations and their events.

Originally we had functionality directly call an automation. Kind of like how you'd pick a behavior from a chooser/dropdown on a widget.

Increasingly, we found it helpful to pass some custom input to automations because it made them more reusable.

That evolved into the event handler KATA, where you can describe multiple automations as handlers on any event, with their priority/order being explicit, with the ability to add any custom input, as well as conditional enable/disable.

automation/onlyTasks:
  uri: cerb:automation:example.tasks
  disabled@bool: {{record_type != 'task' ? 'yes'}}

automation/everythingElse:
  uri: cerb:automation:example.generic

Events have three different modes: run every handler (e.g. project board 'move card' actions/macros), run once / only the first enabled handler (e.g. interactions), or running every handler until the first successful response (e.g. mail routing).

So with something like the mail routing event, you can specify several automation 'handlers' in a preferred order. For instance, checking for support recipients, sales recipients, etc. The first handler can just 'exit' if it doesn't match, and the next handler is checked. Once a handler ends in the 'return' state you have a successful result.

automation/support:
  uri: cerb:automation:cerb.mailRouting.recipientRules
  inputs:
    group_name: Support
    email_patterns@list:
      support@*
      team@cerb.example

automation/sales:
  uri: cerb:automation:cerb.mailRouting.recipientRules
  inputs:
    group_name: Sales
    email_patterns@list:
      sales@cerb.example

automation/dev:
  uri: cerb:automation:cerb.mailRouting.recipientRules
  inputs:
    group_name: Development
    email_patterns@list:
      bugs@cerb.example
      dev@cerb.example
      development@cerb.example

That assumes the handlers need to run to determine if they match.

For something like the 'record changed' event, you can check the record's type directly in the events KATA, and skip over some handlers. That lets you efficiently create handlers that react to every record change on any type (e.g. audit logging), or for any subset of record types.

It also makes development a lot easier. In the events KATA you can bind a higher priority "dev" handler to only your account, and the production workflows continue to run on the second "stable" handler.

Even for something like a webhook listener event, you could have one automation bind for responding to your IP for testing/dev. Once you're satisfied with changes you can promote that to the production handler.

We also don't expect all event handler automations to be written from scratch. For instance, on something like 'mail routing', we provide built-in default handlers for 'route by recipient email address' where you just pick that from a list and add a list of email patterns like team@* or *@company.example. That's the most common scenario, and an admin doesn't have to understand anything about automations to use it.

Event handlers also make it easy for us to provide default built-in functionality, which can be overridden in your environment without altering the original. For instance, the 'Reply' button on ticket profiles is toolbar KATA. If you provide your own interaction in that toolbar with the nickname 'reply', it will take the place of our functionality. If you conditionally disable your interaction, ours will be the fallback. You can always compare or revert your implementation to the baseline.

That event handler approach requires a parent record for all events. That's obvious for automations on things like webhook listeners, maps, reminders, or project boards. But we didn't want to have to create a bunch of new record types for mail filters, mail routing, data query datasources, etc.

So we came up with two new record types that cover almost everything:

• Automation events - Triggered events like 'record changed', 'mail filtering', etc.

• Automation timers - Time-based events like recurring or scheduled workflows.

An automation timer can handle workflows like syncing tweets into a custom record at regular intervals, or sending daily reports.

As a reminder, automations end in an explicit end state: exit (no result), return (success), error (failure), or await (continuations). This explicit state is a huge improvement over bot behaviors.

Only some events support continuations, where there can be any number of input/output cycles.

Timers support continuations, so after they execute they can 'await' a future date/time to resume.

A timer can be linked to specific records (like a ticket or customer), and/or its events KATA can include custom input for the automation to describe the target record (like a ticket ID/mask).

So you can have timers that run once and stop; and you can have timers that intermittently run for days/weeks/months with varying conditional intervals in between actions. One of the more interesting applications of that would be sales automation like on-boarding. An automation timer could run daily for 7 days and then end.

Different daily tips could be sent based on the target (e.g. the usage patterns of a specific client). That workflow is really cumbersome to implement in bot behaviors because they have no inherent permanence. You have to keep track of everything with custom fields, which creates a lot of clutter and noise. Automation timers use continuations. On their next run they resume the same state/dictionary as all past executions. If you set a variable days or months ago, the current execution of the continuation will remember it. It's just like designing a serial workflow and then adding indeterminate waits where you need a piece of external information to continue (e.g. "has the client logged in yet?", "have they tried feature X?").

While typing out some automations KATA and events KATA is a little more technical than point-and-click behaviors, they support conditional scripting and placeholders for every criteria, every action, and every parameter. Their implementation is much simpler and more elegant. And once you learn the basics, they work exactly the same way everywhere.

Automation-based events are also "top down" -- in that you specify handlers for events in a single place. For instance, you can immediately see everything that will happen in mail filtering or routing, in order, in the same events KATA. Bot behaviors were "bottom up", where new handlers could be added democratically, and relied on a non-unique 'priority' where the sequence of event responders was never clear (especially if they had the same priority).

The main work left is adding the helpers so everything is intuitive. Help buttons in event/automation editors, example KATA for everything, available placeholders, etc. Over time we'll work on visual coding tools that generate KATA, but right now that gets back into a situation where it limits the intentional expressiveness of it.

One other really interesting thing with the help and 'insert from library' features in automation editors is that they're interaction automations themselves. You can click something like 'Insert Automation' in the editor menu, and an interaction popup walks you through configuring it, then it returns the text snippet which is pasted into the editor at the cursor. Those are easy to build and maintain, and they provide good references examples and starting points for custom shortcuts.

cerb10_automation_editor_interactions.mov

As for an ETA, we're working hard to get this out ASAP. That's just tempered by the fact that the best time to get the core building blocks done properly is right now. It would be frustrating for users if we made significant changes to the KATA dialects for automations, events, or toolbars, right after we made them convert all their long-time behaviors over. My goal is to have a schema-stable beta out this month. The full release should be Q1.

The roadmap looks like:

• 10.0: Automations
• 10.1: Portal builder
• 10.x: Refining automation capabilities and usability as everyone replaces behaviors (edge cases)
• 11.0: Retiring all bot behaviors

The current plan is that we'll support existing bot behaviors in all 10.x versions, which will likely encompass full year 2021.

The work on 11.0 will likely start shortly after 10.0 or 10.1, where we go through the process of removing behaviors everywhere, and feeding edge cases into the remaining 10.x updates to make that process as painless as possible.

When clients finish converting bot behaviors they can switch over to 11.x early. New clients will start on 11.x so they aren't creating behaviors they'll have to immediately replace, or wasting time learning about behaviors. At this point, the 11.0 build will be about respectfully breaking backwards compatibility on behaviors rather than big new features.

We'll likely version the branches in the cerb/cerb-release repository so no 10.x clients are automatically upgraded to 11.x before they're ready. For now, the 'stable' branch will remain 10.x, but that won't remain the default branch.

That way everyone has at least this entire year to phase out bot behaviors without feeling pressured or feeling like they're missing out on new updates (e.g. portal builder) by taking their time. But we won't have to wait a year to clean out the cruft for 11.0.

Feel free to check out the 10.0-dev build and provide feedback. I think it will be pretty obvious right away, though, why we're drawing more attention to automations before 10.0 is final. Despite 18 months of work in 10.0, automations aren't something you immediately notice unless you're looking for them. Daily workers won't notice much until admins have built out the interactions. We want to have a catalog of examples (like the package library for automations) to get up and running more quickly.