-
Notifications
You must be signed in to change notification settings - Fork 20
Cross devices correlation and closed loop automation (demo with MTU mismatch automatic correction)
The then
clause in Healtboth rules
is used to set actions to be taken.
These actions can be:
- pre-defined-actions (ex: text message to show, color to show, ...)
- User-Defined-Actions (UDA).
UDA invoke functions. These functions are implemented in Python.
So you create your own python code, and healthbot will execute it automatically based on the data collected and the rules you defined.
This directory has python functions.
These python functions can be triggered by Healthbot rules.
When we configure a rule field
, we can use the argument reference
to refer to another field
from another rule
.
We will use one rule to collect data and update the database, and another rule to process/query the database and take actions.
The rule collect-interfaces-mtu.rule uses openconfig to continuously collect interfaces mtu and update the database with this.
The playbook collect-mtu.playbook uses the rule collect-interfaces-mtu.rule.
You need to instanciate the playbook collect-mtu.playbook for a device group.
The playbook collect-mtu.playbook doesnt show any data in Healthbot GUI because the rule collect-interfaces-mtu.rule doesnt use any trigger
Purpose of above rule and playbook is only to collect data and update the database. There is no database processing/queries in the above rule.
The rule compare-interfaces-mtu.rule:
- continuously queries the database updated by the rule collect-interfaces-mtu.rule
- compares the mtu of 2 connected interfaces (cross devices correlation)
- and in case of mtu mismacth, it changes the junos configuration (closed loop automation) to enforce the same mtu across the 2 connected interfaces
The rule compare-interfaces-mtu.rule uses User-Defined-Actions (UDA) (python code).
Here's the UDA change-mtu-config.py fired automatically by the rule compare-interfaces-mtu.rule in order to change the junos configurationto enforce the same mtu across the 2 connected interfaces
The rule compare-interfaces-mtu.rule is a network-rule
The rule compare-interfaces-mtu.rule uses these 6 variables to define a connection between two devices (so which interface of which device of which group is connected to which interface of which device of which group):
device1-group-variable
device1-inter-variable
device1-name-variable
device2-group-variable
device2-inter-variable
device2-name-variable
The rule compare-interfaces-mtu.rule uses the field device1-mtu
and device2-mtu
with the argument reference
to refer to data collected by the rule collect-interfaces-mtu.rule
The playbook compare-mtu.playbook uses the rule compare-interfaces-mtu.rule
You need to instanciate the playbook compare-mtu.playbook for a network group
For this topology with 3 spines and 4 leaves,
We need 1 instance for each connection.
There are 3 spines, each spine is connected to 4 leaves, so it is 3*4 connections to monitor
So we need 12 instances of the playbook compare-mtu.playbook.
Here's the variables for the 12 instances/connections:
spine vMX1 <-> leaves connections:
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/0 / device2-group-variable vmx, device2-group-variable vMX4, device2-inter-variable ge-0/0/0
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/1 / device2-group-variable vmx, device2-group-variable vMX5, device2-inter-variable ge-0/0/0
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/2 / device2-group-variable vmx, device2-group-variable vMX6, device2-inter-variable ge-0/0/0
device1-group-variable vmx, device1-group-variable vMX1, device1-inter-variable ge-0/0/3 / device2-group-variable vmx, device2-group-variable vMX7, device2-inter-variable ge-0/0/0
spine vMX2 <-> leaves connections:
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/0 / device2-group-variable vmx, device2-group-variable vMX4, device2-inter-variable ge-0/0/1
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/1 / device2-group-variable vmx, device2-group-variable vMX5, device2-inter-variable ge-0/0/1
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/2 / device2-group-variable vmx, device2-group-variable vMX6, device2-inter-variable ge-0/0/1
device1-group-variable vmx, device1-group-variable vMX2, device1-inter-variable ge-0/0/3 / device2-group-variable vmx, device2-group-variable vMX7, device2-inter-variable ge-0/0/1
spine vMX3 <-> leaves connections:
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/0 / device2-group-variable vmx, device2-group-variable vMX4, device2-inter-variable ge-0/0/2
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/1 / device2-group-variable vmx, device2-group-variable vMX5, device2-inter-variable ge-0/0/2
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/2 / device2-group-variable vmx, device2-group-variable vMX6, device2-inter-variable ge-0/0/2
device1-group-variable vmx, device1-group-variable vMX3, device1-inter-variable ge-0/0/3 / device2-group-variable vmx, device2-group-variable vMX7, device2-inter-variable ge-0/0/2
The python script enforce_same_mtu.py executes all the required tasks:
$ python enforce_same_mtu.py
loaded healthbot rule collect-interfaces-mtu.rule
loaded healthbot playbook collect-mtu.playbook
loaded healthbot configuration for the device: vMX1
loaded healthbot configuration for the device: vMX2
loaded healthbot configuration for the device: vMX3
loaded healthbot configuration for the device: vMX4
loaded healthbot configuration for the device: vMX5
loaded healthbot configuration for the device: vMX6
loaded healthbot configuration for the device: vMX7
loaded healthbot configuration for the device group: vmx
healthbot configuration commited!
wait 60 sec
loaded healthbot function change-mtu-config.py
loaded healthbot rule compare-interfaces-mtu.rule
loaded healthbot playbook compare-mtu.playbook
loaded healthbot configuration for the network group: MTU
healthbot configuration commited!
$
Healthbot is collecting the MTU on the interfaces between spines and leaves.
There is currently no MTU mismatch.
In case of MTU mismatch on a link, Healthbot will detect the issue (cross devices correlation) and will fix the MTU mismatch (closed loop automation): it will change the junos configuration to enforce the same MTU across the 2 connected interfaces (in that demo, it will enforce the minimum MTU).
let's connect on vMX1 device, and let's change the MTU on its interface to vMX4. This will create an MTU mismatch issue (between vMX1 and vMX4).
$ ssh 100.123.1.0
Password:
Last login: Mon Mar 11 17:23:00 2019 from 100.123.35.0
--- JUNOS 18.2R1.9 Kernel 64-bit JNPR-11.0-20180614.6c3f819_buil
jcluser@vMX-addr-0> edit
Entering configuration mode
[edit]
jcluser@vMX-addr-0# set interfaces ge-0/0/0 mtu 1300
[edit]
jcluser@vMX-addr-0# commit and-quit
commit complete
Exiting configuration mode
jcluser@vMX-addr-0>
Healthbot detected the issue
Healthbot fixed the issue
Let's connect on vMX4 to verify
$ ssh 100.123.1.3
Password:
--- JUNOS 18.2R1.9 Kernel 64-bit JNPR-11.0-20180614.6c3f819_buil
jcluser@vMX-addr-3> show configuration | compare rollback 1
[edit interfaces ge-0/0/0]
+ mtu 1300;
jcluser@vMX-addr-3> show system commit
0 2019-03-12 13:46:41 UTC by jcluser via netconf