Skip to content

Commit

Permalink
Feature/157 improve playbook execution speed by embedding the json sc…
Browse files Browse the repository at this point in the history
…hemas (#199)
  • Loading branch information
MaartendeKruijf authored Aug 2, 2024
1 parent c1217ec commit 06f59c8
Show file tree
Hide file tree
Showing 57 changed files with 3,356 additions and 18 deletions.
3 changes: 0 additions & 3 deletions docs/content/en/docs/core-components/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,6 @@ To change logging for your SOARCA instance you can use the following environment
|LOG_MODE |development \| production |If production is chosen the `LOG_GLOBAL_LEVEL` is used for all modules defaults to `production`
|LOG_FILE_PATH |filepath |Path to the logfile you want to use for all logging. Defaults to `""` (empty string)
|LOG_FORMAT |text \| json |The logging can be in plain text format or in JSON format. Defaults to `json`
|MQTT_BROKER | dns name or ip | The broker address for SOARCA to connect to, for communication with fins default is `localhost`
|MQTT_PORT | port | The broker address for SOARCA to connect to, for communication with fins default is `1883`
|ENABLE_FINS| true \| false | Enable fins in SOARCA defaults to `false`



Expand Down
21 changes: 21 additions & 0 deletions docs/content/en/docs/getting-started/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ LOG_FORMAT: "json"
ENABLE_FINS: false
MQTT_BROKER: "localhost"
MQTT_PORT: 1883
VALIDATION_SCHEMA_URL: ""
{{< /tab >}}
{{< /tabpane >}}

Expand All @@ -113,3 +114,23 @@ make build
cp .env.example .env
./build/soarca
```

### Configuring SOARCA

|variable |content |description
|---|---|---|
|PORT |port |Set the exposed port of SOARCA the default is `8080`
|DATABASE |true \| false | Set if you want to run with external database default is `false`
|MONGODB_URI |uri |Set the Mongo DB uri default is `mongodb://localhost:27017`
|DATABASE_NAME |name |Set the Mongo DB database name when using docker default is `soarca`
|DB_USERNAME |user |Set the Mongo DB database user when using docker default is `root`
|DB_PASSWORD |password |Set the Mongo DB database users password when using docker default is `rootpassword`. IT IS RECOMMENDED TO CHANGE THIS IN PRODUCTION!
|MAX_REPORTERS |number |Set the maximum number of downstream reporters default is `5`
|LOG_GLOBAL_LEVEL |[Log levels] |One of the specified log levels. Defaults to `info`
|LOG_MODE |development \| production |If production is chosen the `LOG_GLOBAL_LEVEL` is used for all modules defaults to `production`
|LOG_FILE_PATH |filepath |Path to the logfile you want to use for all logging. Defaults to `""` (empty string)
|LOG_FORMAT |text \| json |The logging can be in plain text format or in JSON format. Defaults to `json`
|MQTT_BROKER | dns name or ip | The broker address for SOARCA to connect to, for communication with fins default is `localhost`
|MQTT_PORT | port | The broker address for SOARCA to connect to, for communication with fins default is `1883`
|ENABLE_FINS| true \| false | Enable fins in SOARCA defaults to `false`
|VALIDATION_SCHEMA_URL|url| Set a custom validation schema to be used to validate playbooks defaul is `""` to use internal. NOTE: changing this heavily impacts performance.
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ require (
github.com/google/uuid v1.3.1
github.com/joho/godotenv v1.5.1
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.9.0
github.com/swaggo/files v1.0.1
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/eclipse/paho.mqtt.golang v1.4.3 h1:2kwcUGn8seMUfWndX0hGbvH8r7crgcJguQNCyp70xik=
github.com/eclipse/paho.mqtt.golang v1.4.3/go.mod h1:CSYvoAlsMkhYOXh/oKyxa8EcBci6dVkLCbo5tTC1RIE=
github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
Expand Down Expand Up @@ -104,6 +106,8 @@ github.com/rogpeppe/go-internal v1.8.0 h1:FCbCCtXNOY3UtUuHUYaghJg4y7Fd14rXifAYUA
github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE=
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1 h1:lZUw3E0/J3roVtGQ+SCrUrg3ON6NgVqpn3+iol9aGu4=
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1/go.mod h1:uToXkOrWAZ6/Oc07xWQrPOhJotwFIyu2bBVN41fcDUY=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1 h1:PKK9DyHxif4LZo+uQSgXNqs0jj5+xZwwfKHgph2lxBw=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
Expand Down
76 changes: 62 additions & 14 deletions models/validator/schema.go
Original file line number Diff line number Diff line change
@@ -1,23 +1,32 @@
package validator

import (
"embed"
"encoding/json"
"errors"
"io/fs"
"reflect"
"soarca/logger"
"soarca/models/cacao"
"soarca/utils"
"strings"

"github.com/go-playground/validator/v10"
jsonschema "github.com/santhosh-tekuri/jsonschema/v5"
_ "github.com/santhosh-tekuri/jsonschema/v5/httploader"
jsonschema "github.com/santhosh-tekuri/jsonschema/v6"
)

type Empty struct{}

var component = reflect.TypeOf(Empty{}).PkgPath()
var log *logger.Log

var oca_cacao_schemas string = "https://raw.githubusercontent.com/opencybersecurityalliance/cacao-roaster/main/lib/cacao-json-schemas/schemas/playbook.json"
const (
oca_cacao_schemas string = "./schemas/playbook.json"
)

//go:embed schemas/*
var schemas embed.FS

//var oasis_cacao_schemas string = "https://raw.githubusercontent.com/oasis-open/cacao-json-schemas/main/schemas/playbook.json"
//var cyentific_cacao_schemas string = "https://raw.githubusercontent.com/cyentific-rni/cacao-json-schemas/cacao-v2.0-csd03/schemas/playbook.json"
Expand Down Expand Up @@ -45,6 +54,52 @@ func UnmarshalJson[BodyType any](b *[]byte) (any, error) {
return body, nil
}

func validateWithLocalSchema(playbookToValidate map[string]interface{}) error {

compiler := jsonschema.NewCompiler()

err := fs.WalkDir(schemas, ".", func(path string, d fs.DirEntry, err error) error {
isFile := d.Type().IsRegular()

if isFile {
content, _ := fs.ReadFile(schemas, path)
data, err := jsonschema.UnmarshalJSON(strings.NewReader(string(content)))
if err != nil {
return err
}
if err := compiler.AddResource(path, data); err != nil {
return err
}
}
return nil
})
if err != nil {
log.Error(err)
return err
}

sch, err := compiler.Compile(oca_cacao_schemas)
if err != nil {
return err
}

err = sch.Validate(playbookToValidate)
return err
}

func validateWithRemoteSchema(data map[string]interface{}, url string) error {
compiler := jsonschema.NewCompiler()

sch, err := compiler.Compile(url)
if err != nil {
return err
}
if err := sch.Validate(data); err != nil {
return err
}
return nil
}

func IsValidCacaoJson(data []byte) error {

var rawJson map[string]interface{}
Expand All @@ -54,25 +109,18 @@ func IsValidCacaoJson(data []byte) error {

version := rawJson["spec_version"]

compiler := jsonschema.NewCompiler()
compiler.Draft = jsonschema.Draft7

var sch *jsonschema.Schema
var err error
switch version {
case cacao.CACAO_VERSION_1:
return errors.New("you submitted a cacao v1 playbook. at the moment, soarca only supports cacao v2 playbooks")
case cacao.CACAO_VERSION_2:
sch, err = compiler.Compile(oca_cacao_schemas)
if err != nil {
return err
schemaUrl := utils.GetEnv("VALIDATION_SCHEMA_URL", "")
if schemaUrl != "" {
return validateWithRemoteSchema(rawJson, schemaUrl)
} else {
return validateWithLocalSchema(rawJson)
}
default:
return errors.New("unsupported cacao version")
}

if err := sch.Validate(rawJson); err != nil {
return err
}
return nil
}
63 changes: 63 additions & 0 deletions models/validator/schemas/agent-target/agent-target.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"$id": "agent-target",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "agent-target",
"description": "In a CACAO playbook, agents are the entities that execute commands (see section 5) on or against targets. Agents are stored in a dictionary where the ID is the key and the value is an 'agent-target' object (see section 10.1). Targets are stored in a dictionary where the ID is the key and the value is an 'agent-target' object (see section 10.1). Common properties for agents and targets are defined in section 7.1. \n\nAgents can involve either manual or automated processing. For example, an individual may process a command manually, while a firewall may process a command automatically. An agent and target type vocabulary is defined in section 7.2, and each agent and target type is further defined in the rest of the sections. Types include security infrastructure such as firewalls, routers, and threat intelligence platforms, as well as specific network addressable agents like URLs and IPv4/IPv6/MAC addresses. \n\nAgents and targets can use and refer to variables just like other parts of the playbook. For any agent or target property value, the producer may define a variable substitution such that the actual property value is determined at runtime based on the variable assigned to the agent or target. In Example 7.1, an agent is referenced within a workflow step, but the agent's actual values are based on variables (e.g., name, email, phone, location) instead of being hard-coded by the agent itself. \n\nEach object (agent or target) contains base properties that are common across all objects. These properties are defined in the following table. The ID for each object is stored as the key in the agent_definitions dictionary or the target_definitions dictionary.",
"type": "object",
"properties": {
"type": {
"$ref": "#/$defs/agent-target-type-ov",
"description": "The type of object being used. The value of this property SHOULD come from the 'agent-target-type-ov' vocabulary."
},
"name": {
"type": "string",
"description": "The name that represents this object that is meant to be displayed in a user interface or captured in a log message. This property MUST be populated."
},
"description": {
"type": "string",
"description": "More details, context, and possibly an explanation about this object. This property SHOULD be populated."
},
"location": {
"$ref": "../data-types/civic-location.json",
"description": "Physical address information for this object."
},
"agent_target_extensions": {
"minProperties": 1,
"type": "object",
"patternProperties": {
"^extension-definition--[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$": {
"type": "object"
}
},
"description": "This property declares the extensions that are in use on this action or target and contains any of the properties and values that are to be used by that extension. \n\nThe key for each entry in the dictionary MUST be an 'identifier' (see section 10.10 for more information on identifiers) that uniquely identifies the extension. The value for each key is a JSON object that contains the structure as defined in the extension definition's schema property. The actual step extension definition is located in the 'extension_definitions' property found at the Playbook level."
}
},
"required": [
"type",
"name"
],
"$defs": {
"agent-target-type-ov": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [
"group",
"individual",
"location",
"organization",
"sector",
"http-api",
"linux",
"net-address",
"security-category",
"ssh"
]
}
]
}
}
}
27 changes: 27 additions & 0 deletions models/validator/schemas/agent-target/group.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"$id": "agent-target",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "group",
"description": "This type defines a group object and is used for commands that need to be processed or executed by a group. This object inherits the common agent properties. In addition to the inherited properties, this section defines the following additional property that is valid for this type.",
"type": "object",
"allOf": [
{
"$ref": "agent-target.json"
},
{
"properties": {
"type": {
"type": "string",
"description": "The value of this property MUST be 'group'.",
"enum": [
"group"
]
},
"contact": {
"$ref": "../data-types/contact.json",
"description": "Contact information for this agent."
}
}
}
]
}
Loading

0 comments on commit 06f59c8

Please sign in to comment.