preliminary integration of webhooks validator (#224)

* preliminary integration of webhooks

* implement webhook validation in main.go

* go.sum

* added documentation for optional fields

* edited tr1d1um_spruce

* Update deploy/packaging/tr1d1um_spruce.yaml

Co-authored-by: Joel Unzain <joe94@users.noreply.github.com>

* Update deploy/packaging/tr1d1um_spruce.yaml

Co-authored-by: Joel Unzain <joe94@users.noreply.github.com>

* documentation

* Update deploy/packaging/tr1d1um_spruce.yaml

Co-authored-by: Joel Unzain <joe94@users.noreply.github.com>

* documentation

* Update deploy/packaging/tr1d1um_spruce.yaml

Co-authored-by: Joel Unzain <joe94@users.noreply.github.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* Update tr1d1um.yaml

Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

* documentation

* documentation

* changelog and gitignore, added cpuprofile

* changelog

Co-authored-by: Joel Unzain <joe94@users.noreply.github.com>
Co-authored-by: Kristina Pathak <kmspring57@gmail.com>

.gitignore

@@ -53,6 +53,7 @@ _testmain.go
 *.exe
 *.test
 *.prof
+cpuprofile
 
 tr1d1um
 .ignore

CHANGELOG.md

@@ -5,6 +5,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+## [v0.6.0]
+- Integrated webhook validator and added documentation and configuration for it. [#224](https://github.com/xmidt-org/tr1d1um/pull/224)
 - Bump bascule version which includes a security vulnerability fix. [#223](https://github.com/xmidt-org/tr1d1um/pull/223)
 
 ## [v0.5.10]
@@ -105,7 +108,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 - Initial creation.
 
-[Unreleased]: https://github.com/xmidt-org/tr1d1um/compare/v0.5.10...HEAD
+[Unreleased]: https://github.com/xmidt-org/tr1d1um/compare/v0.6.0...HEAD
+[v0.6.0]:https://github.com/xmidt-org/tr1d1um/compare/v0.5.10...v0.6.0
 [v0.5.10]: https://github.com/xmidt-org/tr1d1um/compare/v0.5.9...v0.5.10
 [v0.5.9]: https://github.com/xmidt-org/tr1d1um/compare/v0.5.8...v0.5.9
 [v0.5.8]: https://github.com/xmidt-org/tr1d1um/compare/v0.5.7...v0.5.8

deploy/packaging/tr1d1um_spruce.yaml

@@ -122,6 +122,69 @@ log:
 # webhookStore provides configuration for storing and obtaining webhook
 # information using argus.
 webhook:
+
+  # validation provides options for validating the webhook's URL and TTL 
+  # related fields. Some validation happens regardless of the configuration: 
+  # URLs must be a valid URL structure, the Matcher.DeviceID values must 
+  # compile into regular expressions, and the Events field must have at 
+  # least one value and all values must compile into regular expressions. 
+  validation:
+
+    # url provides options for additional validation of the webhook's 
+    # Config.URL, FailureURL, and Config.AlternativeURLs fields.
+    url:
+      # httpsOnly will allow only URLs with https schemes through if true.
+      # (Optional). Defaults to 'false'.
+      httpsOnly: false
+
+      # allowLoopback will allow any canonical or IP loopback address if 
+      # true. Otherwise, loopback addresses are considered invalid.
+      # (Optional). Defaults to 'false'.
+      allowLoopback: true
+
+      # allowIP allows the different webhook URLs to have IP hostnames if set to true.
+      # (Optional). Defaults to 'false'.
+      allowIP: true
+
+      # allowSpecialUseHosts allows URLs that include reserved domains if set to true.  
+      # Read more here: https://en.wikipedia.org/wiki/Top-level_domain#Reserved_domains
+      # (Optional). Defaults to 'false'.
+      allowSpecialUseHosts: true
+
+      # allowSpecialUseIPs, if set to true, allows URLs that contain or route to IPs that have 
+      # been marked as reserved through various RFCs: rfc6761, rfc6890, rfc8190.
+      # (Optional). Defaults to 'false'.
+      allowSpecialUseIPs: true
+
+      # invalidHosts is a slice that contains strings that we do not want 
+      # allowed in URLs, providing a way to deny certain domains or hostnames.
+      # (Optional). Defaults to an empty slice.
+      invalidHosts: []
+
+      # invalidSubnets is a list of IP subnets.  If a URL contains an 
+      # IP or resolves to an IP in one of these subnets, the webhook is 
+      # considered invalid.
+      # (Optional). Defaults to an empty slice.
+      invalidSubnets: []
+
+    # ttl provides information for what is considered valid for time-related 
+    # fields (Duration and Until) in the webhook.  A webhook set to expire 
+    # too far in the future is considered invalid, while a time in the past 
+    # is considered equivalent to a request to delete the webhook.  
+    # Regardless of this configuration, either Until or Duration must have a 
+    # non-zero value.
+    ttl:
+      # max is the length of time a webhook is allowed to live.  The Duration 
+      # cannot be larger than this value, and the Until value cannot be set 
+      # later than the current time + max + jitter.
+      max: 1m
+
+      # jitter is the buffer time added when checking that the Until value is 
+      # valid.  If there is a slight clock skew between servers or some delay 
+      # in the http request, jitter should help account for that when ensuring 
+      # that Until is not a time too far in the future.
+      jitter: 10s
+
   # JWTParserType establishes which parser type will be used by the JWT token
   # acquirer used by Argus. Options include 'simple' and 'raw'.
   # Simple: parser assumes token payloads have the following structure: https://github.com/xmidt-org/bascule/blob/c011b128d6b95fa8358228535c63d1945347adaa/acquire/bearer.go#L77

go.mod

@@ -12,7 +12,7 @@ require (
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.7.1
 	github.com/stretchr/testify v1.7.0
-	github.com/xmidt-org/ancla v0.2.1
+	github.com/xmidt-org/ancla v0.2.2-0.20210909164415-26b889e7d9ec
 	github.com/xmidt-org/bascule v0.10.2
 	github.com/xmidt-org/candlelight v0.0.5
 	github.com/xmidt-org/webpa-common/v2 v2.0.1

go.sum

@@ -673,6 +673,8 @@ github.com/vmware/govmomi v0.18.0/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59b
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/xmidt-org/ancla v0.2.1 h1:Pwvkfhj636rbAdZCZHyAbmdPct1jx/hK/QOFcPUeAFM=
 github.com/xmidt-org/ancla v0.2.1/go.mod h1:ouFTJFAgisn3DpLAdtkTbOOkYyprkO9ZbXqpAmin/Gg=
+github.com/xmidt-org/ancla v0.2.2-0.20210909164415-26b889e7d9ec h1:eG6mr5GhaKG+6d94EkGpvnd+jh6aHUwWupnmVTwdO/M=
+github.com/xmidt-org/ancla v0.2.2-0.20210909164415-26b889e7d9ec/go.mod h1:ouFTJFAgisn3DpLAdtkTbOOkYyprkO9ZbXqpAmin/Gg=
 github.com/xmidt-org/argus v0.3.9/go.mod h1:mDFS44R704gl9Fif3gkfAyvnZa53SvMepmXjYWABPvk=
 github.com/xmidt-org/argus v0.3.10-0.20201105190057-402fede05764/go.mod h1:lnMCVB/i0gOlUOOd2WbzDDgzTEqP5TipzQ8xKIw+N/I=
 github.com/xmidt-org/argus v0.3.10-0.20201217204602-66f69b12c498/go.mod h1:lnMCVB/i0gOlUOOd2WbzDDgzTEqP5TipzQ8xKIw+N/I=

main.go

@@ -181,7 +181,19 @@ func tr1d1um(arguments []string) (exitCode int) {
 		}
 		defer stopWatch()
 
-		addWebhookHandler := ancla.NewAddWebhookHandler(svc, ancla.HandlerConfig{MetricsProvider: metricsRegistry})
+		var webhookValidationConfig ancla.ValidatorConfig
+		err = v.UnmarshalKey(webhookConfigKey+".validation", &webhookValidationConfig)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to initialize webhook validation config: %s\n", err.Error())
+			return 1
+		}
+		builtValidators, err := ancla.BuildValidators(webhookValidationConfig)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to initialize webhook validators: %s\n", err.Error())
+			return 1
+		}
+
+		addWebhookHandler := ancla.NewAddWebhookHandler(svc, ancla.HandlerConfig{MetricsProvider: metricsRegistry, V: builtValidators})
 		getAllWebhooksHandler := ancla.NewGetAllWebhooksHandler(svc)
 
 		APIRouter.Handle("/hook", authenticate.Then(addWebhookHandler)).Methods(http.MethodPost)

tr1d1um.yaml

@@ -123,6 +123,69 @@ log:
 # information using Argus.
 # Optional: if key is not supplied, webhooks would be disabled.
 webhook:
+
+  # validation provides options for validating the webhook's URL and TTL 
+  # related fields. Some validation happens regardless of the configuration: 
+  # URLs must be a valid URL structure, the Matcher.DeviceID values must 
+  # compile into regular expressions, and the Events field must have at 
+  # least one value and all values must compile into regular expressions. 
+  validation:
+
+    # url provides options for additional validation of the webhook's 
+    # Config.URL, FailureURL, and Config.AlternativeURLs fields.
+    url:
+      # httpsOnly will allow only URLs with https schemes through if true.
+      # (Optional). Defaults to 'false'.
+      httpsOnly: false
+
+      # allowLoopback will allow any canonical or IP loopback address if 
+      # true. Otherwise, loopback addresses are considered invalid.
+      # (Optional). Defaults to 'false'.
+      allowLoopback: true
+
+      # allowIP allows the different webhook URLs to have IP hostnames if set to true.
+      # (Optional). Defaults to 'false'.
+      allowIP: true
+
+      # allowSpecialUseHosts allows URLs that include reserved domains if set to true.  
+      # Read more here: https://en.wikipedia.org/wiki/Top-level_domain#Reserved_domains
+      # (Optional). Defaults to 'false'.
+      allowSpecialUseHosts: true
+
+      # allowSpecialUseIPs, if set to true, allows URLs that contain or route to IPs that have 
+      # been marked as reserved through various RFCs: rfc6761, rfc6890, rfc8190.
+      # (Optional). Defaults to 'false'.
+      allowSpecialUseIPs: true
+
+      # invalidHosts is a slice that contains strings that we do not want 
+      # allowed in URLs, providing a way to deny certain domains or hostnames.
+      # (Optional). Defaults to an empty slice.
+      invalidHosts: []
+
+      # invalidSubnets is a list of IP subnets.  If a URL contains an 
+      # IP or resolves to an IP in one of these subnets, the webhook is 
+      # considered invalid.
+      # (Optional). Defaults to an empty slice.
+      invalidSubnets: []
+
+    # ttl provides information for what is considered valid for time-related 
+    # fields (Duration and Until) in the webhook.  A webhook set to expire 
+    # too far in the future is considered invalid, while a time in the past 
+    # is considered equivalent to a request to delete the webhook.  
+    # Regardless of this configuration, either Until or Duration must have a 
+    # non-zero value.
+    ttl:
+      # max is the length of time a webhook is allowed to live.  The Duration 
+      # cannot be larger than this value, and the Until value cannot be set 
+      # later than the current time + max + jitter.
+      max: 1m
+
+      # jitter is the buffer time added when checking that the Until value is 
+      # valid.  If there is a slight clock skew between servers or some delay 
+      # in the http request, jitter should help account for that when ensuring 
+      # that Until is not a time too far in the future.
+      jitter: 10s
+
   # JWTParserType establishes which parser type will be used by the JWT token
   # acquirer used by Argus. Options include 'simple' and 'raw'.
   # Simple: parser assumes token payloads have the following structure: https://github.com/xmidt-org/bascule/blob/c011b128d6b95fa8358228535c63d1945347adaa/acquire/bearer.go#L77