Fix terminator dynamic cost failure crediting. Fixes #1382

CHANGELOG.md

@@ -4,8 +4,10 @@
 
 ## Component Updates and Bug Fixes
 
-* [Issue #1336](https://github.com/openziti/ziti/issues/1336) - `ziti edge quickstart` did
-  not create the usual edge router/service edge router policy.
+* github.com/openziti/ziti: [v0.30.4 -> v0.30.5](https://github.com/openziti/ziti/compare/v0.30.4...v0.30.5)
+    * [Issue #1336](https://github.com/openziti/ziti/issues/1336) - `ziti edge quickstart` did
+      not create the usual edge router/service edge router policy.
+    * [Issue #1382](https://github.com/openziti/ziti/issues/1382) - service failure costs are not shrinking over time
 
 # Release 0.30.4
 
@@ -83,104 +85,49 @@ to `connectTimeout`.
 
 ## Edge and Fabric Merges
 
-The code from the [fabric](https://github.com/openziti/fabric)
-and [edge](https://github.com/openziti/edge)
+The code from the [fabric](https://github.com/openziti/fabric) and [edge](https://github.com/openziti/edge)
 repositories has been merged into the ziti repository. Once issues have been migrated, those two
 projects will be archived.
 
 ## Component Updates and Bug Fixes
 
-*
-
-github.com/openziti/channel/v2: [v2.0.95 -> v2.0.99](https://github.com/openziti/channel/compare/v2.0.95...v2.0.99)
-
-*
-
-github.com/openziti/edge: [v0.24.404 -> v0.24.420](https://github.com/openziti/edge/compare/v0.24.404...v0.24.420)
-
-* [Issue #1610](https://github.com/openziti/edge/issues/1610) - Don't update identity on auth if
-  sdkInfo and envInfo haven't
-  changed * [Issue #1611](https://github.com/openziti/edge/issues/1611) - Some entity changes events
-  are unattributed when generated from edge client rest
-  endpoint * [Issue #1616](https://github.com/openziti/edge/issues/1616) - host.v1 and host.v2 are
-  out of sync * [Issue #1599](https://github.com/openziti/edge/issues/1599) - entityChange events
-  should mask or redact jwt token * [Issue #1607](https://github.com/openziti/edge/issues/1607) -
-  remove enrollment tokens `not before`
-* [Issue #1310](https://github.com/openziti/ziti/issues/1310) - router run --extend doesn't wait for
-  control channel
-
-*
-
-github.com/openziti/edge-api: [v0.25.33 -> v0.25.37](https://github.com/openziti/edge-api/compare/v0.25.33...v0.25.37)
-
-* [Issue #44](https://github.com/openziti/edge-api/issues/44) - create session can return 404 if
-  service id is invalid
-
-*
-
-github.com/openziti/fabric: [v0.24.23 -> v0.24.36](https://github.com/openziti/fabric/compare/v0.24.23...v0.24.36)
-
-* [Issue #799](https://github.com/openziti/fabric/issues/799) - Alll controllers to advertise
-  capabilities to routers/other
-  controllers * [Issue #796](https://github.com/openziti/fabric/issues/796) - Make link heartbeats
-  configurable, including an unresponive close
-  timeout * [Issue #794](https://github.com/openziti/fabric/issues/794) - Add output file to ziti
-  agent controller snapshot-db
-  command * [Issue #792](https://github.com/openziti/fabric/issues/792) - include raft index in DB
-  snapshot filename * [Issue #791](https://github.com/openziti/fabric/issues/791) - FieldError
-  Reason field not parsed correct when transmitted from cluster node to cluster
-  node * [Issue #789](https://github.com/openziti/fabric/issues/789) - Retransmitting prevents
-  circuit from being idle checked
-
-*
-
-github.com/openziti/foundation/v2: [v2.0.30 -> v2.0.32](https://github.com/openziti/foundation/compare/v2.0.30...v2.0.32)
-
-*
-
-github.com/openziti/identity: [v1.0.61 -> v1.0.63](https://github.com/openziti/identity/compare/v1.0.61...v1.0.63)
-
-*
-
-github.com/openziti/metrics: [v1.2.33 -> v1.2.35](https://github.com/openziti/metrics/compare/v1.2.33...v1.2.35)
-
-*
-
-github.com/openziti/runzmd: [v1.0.30 -> v1.0.32](https://github.com/openziti/runzmd/compare/v1.0.30...v1.0.32)
-
-*
-
-github.com/openziti/sdk-golang: [v0.20.101 -> v0.20.116](https://github.com/openziti/sdk-golang/compare/v0.20.101...v0.20.116)
-
-* [Issue #431](https://github.com/openziti/sdk-golang/issues/431) - sdk can panic when
-  re-authenticating and rebinding at the same
-  time * [Issue #238](https://github.com/openziti/sdk-golang/issues/238) - service binding needs to
-  restart if service is
-  recreated * [Issue #115](https://github.com/openziti/sdk-golang/issues/115) - Don't use hostname
-  for CN in CSR * [Issue #429](https://github.com/openziti/sdk-golang/issues/429) - error handling
-  is broken, so we don't properly retry
-
-*
-
-github.com/openziti/secretstream: [v0.1.11 -> v0.1.12](https://github.com/openziti/secretstream/compare/v0.1.11...v0.1.12)
-
-*
-
-github.com/openziti/storage: [v0.2.14 -> v0.2.18](https://github.com/openziti/storage/compare/v0.2.14...v0.2.18)
-
-* [Issue #31](https://github.com/openziti/storage/issues/31) - Fix ANTLR int overflow error on arm
-  32
-
-*
-
-github.com/openziti/transport/v2: [v2.0.103 -> v2.0.107](https://github.com/openziti/transport/compare/v2.0.103...v2.0.107)
-
-*
-
-github.com/openziti/ziti: [v0.30.3 -> v0.30.4](https://github.com/openziti/ziti/compare/v0.30.3...v0.30.4)
-
-* Fixed an issue causing router configs to be rewritten when docker compose was brought up with
-  existing configs
+* github.com/openziti/channel/v2: [v2.0.95 -> v2.0.99](https://github.com/openziti/channel/compare/v2.0.95...v2.0.99)
+* github.com/openziti/edge: [v0.24.404 -> v0.24.420](https://github.com/openziti/edge/compare/v0.24.404...v0.24.420)
+    * [Issue #1610](https://github.com/openziti/edge/issues/1610) - Don't update identity on auth if sdkInfo and envInfo haven't changed 
+    * [Issue #1611](https://github.com/openziti/edge/issues/1611) - Some entity changes events are unattributed when generated from edge client rest endpoint 
+    * [Issue #1616](https://github.com/openziti/edge/issues/1616) - host.v1 and host.v2 are out of sync 
+    * [Issue #1599](https://github.com/openziti/edge/issues/1599) - entityChange events should mask or redact jwt token 
+    * [Issue #1607](https://github.com/openziti/edge/issues/1607) - remove enrollment tokens `not before`
+    * [Issue #1310](https://github.com/openziti/ziti/issues/1310) - router run --extend doesn't wait for control channel
+
+* github.com/openziti/edge-api: [v0.25.33 -> v0.25.37](https://github.com/openziti/edge-api/compare/v0.25.33...v0.25.37)
+    * [Issue #44](https://github.com/openziti/edge-api/issues/44) - create session can return 404 if service id is invalid
+
+* github.com/openziti/fabric: [v0.24.23 -> v0.24.36](https://github.com/openziti/fabric/compare/v0.24.23...v0.24.36)
+    * [Issue #799](https://github.com/openziti/fabric/issues/799) - Alll controllers to advertise capabilities to routers/other controllers 
+    * [Issue #796](https://github.com/openziti/fabric/issues/796) - Make link heartbeats configurable, including an unresponive close timeout 
+    * [Issue #794](https://github.com/openziti/fabric/issues/794) - Add output file to ziti agent controller snapshot-db command 
+    * [Issue #792](https://github.com/openziti/fabric/issues/792) - include raft index in DB snapshot filename 
+    * [Issue #791](https://github.com/openziti/fabric/issues/791) - FieldError Reason field not parsed correct when transmitted from cluster node to cluster node 
+    * [Issue #789](https://github.com/openziti/fabric/issues/789) - Retransmitting prevents circuit from being idle checked
+
+* github.com/openziti/foundation/v2: [v2.0.30 -> v2.0.32](https://github.com/openziti/foundation/compare/v2.0.30...v2.0.32)
+* github.com/openziti/identity: [v1.0.61 -> v1.0.63](https://github.com/openziti/identity/compare/v1.0.61...v1.0.63)
+* github.com/openziti/metrics: [v1.2.33 -> v1.2.35](https://github.com/openziti/metrics/compare/v1.2.33...v1.2.35)
+* github.com/openziti/runzmd: [v1.0.30 -> v1.0.32](https://github.com/openziti/runzmd/compare/v1.0.30...v1.0.32)
+* github.com/openziti/sdk-golang: [v0.20.101 -> v0.20.116](https://github.com/openziti/sdk-golang/compare/v0.20.101...v0.20.116)
+    * [Issue #431](https://github.com/openziti/sdk-golang/issues/431) - sdk can panic when re-authenticating and rebinding at the same time 
+    * [Issue #238](https://github.com/openziti/sdk-golang/issues/238) - service binding needs to restart if service is recreated 
+    * [Issue #115](https://github.com/openziti/sdk-golang/issues/115) - Don't use hostname for CN in CSR 
+    * [Issue #429](https://github.com/openziti/sdk-golang/issues/429) - error handling is broken, so we don't properly retry
+
+* github.com/openziti/secretstream: [v0.1.11 -> v0.1.12](https://github.com/openziti/secretstream/compare/v0.1.11...v0.1.12)
+* github.com/openziti/storage: [v0.2.14 -> v0.2.18](https://github.com/openziti/storage/compare/v0.2.14...v0.2.18)
+    * [Issue #31](https://github.com/openziti/storage/issues/31) - Fix ANTLR int overflow error on arm 32
+
+* github.com/openziti/transport/v2: [v2.0.103 -> v2.0.107](https://github.com/openziti/transport/compare/v2.0.103...v2.0.107)
+* github.com/openziti/ziti: [v0.30.3 -> v0.30.4](https://github.com/openziti/ziti/compare/v0.30.3...v0.30.4)
+    * Fixed an issue causing router configs to be rewritten when docker compose was brought up with existing configs
 
 # Release 0.30.3
 

controller/xt/failure.go

@@ -28,7 +28,13 @@ func (self *failureCosts) CreditOverTime(credit uint8, period time.Duration) *ti
 	go func() {
 		for range ticker.C {
 			for val := range self.costMap.IterBuffered() {
-				self.successWithCredit(val.Key, uint16(credit))
+				actualCredit := self.successWithCredit(val.Key, uint16(credit))
+				GlobalCosts().UpdateDynamicCost(val.Key, func(u uint16) uint16 {
+					if u < actualCredit {
+						return 0
+					}
+					return u - actualCredit
+				})
 			}
 		}
 	}()

go.mod

@@ -66,7 +66,7 @@ require (
 	github.com/rabbitmq/amqp091-go v1.8.1
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
 	github.com/russross/blackfriday v1.6.0
-	github.com/shirou/gopsutil/v3 v3.23.8
+	github.com/shirou/gopsutil/v3 v3.23.9
 	github.com/sirupsen/logrus v1.9.3
 	github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
 	github.com/spf13/cobra v1.7.0

go.sum

@@ -747,8 +747,8 @@ github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQD
 github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
 github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
-github.com/shirou/gopsutil/v3 v3.23.8 h1:xnATPiybo6GgdRoC4YoGnxXZFRc3dqQTGi73oLvvBrE=
-github.com/shirou/gopsutil/v3 v3.23.8/go.mod h1:7hmCaBn+2ZwaZOr6jmPBZDfawwMGuo1id3C6aM8EDqQ=
+github.com/shirou/gopsutil/v3 v3.23.9 h1:ZI5bWVeu2ep4/DIxB4U9okeYJ7zp/QLTO4auRb/ty/E=
+github.com/shirou/gopsutil/v3 v3.23.9/go.mod h1:x/NWSb71eMcjFIO0vhyGW5nZ7oSIgVjrCnADckb85GA=
 github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
 github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
 github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=

zititest/go.mod

@@ -149,7 +149,7 @@ require (
 	github.com/rodaine/table v1.0.1 // indirect
 	github.com/rs/cors v1.9.0 // indirect
 	github.com/russross/blackfriday v1.6.0 // indirect
-	github.com/shirou/gopsutil/v3 v3.23.8 // indirect
+	github.com/shirou/gopsutil/v3 v3.23.9 // indirect
 	github.com/shoenig/go-m1cpu v0.1.6 // indirect
 	github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e // indirect
 	github.com/speps/go-hashids v2.0.0+incompatible // indirect

zititest/go.sum

@@ -761,8 +761,8 @@ github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQD
 github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
 github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
-github.com/shirou/gopsutil/v3 v3.23.8 h1:xnATPiybo6GgdRoC4YoGnxXZFRc3dqQTGi73oLvvBrE=
-github.com/shirou/gopsutil/v3 v3.23.8/go.mod h1:7hmCaBn+2ZwaZOr6jmPBZDfawwMGuo1id3C6aM8EDqQ=
+github.com/shirou/gopsutil/v3 v3.23.9 h1:ZI5bWVeu2ep4/DIxB4U9okeYJ7zp/QLTO4auRb/ty/E=
+github.com/shirou/gopsutil/v3 v3.23.9/go.mod h1:x/NWSb71eMcjFIO0vhyGW5nZ7oSIgVjrCnADckb85GA=
 github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
 github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
 github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=

zititest/tests/scp_test.go

@@ -108,7 +108,7 @@ func testScp(t *testing.T, hostSelector string, hostType string, encrypted bool)
 			fmt.Sprintf("scp -o StrictHostKeyChecking=no ./fablab/bin/ziti ssh-%s%s.ziti:/tmp/ziti-%s", hostType, nameExtra, uuid.NewString()),
 		}
 
-		o, err := lib.RemoteExecAllWithTimeout(sshConfigFactory, 10*time.Second, cmds...)
+		o, err := lib.RemoteExecAllWithTimeout(sshConfigFactory, 30*time.Second, cmds...)
 		if hostType == "zet" && err != nil {
 			t.Skipf("zet hosted ssh failed [%v]", err.Error())
 			return