Skip to content

Commit

Permalink
Ethanperry/anomaly detection (#33)
Browse files Browse the repository at this point in the history
Introduces the following new tools:
1. EWMA Anomaly Detection Server
2. HTTP/GRPC Shift Calculator Server
3. SQL Callout Server

---------

Co-authored-by: Ethan <ethan@ethansstress.guest.corp.microsoft.com>
  • Loading branch information
ethanperry1 and Ethan authored Nov 17, 2023
1 parent 0e5ce14 commit c020e0e
Show file tree
Hide file tree
Showing 36 changed files with 1,707 additions and 90 deletions.
75 changes: 11 additions & 64 deletions lib/proto/message.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 1 addition & 4 deletions lib/proto/message.proto
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ message Message {
repeated Message array = 6;
}

message Empty {
}

service Sender {
rpc Send(Message) returns (Empty) {}
rpc Send(Message) returns (Message) {}
}
10 changes: 5 additions & 5 deletions lib/proto/message_grpc.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 21 additions & 0 deletions samples/anomaly-detection/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# syntax=docker/dockerfile:1

FROM mcr.microsoft.com/oss/go/microsoft/golang:1.21-cbl-mariner2.0 AS build

COPY ./lib /workdir/lib

COPY ./samples/anomaly-detection /workdir/samples/anomaly-detection

WORKDIR /workdir/samples/anomaly-detection

RUN go mod download

RUN go install github.com/magefile/mage@latest

RUN mage ci

RUN go build -o ./bin/anomaly-detection ./cmd/anomaly-detection

EXPOSE 2112

CMD [ "/bin/anomaly-detection" ]
133 changes: 133 additions & 0 deletions samples/anomaly-detection/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Anomaly Detection Server

## Usage

### Server as a Pod

```sh
# From the root of the anomaly-detection directory.
docker build ../.. -f Dockerfile -t <yourcrhere>.azurecr.io/anomaly-detection:latest

# Or if running from the root of the explore-iot-operations repository.
# docker build . -f ./samples/anomaly-detection/Dockerfile -t <yourcrhere>.azurecr.io/anomaly-detection:latest

# Push or load your newly built image into your cluster, depending on the k8s setup.
# docker push <yourcrhere>.azurecr.io/anomaly-detection:latest # Using AKS + Connected ACR
# minikube load <yourcrhere>.azurecr.io/anomaly-detection:latest # Using minikube
# docker save <yourcrhere>.azurecr.io/anomaly-detection:latest | k3s ctr images import - # Using K3s

kubectl run anomaly-detection --image=<yourcrhere>.azurecr.io/anomaly-detection:latest --stdin < config.yml
```

### Configuration

```yaml
logger: # Logger settings
level: 0
server: # Server settings
route: /anomaly
port: 3333
algorithm: # Algorithm settings
temperature: # Temperature related algorithm settings
lambda: 0.25 # Lambda value for temperature anomaly detection
lFactor: 3 # L factor for temperature anomaly detection
controlT: 90 # Control limit T constant
controlS: 20 # Control limit S constant
controlN: 10 # Control limit N constant
vibration: # Vibration related algorithm settings
lambda: 0.25
lFactor: 3
type: dynamic # Use dynamic algorithm, no requirement for control limit T, S, or N values
humidity: # Humidity related algorithm settings
lambda: 0.25
lFactor: 3
controlT: 80
controlS: 20
controlN: 10
```
## Algorithms
Use the `type: dynamic` property in the configuration to choose an algorithm from the following.

### Algorithm #1 - Control Limit Formula with Estimated Control Limits

$$B = L\frac{S}{\sqrt{n}}\sqrt{\frac{\lambda}{2 - \lambda}[1 - (1 - \lambda)^{2i}]}$$

$$B_u = T + B$$

$$B_l = T - B$$

<center>

| Symbol | Meaning |
|---|---|
| $B$ | Control Limit Value |
| $B_u$ | Upper Control Limit |
| $B_l$ | Lower Control Limit |
| $L$ | L Factor |
| $\lambda$ | Lambda Factor $(0 \leq \lambda \leq 1)$ |
| $i$ | Iteration or Count |
| $T$ | Constant Estimation of Sample Mean |
| $S$ | Constant Estimation of Sample Std Dev |
| $n$ | Constant Estimation of Rational Subgroup Size |

</center>

### Algorithm #2 - Control Limit Formula with Dynamically Evaluated Control Limits (`type: dynamic`)


$$B = L\sigma_i\sqrt{\frac{\lambda}{2 - \lambda}[1 - (1 - \lambda)^{2i}]}$$

$$B_u = \mu_i + B$$


$$B_l = \mu_i - B$$

<center>

| Symbol | Meaning |
|---|---|
| $B$ | Control Limit Value |
| $B_u$ | Upper Control Limit |
| $B_l$ | Lower Control Limit |
| $L$ | L Factor |
| $\lambda$ | Lambda Factor $(0 \leq \lambda \leq 1)$ |
| $i$ | Iteration or Count |
| $\mu_i$ | Sample Mean from Iteration $i$ |
| $\sigma_i$ | Sample Std Dev from Iteration $i$ |

</center>

### EWMA Value and Anomaly Detection Conditions

The following formulas are shared between the two algorithms.

__EWMA Value Formula__

$$z_{i} = \lambda x_i + (1 - \lambda)z_{i-1}$$

__Anomaly Condition__

$$
a=
\begin{cases}
0, & \text{if } B_l \leq z_i \leq B_u \\
1, & \text{otherwise}
\end{cases}
$$


<center>

| Symbol | Meaning |
|---|---|
| $B_u$ | Upper Control Limit |
| $B_l$ | Lower Control Limit |
| $\lambda$ | Lambda Factor $(0 \leq \lambda \leq 1)$ |
| $i$ | Iteration or Count |
| $x_i$ | Observed Value from Iteration $i$ |
| $z_i$ | EWMA Value for Iteration $i$ |
| $a$ | Anomaly Occurance, $1$ Implies an Anomaly |

</center>
24 changes: 24 additions & 0 deletions samples/anomaly-detection/cmd/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
logger:
level: 0
server:
route: /anomaly
port: 3333
algorithm:
temperature:
lambda: 0.25
lFactor: 3
controlT: 90
controlS: 20
controlN: 10
vibration:
lambda: 0.25
lFactor: 3
controlT: 50
controlS: 20
controlN: 10
humidity:
lambda: 0.25
lFactor: 3
controlT: 80
controlS: 20
controlN: 10
Loading

0 comments on commit c020e0e

Please sign in to comment.