reporter api #137

lucamrgs · 2024-05-01T09:57:58Z

Defines the initial API endpoints and route logic for reporting

MaartendeKruijf

Nice start on the api we need to refine a bit but looks good already

MaartendeKruijf · 2024-05-13T13:01:24Z

docs/content/en/docs/core-components/api-reporter.md

+
+```plantuml
+@startjson
+[


just return a list of all executions not only the id's, we could use the /reporter/meta to return the execution-id and playbook-id.

MaartendeKruijf · 2024-05-13T13:59:45Z

docs/content/en/docs/core-components/api-reporter.md

+##### Step execution data
+|field              |content                |type               | description |
+| ----------------- | --------------------- | ----------------- | ----------- |
+|execution_id       |UUID                   |string             |The id of the execution of the playbook where the step resides


Maybe this is redundant in combination with the execution_id of the response. Internally it's included during reporting so it can be matched with the correct execution

Took me forever but now it's done

MaartendeKruijf · 2024-05-13T14:02:48Z

internal/controller/controller.go

@@ -47,6 +49,8 @@ type Controller struct {

 var mainController = Controller{}

+var mainCache = cache.New(&timeUtil.Time{})


We need to discuss mutex for coroutines

MaartendeKruijf · 2024-05-13T14:04:23Z

internal/reporter/downstream_reporter/cache/cache.go

-	Cache        map[string]report.ExecutionEntry // Cached up to max
-	fifoRegister []string                         // Used for O(1) FIFO cache management
+	Cache        map[string]cache_report.ExecutionEntry // Cached up to max
+	fifoRegister []string                               // Used for O(1) FIFO cache management
 }

 func New(timeUtil itime.ITime) *Cache {
 	maxExecutions, _ := strconv.Atoi(utils.GetEnv("MAX_EXECUTIONS", strconv.Itoa(MaxExecutions)))


Remove this and intject it

MaartendeKruijf · 2024-05-13T14:05:10Z

models/api/reporter.go

+type Status uint8
+
+const (
+	SuccessfullyExecuted    = "successfully_executed"


Maybe include a link to the execution status repo in a comment

MaartendeKruijf · 2024-05-13T15:00:20Z

test/unittest/mocks/mock_reporter/mock_downstream_reporter.go

+func (reporter *Mock_Downstream_Reporter) GetExecutionsIds() []string {
+	args := reporter.Called()
+	return args.Get(0).([]string)
+}
+
+func (reporter *Mock_Downstream_Reporter) GetExecutionReport(executionKey uuid.UUID) (cache_report.ExecutionEntry, error) {
+	args := reporter.Called(executionKey)
+	return args.Get(0).(cache_report.ExecutionEntry), args.Error(1)
+}


These should be in a separate mock as they are not in the cache interface

MaartendeKruijf · 2024-05-13T15:01:17Z

test/unittest/reporters/downstream_reporter/cache_test.go

@@ -4,7 +4,7 @@ import (
 	"errors"
 	"soarca/internal/reporter/downstream_reporter/cache"
 	"soarca/models/cacao"
-	"soarca/models/report"
+	report "soarca/models/cache"


Here is a mismatch between models and naming

MaartendeKruijf · 2024-05-13T15:04:07Z

test/integration/api/reporter_api_test.go

Place this in the unittest folder as it's an unittest

MaartendeKruijf · 2024-05-13T15:07:50Z

docs/content/en/docs/core-components/api-reporter.md

+
+##### Step execution data
+|field              |content                |type               | description |
+| ----------------- | --------------------- | ----------------- | ----------- |


Automated execution field is missing here

Also see what other fields would make sense to include

MaartendeKruijf · 2024-05-13T15:58:04Z

docs/content/en/docs/core-components/api-reporter.md

+| successfully_executed | The workflow step was executed successfully (completed). |
+|failed| The workflow step failed. |
+|ongoing| The workflow step is in progress. |
+|server_side_error| A server-side error occurred. |


I've sugested await_user_input cyentific-rni/workflow-status#3 if that goes true we need to add that here

MaartendeKruijf · 2024-05-14T08:29:47Z

docs/content/en/docs/core-components/api-reporter.md

+
+##### Step execution data
+|field              |content                |type               | description |
+| ----------------- | --------------------- | ----------------- | ----------- |


Also see what other fields would make sense to include

MaartendeKruijf · 2024-05-15T07:27:46Z

models/api/reporter.go

+	ExecutionId        string
+	StepId             string
+	Started            string
+	Ended              string


MaartendeKruijf · 2024-05-15T10:12:30Z

test/unittest/routes/reporter_api/reporter_api_test.go

+			Ended:       expectedEnded,
+			StepResults: map[string]cache_model.StepResult{},
+			Error:       nil,
+			Status:      2,


Status 2 is an enum right?

lucamrgs linked an issue May 1, 2024 that may be closed by this pull request

Reporter API definition #136

Closed

lucamrgs changed the title ~~add first reporter api~~ reporter api May 2, 2024

lucamrgs force-pushed the feature/136-reporter-api-definition branch from c5244cc to 267d3d5 Compare May 7, 2024 12:19

lucamrgs marked this pull request as ready for review May 7, 2024 12:44

lucamrgs requested a review from MaartendeKruijf May 7, 2024 12:44

RabbITCybErSeC force-pushed the feature/136-reporter-api-definition branch from 3364acd to df53e97 Compare May 8, 2024 12:23

MaartendeKruijf requested changes May 13, 2024

View reviewed changes

MaartendeKruijf mentioned this pull request May 13, 2024

Feature/add manual sugestion workflow status cyentific-rni/workflow-status#3

Merged

MaartendeKruijf force-pushed the feature/136-reporter-api-definition branch from df53e97 to 6121451 Compare May 14, 2024 08:23

MaartendeKruijf requested changes May 14, 2024

View reviewed changes

lucamrgs added 20 commits May 15, 2024 11:57

add first reporter api documentation

233191e

create api reporter instantiation and links. must test

0baa488

add model for api response

0c8f3b3

add parsing from cache entry to reporter api results

b2c9373

enroll soarca-level cache object in both api and reporting

607dcb8

add template report api testing

de79172

rename report model to cache model

73467f8

add get executions base test

4b0a21a

add reporter api testing

35ad402

split unittest and integration test on reporter api

1610440

trying to solve lint error cannot parse response comment

efe5afd

solving lint error due to scheme specification

df5e1e2

remove schemes specification

6d25758

remove scheme from success specification

9ef4e67

add right scheme to response

b650e83

fix lint error check in controller and recorder body bytes in tests

de82654

change reporter documentation according to review

824b88c

change get executions to return full executions information

7f58da3

fix commented test check

9e3bbc0

remove unused function for cache

b27bd8b

lucamrgs added 18 commits May 15, 2024 11:57

inject cache size from controller

e8266ed

add comment ref to cyentific workflow status repo in reporter api model

b14a27f

move reporter api init setup to reporter api file

3d38c23

use error soarca package in reporter api

bda81e2

change mustparse to parse in reporter api

b92772a

change default request interval in reporter api to const var

6af48b8

fix playbook error var use in reporter api

b89e7c3

change uuid parse to mustparse in cache tests

c2da441

separate mock cache from mock downstram reporter

bc7aa3a

add file mock cache

bb688c1

remove unused fcns from cache mock

0d16403

move report api dynamic tests to unittest

fee2f3a

align reporter api model with cyentific workflow status model

e0e5f33

implement status text

c3cfbcf

fix lint error check

8d6197e

add command types to cacao model

867f444

change reporter api resutls model

841c51a

remove deep copy of cache entries now passed by reference to api

9990771

lucamrgs force-pushed the feature/136-reporter-api-definition branch from acc6ba6 to 9990771 Compare May 15, 2024 09:58

fix lint bool comparison complaint

4c0c736

MaartendeKruijf previously approved these changes May 15, 2024

View reviewed changes

change reporting status int to status type in tests

98b95c1

lucamrgs dismissed MaartendeKruijf’s stale review via 98b95c1 May 15, 2024 10:27

MaartendeKruijf approved these changes May 15, 2024

View reviewed changes

MaartendeKruijf merged commit 332d515 into development May 15, 2024
9 checks passed

MaartendeKruijf deleted the feature/136-reporter-api-definition branch May 15, 2024 11:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reporter api #137

reporter api #137

lucamrgs commented May 1, 2024

MaartendeKruijf left a comment

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

lucamrgs May 14, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 14, 2024

MaartendeKruijf May 13, 2024

MaartendeKruijf May 14, 2024

MaartendeKruijf May 15, 2024

MaartendeKruijf May 15, 2024

		@@ -47,6 +49,8 @@ type Controller struct {

		var mainController = Controller{}

		var mainCache = cache.New(&timeUtil.Time{})

reporter api #137

reporter api #137

Conversation

lucamrgs commented May 1, 2024

MaartendeKruijf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment