OCSF Mappings for OpenSearch #202

Kevlw-AWS · 2024-12-27T01:36:36Z

Description

This PR adds schemas and and assets that customers can use with OpenSearch to ingest and analyse OCSF v1.1.0 logs. It includes mappings, visualisations, and index patterns.

Issues Resolved

No existing issues

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

YANG-DB · 2024-12-27T04:58:20Z

Thanks @Kevlw-AWS this looks great !!
since this is a rather large PR - I'd like to take the weekend + next week to review it
@Swiddis would you also be able to review ?
Thanks again for your work !

YANG-DB · 2024-12-27T04:59:10Z

@Kevlw-AWS can u plz fix the DCO ?
see how to fix this here

…oject#193) * add csv upload file integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update csv_file getting started docker content Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update fluent-bit index names getting started docker content Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update fluent-bit index names getting started docker content Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: Kevin Low <kevlw@amazon.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: Kevin Low <kevlw@amazon.com>

Signed-off-by: Kevin Low <kevlw@amazon.com>

…ated properly Signed-off-by: Kevin Low <kevlw@amazon.com>

remove AWS account ID information. this is a no no! Signed-off-by: Kevin Low <kevlw@amazon.com>

tweaks for the drop Signed-off-by: Kevin Low <kevlw@amazon.com>

Signed-off-by: Kevin Low <kevlw@amazon.com>

Bumps [org.json:json](https://github.com/douglascrockford/JSON-java) from 20210307 to 20231013. - [Release notes](https://github.com/douglascrockford/JSON-java/releases) - [Changelog](https://github.com/stleary/JSON-java/blob/master/docs/RELEASES.md) - [Commits](https://github.com/douglascrockford/JSON-java/commits) --- updated-dependencies: - dependency-name: org.json:json dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Kevin Low <kevlw@amazon.com>

Signed-off-by: Kevin Low <kevlw@amazon.com>

relative path changes Signed-off-by: Kevin Low <kevlw@amazon.com>

Signed-off-by: Kevin Low <kevlw@amazon.com>

Kevlw-AWS · 2024-12-27T15:52:44Z

Hi Yang, I’ve fixed this. Thanks! Kevin Low Security Solutions Architect Amazon Web Services From: YANGDB ***@***.***> Reply to: opensearch-project/opensearch-catalog ***@***.***> Date: Friday, 27 December 2024 at 12:59 PM To: opensearch-project/opensearch-catalog ***@***.***> Cc: "Low, Kevin" ***@***.***>, Mention ***@***.***> Subject: Re: [opensearch-project/opensearch-catalog] OCSF Mappings for OpenSearch (PR #202) @Kevlw-AWS<https://github.com/Kevlw-AWS> can u plz fix the DCO ? see how to fix this here<https://github.com/opensearch-project/opensearch-catalog/pull/202/checks?check_run_id=34901231954> — Reply to this email directly, view it on GitHub<#202 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASHNJBZM6YWUBR4WJSOZE4T2HTNDHAVCNFSM6AAAAABUH5MPM2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRTGMYTMNZVHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

YANG-DB

Hi
I've reviewed the content I could - the schema is something I've not yet dived into...
For a standard integration such as the following apache https

there are a few elements missing:

data : containing a sample.json documents to show the TryMe functionality
info: containing the specific integrations content and resources README.md files
- for example - each OCSF entity can have its own description here
ocsf.json integration metadata file (see apache's integration metadata file )
LMK if you need any help creating the OCSF.json integration metadata file

YANG-DB · 2024-12-29T21:31:43Z

.DS_Store

What is the purpose of this file ?

you can nuke that

this has been nuked

YANG-DB · 2024-12-29T21:32:00Z

integrations/.DS_Store

same here - what's the purpose of this file ?

this has been nuked

YANG-DB · 2024-12-29T21:33:31Z

integrations/security/OCSF_1_1_0/assets/os_init_IAM_auth.py

can you pls elaborate on the usage of this auth policy?

how is it activated

where is it called

who calls it

YANG-DB · 2024-12-29T21:34:30Z

integrations/security/OCSF_1_1_0/assets/os_init_basic_auth.py

can you pls elaborate on the usage of this auth policy?

how is it activated
where is it called
who calls it

YANG-DB · 2024-12-29T21:37:09Z

integrations/security/OCSF_1_1_0/readme.md

+    - OCSF 6007 - Scan Activity
+
+### OpenSearch Ingestion template
+The OpenSearch Ingestion template (`assets/OSI-pipeline.yaml`) provides a template you can use with an OpenSearch Ingestion pipeline to ingest OCSF data. 


please add more context of the actual pipeline workflow and the framework (data-preper) that operates it

add links to data-preper resources

@Kevlw-AWS , lets sync on building more detail into the readme.md. I figured we were pretty light in the explanation of this solution. i can build the arch diagrams and the explanation.

YANG-DB · 2024-12-29T21:42:30Z

integrations/security/OCSF_1_1_0/readme.md

+
+## Installation instructions
+1. Download the index and component template zip files. Upload it to an S3 bucket or save it to your local machine.
+2. Download the right initialization script based on how you would like to authenticate to OpenSearch (basic auth or AWS IAM). 


plz give an example bash here

% cp {initialization script} % edit {variables in the initialization script} % ./script.sh

one script covers all OCSF 1.1.0 standard...since this solution hinges on component templates, there are a good portion of those component templates that are shared across ALL categories...so for now, in this version, it goes as one complete package regardless if a set of class_uids are not used. If we segregate by class_uid, were screwed because each commit that has a shared component template would need to traverse the dependencies in other source check-insif those are misaligned, it breaks others. That would be something we could filter and de-select if the Integrations package was a bit more flexible than the tunnel vision approach it has now...one source, multiple impls on data source side, similar viziz

The script is a bit complex to have it in bash. I'll add clearer instructions on where to modify the variables and to run it in an IDE

ok cool
@Kevlw-AWS
I would also love to see this as a packaged docker compose solution that anyone can just run and see a live containerized example in its local env

YANG-DB · 2024-12-29T21:43:19Z

integrations/security/OCSF_1_1_0/readme.md

+2. Download the right initialization script based on how you would like to authenticate to OpenSearch (basic auth or AWS IAM). 
+3. Modify the variables in the initialization script. You will need to add your OpenSearch cluster endpoint, authentication information, and the location of the index and component templates.
+4. Run the initialization script. 
+5. Log in to the OpenSearch cluster and upload the OpenSearch objects in the **Saved Objects** screen under **Dashboards Management**.  


plz give some screenshots + url on how/where its done in the dashboard management dialog

@Kevlw-AWS - lets record this in Camtasia and ANI GIF the setup and drop as a gif is supported for readme.md

These have been added as screenshots -- I prefer them to Gifs as GIFS are often grainy and show the whole screen, making the text tiny

YANG-DB · 2024-12-29T21:44:31Z

integrations/security/OCSF_1_1_0/readme.md

+
+Component templates (`schemas/component_templates`) are reusable building blocks that contain mapping definitions. Component templates are used as part of index templates. 
+
+The current set of index and component templates are mapped to the OSCF 1.1.0 standard. The repository contains index templates for the following OCSF 1.1.0 categories and classes:


can you add a visual diagram of the different OCSF entities that are presented here ?
and also a link to the OCSF url resources?

we can hotlink this to the spec for each class_uid....beware it could break if the OCSF project reorgs the structure of the presentation on the current site

YANG-DB · 2024-12-29T21:48:17Z

integrations/security/OCSF_1_1_0/schemas/index_templates.zip

what is the content of this zip file ?
if it being used anywhere ?

something Kevin Low and I would like to see in the Integrations project is the ability to wire things not in the dashboard or core APIs. we should be able to wire any plugin endpoint to seed our needs. and have dependency and order adhered to. to really bring elucidation to this issue, i would need to write a 6 pager...kinda hoping I dont have to do that, but the writing is on the wall...things like the dependency of this to work based on a data prepper config...if that config is not set, this solution breaks...however, this solution for OCSF relies on the strict OCSF schema ties...we need static templates for data to actually ingest properly. right now the install script needs to upload a local zip and unwind it so it can call them to install into OpenSearch via template apis

@kffallis
lets use this OCSF integration use case to create the detailed Integration RFC for what we need in terms of functionality and capabilities so we could have a clear path for the integration framework evolution and release features

YANG-DB · 2024-12-29T21:48:52Z

integrations/security/OCSF_1_1_0/schemas/component_templates.zip

what is the content of this zip file ?
if it being used directly anywhere ?

see comments on all things zip above for index templates etc

kffallis · 2024-12-30T02:33:06Z

"I've reviewed the content I could - the schema is something I've not yet dived into..." you will need more than a week on this topic

Signed-off-by: Kevin Low <kevlw@amazon.com>

…ll not working Signed-off-by: Kevin Low <kevlw@amazon.com>

…ile. Integration package can be built and imported but not accessed Signed-off-by: Kevin Low <kevlw@amazon.com>

…e why. OCSF mappings still not working

YANG-DB · 2025-01-14T21:51:26Z

@Swiddis would u be able to review and comment ?
we also need some feedback to the issues ocsf need to be solved to would become fully usable:

[FEATURE]Add custom prefix to integration assets dashboards-observability#2302
[FEATURE]disable mandatory index selection on integration SetUp dashboards-observability#2301
[FEATURE]Allow Integration to disable TryMe button dashboards-observability#2300
Thanks !

Kevlw-AWS requested review from bowenlan-amzn, khushbr, praveensameneni, ananzh, Swiddis, AMoo-Miki and YANG-DB as code owners December 27, 2024 01:36

YANG-DB and others added 21 commits December 27, 2024 23:50

Update README.md (opensearch-project#187)

e35f378

Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: Kevin Low <kevlw@amazon.com>

first commit

12c65cd

Signed-off-by: Kevin Low <kevlw@amazon.com>

first commit

1c5514e

Signed-off-by: Kevin Low <kevlw@amazon.com>

aDded ocsf stuff

a59669d

Signed-off-by: Kevin Low <kevlw@amazon.com>

added init scripts and templates pointing to aliases

032f919

Signed-off-by: Kevin Low <kevlw@amazon.com>

fixed init scripts

4535940

Signed-off-by: Kevin Low <kevlw@amazon.com>

created init script

54a75c6

Signed-off-by: Kevin Low <kevlw@amazon.com>

modified init scriot

6bbea6a

Signed-off-by: Kevin Low <kevlw@amazon.com>

made some changes

163627d

Signed-off-by: Kevin Low <kevlw@amazon.com>

changed init script to put ISM policy first so that it will be associ…

b70bf5f

…ated properly Signed-off-by: Kevin Low <kevlw@amazon.com>

Update OSI-pipeline.yaml

49fa489

remove AWS account ID information. this is a no no! Signed-off-by: Kevin Low <kevlw@amazon.com>

minor changes for shard counts

bf95cd7

tweaks for the drop Signed-off-by: Kevin Low <kevlw@amazon.com>

added readme and json file for integration

adb9b10

Signed-off-by: Kevin Low <kevlw@amazon.com>

fixed init script

3aa9579

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated readme

111e1df

Signed-off-by: Kevin Low <kevlw@amazon.com>

added ndjson

5a9749b

Signed-off-by: Kevin Low <kevlw@amazon.com>

modified readme

2982600

Signed-off-by: Kevin Low <kevlw@amazon.com>

added the cfn scripts

184fc9d

Signed-off-by: Kevin Low <kevlw@amazon.com>

added zip files

0962304

Signed-off-by: Kevin Low <kevlw@amazon.com>

Kevlw-AWS and others added 16 commits December 27, 2024 23:50

added init script with basic auth

1072f5f

Signed-off-by: Kevin Low <kevlw@amazon.com>

cleaned up the scripts folder

bbf10da

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated OSI pipelines

0e39da1

Signed-off-by: Kevin Low <kevlw@amazon.com>

modified script instructions

7475b34

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated zip files

9386637

Signed-off-by: Kevin Low <kevlw@amazon.com>

added readme

0496d44

Signed-off-by: Kevin Low <kevlw@amazon.com>

added images

65bcd71

Signed-off-by: Kevin Low <kevlw@amazon.com>

modified images

f5b5a4b

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated readme

1043a69

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated readme

dbddd2e

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated readme

ff43e80

Signed-off-by: Kevin Low <kevlw@amazon.com>

fixed images

113f264

Signed-off-by: Kevin Low <kevlw@amazon.com>

added list of classes

4883c80

Signed-off-by: Kevin Low <kevlw@amazon.com>

Update readme.md

fe028b0

relative path changes Signed-off-by: Kevin Low <kevlw@amazon.com>

updated folder structure to match the rest of the integrations

6703d01

Signed-off-by: Kevin Low <kevlw@amazon.com>

removed DS_store

be0a7b1

Signed-off-by: Kevin Low <kevlw@amazon.com>

Kevlw-AWS force-pushed the security-lake branch from 6526f34 to be0a7b1 Compare December 27, 2024 15:51

YANG-DB requested changes Dec 29, 2024

View reviewed changes

Kevlw-AWS and others added 6 commits December 30, 2024 18:09

removed .DS_store files from repo

7128342

Signed-off-by: Kevin Low <kevlw@amazon.com>

updated readme with more verbose instructions around instlalation

0451f1e

Signed-off-by: Kevin Low <kevlw@amazon.com>

Merge branch 'opensearch-project:main' into security-lake

f349c37

made changes to conform to integration standard. Integration json sti…

fc4d68a

…ll not working Signed-off-by: Kevin Low <kevlw@amazon.com>

updated format to follow integrations schema. Added OCSF-1.1.0.json f…

000b33f

…ile. Integration package can be built and imported but not accessed Signed-off-by: Kevin Low <kevlw@amazon.com>

modified ocsf.json to work. Requires the mappings from Apache, no clu…

d487b82

…e why. OCSF mappings still not working

YANG-DB approved these changes Jan 14, 2025

View reviewed changes

Swiddis approved these changes Jan 14, 2025

View reviewed changes

YANG-DB merged commit 10872d0 into opensearch-project:main Jan 15, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCSF Mappings for OpenSearch #202

OCSF Mappings for OpenSearch #202

Kevlw-AWS commented Dec 27, 2024 •

edited

Loading

YANG-DB commented Dec 27, 2024

YANG-DB commented Dec 27, 2024

Kevlw-AWS commented Dec 27, 2024 via email

YANG-DB left a comment

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

Kevlw-AWS Dec 30, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

Kevlw-AWS Dec 30, 2024

YANG-DB Dec 29, 2024

YANG-DB Dec 29, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

Kevlw-AWS Dec 30, 2024

YANG-DB Dec 30, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

Kevlw-AWS Dec 30, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

YANG-DB Dec 30, 2024

YANG-DB Dec 29, 2024

kffallis Dec 30, 2024

kffallis commented Dec 30, 2024

YANG-DB commented Jan 14, 2025


		Component templates (`schemas/component_templates`) are reusable building blocks that contain mapping definitions. Component templates are used as part of index templates.

		The current set of index and component templates are mapped to the OSCF 1.1.0 standard. The repository contains index templates for the following OCSF 1.1.0 categories and classes:

OCSF Mappings for OpenSearch #202

OCSF Mappings for OpenSearch #202

Conversation

Kevlw-AWS commented Dec 27, 2024 • edited Loading

Description

Issues Resolved

YANG-DB commented Dec 27, 2024

YANG-DB commented Dec 27, 2024

Kevlw-AWS commented Dec 27, 2024 via email

YANG-DB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kffallis commented Dec 30, 2024

YANG-DB commented Jan 14, 2025

Kevlw-AWS commented Dec 27, 2024 •

edited

Loading