Add security events to SIEM targets

[gberche-orange]

Area(s)

area:log

Is your change request related to a problem? Please describe.

Security information and event management (SIEM) commonly use CEF (Common Event Format) or LEEF (Log Event Extended Format) for formatting security related events.

The CEF specifications are available at https://raffy.ch/blog/wp-content/uploads/2007/06/CEF.pdf

Some introduction material at https://www.splunk.com/en_us/blog/learn/common-event-format-cef.html

Why is CEF important?

CEF is important because it provides a standardized format for logging security-related events, which makes it easier to integrate logs from different sources into a single system. This can be particularly useful for security information and event management (SIEM) solutions, which are designed to collect and analyze logs from multiple sources to detect and respond to security threats.

The LEEF specifications are available at https://www.ibm.com/docs/en/dsm?topic=leef-overview

Examples of events leveraging CEF or LEEF to target SIEM:

• https://learn.microsoft.com/en-us/defender-for-identity/cef-format-sa

  Defender for Identity can forward security alert and health alert events to your SIEM. Alerts and events are in the CEF format. This reference article provides samples of the logs sent to your SIEM.

• https://support.kaspersky.com/KSC/14.2/en-US/151345.htm

You can use the CEF and LEEF formats to export to SIEM systems general events,
The table below shows SIEM systems and the corresponding
formats of export.

SIEM system	Format of export
QRadar	LEEF
ArcSight	CEF
Splunk	CEF

LEEF (Log Event Extended Format)—A customized event format for IBM Security QRadar SIEM. QRadar can integrate, identify, and process LEEF events. LEEF events must use UTF-8 character encoding. You can find detailed information on LEEF protocol in IBM Knowledge Center.
CEF (Common Event Format)—An open log management standard that improves the interoperability of security-related information from different security and network devices and applications. CEF enables you to use a common event log format so that data can easily be integrated and aggregated for analysis by an enterprise management system. CEF events must use UTF-8 character encoding.

• CloudFoundry security logs include fields described by the CEF (Common Event Format) , see https://docs.cloudfoundry.org/loggregator/cc-uaa-logging.html#format

CEF fields used by cloudfoundry security event logs

Cloud Controller logs security events in the Common Event Format (CEF). CEF specifies the following format for log entries:

CEF:Version|Device Vendor|Device Product|Device Version|Signature ID|Name|Severity|Extension

Entries in the Cloud Controller log use the following format:

CEF:CEF_VERSION|cloud_foundry|cloud_controller_ng|CC_API_VERSION|
SIGNATURE_ID|NAME|SEVERITY|rt=TIMESTAMP suser=USERNAME suid=USER_GUID
cs1Label=userAuthenticationMechanism cs1=AUTH_MECHANISM
cs2Label=vcapRequestId cs2=VCAP_REQUEST_ID request=REQUEST
requestMethod=REQUEST_METHOD cs3Label=result cs3=RESULT
cs4Label=httpStatusCode cs4=HTTP_STATUS_CODE src=SOURCE_ADDRESS
dst=DESTINATION_ADDRESS cs5Label=xForwardedFor cs5=X_FORWARDED_FOR_HEADER

See the following list for a description of the properties shown in the Cloud Controller log format:

• CEF_VERSION: The version of CEF used in the logs.
• CC_API_VERSION: The current Cloud Controller API version.
• SIGNATURE_ID: The method and path of the request. For example, GET /v2/app:GUID.
• NAME: The same as SIGNATURE_ID.
• SEVERITY: An integer that reflects the importance of the event.
• TIMESTAMP: The number of milliseconds since the Unix epoch.
• USERNAME: The name of the user who originated the request.
• USER_GUID: The GUID of the user who originated the request.
• AUTH_MECHANISM: The user authentication mechanism. This can be oauth-access-token, basic-auth, or no-auth.
• VCAP_REQUEST_ID: The VCAP request ID of the request.
• REQUEST: The request path and parameters. For example, /v2/info?MY-PARAM=VALUE.
• REQUEST_METHOD. The method of the request. For example, GET.
• RESULT: The meaning of the HTTP status code of the response. For example, success.
• HTTP_STATUS_CODE. The HTTP status code of the response. For example, 200.
• SOURCE_ADDRESS: The IP address of the client who originated the request.
• DESTINATION_ADDRESS: The IP address of the Cloud Controller VM.
• X_FORWARDED_FOR_HEADER: The contents of the X-Forwarded-For
  header of the request. This is empty if the header is not present.

Examples of log entries

The following list provides several example requests with the corresponding Cloud Controller log entries.

• An anonymous GET request:

  CEF:0|cloud_foundry|cloud_controller_ng|2.54.0|GET /v2/info|GET
  /v2/info|0|rt=1460690037402 suser= suid= request=/v2/info
  requestMethod=GET src=127.0.0.1 dst=<%=vars.example_ip_1%>
  cs1Label=userAuthenticationMechanism cs1=no-auth cs2Label=vcapRequestId
  cs2=c4bac383-7cc9-4d9f-b1c0-1iq8c0baa000 cs3Label=result cs3=success
  cs4Label=httpStatusCode cs4=200 cs5Label=xForwardedFor
  cs5=<%=vars.example_ip_2%>
  

/CC @KarstenSchnitter

Security related CEF fields are not currently described in https://github.com/open-telemetry/semantic-conventions/blob/96f8bda9bab363cb01e2441820bc83a5dad15801/docs/attributes-registry/log.md nor in https://github.com/open-telemetry/semantic-conventions/blob/96f8bda9bab363cb01e2441820bc83a5dad15801/docs/attributes-registry/cloudfoundry.md

Describe the solution you'd like

New security event registry fields selecting relevant security fields from CEF and LEEF

Possibly cloudfoundry-specific extensions to CEF

Describe alternatives you've considered

No response

Additional context

No response

[KarstenSchnitter]

Thank you for providing this issue. I think this topic requires a thorough discussion.

I have access to the CEF logs of an entire Cloud Foundry installation and can provide additional examples from other components as well. The example you chose is essentially an access logs. There are existing attributes, e.g., in http.md. Note, that vcapRequestId and xForwardedFor are essentially HTTP headers x-vcap-request-id and x-forwarded-for.

Let me provide a different example to show the diversity of these logs. This one is from the vcap.agent running on all bosh vms, that comprise a Cloud Foundry installation. It is a DNS sync event for the Bosh deployment manager:

2024/10/09 20:06:49 CEF:0|CloudFoundry|BOSH|1|agent_api|sync_dns_with_signed_url|1|duser=director.32a3e9df-257d-4590-a6a7-2b5686d3cb2d.5ec737a2-28fb-4335-87f9-c30c314681b8.518a6cd0-9437-46cf-b5c8-d78143404c9f src=bosh.hcp-staging.internal spt=4222 shost=5ec737a2-28fb-4335-87f9-c30c314681b8

This event looks very different from your example. It has a different tag (CloudFoundry vs cloud_foundry) and very different extensions. Comparing just those two events, one might think, that the third entry cloud_controller_ng and bosh might be a service.name and the next field a service.version. But I would argue, that cloud_controller_ng and bosh are more likely a service.namespace. This becomes clearer from the syslog headers, that are coming with this message:

<7>1 2024-10-09T20:06:49.016608+00:00 10.0.232.2 vcap.agent 5679 - [instance@47450 director="bosh-hcp-staging" deployment="application-logging" group="kafka" az="z3" id="f68fe289-0841-4ac7-b43d-29d645b55801"] 2024/10/09 20:06:49 CEF:0|CloudFoundry|BOSH|1|agent_api|sync_dns_with_signed_url|1|duser=director.32a3e9df-257d-4590-a6a7-2b5686d3cb2d.5ec737a2-28fb-4335-87f9-c30c314681b8.518a6cd0-9437-46cf-b5c8-d78143404c9f src=bosh.hcp-staging.internal spt=4222 shost=5ec737a2-28fb-4335-87f9-c30c314681b8

Following the cloudfoundry resource attributes, we would map vcap.agent to attribute cloudfoundry.system.id and f68fe289-0841-4ac7-b43d-29d645b55801 to cloudfoundry.system.instance.id. We can use those fields fields as service.name and service.instance.id. But we already know, that the service actually is bosh-dns. This is an example, where we might want to diverge and use this as service.name instead.

I think, from the example above it already becomes clear, that there should be a general decision first, whether SIEM formats should always use there own attribute sets or utilize existing attributes. From a brief look on the specifications it seems to me, that a mixture of both might be useful. For OpenTelemetry pipelines it is necessary to understand, that these SIEM events exist not on there own but have additional context as shown by the syslog header above. They will be mapped to the appropriate attributes independent of the event content.