Add investigation guide checks

.github/workflows/investigation-guides.yml

@@ -0,0 +1,33 @@
+name: Investigation Guide Check
+
+on:
+  pull_request:
+    types: [opened, reopened, labeled, synchronize]
+    branches: [ "*" ]
+    paths:
+      - 'rules/**/*.toml'
+
+jobs:
+  investigation_guide_check:
+    runs-on: ubuntu-latest
+    if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip-guide') }}
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip cache purge
+        pip install .[dev]
+
+    - name: Run Investigation Guide Check
+      env:
+        GITHUB_TOKEN: "${{ secrets.PROTECTIONS_MACHINE_TOKEN }}"
+      run: |
+        python -m detection_rules dev check_investigation_guide --pr-number ${{ github.event.pull_request.number }}

detection_rules/devtools.py

@@ -235,6 +235,39 @@ def bump_versions(major_release: bool, minor_release: bool, patch_release: bool,
     save_etc_dump({"package": pkg_data}, "packages.yml")
 
 
+@dev_group.command('check-investigation-guide')
+@click.option("--token", required=True, prompt=get_github_token() is None, default=get_github_token(),
+              help="GitHub token to use for the PR", hide_input=True)
+@click.option("--pr-number", required=True, help="PR number to check", type=int)
+@click.pass_context
+def check_investigation_guide(ctx, token: str, pr_number: int):
+    """Check for rules in a GitHub PR to see if it has an investigation guide."""
+
+    failed = False
+    github = GithubClient(token)
+    client = github.authenticated_client
+    repository = client.get_repo("elastic/detection-rules")
+    pull_request = repository.get_pull(pr_number)
+    files = pull_request.get_files()
+
+    for file in files:
+        if file.filename.startswith('rules/') and file.filename.endswith('.toml'):
+            if file.filename.startswith('rules/_deprecated/'):
+                continue
+
+            rule = RuleCollection().load_file(Path(file.filename))
+            note = rule.contents.data.get('note')
+
+            if note is None or not re.search(rf'### Investigating\s+{re.escape(rule.name)}', note, re.I | re.M):
+                if not failed:
+                    click.echo("Missing investigation guide for:", err=True)
+
+                failed = True
+                click.echo(file.filename, err=True)
+
+    ctx.exit(int(failed))
+
+
 @dataclasses.dataclass
 class GitChangeEntry:
     status: str

rules/integrations/google_workspace/initial_access_google_workspace_object_copied_from_external_drive_and_access_granted_to_custom_application.toml

@@ -4,7 +4,7 @@ integration = ["google_workspace"]
 maturity = "production"
 min_stack_comments = "Breaking changes for Google Workspace integration."
 min_stack_version = "8.4.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -33,7 +33,7 @@ license = "Elastic License v2"
 name = "Google Workspace Object Copied from External Drive and Access Granted to Custom Application"
 note = """## Triage and analysis
 
-### Investigating Google Workspace Resource Copied from External Drive and Access Granted to Custom Application
+### Investigating Google Workspace Object Copied from External Drive and Access Granted to Custom Application
 
 Google Workspace users can share access to Drive objects such as documents, sheets, and forms via email delivery or a shared link. Shared link URIs have parameters like `view` or `edit` to indicate the recipient's permissions. The `copy` parameter allows the recipient to copy the object to their own Drive, which grants the object with the same privileges as the recipient. Specific objects in Google Drive allow container-bound scripts that run on Google's Apps Script platform. Container-bound scripts can contain malicious code that executes with the recipient's privileges if in their Drive.
 

rules/integrations/google_workspace/persistence_google_workspace_api_access_granted_via_domain_wide_delegation_of_authority.toml

@@ -4,7 +4,7 @@ integration = ["google_workspace"]
 maturity = "production"
 min_stack_comments = "Breaking changes for Google Workspace integration."
 min_stack_version = "8.4.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -27,7 +27,7 @@ license = "Elastic License v2"
 name = "Google Workspace API Access Granted via Domain-Wide Delegation of Authority"
 note = """## Triage and analysis
 
-### Investigating API Access Granted via Domain-Wide Delegation of Authority
+### Investigating Google Workspace API Access Granted via Domain-Wide Delegation of Authority
 
 Domain-wide delegation is a feature that allows apps to access users' data across an organization's Google Workspace environment. Only super admins can manage domain-wide delegation, and they must specify each API scope that the application can access. Google Workspace services all have APIs that can be interacted with after domain-wide delegation is established with an OAuth2 client ID of the application. Typically, GCP service accounts and applications are created where the Google Workspace APIs are enabled, thus allowing the application to access resources and services in Google Workspace.
 

rules/linux/command_and_control_connection_attempt_by_non_ssh_root_session.toml

@@ -4,7 +4,7 @@ integration = ["endpoint"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -25,7 +25,7 @@ language = "eql"
 license = "Elastic License v2"
 name = "Suspicious Network Connection Attempt by Root"
 note = """## Triage and analysis
-### Investigating Connection Attempt by Non-SSH Root Session
+### Investigating Suspicious Network Connection Attempt by Root
 Detection alerts from this rule indicate a strange or abnormal outbound connection attempt by a privileged process.  Here are some possible avenues of investigation:
 - Examine unusual and active sessions using commands such as 'last -a', 'netstat -a', and 'w -a'.
 - Analyze processes and command line arguments to detect anomalous process execution that may be acting as a listener.

rules/linux/execution_file_transfer_or_listener_established_via_netcat.toml

@@ -4,7 +4,7 @@ integration = ["endpoint"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -27,7 +27,7 @@ license = "Elastic License v2"
 name = "File Transfer or Listener Established via Netcat"
 note = """## Triage and analysis
 
-### Investigating Netcat Network Activity
+### Investigating File Transfer or Listener Established via Netcat
 
 Netcat is a dual-use command line tool that can be used for various purposes, such as port scanning, file transfers, and connection tests. Attackers can abuse its functionality for malicious purposes such creating bind shells or reverse shells to gain access to the target system.
 

rules/linux/execution_shell_evasion_linux_binary.toml

@@ -4,13 +4,13 @@ integration = ["endpoint"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
 description = """
-Identifies the abuse of a Linux binary to break out of a restricted shell or environment by spawning an interactive 
-system shell. The activity of spawning a shell from a binary is not common behavior for a user or system administrator, 
+Identifies the abuse of a Linux binary to break out of a restricted shell or environment by spawning an interactive
+system shell. The activity of spawning a shell from a binary is not common behavior for a user or system administrator,
 and may indicate an attempt to evade detection, increase capabilities or enhance the stability of an adversary.
 """
 from = "now-9m"
@@ -20,7 +20,7 @@ license = "Elastic License v2"
 name = "Linux Restricted Shell Breakout via Linux Binary(s)"
 note = """## Triage and analysis
 
-### Investigating Shell Evasion via Linux Utilities
+### Investigating Linux Restricted Shell Breakout via Linux Binary(s)
 Detection alerts from this rule indicate that a Linux utility has been abused to breakout of restricted shells or
 environments by spawning an interactive system shell.
 Here are some possible avenues of investigation:
@@ -105,10 +105,10 @@ process where host.os.type == "linux" and event.type == "start" and
 (
   /* launching shell from capsh */
   (process.name == "capsh" and process.args == "--") or
-  
+
   /* launching shells from unusual parents or parent+arg combos */
   (process.name : "*sh" and (
-    (process.parent.name : ("byebug", "ftp", "strace", "zip", "*awk", "git", "tar") and 
+    (process.parent.name : ("byebug", "ftp", "strace", "zip", "*awk", "git", "tar") and
     (
       process.parent.args : "BEGIN {system(*)}" or
       (process.parent.args : ("*PAGER*", "!*sh", "exec *sh") or process.args : ("*PAGER*", "!*sh", "exec *sh")) or
@@ -118,7 +118,7 @@ process where host.os.type == "linux" and event.type == "start" and
         )
       )
     ) or
-    
+
     /* shells specified in parent args */
     /* nice rule is broken in 8.2 */
     (process.parent.args : "*sh" and
@@ -136,7 +136,7 @@ process where host.os.type == "linux" and event.type == "start" and
     (process.parent.name == "crash" and process.parent.args == "-h") or
     (process.name == "sensible-pager" and process.parent.name in ("apt", "apt-get") and process.parent.args == "changelog")
     /* scope to include more sensible-pager invoked shells with different parent process to reduce noise and remove false positives */
-    
+
   )) or
   (process.name == "busybox" and event.action == "exec" and process.args_count == 2 and process.args : "*sh" and not process.executable : "/var/lib/docker/overlay2/*/merged/bin/busybox") or
   (process.name == "env" and process.args_count == 2 and process.args : "*sh") or

rules/linux/persistence_linux_user_added_to_privileged_group.toml

@@ -4,7 +4,7 @@ integration = ["endpoint"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/07/20"
+updated_date = "2023/08/02"
 
 [transform]
 [[transform.osquery]]
@@ -26,7 +26,7 @@ query = "SELECT pid, username, name FROM processes p JOIN users u ON u.uid = p.u
 [rule]
 author = ["Elastic"]
 description = """
-Identifies attempts to add a user to a privileged group. Attackers may add users to a privileged group in order to 
+Identifies attempts to add a user to a privileged group. Attackers may add users to a privileged group in order to
 establish persistence on a system.
 """
 from = "now-9m"
@@ -36,7 +36,7 @@ license = "Elastic License v2"
 name = "Linux User Added to Privileged Group"
 note = """## Triage and analysis
 
-### Investigating Linux User User Added to Privileged Group
+### Investigating Linux User Added to Privileged Group
 
 The `usermod`, `adduser`, and `gpasswd` commands can be used to assign user accounts to new groups in Linux-based operating systems.
 
@@ -93,8 +93,8 @@ process.args in ("root", "admin", "wheel", "staff", "sudo",
                              "disk", "video", "shadow", "lxc", "lxd") and
 (
   process.name in ("usermod", "adduser") or
-  process.name == "gpasswd" and 
-  process.args in ("-a", "--add", "-M", "--members") 
+  process.name == "gpasswd" and
+  process.args in ("-a", "--add", "-M", "--members")
 )
 '''
 

rules/ml/initial_access_ml_linux_anomalous_user_name.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2020/03/25"
 maturity = "production"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
 
@@ -31,7 +31,7 @@ machine_learning_job_id = ["v3_linux_anomalous_user_name"]
 name = "Unusual Linux Username"
 note = """## Triage and analysis
 
-### Investigating an Unusual Linux User
+### Investigating Unusual Linux Username
 Detection alerts from this rule indicate activity for a Linux user name that is rare and unusual. Here are some possible avenues of investigation:
 - Consider the user as identified by the username field. Is this program part of an expected workflow for the user who ran this program on this host? Could this be related to troubleshooting or debugging activity by a developer or site reliability engineer?
 - Examine the history of user activity. If this user only manifested recently, it might be a service account for a new software package. If it has a consistent cadence (for example if it runs monthly or quarterly), it might be part of a monthly or quarterly business process.

rules/ml/initial_access_ml_windows_anomalous_user_name.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2020/03/25"
 maturity = "production"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
 
@@ -31,7 +31,7 @@ machine_learning_job_id = ["v3_windows_anomalous_user_name"]
 name = "Unusual Windows Username"
 note = """## Triage and analysis
 
-### Investigating an Unusual Windows User
+### Investigating Unusual Windows Username
 Detection alerts from this rule indicate activity for a Windows user name that is rare and unusual. Here are some possible avenues of investigation:
 - Consider the user as identified by the username field. Is this program part of an expected workflow for the user who ran this program on this host? Could this be related to occasional troubleshooting or support activity?
 - Examine the history of user activity. If this user only manifested recently, it might be a service account for a new software package. If it has a consistent cadence (for example if it runs monthly or quarterly), it might be part of a monthly or quarterly business process.

rules/ml/initial_access_ml_windows_rare_user_type10_remote_login.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2020/03/25"
 maturity = "production"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
 
@@ -26,7 +26,7 @@ machine_learning_job_id = ["v3_windows_rare_user_type10_remote_login"]
 name = "Unusual Windows Remote User"
 note = """## Triage and analysis
 
-### Investigating an Unusual Windows User
+### Investigating Unusual Windows Remote User
 Detection alerts from this rule indicate activity for a rare and unusual Windows RDP (remote desktop) user. Here are some possible avenues of investigation:
 - Consider the user as identified by the username field. Is the user part of a group who normally logs into Windows hosts using RDP (remote desktop protocol)? Is this logon activity part of an expected workflow for the user?
 - Consider the source of the login. If the source is remote, could this be related to occasional troubleshooting or support activity by a vendor or an employee working remotely?"""

rules/ml/ml_linux_anomalous_network_activity.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2020/03/25"
 maturity = "production"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
 
@@ -22,7 +22,7 @@ machine_learning_job_id = ["v3_linux_anomalous_network_activity"]
 name = "Unusual Linux Network Activity"
 note = """## Triage and analysis
 
-### Investigating Unusual Network Activity
+### Investigating Unusual Linux Network Activity
 Detection alerts from this rule indicate the presence of network activity from a Linux process for which network activity is rare and unusual.  Here are some possible avenues of investigation:
 - Consider the IP addresses and ports. Are these used by normal but infrequent network workflows? Are they expected or unexpected?
 - If the destination IP address is remote or external, does it associate with an expected domain, organization or geography? Note: avoid interacting directly with suspected malicious IP addresses.

rules/ml/ml_windows_anomalous_network_activity.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2020/03/25"
 maturity = "production"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
 
@@ -23,7 +23,7 @@ machine_learning_job_id = ["v3_windows_anomalous_network_activity"]
 name = "Unusual Windows Network Activity"
 note = """## Triage and analysis
 
-### Investigating Unusual Network Activity
+### Investigating Unusual Windows Network Activity
 Detection alerts from this rule indicate the presence of network activity from a Windows process for which network activity is very unusual.  Here are some possible avenues of investigation:
 - Consider the IP addresses, protocol and ports. Are these used by normal but infrequent network workflows? Are they expected or unexpected?
 - If the destination IP address is remote or external, does it associate with an expected domain, organization or geography? Note: avoid interacting directly with suspected malicious IP addresses.

rules/windows/credential_access_mimikatz_powershell_module.toml

@@ -4,7 +4,7 @@ integration = ["windows"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -20,7 +20,7 @@ license = "Elastic License v2"
 name = "Potential Invoke-Mimikatz PowerShell Script"
 note = """## Triage and analysis
 
-### Investigating Mimikatz PowerShell Activity
+### Investigating Potential Invoke-Mimikatz PowerShell Script
 
 [Mimikatz](https://github.com/gentilkiwi/mimikatz) is an open-source tool used to collect, decrypt, and/or use cached credentials. This tool is commonly abused by adversaries during the post-compromise stage where adversaries have gained an initial foothold on an endpoint and are looking to elevate privileges and seek out additional authentication objects such as tokens/hashes/credentials that can then be used to move laterally and pivot across a network.
 

rules/windows/execution_suspicious_psexesvc.toml

@@ -4,7 +4,7 @@ integration = ["endpoint", "windows"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -19,7 +19,7 @@ license = "Elastic License v2"
 name = "Suspicious Process Execution via Renamed PsExec Executable"
 note = """## Triage and analysis
 
-### Investigating PsExec Network Connection
+### Investigating Suspicious Process Execution via Renamed PsExec Executable
 
 PsExec is a remote administration tool that enables the execution of commands with both regular and SYSTEM privileges on Windows systems. It operates by executing a service component `Psexecsvc` on a remote system, which then runs a specified process and returns the results to the local system. Microsoft develops PsExec as part of the Sysinternals Suite. Although commonly used by administrators, PsExec is frequently used by attackers to enable lateral movement and execute commands as SYSTEM to disable defenses and bypass security protections.
 

rules/windows/initial_access_unusual_dns_service_file_writes.toml

@@ -4,7 +4,7 @@ integration = ["endpoint", "windows"]
 maturity = "production"
 min_stack_comments = "New fields added: required_fields, related_integrations, setup"
 min_stack_version = "8.3.0"
-updated_date = "2023/06/22"
+updated_date = "2023/08/02"
 
 [rule]
 author = ["Elastic"]
@@ -19,7 +19,7 @@ license = "Elastic License v2"
 name = "Unusual File Modification by dns.exe"
 note = """## Triage and analysis
 
-### Investigating Unusual File Write
+### Investigating Unusual File Modification by dns.exe
 Detection alerts from this rule indicate potential unusual/abnormal file writes from the DNS Server service process (`dns.exe`) after exploitation from CVE-2020-1350 (SigRed) has occurred. Here are some possible avenues of investigation:
 - Post-exploitation, adversaries may write additional files or payloads to the system as additional discovery/exploitation/persistence mechanisms.
 - Any suspicious or abnormal files written from `dns.exe` should be reviewed and investigated with care.