added --include-status option

[hitenkoku]

What Changed

• added --include-status option

[hitenkoku]

Evidence

help

> ./1193.exe csv-timeline -h
Hayabusa v2.14.0 - Dev Build
Yamato Security (https://github.com/Yamato-Security/hayabusa - @SecurityYamato)

Usage:
  hayabusa.exe csv-timeline <INPUT> [OPTIONS]

Options:
  -h, --help  Print help

Input:
  -d, --directory <DIR>  Directory of multiple .evtx files
  -f, --file <FILE>      File path to one .evtx file
  -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
  -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
  -x, --recover-records  Carve evtx records from slack space (default: disabled)

Filtering:
  -E, --EID-filter                      Scan only common EIDs for faster speed (./rules/config/target_event_IDs.txt)
  -D, --enable-deprecated-rules         Enable rules with a status of deprecated
  -n, --enable-noisy-rules              Enable rules set to noisy (./rules/config/noisy_rules.txt)
  -u, --enable-unsupported-rules        Enable rules with a status of unsupported
  -e, --exact-level <LEVEL>             Only load rules with a specific level (informational, low, medium, high, critical)
      --exclude-category <CATEGORY...>  Do not load rules with specified logsource categories (ex: process_creation,pipe_created)
      --exclude-computer <COMPUTER...>  Do not scan specified computer names (ex: ComputerA) (ex: ComputerA,ComputerB)
      --exclude-eid <EID...>            Do not scan specific EIDs for faster speed (ex: 1) (ex: 1,4688)
      --exclude-status <STATUS...>      Do not load rules according to status (ex: experimental) (ex: stable,test)
      --exclude-tag <TAG...>            Do not load rules with specific tags (ex: sysmon)
      --include-category <CATEGORY...>  Only load rules with specified logsource categories (ex: process_creation,pipe_created)
      --include-computer <COMPUTER...>  Scan only specified computer names (ex: ComputerA) (ex: ComputerA,ComputerB)
      --include-eid <EID...>            Scan only specified EIDs for faster speed (ex: 1) (ex: 1,4688)
      --include-status <STATUS...>      Only load rules with specific status (ex: experimental) (ex: stable,test)
      --include-tag <TAG...>            Only load rules with specific tags (ex: attack.execution,attack.discovery)
  -m, --min-level <LEVEL>               Minimum level for rules to load (default: informational)
  -P, --proven-rules                    Scan with only proven rules for faster speed (./rules/config/proven_rules.txt)
      --timeline-end <DATE>             End time of the event logs to load (ex: "2022-02-22 23:59:59 +09:00")
      --timeline-offset <OFFSET>        Scan recent events based on an offset (ex: 1y, 3M, 30d, 24h, 30m)
      --timeline-start <DATE>           Start time of the event logs to load (ex: "2020-02-22 00:00:00 +09:00")

Output:
  -G, --GeoIP <MAXMIND-DB-DIR>       Add GeoIP (ASN, city, country) info to IP addresses
  -H, --HTML-report <FILE>           Save Results Summary details to an HTML report (ex: results.html)
  -M, --multiline                    Output event field information in multiple rows
  -F, --no-field-data-mapping        Disable field data mapping
      --no-pwsh-field-extraction     Disable field extraction of PowerShell classic logs
  -o, --output <FILE>                Save the timeline in CSV format (ex: results.csv)
  -p, --profile <PROFILE>            Specify output profile
  -R, --remove-duplicate-data        Duplicate field data will be replaced with "DUP"
  -X, --remove-duplicate-detections  Remove duplicate detections (default: disabled)

Display Settings:
      --no-color            Disable color output
  -N, --no-summary          Do not display Results Summary for faster speed
  -q, --quiet               Quiet mode: do not display the launch banner
  -v, --verbose             Output verbose information
  -T, --visualize-timeline  Output event frequency timeline (terminal needs to support unicode)

General Options:
  -C, --clobber                        Overwrite files when saving
  -h, --help                           Show the help menu
  -w, --no-wizard                      Do not ask questions. Scan for all events and alerts
  -Q, --quiet-errors                   Quiet errors mode: do not save error logs
  -r, --rules <DIR/FILE>               Specify a custom rule directory or file (default: ./rules)
  -c, --rules-config <DIR>             Specify custom rule config directory (default: ./rules/config)
      --target-file-ext <FILE-EXT...>  Specify additional evtx file extensions (ex: evtx_data)
  -t, --threads <NUMBER>               Number of threads (default: optimal number for performance)

Time Format:
      --European-time     Output timestamp in European time format (ex: 22-02-2022 22:00:00.123 +02:00)
      --ISO-8601          Output timestamp in ISO-8601 format (ex: 2022-02-22T10:10:10.1234567Z) (Always UTC)
      --RFC-2822          Output timestamp in RFC 2822 format (ex: Fri, 22 Feb 2022 22:00:00 -0600)
      --RFC-3339          Output timestamp in RFC 3339 format (ex: 2022-02-22 22:00:00.123456-06:00)
      --US-military-time  Output timestamp in US military time format (ex: 02-22-2022 22:00:00.123 -06:00)
      --US-time           Output timestamp in US time format (ex: 02-22-2022 10:00:00.123 PM -06:00)
  -U, --UTC               Output time in UTC format (default: local time)

Execute result

> ./1193.exe csv-timeline -d ..\hayabusa-sample-evtx\ --include-status experimental,stable  -w
...
Total event log files: 584
Total file size: 137.1 MB

Loading detection rules. Please wait.

Excluded rules: 2926
Noisy rules: 12 (Disabled)

Experimental rules: 1118 (82.33%)
Stable rules: 240 (17.67%)

Hayabusa rules: 135
Sigma rules: 1223
Total enabled detection rules: 1358

Output profile: standard
...

case: not found status

> ./1193.exe csv-timeline -d ..\hayabusa-sample-evtx\ --include-status notfound  -w           
...
Total event log files: 584
Total file size: 137.1 MB

Loading detection rules. Please wait.

Excluded rules: 4284
Noisy rules: 12 (Disabled)


Total enabled detection rules: 0

Output profile: standard

Scanning in progress. Please wait.

[ERROR] No rules were loaded. Please download the latest rules with the update-rules command.


Elapsed time: 00:00:00.332

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls

default

> ./1193.exe csv-timeline -d ..\hayabusa-sample-evtx\  -w                          
...
Total event log files: 584
Total file size: 137.1 MB

Loading detection rules. Please wait.

Excluded rules: 26
Noisy rules: 12 (Disabled)

Deprecated rules: 193 (4.80%) (Disabled)
Experimental rules: 1118 (27.81%)
Stable rules: 240 (5.97%)
Test rules: 2662 (66.22%)
Unsupported rules: 45 (1.12%) (Disabled)

Hayabusa rules: 161
Sigma rules: 3859
Total enabled detection rules: 4020

Output profile: standard
...

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (0fb8690) 81.49% compared to head (3687b02) 81.54%.

Files	Patch %	Lines
src/detections/configs.rs	92.10%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1280      +/-   ##
==========================================
+ Coverage   81.49%   81.54%   +0.04%     
==========================================
  Files          27       27              
  Lines       24220    24293      +73     
==========================================
+ Hits        19739    19809      +70     
- Misses       4481     4484       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fukusuket]

I confirmed that the number of items when test, stable, and experimental are specified in include-status matches the number of items when loaded by default :)

I also confirmed that integration-test was also successful. LGTM!!🚀
https://github.com/Yamato-Security/hayabusa/actions/runs/7902568990/job/21568517716

[YamatoSecurity]

@hitenkoku LGTM! Thanks so much!