added prompt that the user which sigma rules they want to enable by default

[hitenkoku]

What Changed

• added prompt that the user which sigma rules they want to enable by default.
• added -a, --no-asking option

I would appreciate it if you could review when you have time.

[codecov]

Codecov Report

Attention: 85 lines in your changes are missing coverage. Please review.

Comparison is base (da093e4) 83.76% compared to head (a2971d3) 83.50%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1191      +/-   ##
==========================================
- Coverage   83.76%   83.50%   -0.26%     
==========================================
  Files          26       26              
  Lines       23463    23696     +233     
==========================================
+ Hits        19653    19788     +135     
- Misses       3810     3908      +98     

Files	Coverage Δ
src/detections/detection.rs	75.66% <100.00%> (+0.06%)	⬆️
src/detections/rule/condition_parser.rs	97.01% <100.00%> (+0.01%)	⬆️
src/detections/rule/count.rs	93.63% <100.00%> (+<0.01%)	⬆️
src/detections/rule/matchers.rs	97.08% <100.00%> (+<0.01%)	⬆️
src/detections/rule/mod.rs	94.78% <100.00%> (+0.04%)	⬆️
src/detections/rule/selectionnodes.rs	92.32% <100.00%> (+0.01%)	⬆️
src/detections/utils.rs	92.84% <100.00%> (+0.04%)	⬆️
src/options/htmlreport.rs	98.65% <100.00%> (+0.01%)	⬆️
src/options/profile.rs	85.07% <100.00%> (+0.08%)	⬆️
src/yaml.rs	88.44% <100.00%> (+0.32%)	⬆️
... and 3 more

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fukusuket]

@hitenkoku
When Core/Core+/Core++/All alert rules is selected by default as shown below, it seems that med, low, and info levels are also detected. Could you please confirm?

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -o t-new-core.csv -C -q
Start time: 2023/10/19 00:20

Total event log files: 583
Total file size: 137.1 MB

Detection rule sets:

✔ Which set of detection rules would you like to load? · 1. Core ( status: testing, stable | level: high, critical )
✔ Include Emerging Threats rules?(Y/n): · yes
✔ Include Threat Hunting rules?(y/N): · no

Loading detection rules. Please wait.

Excluded rules: 197
Noisy rules: 12 (Disabled)

Deprecated rules: 182 (7.64%) (Disabled)
Experimental rules: 905 (38.01%)
Stable rules: 179 (7.52%)
Test rules: 1297 (54.47%)
Unsupported rules: 45 (1.89%) (Disabled)

Hayabusa rules: 159
Sigma rules: 2222
Total enabled detection rules: 2381

Output profile: standard

Scanning in progress. Please wait.

[00:00:04] 583 / 583   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
...
Results Summary:

Events with hits / Total events: 19,650 / 47,465 (Data reduction: 27,815 events (58.60%))

Total | Unique detections: 32,231 | 566
Total | Unique critical detections: 53 (0.16%) | 19 (3.36%)
Total | Unique high detections: 5,936 (18.42%) | 236 (41.70%)
Total | Unique medium detections: 1,970 (6.11%) | 194 (34.28%)
Total | Unique low detections: 6,044 (18.75%) | 62 (10.95%)
Total | Unique informational detections: 18,228 (56.55%) | 55 (9.72%)

Dates with most total detections:
critical: 2019-07-19 (12), high: 2016-09-20 (3,652), medium: 2019-05-19 (167), low: 2016-09-20 (3,708), informational: 2016-08-19 (2,104)

Top 5 computers with most unique detections:
critical: MSEDGEWIN10 (7), IEWIN7 (3), FS03.offsec.lan (2), IE10Win7 (2), rootdc1.offsec.lan (2)
high: MSEDGEWIN10 (116), IEWIN7 (69), fs03vuln.offsec.lan (26), FS03.offsec.lan (26), IE10Win7 (23)
medium: MSEDGEWIN10 (82), IEWIN7 (49), FS03.offsec.lan (23), fs03vuln.offsec.lan (20), rootdc1.offsec.lan (16)
low: MSEDGEWIN10 (37), FS03.offsec.lan (19), IEWIN7 (17), fs03vuln.offsec.lan (17), fs01.offsec.lan (11)
informational: IEWIN7 (17), MSEDGEWIN10 (17), PC01.example.corp (15), fs01.offsec.lan (15), FS03.offsec.lan (13)

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                           Top high alerts:                                 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Sticky Key Like Backdoor Usage - Registry (8)                  Metasploit SMB Authentication (3,562)            │
│ CobaltStrike Service Installations - System (6)                Malicious Svc Possibly Installed (271)           │
│ Active Directory Replication from Non Machine Account (6)      Susp Svc Installed (257)                         │
│ Meterpreter or Cobalt Strike Getsystem Service Instal... (6)   Suspicious Service Installation Script (250)     │
│ Defender Alert (Severe) (4)                                    PowerShell Scripts Installed as Services (250)   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                             Top low alerts:                                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Potentially Malicious PwSh (235)                               Logon Failure (Wrong Password) (3,564)           │
│ Proc Injection (104)                                           Susp CmdLine (Possible LOLBIN) (1,418)           │
│ Reg Key Value Set (Sysmon Alert) (103)                         Non Interactive PowerShell Process Spawned (325) │
│ Remote Thread Creation Via PowerShell (93)                     Proc Access (156)                                │
│ Log File Cleared (87)                                          DLL Loaded (Sysmon Alert) (108)                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Proc Exec (11,173)                                             Svc Installed (331)                              │
│ NetShare File Access (2,564)                                   Explicit Logon (304)                             │
│ PwSh Scriptblock (789)                                         New Non-USB PnP Device (268)                     │
│ PwSh Pipeline Exec (680)                                       Net Conn (243)                                   │
│ NetShare Access (433)                                          File Created (210)                               │
╰──────────────────────────────────────────────────────────────╌──────────────────────────────────────────────────╯

Saved file: t-new-core.csv (31.4 MB)

Elapsed time: 00:00:10.164

[YamatoSecurity]

@hitenkoku The prompt looks great! Better than what I was imagining.

One small thing: When a user chooses 1-3 there is a space before Loading detection rules:
Ex:

✔ Which set of detection rules would you like to load? · 1. Core ( status: testing, stable | level: high, critical )
✔ Include Emerging Threats rules?(Y/n): · yes
✔ Include Threat Hunting rules?(y/N): · yes

Loading detection rules. Please wait.

But if the user chooses 4 or 5, then there is no space:

✔ Which set of detection rules would you like to load? · 4. All alert rules ( status: * | level: low+ )
Loading detection rules. Please wait.

Can you add a space like this:

✔ Which set of detection rules would you like to load? · 4. All alert rules ( status: * | level: low+ )

Loading detection rules. Please wait.

and

✔ Which set of detection rules would you like to load? · 5. All event and alert rules ( status: * | level: informational+ )

Loading detection rules. Please wait.

[hitenkoku]

@fukusuket Thanks for your check.
I forgot adding level processing. I fixed following problem in 2a78cd6 .

Could you check it?

@hitenkoku When Core/Core+/Core++/All alert rules is selected by default as shown below, it seems that med, low, and info levels are also detected. Could you please confirm?

• check result in my env

> ./1188.exe csv-timeline -d ..\hayabusa-sample-evtx\ -o 1188.csv -C
...
Total event log files: 584
Total file size: 137.1 MB

Detection rule sets:

✔ Which set of detection rules would you like to load? · 2. Core+ ( status: testing, stable | level: medium, high, critical )
✔ Include Emerging Threats rules?(Y/n): · no
✔ Include Threat Hunting rules?(y/N): · no

Loading detection rules. Please wait.

Excluded rules: 228
Noisy rules: 12 (Disabled)

Deprecated rules: 182 (7.74%) (Disabled)
Experimental rules: 970 (41.28%)
Test rules: 1380 (58.72%)
Unsupported rules: 45 (1.91%) (Disabled)

Hayabusa rules: 31
Sigma rules: 2319
Total enabled detection rules: 2350

Output profile: standard

Scanning in progress. Please wait.

[00:00:05] 584 / 584   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors: 
...
Results Summary:

Events with hits / Total events: 5,160 / 47,472 (Data reduction: 42,312 events (89.13%))

Total | Unique detections: 6,734 | 406
Total | Unique critical detections: 56 (0.83%) | 20 (4.93%)
Total | Unique high detections: 5,474 (81.29%) | 216 (53.20%)
Total | Unique medium detections: 1,204 (17.88%) | 170 (41.87%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 0 (0.00%) | 0 (0.00%)

Dates with most total detections:
critical: 2019-07-19 (15), high: 2016-09-20 (3,646), medium: 2020-08-02 (89), low: n/a, informational: n/a

Top 5 computers with most unique detections:
critical: MSEDGEWIN10 (8), IEWIN7 (3), FS03.offsec.lan (2), IE10Win7 (2), rootdc1.offsec.lan (2)       
high: MSEDGEWIN10 (103), IEWIN7 (61), fs03vuln.offsec.lan (23), FS03.offsec.lan (22), IE10Win7 (20)    
medium: MSEDGEWIN10 (71), IEWIN7 (42), IE10Win7 (15), rootdc1.offsec.lan (14), PC01.example.corp (13)  
low: n/a
informational: n/a

╭────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                        Top high alerts:                           │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Sticky Key Like Backdoor Usage - ... (8)    Metasploit SMB Authentication (3,562)      │
│ CobaltStrike Service Installation... (6)    Malicious Svc Possibly Installed (271)     │
│ Active Directory Replication from... (6)    Suspicious Service Installation S... (250) │
│ Meterpreter or Cobalt Strike Gets... (6)    PowerShell Scripts Installed as S... (250) │
│ WannaCry Ransomware Activity (4)            Remote Thread Creation In Uncommo... (95)  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                          Top low alerts:                            │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Remote Thread Creation Via PowerS... (93)   n/a                                        │
│ Suspicious CMD Shell Output Redir... (71)   n/a                                        │
│ WSF/JSE/JS/VBA/VBE File Execution... (62)   n/a                                        │
│ Password Policy Enumerated (46)             n/a                                        │
│ Suspicious PowerShell Invocations... (43)   n/a                                        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                         n/a                                        │
│ n/a                                         n/a                                        │
│ n/a                                         n/a                                        │
│ n/a                                         n/a                                        │
│ n/a                                         n/a                                        │
╰───────────────────────────────────────────╌────────────────────────────────────────────╯

Saved file: 1188.csv (5.9 MB)

Elapsed time: 00:00:11.508

[hitenkoku]

@YamatoSecurity Thanks for your review.

I fixed new line position in df6e009.

Could you check it?

[fukusuket]

@hitenkoku
I confirmed that options 1 ~ 5 filter as expected. LGTM!!🚀

[YamatoSecurity]

@hitenkoku
I updated some of the wording.
Sorry for a few things:

1. -a, --no-asking to -w, --no-wizard?
   I think it will be better to refer to this as the scan wizard.

2. I am sorry I think i wrote status: testing in the issue, but the correct status is test, not testing.

I tried to change testing to test in vec! but it did not load any rules:

("1. Core ( status: test, stable | level: high, critical )", (vec!["testing", "stable"], "high")),
("2. Core+ ( status: test, stable | level: medium, high, critical )", (vec!["testing", "stable"], "medium")),
("3. Core++ ( status: experimental, test, stable | level: medium, high, critical )", (vec!["experimental", "testing", "stable"], "medium")),

Can you fix this?

3. When I enable Core, Core++ rules, etc.. stable rules should be loaded but it seems that only test rules get loaded:
   

4. Since levels, status and tags are being overridden by the wizard, i think we should add a requirement for the --exclude-status, --include-tag, --exclude-tag, --min-level and --exact-level options so that they require that -w, --no-wizard is enabled. What do you think?

[hitenkoku]

@YamatoSecurity Thanks for your review.

I fixed following your point out. Could you check it?

1. -a, --no-asking to -w, --no-wizard?
   I think it will be better to refer to this as the scan wizard.

I fixed in 5225290.

2. I am sorry I think i wrote status: testing in the issue, but the correct status is test, not testing.

I tried to change testing to test in vec! but it did not load any rules:

("1. Core ( status: test, stable | level: high, critical )", (vec!["testing", "stable"], "high")),
("2. Core+ ( status: test, stable | level: medium, high, critical )", (vec!["testing", "stable"], "medium")),
("3. Core++ ( status: experimental, test, stable | level: medium, high, critical )", (vec!["experimental", "testing", "stable"], "medium")),

Can you fix this?
3. When I enable Core, Core++ rules, etc.. stable rules should be loaded but it seems that only test rules get loaded:

I fixed in eab295a.

4. Since levels, status and tags are being overridden by the wizard, i think we should add a requirement for the --exclude-status, --include-tag, --exclude-tag, --min-level and --exact-level options so that they require that -w, --no-wizard is enabled. What do you think?

I agree you.
I fixed in dac57ae.
I want to check your opinion.
pivot-keywords-list command does not have --include-tag and --exclude-tag .
Do you think better that their option is added in this command?

[YamatoSecurity]

@hitenkoku Thank you! I checked that those are fixed.
Just 2 things:

1. The Total enabled detection rules: count does not change with Core and Core+. There maybe a bug when counting the medium rules?

2. When i choose 5. All event and alert rules ( status: * | level: informational+ ) it gives me the error: [ERROR] No rules were loaded. Please download the latest rules with the update-rules command.

[YamatoSecurity]

@hitenkoku

pivot-keywords-list command does not have --include-tag and --exclude-tag .
Do you think better that their option is added in this command?

Yes, maybe in a different issue we can add this.

[hitenkoku]

I created issue to following comment in #1195.

pivot-keywords-list command does not have --include-tag and --exclude-tag .
Do you think better that their option is added in this command?

Yes, maybe in a different issue we can add this.

[hitenkoku]

@YamatoSecurity Thank you for your comment.
I fixed following your point out. Could you check it?

1. The Total enabled detection rules: count does not change with Core and Core+. There maybe a bug when counting the medium rules?

2. When i choose `5. All event and alert rules ( status: * | level: informational+ )` it gives me the error: `[ERROR] No rules were loaded. Please download the latest rules with the update-rules command.`

[YamatoSecurity]

@YamatoSecurity Thank you for your comment.
I fixed following your point out. Could you check it?

LGTM! thank you!

[YamatoSecurity]

LGTM

[hitenkoku]

@fukusuket @YamatoSecurity I appriciate your continuous review.

since already checked in #1196, I will merge it.