Where is inbound anomaly score ? #662

Barnoux · 2023-02-17T20:20:22Z

Barnoux
Feb 17, 2023

Hello,

Based on this super article : https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/#step_3_the_first_batch_of_rule_exclusions

How can we have the total of an anomaly score for a request, like the autor of the blog work with it in access of the Apache's log ?

I would like to have it in the caddy logs. i think it will be so much easier to analyse the data.

For the exemple bellow, i have for one request, 5 coraza messages with each of a critical severity. But in the access log of caddy for this request, i don't have the total of the inbound anomaly score.

jcchavezs · 2023-02-18T11:18:47Z

jcchavezs
Feb 18, 2023
Maintainer

Ping @M4tteoP

…

On Fri, 17 Feb 2023, 21:20 Barnoux, ***@***.***> wrote: Hello, Based on this super article : https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/#step_3_the_first_batch_of_rule_exclusions How can we have the total of an anomaly score for a request, like the autor of the blog work with it in acess of the Apache's log ? I would like to have it in the caddy logs. i think it will be so much easier to analyse the data. For the exemple bellow, i have for one request, 5 coraza messages with each of a critical severity. But in the access log of caddy for this request, i don't have the total of the inbound anomaly score. [image: image] <https://user-images.githubusercontent.com/47791676/219783868-7ec00b85-aa4f-4c4a-ba13-ab94f641a33d.png> — Reply to this email directly, view it on GitHub <#662>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAWRUZESIHW66YFAPN3WX7MRDANCNFSM6AAAAAAU72U5PE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

0 replies

M4tteoP · 2023-03-03T16:51:00Z

M4tteoP
Mar 3, 2023
Collaborator

Hi @Barnoux! Please, take a look at #684. It might not completely solve your question (To the best of my knowledge customizing access logs would require a proper support for each connector), but I think that fixed correlation rules with a high reporting level may come handy printing in the error logs, ,among other things, the total of anomaly scores

7 replies

Barnoux Mar 28, 2023
Author

Hey @M4tteoP 😄 Now, i have something by enabling log by default log on phase 5!
I configure "tx.reporting_level=5" in crs-setup.conf. but i don't have the msg doesn't gave us the total of anomaly score in the msg.

{
  "level": "error",
  "ts": 1680024189.2647123,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. HTTP Parameter Pollution (utf-8) [file \"/ruleset/coreruleset/rules/REQUEST-921-PROTOCOL-ATTACK.conf\"] [line \"0\"] [id \"921180\"] [rev \"\"] [msg \"HTTP Parameter Pollution (utf-8)\"] [data \"Matched Data: sec-fetch-dest found within TX:paramcounter_args_names: 3\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/137/15/460\"] [tag \"paranoia-level/3\"] [hostname \"\"] [uri \"/identity/connect/token\"] [unique_id \"SJCANpvmLJlOelUj\"]\n[client \"192.168.1.1\"] Coraza: Warning. HTTP Parameter Pollution (utf-8) [file \"/ruleset/coreruleset/rules/REQUEST-921-PROTOCOL-ATTACK.conf\"] [line \"0\"] [id \"921180\"] [rev \"\"] [msg \"\"] [data \"\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/137/15/460\"] [tag \"paranoia-level/3\"] [hostname \"\"] [uri \"/identity/connect/token\"] [unique_id \"SJCANpvmLJlOelUj\"]\n"
}

{
  "level": "error",
  "ts": 1680024189.2974505,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning.  [file \"/ruleset/coreruleset/rules/RESPONSE-980-CORRELATION.conf\"] [line \"0\"] [id \"980170\"] [rev \"\"] [msg \"\"] [data \"\"] [severity \"emergency\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"reporting\"] [hostname \"\"] [uri \"/identity/connect/token\"] [unique_id \"SJCANpvmLJlOelUj\"]\n"
}

the rule that is triggered is this one

SecAction \
    "id:980170,\
    phase:5,\
    pass,\
    t:none,\
    noauditlog,\
    msg:'Anomaly Scores: \
(Inbound Scores: blocking=%{tx.blocking_inbound_anomaly_score}, detection=%{tx.detection_inbound_anomaly_score}, per_pl=%{tx.inbound_anomaly_score_pl1}-%{tx.inbound_anomaly_score_pl2}-%{tx.inbound_anomaly_score_pl3}-%{tx.inbound_anomaly_score_pl4}, threshold=%{tx.inbound_anomaly_score_threshold}) - \
(Outbound Scores: blocking=%{tx.blocking_outbound_anomaly_score}, detection=%{tx.detection_outbound_anomaly_score}, per_pl=%{tx.outbound_anomaly_score_pl1}-%{tx.outbound_anomaly_score_pl2}-%{tx.outbound_anomaly_score_pl3}-%{tx.outbound_anomaly_score_pl4}, threshold=%{tx.outbound_anomaly_score_threshold}) - \
(SQLI=%{tx.sql_injection_score}, XSS=%{tx.xss_score}, RFI=%{tx.rfi_score}, LFI=%{tx.lfi_score}, RCE=%{tx.rce_score}, PHPI=%{tx.php_injection_score}, HTTP=%{tx.http_violation_score}, SESS=%{tx.session_fixation_score}, COMBINED_SCORE=%{tx.anomaly_score})',\
    tag:'reporting',\
    ver:'OWASP_CRS/4.0.0-rc1'"

M4tteoP Mar 29, 2023
Collaborator

We are getting closer :D. The last step is using a caddy connector with an updated Coraza. I think you are facing the empty message problem that I recently fixed in #684 (see here). The corazawaf/coraza-caddy#49 open PR should fix it. Hopefully we will manage to merge it soon 🤞

Barnoux Mar 30, 2023
Author

Alright thanks for the hard work, i need to be patient now 😄

M4tteoP Mar 31, 2023
Collaborator

Hey! We merged a pretty big pr with the middleware fully refactored and pointing to a recent Coraza version. Any early feedback and updates about this issue are super welcome!
There is not a release yet, be sure to build the specific latest commit 34daaf87f9ddaca2833461de59ebada21c902598
Thanks :)

Barnoux Mar 31, 2023
Author

Hey 😄 with the command:
xcaddy build --with github.com/corazawaf/coraza-caddy@master

i got the following error:
Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: invalid WAF config from file: failed to readfile: open path: invalid argument
i don't have it with the coraza-caddy version v1.2.2

jcchavezs · 2023-03-31T19:32:16Z

jcchavezs
Mar 31, 2023
Maintainer

Could you please try with the specific commit?

…

On Fri, 31 Mar 2023, 20:46 Barnoux, ***@***.***> wrote: Hey 😄 with the command: xcaddy build --with ***@***.*** i got the following error: Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: invalid WAF config from file: failed to readfile: open path: invalid argument i don't have it with the coraza-caddy version v1.2.2 — Reply to this email directly, view it on GitHub <#662 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAQPWYK72AZR4Y7VDW3W64RBPANCNFSM6AAAAAAU72U5PE> . You are receiving this because you commented.Message ID: ***@***.***>

4 replies

Barnoux Mar 31, 2023
Author

Hey 😄
I used this command xcaddy build --with github.com/corazawaf/coraza-caddy@34daaf87f9ddaca2833461de59ebada21c902598
and i got this error:
Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: invalid WAF config from file: failed to readfile: open path: invalid argument

jcchavezs Apr 17, 2023
Maintainer

How does your caddy config look like?

Barnoux Apr 17, 2023
Author

There you go! 😸

# Set up configuration for coraza_waf middleware
# `order coraza_waf first` ensures that the coraza_waf middleware is executed before any other middleware
# `persist_config off` disables configuration persistence across restarts
# `log` sets up logging configuration for Caddy
# `servers` sets up server-level settings for all servers defined in this file
{
        order coraza_waf first
        persist_config off
        log {
                level ERROR    # sets logging level to ERROR
                format json    # logs in JSON format
                output file /var/log/caddy/access.log {    # logs to file
                        roll_uncompressed    # ensures log files are uncompressed
                }
        }
        servers {
                timeouts {    # sets timeouts for various actions
                        idle 10s    # timeout for idle connections
                        read_body 10s    # timeout for reading request body
                        read_header 10s    # timeout for reading request header
                }
                max_header_size 4KB    # sets maximum header size for requests
        }
}

# Default server block that will respond with a "Not found" message for all requests to port 443
:443 {
        respond "Not found" 404
}

# Server block for {$DOMAIN} that includes the coraza_waf middleware, sets up TLS with {$EMAIL}, and defines various headers
{$DOMAIN}:443 {
        coraza_waf {
                include /ruleset/coraza.conf    # includes coraza_waf configuration file
                include /ruleset/sites/*.conf    # includes configuration files for individual sites
                include /ruleset/coreruleset/rules/*.conf    # includes core ruleset files
        }
        request_body {
                max_size 100KB    # sets maximum request body size
        }
        tls {$EMAIL}    # sets up TLS with email address
        header {    # sets various headers
                Strict-Transport-Security "max-age=15768000;"    # sets HSTS header
                X-XSS-Protection "1; mode=block"    # sets XSS protection header
                Referrer-Policy "no-referrer"    # sets referrer policy header
                X-Content-Type-Options "nosniff"    # sets content type options header
                X-Frame-Options "SAMEORIGIN"    # sets frame options header
                X-Robots-Tag "none"    # sets robots tag header
                -Server    # disables server header
        }
        @insecureadmin {    # defines named matcher for insecure admin requests
                not remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8    # matches requests not coming from specified IP ranges
                path /admin*    # matches requests to /admin* path
        }
        redir @insecureadmin /    # redirects insecure admin requests to root path
        encode gzip    # enables gzip compression
        reverse_proxy /notifications/hub vaultwarden:3012    # sets up reverse proxy for /notifications/hub path to vaultwarden:3012
        reverse_proxy vaultwarden:60278 {    # sets up reverse proxy for vaultwarden:60278
                trusted_proxies private_ranges    # trusts private IP ranges for X-Real-IP header
                header_up X-Real-IP {remote_host}    # sets X-Real-IP header to remote host
        }
}

Barnoux Jun 1, 2023
Author

Do you have any update about my problem ?

jcchavezs · 2023-06-01T12:12:40Z

jcchavezs
Jun 1, 2023
Maintainer

Could you please try again with latest coraza-caddy version?

…

On Thu, 1 Jun 2023, 12:52 Barnoux, ***@***.***> wrote: Do you have any update about my problem ? — Reply to this email directly, view it on GitHub <#662 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAVKUEMP7IZWZ5ISVKTXJBX7FANCNFSM6AAAAAAU72U5PE> . You are receiving this because you commented.Message ID: ***@***.***>

2 replies

Barnoux Jun 3, 2023
Author

I think it's working :) I need to test by triggering th e WAF with a XSS or OS command injection to see if the total inbound score of the request is logged and correct 👍

Barnoux Jun 3, 2023
Author

I tested with an OS command injection and it's working!! i have the total inbound score :) But one things is weird, i have duplicated coraza log for each line.

PL4 alert duplicated - score calculeted by correlation rule = 10

{
  "level": "error",
  "ts": 1685797070.720458,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. Invalid character in request (outside of very strict set) [file \"/ruleset/coreruleset/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"3479\"] [id \"920273\"] [rev \"\"] [msg \"Invalid character in request (outside of very strict set)\"] [data \"ARGS:foo=/etc/passwd\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/210/272\"] [tag \"paranoia-level/4\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n[client \"192.168.1.1\"] Coraza: Warning. Invalid character in request (outside of very strict set) [file \"/ruleset/coreruleset/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"3479\"] [id \"920273\"] [rev \"\"] [msg \"Invalid character in request (outside of very strict set)\"] [data \"ARGS:bar=/bin/sh\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/210/272\"] [tag \"paranoia-level/4\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

PL1 alert not duplicated - score calculeted by correlation rule = 5

{
  "level": "error",
  "ts": 1685797070.7224216,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. OS File Access Attempt [file \"/ruleset/coreruleset/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"4063\"] [id \"930120\"] [rev \"\"] [msg \"OS File Access Attempt\"] [data \"Matched Data: etc/passwd found within ARGS:foo: /etc/passwd\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [tag \"PCI/6.5.4\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

PL1 alert duplicated - score calculeted by correlation rule = 10

{
  "level": "error",
  "ts": 1685797070.723283,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. Remote Command Execution: Unix Shell Code Found [file \"/ruleset/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"4759\"] [id \"932160\"] [rev \"\"] [msg \"Remote Command Execution: Unix Shell Code Found\"] [data \"Matched Data: etc/passwd found within ARGS:foo: /etc/passwd\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/248/88\"] [tag \"PCI/6.5.2\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n[client \"192.168.1.1\"] Coraza: Warning. Remote Command Execution: Unix Shell Code Found [file \"/ruleset/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"4759\"] [id \"932160\"] [rev \"\"] [msg \"Remote Command Execution: Unix Shell Code Found\"] [data \"Matched Data: bin/sh found within ARGS:bar: /bin/sh\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/248/88\"] [tag \"PCI/6.5.2\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

Correlation alert - total inbound score = 25, without the duplication it should be = 15

{
  "level": "error",
  "ts": 1685797070.7382822,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. Anomaly Scores: (Inbound Scores: blocking=25, detection=25, per_pl=15-0-0-10, threshold=1000) - (Outbound Scores: blocking=0, detection=0, per_pl=0-0-0-0, threshold=1000) - (SQLI=0, XSS=0, RFI=0, LFI= [file \"/ruleset/coreruleset/rules/RESPONSE-980-CORRELATION.conf\"] [line \"11046\"] [id \"980170\"] [rev \"\"] [msg \"Anomaly Scores: (Inbound Scores: blocking=25, detection=25, per_pl=15-0-0-10, threshold=1000) - (Outbound Scores: blocking=0, detection=0, per_pl=0-0-0-0, threshold=1000) - (SQLI=0, XSS=0, RFI=0, LFI=\"] [data \"\"] [severity \"emergency\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"reporting\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

Barnoux · 2023-06-03T15:37:37Z

Barnoux
Jun 3, 2023
Author

Thanks a lot for the help. The initial question that i have is now resolved.

The duplicated modsecurity fields's in caddy logs's will be followed here: corazawaf/coraza-caddy#32

0 replies

jcchavezs · 2023-06-03T16:09:27Z

jcchavezs
Jun 3, 2023
Maintainer

Awesome!

…

On Sat, 3 Jun 2023, 17:37 Barnoux, ***@***.***> wrote: Closed #662 <#662> as resolved. — Reply to this email directly, view it on GitHub <#662>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAUPSWSRTCM7CZTOEU3XJNK4ZANCNFSM6AAAAAAU72U5PE> . You are receiving this because you commented.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Where is inbound anomaly score ? #662

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 13 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Where is inbound anomaly score ? #662

Barnoux Feb 17, 2023

Replies: 6 comments · 13 replies

jcchavezs Feb 18, 2023 Maintainer

M4tteoP Mar 3, 2023 Collaborator

Barnoux Mar 28, 2023 Author

M4tteoP Mar 29, 2023 Collaborator

Barnoux Mar 30, 2023 Author

M4tteoP Mar 31, 2023 Collaborator

Barnoux Mar 31, 2023 Author

jcchavezs Mar 31, 2023 Maintainer

Barnoux Mar 31, 2023 Author

jcchavezs Apr 17, 2023 Maintainer

Barnoux Apr 17, 2023 Author

Barnoux Jun 1, 2023 Author

jcchavezs Jun 1, 2023 Maintainer

Barnoux Jun 3, 2023 Author

Barnoux Jun 3, 2023 Author

Barnoux Jun 3, 2023 Author

jcchavezs Jun 3, 2023 Maintainer

Barnoux
Feb 17, 2023

Replies: 6 comments 13 replies

jcchavezs
Feb 18, 2023
Maintainer

M4tteoP
Mar 3, 2023
Collaborator

Barnoux Mar 28, 2023
Author

M4tteoP Mar 29, 2023
Collaborator

Barnoux Mar 30, 2023
Author

M4tteoP Mar 31, 2023
Collaborator

Barnoux Mar 31, 2023
Author

jcchavezs
Mar 31, 2023
Maintainer

Barnoux Mar 31, 2023
Author

jcchavezs Apr 17, 2023
Maintainer

Barnoux Apr 17, 2023
Author

Barnoux Jun 1, 2023
Author

jcchavezs
Jun 1, 2023
Maintainer

Barnoux Jun 3, 2023
Author

Barnoux Jun 3, 2023
Author

Barnoux
Jun 3, 2023
Author

jcchavezs
Jun 3, 2023
Maintainer