First report: #3266 (reporter: @leveryd)

Describe the bug

Rules with the rx operator matching the "start of line" metacharacter (^) behave differently based on the WAF engine. It depends if the multiline modifier is enabled by default for each regex or not.
It leads to:

• Possible false positives
• Extra computational time for computing checks not intended by the rule(s) design.

Default Modifiers

Not an expert on all the codebases, but to the best of what I understood:

ModSec2 (v2.9.7) (for Apache)

• Comes with PCRE2_DOTALL | PCRE2_DOLLAR_ENDONLY (see here).
• Under standalone/regex.c there also is some code that includes PCRE_MULTILINE if cflags comes with AP_REG_NEWLINE. I guess it is at compilation time. But on this file, there is not even mention of PCRE2, which should have been added to 2.9.* 🧐

Libmodsecurity (v.3.0.9) (E.g. Nginx)

• Comes with PCRE2_DOTALL|PCRE2_MULTILINE (see here).

Coraza (v3.0.3):

• Comes with "(?sm)%s", options.Arguments. Since #732, rx operator has been aligned to Libmodsecurity with both dotall and multiline.
• Notably without that PR 942522-7 test New line bypass was failing. That being said, it could be just a matter of DOTALL and not MULTILINE.

Steps to reproduce

Example rule attempting to catch the closing curly bracket at the beginning of the line: SecRule RESPONSE_BODY "@rx ^}" "id:911,phase:4, deny, status:403, capture,log, t:none, msg:'multiline active'"
Request: curl 'localhost:80/anything'
Expected server response:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Host": "localhost",
    "User-Agent": "curl/7.87.0"
  },
  "json": null,
  "method": "GET",
  "origin": "192.168.48.1",
  "url": "http://localhost/anything"
} 

Results:

• ModSec 2.9 on Apache: does not trigger this rule
• libModSec 3 on Nginx: triggers the rule, and writes the log, but can not enforce the deny (headers already sent; known and accepted limitation of the platform)
• Caddy-Coraza: triggers the rule and denies the request

Possible solutions

The goal is to achieve consistency across WAF Engines that supports CRS bringing their behavior in line with the reference platform (Apache) (see August 2023 Monthly Chat Decisions).
Possible solutions are:

• Work with engine devs to align the default modifiers.
• Make regexes not dependent on the multiline modifier:
  • Exploring solutions provided at [QUESTION] Can modsecurity support check response body first ten bytes? owasp-modsecurity/ModSecurity#2921
  • Using (?-m) (@azurit's proposal).