Extend allowlist & handle extend when validating #1524

rgmz · 2024-09-23T20:27:08Z

Description:

This fixes #1523.

Checklist:

Does your PR pass tests?
Have you written new tests for your changes?
Have you lint your code locally prior to submission?

zricethezav · 2024-09-25T13:17:56Z

config/config.go

 			c.OrderedRules = append(c.OrderedRules, ruleID)
+		} else {
+			// Rule exists, merge our changes into the base.


nice, ty for this

Do you think there are any other fields we should append or overwrite?

I would suggest adding RegexTarget as well:

diff --git a/config/config.go b/config/config.go index caea22f..34e5f98 100644 --- a/config/config.go +++ b/config/config.go @@ -252,6 +252,7 @@ func (c *Config) extend(extensionConfig Config) { baseRule.Allowlist.Commits = append(baseRule.Allowlist.Commits, currentRule.Allowlist.Commits...) baseRule.Allowlist.Paths = append(baseRule.Allowlist.Paths, currentRule.Allowlist.Paths...) baseRule.Allowlist.Regexes = append(baseRule.Allowlist.Regexes, currentRule.Allowlist.Regexes...) + baseRule.Allowlist.RegexTarget = currentRule.Allowlist.RegexTarget baseRule.Allowlist.StopWords = append(baseRule.Allowlist.StopWords, currentRule.Allowlist.StopWords...) delete(c.Rules, ruleID)

I have a case where I must use line and the change above would enable extend for us.

I just made #1536 to cover that case

I'm surprised this didn't get caught earlier; I believe this makes it impossible to use most of the rule attributes documented in the README:

# An array of tables that contain information that define instructions # on how to detect secrets [[rules]] # Unique identifier for this rule id = "awesome-rule-1" # Short human readable description of the rule. description = "awesome rule 1" # Golang regular expression used to detect secrets. Note Golang's regex engine # does not support lookaheads. regex = '''one-go-style-regex-for-this-rule''' # Golang regular expression used to match paths. This can be used as a standalone rule or it can be used # in conjunction with a valid `regex` entry. path = '''a-file-path-regex''' # Array of strings used for metadata and reporting purposes. tags = ["tag","another tag"] # Int used to extract secret from regex match and used as the group that will have # its entropy checked if `entropy` is set. secretGroup = 3 # Float representing the minimum shannon entropy a regex group must have to be considered a secret. entropy = 3.5

https://github.com/gitleaks/gitleaks?tab=readme-ov-file#configuration

9999years · 2024-10-09T19:03:32Z

Hello, this broke the ability to override the entropy of rules.

With .gitleaks.toml:

[extend]
useDefault = true

[[rules]]
id = "finicity-client-secret"
entropy = 4

$ echo "newtype FinicityDataResponse = FinicityDataResponse" | gitleaks --config=.gitleaks.toml --redact=0 --no-banner --verbose --log-level trace stdin
11:54AM DBG using gitleaks config .gitleaks.toml from `--config`
11:54AM DBG extending config with default config

Finding:     newtype FinicityDataResponse = FinicityDataResponse
Secret:      FinicityDataResponse
RuleID:      finicity-client-secret
Entropy:     3.584184

11:54AM INF scan completed in 1.72ms
11:54AM WRN leaks found: 1

rgmz · 2024-10-09T19:12:01Z

Hello, this broke the ability to override the entropy of rules.

Hey @9999years, can you elaborate on what you expect to happen and which version you upgraded from? As far as I'm aware, that would have never worked.

Prior this this PR, extend only did did two things:

Merged your custom rules with the extended config, provided there were no ID conflicts
Appended values to the global allowlist.

https://github.com/gitleaks/gitleaks/blob/v8.19.2/config/config.go#L237

Edit: prior to v8.19.3, the example you provided makes finicity-client-secret a no-op rule:

 "finicity-client-secret": {
 	RuleID:      "finicity-client-secret",
- 	Description: "Identified a Finicity Client Secret, which could lead to compromised financial service integrations and data breaches.",
+ 	Description: "",
- 	Regex: s`(?i)(?:finicity)(?:[0-9a-z\-_\t .]{0,20})(?:[\s|']|[\s|"]){0,3}(?:=|>|:{1,3}=|\|\|:|<=|=>|:|\?=)(?:'|\"|\s|=|\x60){0,5}([a-z0-9]`...,
- 	Regex: nil.
+ 	Entropy:  4,
- 	Keywords: []string{"finicity"},
+ 	Keywords: []string{},
},

9999years · 2024-10-09T19:46:11Z

@rgmz Just a minimal example. Our actual configuration looks like this:

[[rules]]
description = "Finicity Client Secret"
id = "finicity-client-secret"
regex = '''(?i)(?:finicity)(?:[0-9a-z\-_\t .]{0,20})(?:[\s|']|[\s|"]){0,3}(?:=|>|:{1,3}=|\|\|:|<=|=>|:|\?=)(?:'|\"|\s|=|\x60){0,5}([a-z0-9]{20})(?:['|\"|\n|\r|\s|\x60|;]|$)'''
secretGroup = 1
entropy = 4
keywords = ["finicity"]

PR gitleaks#1524 broke the ability to specify a `[[rules]]` with the same ID as a built-in rule in order to extend the default configuration. This PR restores this ability, but merges the provided configuration with the default rule instead of overwriting the default rule wholesale.

* Reimplement the ability to override built-in rules PR #1524 broke the ability to specify a `[[rules]]` with the same ID as a built-in rule in order to extend the default configuration. This PR restores this ability, but merges the provided configuration with the default rule instead of overwriting the default rule wholesale. * Add tests * Append keywords and tags

…ks#1524)

* Reimplement the ability to override built-in rules PR gitleaks#1524 broke the ability to specify a `[[rules]]` with the same ID as a built-in rule in order to extend the default configuration. This PR restores this ability, but merges the provided configuration with the default rule instead of overwriting the default rule wholesale. * Add tests * Append keywords and tags

This comment was marked as outdated.

Sign in to view

rgmz force-pushed the fix/extend-validate branch 2 times, most recently from d825dd8 to 3e99a86 Compare September 23, 2024 21:15

rgmz marked this pull request as ready for review September 23, 2024 21:16

fix(config): extend allowlist & handle extend when validating

87bec7b

rgmz force-pushed the fix/extend-validate branch from 3e99a86 to 87bec7b Compare September 23, 2024 21:17

zricethezav reviewed Sep 25, 2024

View reviewed changes

zricethezav merged commit ed19c4e into gitleaks:master Sep 25, 2024
1 check passed

rgmz deleted the fix/extend-validate branch September 25, 2024 13:20

bufferoverflow mentioned this pull request Sep 28, 2024

fix: use regexTarget for extend config #1536

Merged

3 tasks

9999years mentioned this pull request Oct 9, 2024

Fix rule extension #1556

Merged

3 tasks

alayne222 pushed a commit to alayne222/gitleaks that referenced this pull request May 28, 2025

fix(config): extend allowlist & handle extend when validating (gitlea…

d35136e

…ks#1524)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Extend allowlist & handle extend when validating #1524

Extend allowlist & handle extend when validating #1524

Uh oh!

rgmz commented Sep 23, 2024 •

edited

Loading

Uh oh!

This comment was marked as outdated.

zricethezav Sep 25, 2024

Uh oh!

rgmz Sep 25, 2024

Uh oh!

bufferoverflow Sep 28, 2024

Uh oh!

bufferoverflow Sep 28, 2024

Uh oh!

9999years Oct 9, 2024

Uh oh!

Uh oh!

9999years commented Oct 9, 2024

Uh oh!

rgmz commented Oct 9, 2024 •

edited

Loading

Uh oh!

9999years commented Oct 9, 2024

Uh oh!

Uh oh!

Uh oh!

Extend allowlist & handle extend when validating #1524

Extend allowlist & handle extend when validating #1524

Uh oh!

Conversation

rgmz commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Checklist:

Uh oh!

This comment was marked as outdated.

zricethezav Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

rgmz Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

bufferoverflow Sep 28, 2024

Choose a reason for hiding this comment

Uh oh!

bufferoverflow Sep 28, 2024

Choose a reason for hiding this comment

Uh oh!

9999years Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

9999years commented Oct 9, 2024

Uh oh!

rgmz commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

9999years commented Oct 9, 2024

Uh oh!

Uh oh!

rgmz commented Sep 23, 2024 •

edited

Loading

rgmz commented Oct 9, 2024 •

edited

Loading