Skip to content

Conversation

rgmz
Copy link
Contributor

@rgmz rgmz commented Nov 23, 2024

Description:

This creates a new command to show the effective config that Gitleaks actually uses, to help troubleshoot issues where

$ ./gitleaks config
title = 'Gitleaks development'

[extend]
  useDefault = true

[[rules]]
  id = '1password-service-account-token'
  description = 'Uncovered a possible 1Password service account token, potentially compromising access to secrets in vaults.'
  regex = 'ops_eyJ[a-zA-Z0-9+/]{250,}={0,3}'
  entropy = 4.0
  keywords = ['ops_']

...

[allowlist]
  condition = 'OR'
  paths = ['(^|/)cmd/generate/config/rules', 'gitleaks\.toml', '(?i)\.(bmp|gif|jpe?g|svg|tiff?)$', '\.(eot|[ot]tf|woff2?)$', '(.*?)(doc|docx|zip|xls|pdf|bin|socket|vsidx|v2|suo|wsuo|.dll|pdb|exe|gltf)$', 'go\.(mod|sum|work(\.sum)?)$', '(^|/)vendor/modules\.txt$', '(^|/)vendor/(github\.com|golang\.org/x|google\.golang\.org|gopkg\.in|istio\.io|k8s\.io|sigs\.k8s\.io)/.*$', '(^|/)gradlew(\.bat)?$', '(^|/)gradle\.lockfile$', '(^|/)mvnw(\.cmd)?$', '(^|/)\.mvn/wrapper/MavenWrapperDownloader\.java$', '(^|/)node_modules/.*?$', '(^|/)package-lock\.json$', '(^|/)yarn\.lock$', '(^|/)pnpm-lock\.yaml$', '(^|/)npm-shrinkwrap\.json$', '(^|/)bower_components/.*?$', '(^|/)(angular|jquery(-?ui)?|plotly|swagger-?ui)[a-zA-Z0-9.-]*(\.min)?\.js(\.map)?$', '(^|/)(Pipfile|poetry)\.lock$', '(?i)/?(v?env|virtualenv)/lib(64)?/.+$', '(?i)(^|/)(lib(64)?/python[23](\.\d{1,2})+/|python/[23](\.\d{1,2})+/lib(64)?/).+$', '(?i)(^|/)[a-z0-9_.]+-[0-9.]+\.dist-info/.+$', '(^|/)vendor/(bundle|ruby)/.*?$', '\.gem$', 'verification-metadata.xml', 'Database.refactorlog']
  regexes = ['(?i)^true|false|null$', '^(?i:a+|b+|c+|d+|e+|f+|g+|h+|i+|j+|k+|l+|m+|n+|o+|p+|q+|r+|s+|t+|u+|v+|w+|x+|y+|z+|\*+|\.+)$', '^\$(\d+|{\d+})$', '^\$([A-Z_]+|[a-z_]+)$', '^\${([A-Z_]+|[a-z_]+)}$', '^\{\{[ \t]*[\w ().|]+[ \t]*}}$', "^\\$\\{\\{[ \\t]*((env|github|secrets|vars)(\\.[A-Za-z]\\w+)+[\\w \"'&./=|]*)[ \\t]*}}$", '^%([A-Z_]+|[a-z_]+)%$', '^%[+\-# 0]?[bcdeEfFgGoOpqstTUvxX]$', '^\{\d{0,2}}$', '^@([A-Z_]+|[a-z_]+)@$']

Questions

  • Show the command take a source + config command like others, or take a single argument pointing to a config file?
  • Should empty fields be rendered or hidden? A common issue I've seen is people writing Files or Path which don't get picked up; explicit empty fields would make that obvious, but add a lot of noise.
  • Is the TOML it generates actually valid? Specifically, the quotation marks around regexes.

Checklist:

  • Does your PR pass tests?
  • Have you written new tests for your changes?
  • Have you lint your code locally prior to submission?

@rgmz rgmz force-pushed the feat/show-config branch 5 times, most recently from d94e6c9 to 75f66bb Compare November 23, 2024 20:20
@rgmz rgmz force-pushed the feat/show-config branch from 75f66bb to 98a6b18 Compare December 5, 2024 23:11
@rgmz rgmz force-pushed the feat/show-config branch 2 times, most recently from 283d329 to 34d00ad Compare January 10, 2025 12:12
@rgmz rgmz marked this pull request as ready for review January 10, 2025 12:13
@rgmz rgmz changed the title (WIP) Create command to print config Create command to print config Jan 15, 2025
@rgmz rgmz force-pushed the feat/show-config branch from 34d00ad to b855095 Compare January 15, 2025 13:03
@rgmz rgmz force-pushed the feat/show-config branch from b855095 to 5dd11a7 Compare April 6, 2025 16:51
@rgmz rgmz force-pushed the feat/show-config branch 3 times, most recently from 06bc6ed to fe3993e Compare April 12, 2025 16:16
@rgmz rgmz marked this pull request as draft April 12, 2025 16:17
@rgmz rgmz force-pushed the feat/show-config branch from fe3993e to e6b7767 Compare April 12, 2025 18:24
@rgmz rgmz mentioned this pull request Apr 14, 2025
3 tasks
@rgmz rgmz force-pushed the feat/show-config branch from e6b7767 to 005d8f8 Compare April 30, 2025 12:36
@zricethezav
Copy link
Collaborator

@rgmz

Show the command take a source + config command like others, or take a single argument pointing to a config file?

I'm thinking gitleaks config --show {source}. Requiring a --show flag gives us room to experiment with other config features (loading from a registry for example). Of course, this should be labeled experimental in case we want to change it which I don't think will be an issue since this is mostly a troubleshooting tool. WDYT?

Should empty fields be rendered or hidden? A common issue I've seen is people writing Files or Path which don't get picked up; explicit empty fields would make that obvious, but add a lot of noise.

I think they should be hidden. We want to display what the user sees when editing a config file (partial config if they are using extended configs)

@zricethezav
Copy link
Collaborator

quick sanity test too: gitleaks config --show {source} > out.toml && gitleaks dir --config=out.toml.

Wherever this feature lands, the above commands should work

@zricethezav
Copy link
Collaborator

@rgmz

diff --git a/cmd/config.go b/cmd/config.go
index 0a0d2d4..668714f 100644
--- a/cmd/config.go
+++ b/cmd/config.go
@@ -3,6 +3,8 @@ package cmd
 import (
        "bytes"
        "fmt"
+       "regexp"
+       "strconv"

        "github.com/pelletier/go-toml/v2"
        "github.com/rs/zerolog/log"
@@ -45,7 +47,22 @@ func runShowConfig(cmd *cobra.Command, args []string) {
        if err := enc.Encode(outputConfig); err != nil {
                log.Fatal().Err(err).Msg("could not encode config")
        }
-       fmt.Println(buf.String())
+       out := buf.String()
+       re := regexp.MustCompile(`(?m)^(\s*regex\s*=\s*)"(.+)"$`)
+       out = re.ReplaceAllStringFunc(out, func(m string) string {
+               parts := re.FindStringSubmatch(m)
+               // parts[1] == leading "    regex = "
+               // parts[2] == the raw inner escaped content
+               unq, err := strconv.Unquote(`"` + parts[2] + `"`)
+               if err != nil {
+                       // fallback to the escaped form
+                       unq = parts[2]
+               }
+               // wrap in a TOML multi-line literal (no escaping ever)
+               // see TOML spec: multi-line literal = '''…''' :contentReference[oaicite:1]{index=1}
+               return parts[1] + "'''" + unq + "'''"
+       })
+       fmt.Println(out)
 }

 // effectiveConfig matches the structure of `gitleaks.toml`.
@@ -54,5 +71,5 @@ type effectiveConfig struct {
        Title      string              `toml:"title,omitempty"`
        Extend     config.Extend       `toml:"extend,omitempty"`
        Rules      []config.Rule       `toml:"rules,omitempty"`
-       Allowlists []*config.Allowlist `toml:"allowlist,omitempty"`
+       Allowlists []*config.Allowlist `toml:"allowlists,omitempty"`
 }

This passes my simple gitleaks config --show {source} > out.toml && gitleaks dir --config=out.toml test

@rgmz rgmz force-pushed the feat/show-config branch from 005d8f8 to ee8c7b8 Compare May 3, 2025 14:19
@rgmz
Copy link
Contributor Author

rgmz commented May 3, 2025

This passes my simple gitleaks config --show {source} > out.toml && gitleaks dir --config=out.toml test

It's better but can still fail for rule.path, allowlist.regexes, or allowlist.paths. There doesn't seem to be conditional support for multi-line literals in https://github.com/pelletier/go-toml, unless I'm missing something.

@rgmz rgmz force-pushed the feat/show-config branch from ee8c7b8 to 7732b63 Compare June 1, 2025 17:19
@rgmz rgmz force-pushed the feat/show-config branch from 7732b63 to ea7972f Compare July 24, 2025 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants