Skip to content

Conversation

takumin
Copy link
Contributor

@takumin takumin commented Apr 24, 2025

projectdiscovery/katana: A next-generation crawling and spidering framework

$ aqua g -i projectdiscovery/katana

Check List

How to confirm if this package works well

Reviewers aren't necessarily familiar with this package, so please describe how to confirm if this package works well.
Please confirm if this package works well yourself as much as possible.

Command and output

$ cmdx con linux amd64
+ bash scripts/connect.sh
[INFO] Connecting to the container aqua-registry (linux/amd64)
root@bd8d9067be76:/workspace# katana -h
Katana is a fast crawler focused on execution in automation
pipelines offering both headless and non-headless crawling.

Usage:
  katana [flags]

Flags:
INPUT:
   -u, -list string[]     target url / list to crawl
   -resume string         resume scan using resume.cfg
   -e, -exclude string[]  exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)

CONFIGURATION:
   -r, -resolvers string[]       list of custom resolver (file or comma separated)
   -d, -depth int                maximum depth to crawl (default 3)
   -jc, -js-crawl                enable endpoint parsing / crawling in javascript file
   -jsl, -jsluice                enable jsluice parsing in javascript file (memory intensive)
   -ct, -crawl-duration value    maximum duration to crawl the target for (s, m, h, d) (default s)
   -kf, -known-files string      enable crawling of known files (all,robotstxt,sitemapxml), a minimum depth of 3 is required to ensure all known files are properly crawled.
   -mrs, -max-response-size int  maximum response size to read (default 4194304)
   -timeout int                  time to wait for request in seconds (default 10)
   -time-stable int              time to wait until the page is stable in seconds (default 1)
   -aff, -automatic-form-fill    enable automatic form filling (experimental)
   -fx, -form-extraction         extract form, input, textarea & select elements in jsonl output
   -retry int                    number of times to retry the request (default 1)
   -proxy string                 http/socks5 proxy to use
   -td, -tech-detect             enable technology detection
   -H, -headers string[]         custom header/cookie to include in all http request in header:value format (file)
   -config string                path to the katana configuration file
   -fc, -form-config string      path to custom form configuration file
   -flc, -field-config string    path to custom field configuration file
   -s, -strategy string          Visit strategy (depth-first, breadth-first) (default "depth-first")
   -iqp, -ignore-query-params    Ignore crawling same path with different query-param values
   -tlsi, -tls-impersonate       enable experimental client hello (ja3) tls randomization
   -dr, -disable-redirects       disable following redirects (default false)

DEBUG:
   -health-check, -hc        run diagnostic check up
   -elog, -error-log string  file to write sent requests error log
   -pprof-server             enable pprof server

HEADLESS:
   -hl, -headless                    enable headless hybrid crawling (experimental)
   -sc, -system-chrome               use local installed chrome browser instead of katana installed
   -sb, -show-browser                show the browser on the screen with headless mode
   -ho, -headless-options string[]   start headless chrome with additional options
   -nos, -no-sandbox                 start headless chrome in --no-sandbox mode
   -cdd, -chrome-data-dir string     path to store chrome browser data
   -scp, -system-chrome-path string  use specified chrome browser for headless crawling
   -noi, -no-incognito               start headless chrome without incognito mode
   -cwu, -chrome-ws-url string       use chrome browser instance launched elsewhere with the debugger listening at this URL
   -xhr, -xhr-extraction             extract xhr request url,method in jsonl output

SCOPE:
   -cs, -crawl-scope string[]       in scope url regex to be followed by crawler
   -cos, -crawl-out-scope string[]  out of scope url regex to be excluded by crawler
   -fs, -field-scope string         pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn")
   -ns, -no-scope                   disables host based default scope
   -do, -display-out-scope          display external endpoint from scoped crawling

FILTER:
   -mr, -match-regex string[]       regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[]      regex or list of regex to filter on output url (cli, file)
   -f, -field string                field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -sf, -store-field string         field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -em, -extension-match string[]   match output for given extension (eg, -em php,html,js)
   -ef, -extension-filter string[]  filter output for given extension (eg, -ef png,css)
   -mdc, -match-condition string    match response with dsl based condition
   -fdc, -filter-condition string   filter response with dsl based condition

RATE-LIMIT:
   -c, -concurrency int          number of concurrent fetchers to use (default 10)
   -p, -parallelism int          number of concurrent inputs to process (default 10)
   -rd, -delay int               request delay between each request in seconds
   -rl, -rate-limit int          maximum requests to send per second (default 150)
   -rlm, -rate-limit-minute int  maximum number of requests to send per minute

UPDATE:
   -up, -update                 update katana to latest version
   -duc, -disable-update-check  disable automatic katana update check

OUTPUT:
   -o, -output string                file to write output to
   -sr, -store-response              store http requests/responses
   -srd, -store-response-dir string  store http requests/responses to custom directory
   -ncb, -no-clobber                 do not overwrite output file
   -sfd, -store-field-dir string     store per-host field to custom directory
   -or, -omit-raw                    omit raw requests/responses from jsonl output
   -ob, -omit-body                   omit response body from jsonl output
   -j, -jsonl                        write output in jsonl format
   -nc, -no-color                    disable output content coloring (ANSI escape codes)
   -silent                           display output only
   -v, -verbose                      display verbose output
   -debug                            display debug output
   -version                          display project version

If files such as configuration file are needed, please share them.

Reference

@suzuki-shunsuke suzuki-shunsuke added the enhancement New feature or request label Apr 24, 2025
Comment on lines +1 to +9
---
# yaml-language-server: $schema=https://raw.githubusercontent.com/aquaproj/aqua/main/json-schema/aqua-generate-registry.json
# aqua - Declarative CLI Version Manager
# https://aquaproj.github.io/
# Other than name is optional. All initial values are just examples.
name: projectdiscovery/katana
# version_filter: not (Version matches "-rc$")
# version_prefix: cli-
# all_assets_filter: not (Asset matches "-cli")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to use scaffold.yaml.

@suzuki-shunsuke suzuki-shunsuke added this to the v4.353.0 milestone Apr 24, 2025
@suzuki-shunsuke suzuki-shunsuke merged commit 33d9594 into aquaproj:main Apr 24, 2025
19 checks passed
@github-project-automation github-project-automation bot moved this to Done in main Apr 24, 2025
@takumin takumin deleted the feat/projectdiscovery/katana branch April 24, 2025 22:50
@suzuki-shunsuke
Copy link
Member

🎉 https://github.com/aquaproj/aqua-registry/releases/tag/v4.353.0

tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Apr 26, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [aquaproj/aqua-registry](https://github.com/aquaproj/aqua-registry) | minor | `v4.346.1` -> `v4.353.0` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>aquaproj/aqua-registry (aquaproj/aqua-registry)</summary>

### [`v4.353.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.353.0)

[Compare Source](aquaproj/aqua-registry@v4.352.0...v4.353.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.353.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.353.0) | aquaproj/aqua-registry@v4.352.0...v4.353.0

#### 🎉 New Packages

[#&#8203;35522](aquaproj/aqua-registry#35522) [Nukesor/pueue/pueue](https://github.com/Nukesor/pueue): Pueue is a command-line task management tool for sequential and parallel execution of long-running tasks [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35523](aquaproj/aqua-registry#35523) [Nukesor/pueue/pueued](https://github.com/Nukesor/pueue): Pueue is a command-line task management tool for sequential and parallel execution of long-running tasks [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35321](aquaproj/aqua-registry#35321) [cri-o/cri-o](https://github.com/cri-o/cri-o): Open Container Initiative-based implementation of Kubernetes Container Runtime Interface [@&#8203;takumin](https://github.com/takumin)
[#&#8203;24089](aquaproj/aqua-registry#24089) [ip7z/7zip](https://github.com/ip7z/7zip): 7-Zip is a file archiver with a high compression ratio [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35060](aquaproj/aqua-registry#35060) [oracle/truffleruby/community-native](https://github.com/oracle/truffleruby): A high performance implementation of the Ruby programming language, built on GraalVM [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35542](aquaproj/aqua-registry#35542) [projectdiscovery/katana](https://github.com/projectdiscovery/katana): A next-generation crawling and spidering framework [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35075](aquaproj/aqua-registry#35075) [project-zot/zot](https://github.com/project-zot/zot): zot - A scale-out production-ready vendor-neutral OCI-native container image/artifact registry (purely based on OCI Distribution Specification) [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35467](aquaproj/aqua-registry#35467) [sagiegurari/cargo-make](https://github.com/sagiegurari/cargo-make): Rust task runner and build tool [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35513](aquaproj/aqua-registry#35513) [str4d/rage](https://github.com/str4d/rage): A simple, secure and modern file encryption tool (and Rust library) with small explicit keys, no config options, and UNIX-style composability [@&#8203;takumin](https://github.com/takumin)

#### Re-scaffold

Re-scaffolded 35 packages to improve the quality.

### [`v4.352.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.352.0)

[Compare Source](aquaproj/aqua-registry@v4.351.0...v4.352.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.352.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.352.0) | aquaproj/aqua-registry@v4.351.0...v4.352.0

#### 🎉 New Packages

[#&#8203;35179](aquaproj/aqua-registry#35179) [containerd/containerd/static](https://github.com/containerd/containerd): An open and reliable container runtime (static binary) [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35146](aquaproj/aqua-registry#35146) [zmap/zlint](https://github.com/zmap/zlint): X.509 Certificate Linter focused on Web PKI standards and requirements [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35094](aquaproj/aqua-registry#35094) [cubefs/cubefs](https://github.com/cubefs/cubefs): cloud-native distributed storage [@&#8203;takumin](https://github.com/takumin)
[#&#8203;35055](aquaproj/aqua-registry#35055) [oracle/truffleruby/oracle-native](https://github.com/oracle/truffleruby): A high performance implementation of the Ruby programming language, built on GraalVM [@&#8203;takumin](https://github.com/takumin)

#### Security

[#&#8203;35200](aquaproj/aqua-registry#35200) Re-scaffold containerd/containerd (support github_artifact_attestations) [@&#8203;takumin](https://github.com/takumin)

#### Improvement

[#&#8203;35058](aquaproj/aqua-registry#35058) dyne/slangroom-exec: Add a command `slexfe` [@&#8203;matteo-cristino](https://github.com/matteo-cristino)

#### Re-scaffold

Re-scaffolded about 70 packages to improve the quality.

### [`v4.351.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.351.0)

[Compare Source](aquaproj/aqua-registry@v4.350.0...v4.351.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.351.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.351.0) | aquaproj/aqua-registry@v4.350.0...v4.351.0

#### 🎉 New Packages

[#&#8203;34979](aquaproj/aqua-registry#34979) [bahdotsh/wrkflw](https://github.com/bahdotsh/wrkflw): Validate and execute GitHub Actions workflows locally
[#&#8203;34967](aquaproj/aqua-registry#34967) [stunnel/static-curl](https://github.com/stunnel/static-curl): static builds cURL with HTTP3 [@&#8203;takumin](https://github.com/takumin)

#### Re-scaffold

Re-scaffolded over 20 packages to improve the quality.

### [`v4.350.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.350.0)

[Compare Source](aquaproj/aqua-registry@v4.349.0...v4.350.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.350.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.350.0) | aquaproj/aqua-registry@v4.349.0...v4.350.0

#### 🎉 New Packages

[#&#8203;34920](aquaproj/aqua-registry#34920) [kubernetes-sigs/controller-runtime/setup-envtest](https://github.com/kubernetes-sigs/controller-runtime/tree/main/tools/setup-envtest): A small tool that manages binaries for envtest. It can be used to download new binaries, list currently installed and available ones, and clean up versions [@&#8203;arosh](https://github.com/arosh)
[#&#8203;34908](aquaproj/aqua-registry#34908) [in-toto/witness](https://github.com/in-toto/witness): Witness is a pluggable framework for software supply chain risk management.  It automates, normalizes, and verifies software artifact provenance

#### Re-scaffold

Re-scaffolded over 130 packages to improve the quality.

### [`v4.349.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.349.0)

[Compare Source](aquaproj/aqua-registry@v4.348.0...v4.349.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.349.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.349.0) | aquaproj/aqua-registry@v4.348.0...v4.349.0

#### 🎉 New Packages

[#&#8203;34634](aquaproj/aqua-registry#34634) [samuel-lucas6/Kryptor](https://github.com/samuel-lucas6/Kryptor): A simple, modern, and secure encryption and signing tool that aims to be a better version of age and Minisign

#### Fixes

[#&#8203;34641](aquaproj/aqua-registry#34641) ouch-org/ouch: Follow up changes of ouch 0.6.0
[#&#8203;34642](aquaproj/aqua-registry#34642) fish-shell/fish-shell: Support fish-shell 4.0.2

### [`v4.348.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.348.0)

[Compare Source](aquaproj/aqua-registry@v4.347.3...v4.348.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.348.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.348.0) | aquaproj/aqua-registry@v4.347.3...v4.348.0

#### 🎉 New Packages

[#&#8203;34623](aquaproj/aqua-registry#34623) [block/goose](https://github.com/block/goose): an open source, extensible AI agent that goes beyond code suggestions - install, execute, edit, and test with any LLM [@&#8203;yanolab](https://github.com/yanolab)

### [`v4.347.3`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.347.3)

[Compare Source](aquaproj/aqua-registry@v4.347.2...v4.347.3)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.347.3) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.347.3) | aquaproj/aqua-registry@v4.347.2...v4.347.3

#### Fixes

[#&#8203;34620](aquaproj/aqua-registry#34620) k1LoW/octocov: Support linux/arm64 [@&#8203;yanolab](https://github.com/yanolab)
[#&#8203;34621](aquaproj/aqua-registry#34621) Re-scaffold k1LoW/octocov

### [`v4.347.2`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.347.2)

[Compare Source](aquaproj/aqua-registry@v4.347.1...v4.347.2)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.347.2) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.347.2) | aquaproj/aqua-registry@v4.347.1...v4.347.2

#### Fixes

[#&#8203;34605](aquaproj/aqua-registry#34605) theseus-rs/postgresql-binaries: Replace musl with gnu

### [`v4.347.1`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.347.1)

[Compare Source](aquaproj/aqua-registry@v4.347.0...v4.347.1)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.347.1) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.347.1) | aquaproj/aqua-registry@v4.347.0...v4.347.1

#### Fixes

[#&#8203;34601](aquaproj/aqua-registry#34601) ouch-org/ouch: Re-scaffold
[#&#8203;34598](aquaproj/aqua-registry#34598) pnpm/pnpm: Create a hard link

### [`v4.347.0`](https://github.com/aquaproj/aqua-registry/releases/tag/v4.347.0)

[Compare Source](aquaproj/aqua-registry@v4.346.1...v4.347.0)

[Issues](https://github.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.347.0) | [Merge Requests](https://github.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.347.0) | aquaproj/aqua-registry@v4.346.1...v4.347.0

#### 🎉 New Packages

[#&#8203;34585](aquaproj/aqua-registry#34585) [foundry-rs/foundry](https://github.com/foundry-rs/foundry) - Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust [@&#8203;jessepinkman9900](https://github.com/jessepinkman9900)
[#&#8203;34580](aquaproj/aqua-registry#34580) [ctron/oidc-cli](https://github.com/ctron/oidc-cli): A command line tool to work with OIDC tokens [@&#8203;takumin](https://github.com/takumin)

#### Improvement

[#&#8203;34595](aquaproj/aqua-registry#34595) Re-scaffold tmknom/actdocs
[#&#8203;34596](aquaproj/aqua-registry#34596) Re-scaffold barnybug/cli53

#### Security

[#&#8203;34543](aquaproj/aqua-registry#34543) zigtools/zls: configure Minisign

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4yNTMuMCIsInVwZGF0ZWRJblZlciI6IjM5LjI1Ny4zIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants