← All tools

xcrawl3r

https://github.com/hueristiq/xcrawl3r

Open Source

Categories

Reconnaissance Web Application

Description

A command-line utility designed to recursively spider webpages for URLs. It works by actively traversing websites - following links embedded in webpages, parsing resources like sitemaps and robots.txt files, and even processing local files - to uncover every URL.

Keywords

bug-bounty bug-bounty-tools contentdiscovery crawler ethical-hacking ethical-hacking-tools go golang penetration-testing penetration-testing-tools reconnaissance red-teaming red-teaming-tools web-security

Example Usage

xcrawl3r -h

See also

awesome-osint-arsenal

Open Source Kali

🔍 Curated OSINT & recon toolkit for Kali Linux — 100+ tools, one-command installer, covering SOCMINT…

-Ultimate-Cybersecurity-Roadmap

Free

Ultimate Cybersecurity Roadmap (2025 Edition) | Beginner to Advanced Guide | Learn Ethical Hacking, …

public-skills-builder

Free

Generate Claude Code bug bounty skills from public HackerOne reports and GitHub writeups — 18 vuln c…

httpx

Open Source Kali

Fast and multi-purpose HTTP toolkit from ProjectDiscovery. Probes hosts for live web services, extra…

dnsx

Open Source

Fast and multi-purpose DNS toolkit from ProjectDiscovery. Resolves domains, performs wildcard filter…

API-Pentesting-Tools

Open Source

API Pentesting Tools are specialized security tools used to test and analyze the security of Applica…