Ultimate Hakrawler Cheat Sheet

Samir KC

September 13, 2026 September 13, 2026

Ultimate Hakrawler Cheat Sheet

Fast and simple web crawler for discovering endpoints, URLs, and attack surface from web applications.

1. Basic Crawling

Discover URLs and endpoints from web applications.

Single URL Crawl

echo "https://example.com" | hakrawler

Direct URL Input

hakrawler -url https://example.com

From File

cat urls.txt | hakrawler

Multiple URLs

printf "https://example.com\nhttps://example.org" | hakrawler

Verbose Output

echo "https://example.com" | hakrawler -v

Save Results

echo "https://example.com" | hakrawler > urls.txt

Input Methods

stdin : Pipe URLs
-url <url> : Direct URL
cat file | hakrawler : File input
printf : Multiple URLs

Basic Flags

-v : Verbose mode
-d <depth> : Crawl depth
-s : Show source
-u : Show only unique

2. Depth & Scope

Set Crawl Depth

echo "https://example.com" | hakrawler -d 5

Infinite Depth

echo "https://example.com" | hakrawler -d 0

Scope by Domain

echo "https://example.com" | hakrawler -scope example.com

Scope by Subdomain

echo "https://example.com" | hakrawler -scope "*.example.com"

Exclude Scope

echo "https://example.com" | hakrawler -exclude "*.test.example.com"

Follow Subdomains

echo "https://example.com" | hakrawler -subs

Flag	Description	Example
`-d <num>`	Crawl depth	`-d 5`
`-scope <domain>`	Limit scope	`-scope example.com`
`-exclude <domain>`	Exclude scope	`-exclude test.com`
`-subs`	Include subdomains	`-subs`
`-u`	Unique URLs only	`-u`

Pro Tip: Scope Control
Always define scope to avoid crawling external resources:
echo "https://example.com" | hakrawler -d 5 -scope example.com
Include subdomains for broader coverage:
echo "https://example.com" | hakrawler -d 3 -scope "*.example.com" -subs

3. Headers & Auth

Set Custom Header

echo "https://example.com" | hakrawler -h "Authorization: Bearer TOKEN"

Set Cookie

echo "https://example.com" | hakrawler -h "Cookie: session=abc123"

Set User-Agent

echo "https://example.com" | hakrawler -h "User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1)"

Multiple Headers

echo "https://example.com" | hakrawler -h "Cookie: a=b" -h "X-API-Key: 123"

Use Proxy

echo "https://example.com" | hakrawler -proxy http://127.0.0.1:8080

Set Timeout

echo "https://example.com" | hakrawler -t 30

Insecure Mode (Skip TLS)

echo "https://example.com" | hakrawler -insecure

Header Flags

-h <header> : Custom header
-proxy <url> : Proxy server
-t <sec> : Timeout
-insecure : Skip TLS verify

Auth Examples

Bearer token
Basic auth
Session cookies
API keys

4. Output & Filters

Unique URLs Only

echo "https://example.com" | hakrawler -u

Show Source

echo "https://example.com" | hakrawler -s

Filter JavaScript

echo "https://example.com" | hakrawler | grep "\.js$"

Filter Parameters

echo "https://example.com" | hakrawler | grep "?" | sort -u

Filter API Endpoints

echo "https://example.com" | hakrawler | grep -E "(api|v[0-9]|endpoint)"

Extract Unique Paths

echo "https://example.com" | hakrawler | cut -d"?" -f1 | sort -u

Output Flags

-u : Unique URLs
-s : Show source
-v : Verbose
-json : JSON output

Processing Tools

grep : Filter results
sort -u : Deduplicate
cut : Extract fields
jq : JSON processing

5. Advanced Features

JavaScript Parsing

echo "https://example.com" | hakrawler -js

Extract Links from JS

echo "https://example.com" | hakrawler -js -d 3

Form Discovery

echo "https://example.com" | hakrawler -forms

Robots.txt Parsing

echo "https://example.com" | hakrawler -robots

Sitemap Parsing

echo "https://example.com" | hakrawler -sitemap

All Features Combined

echo "https://example.com" | hakrawler -d 5 -js -forms -robots -sitemap -u

Flag	Description	Use Case
`-js`	Parse JavaScript	Find endpoints in JS
`-forms`	Extract forms	Discover input points
`-robots`	Parse robots.txt	Hidden paths
`-sitemap`	Parse sitemap.xml	All pages
`-json`	JSON output	Structured data

Pro Tip: Comprehensive Crawling
Combine all features for maximum coverage:
echo "https://example.com" | hakrawler -d 5 -js -forms -robots -sitemap -u -subs
This discovers URLs from HTML, JavaScript, forms, robots.txt, and sitemaps.

6. Automation Scripts

Endpoint Discovery Pipeline

#!/bin/bash # endpoint-discovery.sh - Complete endpoint discovery DOMAIN=$1 OUTPUT="recon-$DOMAIN" mkdir -p $OUTPUT echo "[+] Finding subdomains..." subfinder -d "$DOMAIN" | httprobe > $OUTPUT/live-subs.txt echo "[+] Crawling with Hakrawler..." cat $OUTPUT/live-subs.txt | hakrawler -d 5 -js -u > $OUTPUT/endpoints.txt echo "[+] Extracting URLs with parameters..." grep "?" $OUTPUT/endpoints.txt | sort -u > $OUTPUT/parameterized.txt echo "[+] Complete! Results in $OUTPUT/"

JavaScript Endpoint Finder

#!/bin/bash # js-finder.sh - Find endpoints in JavaScript URL=$1 echo "[+] Crawling $URL..." echo "$URL" | hakrawler -js -u | grep "\.js$" > js-files.txt echo "[+] Analyzing JavaScript files..." while read -r jsfile; do curl -s "$jsfile" | grep -oE '["'"'"'](/[a-zA-Z0-9/_?&=-]+)["'"'"']' >> js-endpoints.txt done < js-files.txt echo "[+] Found $(wc -l < js-endpoints.txt) potential endpoints" sort -u js-endpoints.txt -o js-endpoints.txt

Parameter Discovery

#!/bin/bash # param-discovery.sh - Discover URLs with parameters while read -r url; do echo "[+] Crawling $url..." echo "$url" | hakrawler -d 3 -u | grep "?" >> parameterized-urls.txt done < urls.txt echo "[+] Extracting parameter names..." grep -oE "[?&][a-zA-Z0-9_]+=" parameterized-urls.txt | sed 's/[?&]//;s/=//' | sort -u > parameter-names.txt echo "[+] Found $(wc -l < parameter-names.txt) unique parameters"

XSS Target Finder

#!/bin/bash # xss-targets.sh - Find potential XSS targets DOMAIN=$1 echo "[+] Crawling for XSS targets..." echo "https://$DOMAIN" | hakrawler -d 5 -u | grep "?" | while read -r url; do dalfox url "$url" --silence --format json >> xss-results.json done echo "[+] XSS scanning complete" jq -r '.[] | "\(.type): \(.param)"' xss-results.json 2>/dev/null

Full Recon Pipeline

#!/bin/bash # full-recon.sh - Complete reconnaissance DOMAIN=$1 OUTPUT="recon-$DOMAIN" mkdir -p $OUTPUT # Subdomain enumeration echo "[+] Enumerating subdomains..." subfinder -d "$DOMAIN" -o $OUTPUT/subs.txt assetfinder --subs-only "$DOMAIN" >> $OUTPUT/subs.txt sort -u $OUTPUT/subs.txt -o $OUTPUT/subs.txt # Check live hosts echo "[+] Checking live hosts..." cat $OUTPUT/subs.txt | httprobe > $OUTPUT/live.txt # Crawl with Hakrawler echo "[+] Crawling live hosts..." cat $OUTPUT/live.txt | hakrawler -d 5 -js -forms -robots -sitemap -u > $OUTPUT/urls.txt # Fetch historical URLs echo "[+] Fetching historical URLs..." gau --subs "$DOMAIN" > $OUTPUT/historical.txt # Combine all URLs echo "[+] Combining URLs..." cat $OUTPUT/urls.txt $OUTPUT/historical.txt | sort -u > $OUTPUT/all-urls.txt echo "[+] Complete! Results in $OUTPUT/"

Continuous Crawling

#!/bin/bash # continuous.sh - Monitor for new endpoints URL=$1 PREVIOUS="previous.txt" CURRENT="current.txt" while true; do echo "$URL" | hakrawler -d 5 -js -u | sort -u > "$CURRENT" if [ -f "$PREVIOUS" ]; then echo "[+] New endpoints found:" diff "$PREVIOUS" "$CURRENT" | grep "^>" | sed 's/^> //' fi mv "$CURRENT" "$PREVIOUS" sleep 86400 done

Pro Tip: Integration with Other Tools

⇧

Web Application Security