Ultimate Gau Cheat Sheet

GetAllUrls - Fetch known URLs from AlienVault's Open Threat Exchange, Wayback Machine, Common Crawl, and URLScan.

Jump To Section

• Basic Usage
• Data Providers
• Filters & Processing
• Output Formats
• Advanced Features
• Automation Scripts

1. Basic Usage

Fetch historical URLs from multiple sources.

Single Domain

gau example.com

With Subdomains

gau --subs example.com

Multiple Domains

gau example.com example.org example.net

From File

cat domains.txt | gau

Save to File

gau example.com --o urls.txt

Verbose Output

gau --verbose example.com

Basic Flags

• --subs : Include subdomains
• --o <file> : Output file
• --verbose : Verbose mode
• --version : Show version

Quick Examples

• gau tesla.com
• gau --subs google.com
• gau apple.com --o urls.txt
• cat domains.txt | gau --subs

2. Data Providers

All Providers (Default)

gau example.com

Specific Provider

gau --providers wayback example.com

Multiple Providers

gau --providers wayback,otx,commoncrawl example.com

Exclude Provider

gau --providers wayback,otx --exclude urlscan example.com

Provider List

gau --providers wayback,otx,commoncrawl,urlscan example.com

Provider	Description	Data Type
wayback	Wayback Machine	Historical URLs
otx	AlienVault OTX	Threat intel URLs
commoncrawl	Common Crawl	Web crawl data
urlscan	URLScan.io	Scanned URLs

Pro Tip: Provider Selection
Use all providers for maximum coverage:
gau --providers wayback,otx,commoncrawl,urlscan example.com
For speed, use only Wayback Machine:
gau --providers wayback example.com

3. Filters & Processing

Filter by Extension

gau example.com | grep -E "\.(php|asp|aspx|jsp)$"

Filter Parameters

gau example.com | grep "?" | sort -u

Filter JavaScript

gau example.com | grep "\.js$" | sort -u

Filter API Endpoints

gau example.com | grep -E "(api|v[0-9]|endpoint)" | sort -u

Extract Unique Paths

gau example.com | cut -d"?" -f1 | sort -u

Extract Parameters

gau example.com | grep -oE "\?[a-zA-Z0-9_=&-]+" | sort -u

Common Filters

• grep "\.js$" : JavaScript files
• grep "?" : URLs with params
• grep -E "(api)" : API endpoints
• sort -u : Deduplicate

Processing Tools

• cut : Extract fields
• awk : Text processing
• sed : Stream editing
• jq : JSON processing

4. Output Formats

Default Output

gau example.com

JSON Output

gau --json example.com

JSON with Output File

gau --json --o results.json example.com

Process JSON Output

gau --json example.com | jq -r '.[].url'

Extract URLs from JSON

gau --json example.com | jq -r '.[] | "\(.url) \(.provider)"'

Count URLs

gau example.com | wc -l

Output Flags

• --o <file> : Save output
• --json : JSON format
• --verbose : Verbose mode
• --version : Version info

JSON Fields

• url : Found URL
• provider : Source provider
• timestamp : Discovery time
• domain : Domain name

5. Advanced Features

With Threads

gau --threads 10 example.com

With Retries

gau --retries 3 example.com

With Timeout

gau --timeout 30 example.com

With Proxy

gau --proxy http://127.0.0.1:8080 example.com

With User-Agent

gau --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1)" example.com

With Cookies

gau --cookie "session=abc123" example.com

Flag	Description	Example
--threads <num>	Concurrent requests	--threads 10
--retries <num>	Retry attempts	--retries 3
--timeout <sec>	Request timeout	--timeout 30
--proxy <url>	Proxy server	--proxy http://ip:port
--user-agent <ua>	Custom UA	--user-agent "UA"
--cookie <cookies>	Session cookie	--cookie "a=b"

6. Automation Scripts

URL Discovery Pipeline

#!/bin/bash # url-discovery.sh - Complete URL discovery DOMAIN=$1 OUTPUT="urls-$DOMAIN" mkdir -p $OUTPUT echo "[+] Fetching URLs with Gau..." gau --subs "$DOMAIN" > $OUTPUT/gau-urls.txt echo "[+] Fetching URLs with Katana..." katana -u "https://$DOMAIN" -silent > $OUTPUT/katana-urls.txt echo "[+] Combining and deduplicating..." cat $OUTPUT/*.txt | sort -u > $OUTPUT/all-urls.txt echo "[+] Found $(wc -l < $OUTPUT/all-urls.txt) unique URLs" echo "[+] Results in $OUTPUT/"

Parameter Extraction

#!/bin/bash # param-extract.sh - Extract URLs with parameters DOMAIN=$1 echo "[+] Fetching URLs..." gau --subs "$DOMAIN" | grep "?" | sort -u > parameterized-urls.txt echo "[+] Extracting parameter names..." grep -oE "[?&][a-zA-Z0-9_]+=" parameterized-urls.txt | sed 's/[?&]//;s/=//' | sort -u > parameter-names.txt echo "[+] Found $(wc -l < parameter-names.txt) unique parameters" cat parameter-names.txt

JavaScript Endpoint Finder

#!/bin/bash # js-endpoints.sh - Find endpoints in JavaScript files DOMAIN=$1 echo "[+] Fetching JavaScript files..." gau --subs "$DOMAIN" | grep "\.js$" | sort -u > js-files.txt echo "[+] Analyzing JavaScript files..." while read -r jsfile; do curl -s "$jsfile" | grep -oE "(api|endpoint|url|path)[\"']?\s*[:=]\s*[\"'][^\"']+[\"']" >> js-endpoints.txt done < js-files.txt echo "[+] Found $(wc -l < js-endpoints.txt) potential endpoints" sort -u js-endpoints.txt -o js-endpoints.txt

XSS Target Discovery

#!/bin/bash # xss-targets.sh - Find potential XSS targets DOMAIN=$1 echo "[+] Fetching URLs with parameters..." gau --subs "$DOMAIN" | grep "?" | sort -u > xss-targets.txt echo "[+] Testing for XSS..." while read -r url; do dalfox url "$url" --silence --format json >> xss-results.json done < xss-targets.txt echo "[+] XSS testing complete" jq -r '.[] | "\(.type): \(.param) at \(.url)"' xss-results.json 2>/dev/null

Full Recon Pipeline

#!/bin/bash # full-recon.sh - Complete reconnaissance DOMAIN=$1 OUTPUT="recon-$DOMAIN" mkdir -p $OUTPUT # Subdomain enumeration echo "[+] Enumerating subdomains..." subfinder -d "$DOMAIN" -o $OUTPUT/subs.txt assetfinder --subs-only "$DOMAIN" >> $OUTPUT/subs.txt sort -u $OUTPUT/subs.txt -o $OUTPUT/subs.txt # Check live hosts echo "[+] Checking live hosts..." cat $OUTPUT/subs.txt | httprobe > $OUTPUT/live.txt # Fetch historical URLs echo "[+] Fetching historical URLs..." cat $OUTPUT/subs.txt | gau --subs > $OUTPUT/historical-urls.txt # Crawl live hosts echo "[+] Crawling live hosts..." katana -list $OUTPUT/live.txt -silent > $OUTPUT/crawled-urls.txt # Combine all URLs echo "[+] Combining URLs..." cat $OUTPUT/historical-urls.txt $OUTPUT/crawled-urls.txt | sort -u > $OUTPUT/all-urls.txt # Extract interesting endpoints echo "[+] Extracting interesting endpoints..." grep -E "(admin|api|upload|login|dashboard|config)" $OUTPUT/all-urls.txt > $OUTPUT/interesting.txt echo "[+] Complete! Results in $OUTPUT/"

Continuous URL Monitoring

#!/bin/bash # monitor.sh - Monitor for new URLs DOMAIN=$1 PREVIOUS="previous-urls.txt" CURRENT="current-urls.txt" while true; do gau --subs "$DOMAIN" | sort -u > "$CURRENT" if [ -f "$PREVIOUS" ]; then echo "[+] New URLs discovered:" diff "$PREVIOUS" "$CURRENT" | grep "^>" | sed 's/^> //' fi mv "$CURRENT" "$PREVIOUS" sleep 86400 done

Pro Tip: Integration with Other Tools

Combine Gau with other recon tools for comprehensive coverage:
gau --subs example.com | sort -u | tee urls.txt
Then check live URLs:
cat urls.txt | httpx -status-code -content-length
And scan for vulnerabilities:
cat urls.txt | nuclei -t ~/nuclei-templates/

⇧