Tools directory

103 open-source tools for scraping, crawling, and OSINT — organized by category.

Recommended stack

A modern combination that covers static sites, JavaScript apps, APIs, PDFs, news, e-commerce, blogs, and AI/RAG pipelines — all free and open source.

Scrapy
Large-scale crawling
Python
Playwright
JavaScript rendering
Multi
Crawlee
Production orchestration
Node/Python
Trafilatura
Clean text extraction
Python
FirecrawlAI
LLM-ready Markdown/JSON
Node
Crawl4AIAI
AI-ready web crawler
Python
Colly
Ultra-fast Go crawler
Go
Katana
URL discovery & recon
Go
103 of 103 tools

Web scraping engines

20 tools

General-purpose crawlers and HTML parsers.

Scrapy
Large-scale web crawling
PythonBSD-3
Playwright
JavaScript-heavy websites
Python/JS/Java/.NETApache 2.0
Puppeteer
Chrome automation & scraping
Node.jsApache 2.0
Crawlee
Production scraping framework
Node/PythonApache 2.0
Beautiful Soup 4
HTML parsing
PythonMIT
lxml
Fast XML/HTML parsing
PythonBSD
Selenium
Browser automation
Multi-languageApache 2.0
Colly
High-speed crawling
GoApache 2.0
Katana
Fast website crawler
GoMIT
Apache Nutch
Distributed crawling
JavaApache 2.0
StormCrawler
Streaming crawlers
JavaApache 2.0
Heritrix
Internet-scale archiving
JavaApache 2.0
MechanicalSoup
Form automation
PythonMIT
Requests-HTML
Simple JS scraping
PythonMIT
PySpider
Distributed scraping
PythonApache 2.0
Newspaper3k
News article extraction
PythonMIT
Trafilatura
Clean article extraction
PythonApache 2.0
Goquery
jQuery-style HTML parsing
GoBSD
Ferret
Declarative scraping
GoApache 2.0
Gocolly Extensions
Colly enhancements
GoApache 2.0

AI-ready scraping

4 tools

Optimized for LLMs, RAG, and structured extraction.

Firecrawl
AI
Crawl websites into Markdown/JSON
Crawl4AI
AI
AI-ready web crawler
ScrapeGraphAI
AI
LLM-powered structured extraction
Browser Use
AI
Browser automation for AI agents

Awesome lists

3 tools

Curated collections of scraping tools and resources.

awesome-web-scraping
Curated scraping tools on GitHub
awesome-scraping
Community-maintained scraping list
awesome-crawler
Crawler-focused resources

Specialized scrapers

6 tools

Purpose-built downloaders for specific platforms.

youtube-dl
Video metadata / downloader
yt-dlp
Actively maintained yt-dl fork
gallery-dl
Image gallery downloader
snscrape
Social media scraping
instaloader
Instagram data
Wayback Machine Downloader
Archive downloads

OSINT frameworks

5 tools

Full frameworks that aggregate many OSINT sources.

OSINT Framework
Category-indexed link directory
Awesome OSINT
Curated OSINT collection on GitHub
Recon-ng
Web reconnaissance framework
SpiderFoot CE
Automated OSINT collection
BBOT
Recursive recon multitool

Username & social media

7 tools

Find accounts by handle across many platforms.

Sherlock
Username search across 400+ sites
Maigret
Sherlock-style profile hunter
WhatsMyName
Enumeration project
Social Analyzer
Profile analysis toolkit
User-Scanner
Multi-platform username scan
Twint
Twitter scraping (legacy)
Instaloader
Instagram profile data

Email investigation

4 tools

Address enumeration and account discovery.

Holehe
Check if email is registered on sites
theHarvester
Email/domain enumeration
GHunt
Google account recon
EmailHarvester
Public email discovery

Domain, IP & DNS

6 tools

Enumerate infrastructure and subdomains.

Amass
Attack-surface mapping
Subfinder
Passive subdomain enumeration
Findomain
Fast subdomain finder
DNSRecon
DNS enumeration
Assetfinder
Related-domain discovery
Katana
Crawl-based URL discovery

Web intelligence

4 tools

Fingerprint hosts, capture screenshots, probe endpoints.

Aquatone
HTTP screenshots at scale
httpx
Fast HTTP toolkit
EyeWitness
Screenshot / info recon
Nuclei
Template-driven scanning

Phone numbers

2 tools

Enrich and validate telephone numbers.

PhoneInfoga
OSINT for phone numbers
NumVerify
Number lookup (free tier)

Geolocation

4 tools

Place, map, and satellite intelligence.

GeoSpy
Image-to-location AI
GeoNames
Global place-name database
OpenStreetMap
Community-maintained maps
Google Earth Pro
Free 3D imagery

Image & metadata

4 tools

EXIF, OCR, and image cleanup utilities.

ExifTool
Read/write metadata
ExifCleaner
Strip metadata
OCRmyPDF
Add OCR text layer to PDFs
Tesseract OCR
Open-source OCR engine

Video & YouTube

3 tools

Media download and archiving.

yt-dlp
Modern yt-dl fork
youtube-dl
Original downloader
gallery-dl
Gallery / image scraper

Search engines

7 tools

General and privacy-focused engines for pivots.

Google
Advanced dorking
Bing
Alternate index
DuckDuckGo
Privacy-first
Brave Search
Independent index
Yandex
Strong reverse image
Mojeek
Independent crawler
SearXNG
Self-hosted meta-search

Documents

3 tools

Extract metadata and secrets from documents.

FOCA
Metadata analysis
Metagoofil
Public doc harvester
truffleHog
Secret scanning

Cloud & Git

3 tools

Find leaked credentials in repos and buckets.

GitLeaks
Secret scanning for repos
TruffleHog
Deep secret search
Gitrob
Sensitive-file discovery

Threat intelligence

4 tools

Share and match indicators of compromise.

MISP
Threat sharing platform
YARA
Pattern matching for malware
OpenCTI
CTI knowledge platform
OpenIOC
IOC framework

Dark web

4 tools

Access and index Tor-hidden services.

Tor Browser
Anonymous browsing
Ahmia
Hidden service search
OnionSearch
Onion aggregator
OnionScan
.onion site scanner

Blockchain

2 tools

On-chain analysis and tracing.

BlockSci
Blockchain analysis
Orbit
Address graph tracing

Digital forensics

4 tools

Disk, memory, and timeline analysis.

Autopsy
Disk forensics GUI
Volatility 3
Memory analysis
Plaso
Timeline construction
Bulk Extractor
Feature extraction

Utility

4 tools

Data-wrangling and transformation helpers.

jq
JSON on the command line
csvkit
CSV toolkit
OpenRefine
Data cleanup
CyberChef
Encoding/decoding swiss army
Names and descriptions link back to the maintainers' own repositories and sites.