Automation Frameworks

Learning Objectives

• After this, students can describe what theHarvester, SpiderFoot, Recon-ng, and Maltego CE each do and when to reach for each.
• After this, students can run a basic collection with theHarvester and Recon-ng on an authorized target.
• After this, students can explain the role of API keys in these frameworks.
• After this, students can articulate why automation never replaces human analysis.

Why Frameworks Exist

The earlier files in this module taught individual techniques: dorking, username enumeration, DNS and certificate lookups, Wayback mining. Automation frameworks stitch many of those into one run — you give a domain, and the tool queries dozens of sources and assembles the results. They save enormous time and reduce the chance you forget a source. What they do not do is judge, verify, or understand. A framework produces a pile of candidate data; turning that into intelligence is still your job (Module 7).

All four tools below are pre-installed or one command away on Kali Linux. Point them only at domains you own or are authorized to assess (Module 2).

theHarvester — fast, focused email/subdomain/host gathering

theHarvester is the quickest way to answer “what emails, subdomains, and hosts are publicly associated with this domain?” It queries many search engines and data sources and prints a consolidated list. It is passive and lightweight — a great first pass.

Basic run (search DuckDuckGo, limit results):

theHarvester -d companyx.com -l 500 -b duckduckgo

theHarvester -d companyx.com -b crtsh,duckduckgo,otx,rapiddns -f companyx-report

The -b source list is long and includes crtsh, duckduckgo, bing, otx, rapiddns, hackertarget, certspotter, urlscan, virustotal, github-code, shodan, hunter, intelx, securityTrails, and many more. Some sources (Shodan, Hunter, VirusTotal, SecurityTrails, IntelX…) need an API key configured before they’ll return data.

SpiderFoot — broad automated correlation

SpiderFoot is the “cast a wide net” tool. It has 200+ modules that pull from an enormous range of sources and, crucially, correlate results — feeding one finding (a domain) into modules that discover related findings (subdomains, emails, breaches, IPs, linked accounts) automatically. It runs as a local web app you drive from the browser.

Start the web UI:

# on Kali it's packaged:
spiderfoot -l 127.0.0.1:5001
# then open http://127.0.0.1:5001

git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot
pip3 install -r requirements.txt
python3 ./sf.py -l 127.0.0.1:5001

spiderfoot -s companyx.com -u passive -o json

Recon-ng — a modular framework you drive like a console

Recon-ng feels like Metasploit for OSINT: an interactive console with a marketplace of modules, a workspace-based database that stores results, and reporting modules that export them. It rewards a deliberate, step-by-step workflow.

Start it in a named workspace (keeps each engagement’s data separate):

recon-ng -w companyx

[recon-ng][companyx] > marketplace search subdomains
[recon-ng][companyx] > marketplace install recon/domains-hosts/certificate_transparency
[recon-ng][companyx] > modules load recon/domains-hosts/certificate_transparency
[recon-ng][companyx][certificate_transparency] > options set SOURCE companyx.com
[recon-ng][companyx][certificate_transparency] > run

[recon-ng][companyx] > show hosts
[recon-ng][companyx] > db query SELECT DISTINCT host FROM hosts
[recon-ng][companyx] > marketplace install reporting/html
[recon-ng][companyx] > modules load reporting/html
[recon-ng][companyx][html] > run

Maltego CE — visual link analysis

Maltego is different in kind: it’s a graph tool. You drop an entity on a canvas (a domain, email, person, IP) and run transforms that expand it into connected entities, building a visual map of relationships. Seeing the links laid out often reveals connections a text list hides — which accounts share an email, which hosts share an IP, how a person connects to an organization.

Maltego offers a free Community Edition (CE) (the current free tier is called the Basic plan). It requires registration and is capped — CE limits how many entities a transform returns and is for non-commercial use, so it’s fine for learning but not for large engagements. Users who register with an official government/organizational email may qualify for an upgraded free tier with more monthly credits. Maltego’s data sources (“hubs”/transforms) range from free community ones to paid subscriptions.

Choosing a Tool

In practice they combine: theHarvester or SpiderFoot for the fast wide pass, Recon-ng for structured collection you’ll report from, Maltego to make sense of the connections.

API Keys: the Force Multiplier

Every framework here gets dramatically more powerful with API keys. Sources like Shodan, VirusTotal, Hunter.io, SecurityTrails, and HaveIBeenPwned expose far more through their APIs than through their websites, and the frameworks are built to plug those in. Many offer free tiers with limited monthly queries — enough for class. Manage keys carefully: they are credentials. Don’t commit them to Git, don’t paste them into shared docs, and store them in each tool’s key store (keys add in Recon-ng, the settings/config file in SpiderFoot and theHarvester). This is an OPSEC and hygiene point (Module 8).

Why Humans Still Do the Analysis

Automation is collection, not intelligence. Frameworks:

• Produce false positives. A subdomain list includes dead hosts; a username hit may be a different person. You verify (Module 7).
• Lack judgment. A tool can’t tell which of 300 findings actually matters to the objective. You prioritize.
• Miss context. The insight — “these three facts together mean the CFO’s assistant is the phishing path” — comes from a person connecting dots, not a scanner.
• Can drift out of scope. An aggressive tool run can wander into assets or people you’re not authorized to touch. You set and enforce the boundaries (Module 2).

Treat frameworks as tireless research assistants: they gather fast and never forget a source, but you decide what the pile means. The methodology in Module 3 — plan, collect, process, analyze, report — is the human loop that these tools plug into at the “collect” step.

Key Terms

• Framework: a tool that automates and orchestrates many collection techniques and sources.
• Module / transform: a single automated action within a framework (Recon-ng “module”, Maltego “transform”).
• Workspace: an isolated database in Recon-ng that stores one engagement’s results.
• Use case (SpiderFoot): a scan profile (all/footprint/investigate/passive) that controls which modules run.
• API key: a credential that unlocks a data source’s fuller results for a framework.
• Community Edition (CE): Maltego’s free, registration-gated, capped, non-commercial tier.

Key Takeaways

• Frameworks automate the collection step — they don’t verify, prioritize, or understand.
• theHarvester = fast emails/subdomains/hosts; SpiderFoot = broad correlation; Recon-ng = structured, database-backed workflow; Maltego CE = visual link analysis.
• Pick SpiderFoot’s passive use case (and passive sources generally) to avoid touching the target.
• API keys turn these tools from useful to powerful — and they’re credentials, so guard them.
• Automation produces candidate data; humans turn it into intelligence.
• Point every tool only at authorized targets — automation makes scope creep easy.

References

• theHarvester — Kali Tools
• SpiderFoot — smicallef (GitHub)
• SpiderFoot — Kali Tools
• Recon-ng — Kali Tools
• Recon-ng wiki: Features
• Maltego pricing / editions