XFTFSecurity For the Folks

Lesson 5.1

Search Engines

Learning Objectives

  • Explain why different search engines return different results, and why investigators use more than one
  • Describe what makes Google, Bing, DuckDuckGo, Yandex, Baidu, Brave Search, and Mojeek useful for OSINT
  • Recognize recent changes (removed operators, shut-down APIs) that break older tutorials and tools
  • Use the Google Hacking Database as a reference library of proven search queries
  • Understand the risks of relying on AI search summaries instead of the underlying sources

Why Search Engines Are Still the Starting Point

Almost every OSINT investigation starts with a search box. Search engines have already crawled billions of pages, so a few well-built queries can surface documents, profiles, and exposed files that would take days to find by browsing.

The key idea for students: a search engine is not “the internet.” It is one company’s index of the internet: a copy of the pages its crawler visited, ranked by its own algorithm, filtered by its own policies and the laws of the countries it operates in. Two engines with different crawlers will know about different pages. That’s why experienced investigators run the same query across several engines and compare.

There are only a handful of truly independent, large-scale web indexes. Many “alternative” search engines actually show results from Bing or Google with a different interface. When you pick engines to compare, pick ones with different underlying indexes, or you’re just seeing the same results twice.

The Major Engines at a Glance

Engine Own index? Strengths for OSINT Cost
Google Yes Largest index; most useful operators; strong document and file indexing Free
Bing Yes Different index; unique operators like ip: Free
DuckDuckGo Mostly Bing results Privacy (no search history tied to you); “bangs” shortcuts Free
Yandex Yes Russian/Eastern European content; historically the strongest reverse image and face matching Free
Baidu Yes Chinese-language web Free
Brave Search Yes (fully independent since 2023) Independent Western index; privacy-focused Free (paid API)
Mojeek Yes (UK-based) Small independent index that never resold Google/Bing results; surfaces pages big engines bury Free (API available)

Google

  • What it’s good for: the biggest index, the best support for advanced operators, and deep indexing of PDFs, Office documents, and spreadsheets. This is the engine most “Google dorking” is built around.
  • Key operators (full dorking walkthrough is in Module 4):
  • site: limit to a domain or subdomain (site:companyx.com, site:*.companyx.com)
  • filetype: / ext: limit to a file type (filetype:pdf)
  • intitle:, inurl:, intext: match words in the title, URL, or body
  • "exact phrase", -exclude, OR
  • before: / after: limit by date (e.g., after:2025-01-01)
  • Recent changes to know about:
  • The cache: operator is gone. Google removed cached-page links from results in early 2024 and fully retired the cache: operator later that year. Google now points people to the Internet Archive’s Wayback Machine from its “About this result” panel instead. Older tutorials that say “just check Google’s cache” are out of date; use web archives (see 5.6-archives-and-historical-data.md).
  • Operators are not guaranteed. Google quietly changes or drops operators, and some return inconsistent results. Always sanity-check a query by looking at what comes back.
  • Tips:
  • Results are personalized and localized. For consistent results, search while logged out in a clean browser profile (this also helps OPSEC; see Module 8).
  • Country-specific versions (google.de, google.co.jp) and the language/region settings surface regional content that google.com ranks lower.
  • Google Images, Google News, and Google Scholar are separate indexes worth searching on their own.

Bing

  • What it’s good for: a completely separate index from Google, so it regularly finds pages Google doesn’t show.
  • Unique operators:
  • ip: finds pages hosted on a given IP address. This is useful for discovering other websites that share a server with a target (e.g., ip:203.0.113.10).
  • inbody: matches words in the page body.
  • contains: finds pages that link to a given file type.
  • site: and filetype: work much like Google’s.
  • Recent change: Microsoft retired its public Bing Search API in August 2025. Many OSINT tools (and some metadata tools that search for documents through Bing) relied on that API, so tools may silently return fewer results or fail. Test tools before class.
  • Tip: Yahoo and several privacy search engines use Bing’s index, so “checking Yahoo” usually isn’t an independent second opinion.

DuckDuckGo

  • What it’s good for: privacy. It doesn’t build a search history profile on you, which matters when you don’t want your research tied to your identity.
  • Index: its web results come largely from Bing, blended with its own ranking and other sources. Treat it as a privacy-friendly way to see Bing-like results, not as an independent index.
  • Bangs: typing !g, !b, !yandex, !gh (GitHub), !w (Wikipedia), etc. sends your query straight to that site. It’s a fast way to pivot a single query across many sources.

Yandex

  • What it’s good for:
  • Russian-language and Eastern European websites and social platforms, which Western engines index poorly
  • Reverse image search, especially for landmarks, buildings, and faces. OSINT and fact-checking communities have long rated it among the strongest image matchers, partly because it applies fewer privacy filters to face matching than Google does.
  • Ethics and law: the fact that Yandex will match faces does not make it acceptable to identify private people with it. Face matching of non-consenting people is off-limits in this class (see Module 2). Some jurisdictions restrict facial recognition use.
  • OPSEC: Yandex is a Russian company subject to Russian law. Think about what you upload (e.g., don’t upload sensitive client images).

Baidu

  • What it’s good for: China’s dominant search engine, with the best coverage of Chinese-language sites. Useful for researching Chinese companies, products, and supply chains.
  • Tips: search in Simplified Chinese (machine translation of key terms works reasonably well). Results are subject to Chinese censorship, so absence of information on Baidu tells you little.
  • What it’s good for: an independent Western index (it moved off Bing results and became fully independent in 2023), so it’s a genuine third opinion alongside Google and Bing.
  • Access: free to use; the Brave Search API is a paid service that many AI and automation tools now use, since Bing’s API was retired.

Mojeek

  • What it’s good for: a UK-based engine with its own crawler and index that has never resold Google or Bing results. Its index is much smaller, but it ranks pages differently and without personalization, so it sometimes surfaces small or older pages that big engines bury.
  • Access: free; an API is available.

Other Engines Worth Knowing

  • Startpage: shows Google results through a privacy proxy. Good for getting Google-like results without being logged in to Google (not an independent index).
  • Kagi: a paid, ad-free search engine with custom ranking controls (e.g., boosting or blocking specific sites).
  • Regional engines: Naver (South Korea) and Seznam (Czech Republic) are examples of engines that dominate their home markets and index local content better than global engines.
  • Specialized “search engines”: Shodan and Censys (internet-connected devices), crt.sh (certificates), grep.app (source code), and the Wayback Machine (history) are all search engines for specific data types. They’re covered in files 04–06 of this module.

The Google Hacking Database (GHDB)

  • What it is: a free, curated library of “dorks” (search queries that reveal exposed or sensitive information), maintained by OffSec on Exploit-DB since 2010: https://www.exploit-db.com/google-hacking-database.
  • How it’s organized: thousands of community-submitted, reviewed queries in categories such as:
  • Files Containing Passwords / Usernames
  • Sensitive Directories
  • Vulnerable Files and Vulnerable Servers
  • Error Messages
  • Pages Containing Login Portals
  • Various Online Devices (cameras, printers, industrial systems)
  • Network or Vulnerability Data
  • Footholds and Web Server Detection
  • How attackers use it: copy a dork, and they get a list of sites exposing something useful. Dorks are often run at scale to find any vulnerable victim, not a specific one.
  • How defenders use it: add site:yourcompany.com to relevant GHDB dorks to check what your own organization is exposing, then fix it and request removal from search engines.
  • Ethics reminder: a dork that returns someone else’s exposed login page or password file shows you a problem; it does not give you permission to use it. Look, document (if authorized), report; don’t log in or download.

A Word on AI Search and Chatbots

AI assistants and AI search features now answer many questions directly. They’re useful for brainstorming search terms and summarizing, but for OSINT:
- They can hallucinate: confidently state wrong names, dates, or links
- They may mix up people with similar names (a serious harm in person-focused investigations)
- Their summaries are not a source. Always click through to the underlying page, verify it, and cite that page (see Module 7).
- Be careful what you paste in. Queries and uploaded data may be stored by the provider (an OPSEC concern).

Practical Tips

  • Use at least three independent indexes for anything important (e.g., Google, Bing, and Brave or Yandex).
  • Search in the target’s language and region. Translate key terms; use local engines.
  • Log your queries. Record the exact query, engine, and date (see Module 7). Search results change daily, and someone else should be able to repeat your work.
  • Don’t trust result counts. “About 1,200,000 results” is an estimate and is often wildly wrong.
  • Watch for sponsored results and SEO spam that impersonate real organizations.

Key Terms

  • Index: a search engine’s stored copy of the web pages its crawler has visited
  • Crawler (spider/bot): software that automatically visits and downloads web pages for indexing
  • Search operator: a special keyword (like site: or filetype:) that narrows search results
  • Dork / Google dorking: a search query using operators to find exposed or sensitive information
  • GHDB: the Google Hacking Database, a categorized public library of dorks on Exploit-DB
  • Bang (DuckDuckGo): a ! shortcut that sends a query directly to another site’s search

Key Takeaways

  • A search engine is one company’s index of the web, not the whole web
  • Use multiple engines with different indexes: Google, Bing, Brave, Yandex, Mojeek
  • Google’s cache: is gone (2024); use web archives instead
  • Bing’s ip: operator finds other sites on the same server
  • Yandex and Baidu cover regions Western engines miss
  • The GHDB is a free library of proven dorks, useful for defenders auditing their own exposure
  • AI search summaries are not sources; always verify the original page

References

Free to use, copy, change and teach. Security For the Folks dedicates this course to the public domain under CC0 1.0 Universal: no permission or credit needed. Quoted material and the sources it links to belong to their own authors.