How to find orphan pages in Screaming Frog

Screaming Frog is a link-following crawler. It starts at the URL you give it, extracts every href on the page, queues those, and repeats. That is the same discovery model Googlebot uses, and it has the same blind spot: a URL that nothing links to is unreachable by following links, no matter how long you let the crawl run.

So the crawl frontier and the orphan set are mutually exclusive by construction. Everything the spider found has at least one inbound internal link — that is how it was found. If you want the pages it didn't find, you need a list of URLs that exists independently of your internal linking, and then you subtract.

All three sources have to be configured before the crawl starts, not after. Screaming Frog needs them present during collection so it can attribute each URL correctly.

XML sitemap

1. Open Configuration → Spider → Crawl.
2. Under XML Sitemaps, tick “Crawl Linked XML Sitemaps”.
3. If your sitemap is not referenced in robots.txt, tick “Crawl These Sitemaps” and paste the URL directly.

This is the source most people already have, and it is the weakest of the three. A sitemap is generated by your CMS from the same database the site is built from, so it tends to list pages your CMS knows about — which usually includes the orphans, but also includes URLs that were never meant to be indexed.

Google Analytics and Search Console

1. Open API Access → Google Analytics (or Google Search Console).
2. Connect the account and select the correct property and view.
3. Set the date range wide — 12 months catches seasonal pages a 30-day window misses.
4. For Search Console, confirm the property matches the protocol and subdomain you are crawling; a mismatch silently returns nothing.

These two sources are stronger than the sitemap because they are behavioural rather than declarative: a URL appears because a real person landed on it or because it drew an impression in search. A page that gets traffic while having no internal links is the highest-value orphan you can find — it is already earning, and it is earning despite your structure rather than because of it.

Crawl Analysis is a separate post-processing pass, and this is the step people skip. When the crawl finishes, the orphan filters stay empty until you run it.

1. Wait for the crawl to reach 100%.
2. Open Crawl Analysis → Configure.
3. Tick Sitemaps, Analytics, and Search Console (the sources you connected).
4. Optionally tick “Auto Analyse At End of Crawl” so you never forget again.
5. Click Start and wait for the second pass to finish.

Results appear per source, not in one combined list. Each tab has an “Orphan URLs” filter in the dropdown at the top:

• chevron_rightSitemaps tab → Orphan URLs — in your sitemap, never reached by the crawl.
• chevron_rightAnalytics tab → Orphan URLs — received sessions, never reached by the crawl.
• chevron_rightSearch Console tab → Orphan URLs — drew impressions or clicks, never reached by the crawl.

Export each separately and deduplicate in a spreadsheet. A URL appearing in all three is a confirmed orphan that is also earning traffic — start there. A URL appearing only in the sitemap is the weakest signal and the most likely false positive.

  source            proves                          orphan signal
  ---------------   -----------------------------   -------------
  XML sitemap       your CMS knows the URL          weak
  Analytics         a human reached it              strong
  Search Console    Google indexed and served it    strongest

Not every URL in the orphan filter is a problem, and treating the export as a to-do list wastes a day. Filter these out first:

• chevron_rightPaginated URLs (?page=2) and faceted parameter URLs — usually intentional, often deliberately unlinked.
• chevron_rightRedirected URLs still sitting in a stale sitemap — fix the sitemap, not the link graph.
• chevron_rightPages blocked in robots.txt — the crawl could not reach them for a different reason entirely.
• chevron_rightJavaScript-rendered navigation — if your menu builds client-side and you crawled in text-only mode, everything below it looks orphaned. Re-crawl with JavaScript rendering enabled before believing the list.
• chevron_rightLogin-gated or noindexed utility pages that were never meant to be linked.

The Google API connections require a paid Screaming Frog licence, and plenty of people hitting this problem do not have one. Sitemap-only orphan detection still works — it is just weaker, and it is worth knowing exactly how.

Configure the sitemap as described above, crawl, run Crawl Analysis with only Sitemaps ticked, then read the Sitemaps tab. What you get is every URL your CMS declared that the crawl could not reach by following links. That is a genuine orphan list. What you lose is the ability to tell which of those orphans anyone actually visits, so you cannot rank the fixes by value — you get the what without the so-what.

Supplying your own URL list instead

If your sitemap is unreliable — plenty are incomplete, stale, or exclude entire post types — you can substitute a list exported straight from your CMS or database. Any list of URLs that exists independently of your internal linking works as the source of truth for the subtraction.

1. Export every published URL from your CMS (a database query, an admin export, or a platform API).
2. Crawl the site normally in Spider mode, starting from the homepage.
3. Export the Internal tab as your reached set.
4. Diff the two lists in a spreadsheet. Everything in the export and absent from the crawl is an orphan candidate.

Orphan pages are not a one-off cleanup. Every publishing batch, plugin update and template change creates more, so a single audit is a snapshot with a short shelf life.

Screaming Frog's scheduling (File → Scheduling) can run the crawl on a timer, export the tabs you choose, and — importantly — auto-run Crawl Analysis if you enabled that in the configuration. Point the exports at a dated folder and you get a comparable series rather than a single reading.

The number to watch is not the orphan count on its own. It is the orphan count as a share of total URLs, tracked over time. A site that publishes steadily will always create new orphans; what matters is whether your remediation is outpacing the rate you create them. See how often to audit internal links for a sensible cadence.

The orphan export tells you a URL was never reached. It does not tell you whether the URL still works, redirects, or was deleted six months ago — and a dead URL in a stale sitemap is a sitemap problem, not a linking one. List mode resolves this in one pass.

1. Switch Mode → List.
2. Paste or upload your deduplicated orphan candidates (Upload → Paste, or Enter Manually).
3. Crawl. Because list mode does not follow links, every row is the URL you supplied.
4. Sort by Status Code and Indexability.

Now the list splits cleanly. A 200 that is indexable is a genuine orphan worth linking. A 301 means the sitemap is stale — regenerate it. A 404 means the URL should be removed from wherever it is still listed. A 200 marked non-indexable by a noindex tag was probably meant to be unlinked, and belongs in the ignore pile.

Screaming Frog draws a binary line — reached or not reached — and that line understates the problem. Sort the Internal tab by the Inlinks column ascending and you will usually find a much larger group sitting at one or two inbound links, every one of them from a sitewide footer or a paginated archive.

Functionally those pages behave like orphans. A single link from page 9 of a blog archive passes almost nothing, and a footer link that appears on every page passes less per link than a contextual one in a paragraph — the distinction covered in contextual vs navigation links. A site with 4 true orphans and 180 pages on one archive link does not have a small orphan problem; it has a large distribution problem that the orphan filter is not built to show.

  inlinks   what it usually means               priority
  -------   ---------------------------------   --------
  0         true orphan                         high
  1         archive pagination or footer only   high
  2-3       one nav path, no editorial links    medium
  4+        genuinely supported                 low

You now have a list of URLs with no internal links. The next question is the one the export cannot answer: which page should link to each of them, with what anchor text, and which fix matters most?

Answering that means modelling how authority moves through your link graph — which of your existing pages actually has authority to pass, and which target would gain most from receiving it. That is a different computation from listing unreached URLs, and it is the gap between a crawl export and a plan. RankForge does the same subtraction against its own crawl and sitemap, then ranks the fixes by how much authority each link would move, naming the source page, target page, anchor and paragraph.

Once you have the list, how to fix orphan pages covers the remediation order — and it is not simply “add a link from the sitemap page to everything”.