What Is Technical SEO? A Practical Guide for Marketers

Ask ten agencies what technical SEO covers and you'll get ten different invoices. Ask Google, and the list gets a lot shorter. Its technical requirements page names three things a page needs to be eligible for indexing: Googlebot (Google's crawler) isn't blocked from it, the page returns a working response, and the page has indexable content. That's the bar. Everything past it is optimization, and optimization has diminishing returns.

This guide walks through what technical SEO is, how crawling and indexing really work, the exact Core Web Vitals thresholds, and the problems no amount of engineering will solve. You'll also get a list of tactics that Google's own documentation contradicts. Those are the ones you can stop funding.

What You'll Learn

• Google lists just three technical requirements for indexing, and says indexing still isn't guaranteed (Google Search Central).
• Core Web Vitals pass at LCP 2.5 s, INP 200 ms, CLS 0.1, at the 75th percentile.
• robots.txt controls crawling, not indexing. The two get confused constantly.
• Crawl budget guidance is scoped to sites with 1M+ pages. Most sites can skip it.

What is technical SEO, and what does it actually fix?

Technical SEO is the work that makes your pages findable, fetchable, and eligible for indexing. Google Search Central lists exactly three technical requirements: Googlebot isn't blocked, the page works (an HTTP 200 success response), and the page has indexable content. The same page adds a warning worth quoting: "Indexing isn't guaranteed" (Google Search Central).

That's a low, mostly fixed bar. It's the opposite of how technical SEO usually gets sold, which is as an endless surface of things to improve. Once a page clears those three requirements, more technical work doesn't make it more eligible. It's already eligible. What further work buys you is holding that eligibility at scale, across thousands of URLs, template changes, and migrations.

So it helps to split the discipline into four jobs:

• Access. Can crawlers reach and fetch your pages at all?
• Interpretation. Do they see the same content your visitors see?
• Instruction. Do your robots.txt, sitemaps, and canonical tags say what you think they say?
• Experience. Are pages fast and stable enough that you don't lose a close call?

Notice what's missing: relevance, quality, and demand. Google's guidance on helpful content puts it plainly, saying that "SEO can be a helpful activity when it is applied to people-first content, rather than search engine-first content" (Google Search Central). Technical SEO is a multiplier on content that deserves to rank. Multiply zero and you still have zero.

How do crawling, rendering, and indexing work?

Google describes Search as three stages: crawling, indexing, and serving (Google Search Central). For JavaScript-heavy sites, a fourth job hides inside: rendering. Each stage is a separate system with a separate failure mode, and Google is blunt that clearing them all still doesn't force indexing.

Diagnosing SEO problems gets much easier once you know which of the three stages broke. A page that was never crawled and a page that was crawled but not indexed need completely different fixes.

Crawling: can Googlebot fetch the URL?

Crawling is a fetch request, and the HTTP status code you return decides what happens next. Google documents the effects clearly (Google Search Central). A 301 permanent redirect is a strong canonical signal, while a 302 temporary redirect is a weak one. Googlebot follows up to 10 redirect hops before giving up.

404 and 410 are treated the same way, and both lead to de-indexing over time. That surprises people who expect 410 ("gone") to work faster. Server errors matter more: 5xx responses slow crawling and eventually drop URLs from the index. A 429 ("too many requests") counts as a server error too, so aggressive rate limiting can quietly cost you coverage.

Rendering: does Googlebot see what your visitors see?

Googlebot processes JavaScript in three phases: crawl, render, then index. The important detail is that rendering is deferred to a queue and executed by headless Chromium when resources allow (Google Search Central). It isn't instant, and Google publishes no service-level commitment for how long the queue takes.

That has a practical consequence. If your main content, links, or canonical tags only exist after JavaScript runs, they enter Search on a delay you don't control. For a news site or a large catalogue with frequent changes, that delay is a real cost. For a 30-page brochure site that updates twice a year, it usually isn't worth re-platforming over.

Indexing: is the page stored and eligible to serve?

Indexing is where Google analyzes the page and decides whether to store it. Eligibility is not a guarantee, and Google says so on the record. Pages can clear every technical requirement and still be left out, most often because Google judged them duplicative or not worth storing. When that happens, the fix is editorial, not technical.

The three files that control what search engines do

Three instruction layers do most of the work: robots.txt, XML sitemaps, and rel=canonical. Only one of them is a rule. Google calls sitemap submission a hint with no guarantee it downloads the file, and calls rel=canonical a hint it may overrule. robots.txt is binding, but only for crawling.

robots.txt controls crawling, not indexing

This is the single most expensive misunderstanding in technical SEO. Google is explicit that robots.txt is a crawl-management tool, not an index-control tool. A disallowed URL can still be indexed if other sites link to it, and it'll typically appear in results without a description (Google Search Central).

The spec details matter more than most marketers expect. Google parses a maximum of 500 kibibytes of robots.txt and ignores everything past that. Rules apply per host, protocol, and port, so https://example.com and https://shop.example.com need separate files. A 4xx response (except 429) is treated as no robots.txt at all, meaning Googlebot crawls freely. A 5xx response halts crawling for the first 12 hours, then Google falls back to your last good copy for about 30 days while it keeps retrying (Google Search Central).

None of this is Google-specific trivia. RFC 9309, the Standards Track specification for the Robots Exclusion Protocol, defines the file location as scheme://authority/robots.txt, says crawlers should impose a parse limit and that any such limit must allow at least 500 kibibytes, requires crawlers to "assume complete disallow" on 5xx errors, and resolves competing rules by the most octets matched, with allow winning exact ties (IETF). That last rule is why a broad Disallow plus a narrow Allow behaves the way it does.

The trap worth memorizing: if you put a noindex tag on a page and block that same URL in robots.txt, the two cancel each other out. Googlebot never fetches the page, so it never reads the noindex, and the URL can still appear in search results. Google documents this directly (Google Search Central). To remove a page, allow the crawl so Googlebot can actually see the noindex.

XML sitemaps have hard limits and soft power

The sitemap protocol caps a single file at 50,000 URLs or 50MB uncompressed. Past either limit, you split the file and use a sitemap index (sitemaps.org). Only <loc> is required. <lastmod>, <changefreq>, and <priority> are all optional, and the spec calls <changefreq> "a hint and not a command."

Read the spec's own language on <priority>: it is "not likely to influence the position of your URLs." That's the protocol authors telling you the field doesn't do what agencies have sold it as for years. Google adds its own caveat: submitting a sitemap is a hint, with no guarantee it downloads the file or crawls the URLs inside (Google Search Central).

Sitemaps still earn their keep on large sites, sites with weak internal linking, and fresh URLs with few inbound links. On a well-linked 50-page site, the sitemap is housekeeping, not strategy.

Canonical tags are a hint, and signals stack

rel=canonical tells Google which version of a duplicate you prefer. Google may pick a different one anyway (Google Search Central). Its documentation ranks the strength of canonicalization signals: redirects and rel=canonical are both strong, while sitemap inclusion is weak. Signals stack, which is the actionable part.

So don't rely on one tag. Point the redirect, the canonical tag, the internal links, and the sitemap entry at the same URL. When those four agree, Google usually agrees with you. When they contradict each other, Google breaks the tie and you may not like the answer.

What are the Core Web Vitals thresholds?

Three metrics, three numbers. A page passes at Largest Contentful Paint of 2.5 seconds or less, Interaction to Next Paint of 200 milliseconds or less, and Cumulative Layout Shift of 0.1 or less (web.dev). All three are assessed at the 75th percentile of page loads, segmented across mobile and desktop.

Metric	Good	Needs improvement	Poor
LCP (Largest Contentful Paint)	2.5 s or less	Over 2.5 s up to 4.0 s	Over 4.0 s
INP (Interaction to Next Paint)	200 ms or less	Over 200 ms up to 500 ms	Over 500 ms
CLS (Cumulative Layout Shift)	0.1 or less	Over 0.1 up to 0.25	Over 0.25

Source: thresholds per web.dev, "Web Vitals". Core Web Vitals are assessed at the 75th percentile of page loads, segmented across mobile and desktop.

What each metric actually measures

The names are opaque, so here's what's really being timed. LCP measures the render time of the largest image, text block, or video visible in the viewport, relative to when the user first navigated (web.dev). It's a proxy for "when did this page look loaded to a human?"

INP measures latency across every click, tap, and keyboard interaction for the whole page lifecycle, then reports the longest one while excluding outliers (web.dev). Scrolling, hovering, and zooming don't count. INP replaced First Input Delay as a stable Core Web Vital, so treat FID as retired.

CLS measures the largest burst of unexpected layout shift over the page lifecycle, scored as impact fraction multiplied by distance fraction (web.dev). It's the metric that catches ads and images pushing content around under someone's thumb.

Why the 75th percentile?

Because averages hide your worst visits. web.dev explains that the 75th percentile "strikes a reasonable balance," ensuring three out of four visits hit the target while resisting outliers (web.dev). Your median user can be fine while a quarter of your traffic suffers, and the median won't tell you.

The thresholds themselves weren't picked by feel. Google set them against three criteria: grounded in human-perception research, achievable by at least 10% of existing origins, and unified across devices (web.dev). That third criterion is why you don't get an easier mobile target.

What technical SEO can't fix, in Google's own words

There is no single page experience ranking signal. Google's page experience documentation says core ranking systems look at a variety of signals, and that Google "always seeks to show the most relevant content, even if the page experience is sub-par" (Google Search Central). Page experience is a tiebreaker, not a lever.

Read that sentence again if you're about to sign off on a six-figure performance project pitched on rankings. Google's framing is that page experience helps when lots of helpful content already exists and something has to break the tie. If your content isn't in that pool, speed doesn't get you in.

The same page is direct about Core Web Vitals: good scores don't guarantee ranking, because "there's more to great page experience than Core Web Vitals scores alone." So here's the honest list of what technical work will not do for you:

• It won't create relevance. Google shows the most relevant result even when the experience is sub-par.
• It won't fix thin or search-engine-first content. Google's helpful content guidance frames SEO as useful when applied to people-first content.
• It won't manufacture trust. Within E-E-A-T, Google calls trust "the most important member of the family," and notes the others contribute to trust rather than being required individually (Google Search Central).
• It won't guarantee rich results. Correct structured data that passes the Rich Results Test still doesn't guarantee rich results, and marking up content invisible to readers violates Google's guidelines (Google Search Central).

That last one catches teams often. Passing validation and earning a rich result are two different outcomes, and only one of them is under your control.

Which myths can you stop paying for?

Start with crawl budget. Google scopes its crawl budget guide to sites with more than 1 million unique pages changing about weekly, or more than 10,000 unique pages changing daily. If your site is smaller and indexing fine, Google says you don't need the guide (Google Search Central).

"Crawl budget matters for every site." It doesn't. The third qualifying case is a site with many URLs stuck in "Discovered, currently not indexed" in Search Console. If you're not in one of those three buckets, crawl budget work is a line item with no documented payoff. Check the report before you scope the project.

"robots.txt keeps pages out of Google." It's crawl control only. If you need a page gone, you need noindex on a crawlable URL, or authentication, or removal. Blocking is the wrong tool, and combined with noindex it's actively counterproductive.

"Sitemap priority and changefreq help rankings." The sitemaps.org protocol says <priority> is "not likely to influence the position of your URLs." Both fields are optional hints. If someone's billing you to tune them, that's billable time producing nothing measurable.

"A Lighthouse score of 100 improves rankings." Lighthouse produces lab data from a simulated load. Core Web Vitals assessment uses field data at the 75th percentile from real visits. They answer different questions, and Google has never linked a Lighthouse score to ranking. Lighthouse is a useful debugging tool. It isn't a scoreboard.

"Page speed is the Nth most important ranking factor." Neither Google nor any standards body quantifies the weight of any signal, and Google says there's no single page experience signal to weight in the first place. Any specific ranking number you've been shown is someone's estimate wearing a lab coat.

"Google renders JavaScript instantly." Rendering is queued and runs when resources allow. Google says it can take longer and publishes no SLA. Plan for delay rather than assuming parity with your browser.

Where should you start when you can't fix everything?

Fix eligibility before experience. Google's three technical requirements are effectively binary: a blocked, broken, or empty page is worth nothing regardless of how fast it loads. Core Web Vitals only act as a tiebreaker once relevant content already exists (Google Search Central). Work the list in that order.

1. Confirm your revenue pages are indexable. Check the three requirements on the 20 URLs that actually earn money. Not blocked, returning 200, real content in the HTML. This takes an afternoon and catches more damage than a full-site crawl.
2. Check status codes by template, not by sample. One broken template can take out thousands of URLs at once. Look for 5xx responses and 429s first, since those slow crawling and eventually drop URLs.
3. Read robots.txt like a crawler. Verify one file per host, protocol, and port. Look for any URL carrying both Disallow and noindex. Confirm the file is under 500 kibibytes, which matters mainly for large e-commerce sites with generated rules.
4. Make your canonical signals agree. Redirects, canonical tags, internal links, and sitemap entries should all point at the same URL. Contradictions are common after migrations and replatforms, and they're cheap to fix once found.
5. Test whether rendering hides your content. Compare the raw HTML with the rendered page. This is urgent for JavaScript-heavy catalogues and news sites, and close to irrelevant for a small server-rendered site.
6. Then chase Core Web Vitals, in the field. Use field data at the 75th percentile, not a lab score. Prioritize templates by traffic, not by which page is easiest to fix.
7. Skip crawl budget unless you qualify. 1M+ pages, or 10,000+ changing daily, or a pile of "Discovered, currently not indexed" URLs. Otherwise, spend the money on content.

In our experience, the biggest wins on mid-sized sites come from steps 1 through 4, and they cost days rather than quarters. That's a judgment call from repeated audits, not something Google documents. What Google does document is the ceiling: clear the three requirements, keep the instructions consistent, and accept that indexing was never promised to anyone.

Frequently asked questions

Is technical SEO worth paying for on a small site?

Usually a one-time audit is enough, then periodic checks after redesigns. Google's own framing is that you don't need its crawl budget guide if your pages get crawled the same day you publish them. Small sites should verify the three technical requirements, keep canonical signals consistent, and put the remaining budget into content that earns links.

Will blocking a page in robots.txt remove it from Google?

No. Google states that robots.txt is a crawl-management tool, not an index-control tool. A blocked URL can still be indexed if other sites link to it, usually appearing without a description. To remove a page, allow crawling and serve a noindex tag, or use authentication or the removal tools.

Do I need a perfect Lighthouse score?

Google has never connected a Lighthouse score to rankings. Lighthouse reports lab data from a simulated load, while Core Web Vitals are assessed on field data at the 75th percentile of real page loads. Use Lighthouse to diagnose specific problems, then verify improvements in field data before claiming a win.

How long does Google take to render my JavaScript?

Google doesn't publish a number. Its JavaScript SEO documentation says rendering is deferred to a queue and executed by headless Chromium when resources allow, and that it can take longer than crawling. Any vendor quoting a specific rendering time in seconds is guessing. Server-render critical content if timing matters to your business.

Does passing Core Web Vitals guarantee better rankings?

It doesn't. Google says there's no single page experience ranking signal, that "there's more to great page experience than Core Web Vitals scores alone," and that it shows the most relevant content even when page experience is sub-par. Treat passing scores as removing a disadvantage, not as buying an advantage.