A useful technical SEO audit determines whether priority URLs can be crawled, return appropriate responses, expose indexable content, and present consistent canonical, rendered, mobile, and performance signals. Start by comparing evidence per URL rather than chasing a site-wide health score. A product page can return 200 and appear in a sitemap while carrying a noindex directive. Each observation is valid, but together they reveal a policy conflict that requires a decision.
Google describes Search through crawling, indexing, and serving. JavaScript pages may also be queued for rendering after crawling, and that work can happen later. Technical eligibility does not guarantee crawling, indexing, rankings, or visibility in AI-generated answers. The audit should make evidence, impact, and conflicts clear enough for an accountable next step. See Google’s overview of how Search works and its minimum technical requirements.
This workflow is an implementation design for collecting and reconciling evidence. It does not claim a direct API connection, automated deployment, or guaranteed indexing outcome.
An audit should show what each important URL returned, declared, and exposed, not merely whether a tool labeled the site healthy.
What should a technical SEO audit establish?
For each URL that matters to the business, establish four things: whether compliant crawlers can access it, whether it returns an appropriate HTTP response, whether its content and directives allow indexing, and whether its canonical, rendered, and mobile signals agree with the intended page. These are related checks, not a guarantee of search visibility.
Set the scope before crawling. Select priority page types and URLs such as product pages, key category pages, revenue landing pages, and high-value articles. Record why each URL matters. A 404 on an intentionally retired campaign page may be acceptable; the same response on a current revenue page needs investigation. Assess findings by URL and business impact rather than compressing them into one site-wide score.
Build an auditable URL inventory
Assemble candidate URLs from a crawler, XML sitemaps, internal links, Search Console exports, server logs where available, and manually selected priority pages. Preserve every discovery source. A URL found in a sitemap but never through internal links tells a different story from one linked throughout the site. Google recommends that sitemaps contain the canonical URLs a site owner wants shown in search. A sitemap communicates preferences but does not guarantee crawling or indexing. See Google’s sitemap guidance.
Normalize hostnames and protocols for comparison, but inspect query parameters before removing them. Parameters can be tracking noise, or they can change the page content. For each URL observation, record the requested URL, final URL, HTTP status, redirect chain, canonical target, robots meta or X-Robots-Tag, robots.txt access result, sitemap membership, discovery source, page type, business priority, and crawl timestamp. Retain initial HTML and rendered DOM evidence where rendering could affect important content, directives, or links.
Keep different data grains separate. Store one record for the audit run, one URL observation per run, one issue observation per URL and issue type, and one remediation event per approved change. Preserve historical observations for comparison. A URL can have several issues at once, so avoid a single issue flag that erases detail.
Resolve crawlability and indexability conflicts
Robots.txt, noindex, canonical tags, redirects, and sitemap inclusion answer different questions. Robots.txt controls crawling for compliant crawlers; it is not a privacy or access-control mechanism. A noindex directive tells a crawler that can access the page not to index it. A canonical signals which URL is preferred among duplicate or similar pages. A redirect sends visitors and crawlers to another URL. Google weighs canonical signals and may select a different canonical from the one a site indicates.
Check these signals together. A URL in a sitemap that returns 200, is indexable, and self-canonicalizes is structurally consistent. A sitemap URL with noindex or a redirect conflicts with the sitemap’s stated purpose and should be reviewed. A URL that canonicalizes elsewhere is not automatically defective because it may represent an intentional duplicate policy. A blocked URL may prevent Google from seeing its noindex directive, so robots.txt and noindex are not interchangeable.
| Observed state | Interpretation | Review action |
|---|---|---|
| Sitemap, 200, indexable, self-canonical | Consistent signals | Keep under normal monitoring. |
| Sitemap and noindex | Indexability conflict | Confirm policy, then remove it from the sitemap or revise the approved directive. |
| Sitemap URL redirects | Stale sitemap entry | Review the destination and update the sitemap to the intended canonical URL. |
| 200 response, canonical points elsewhere | Possible duplicate policy | Review content relationship, internal links, and intended canonical before changing anything. |
| Robots.txt blocks the URL | Google may not inspect page directives | Confirm the intended crawl and index policy before changing access rules. |
If the goal is to keep a crawlable page out of search, verify the noindex approach. Blocking the page in robots.txt can prevent Google from seeing that directive, so a crawl block is not a substitute for indexation control.
A 301 redirect is a strong canonicalization signal, not an absolute guarantee that Google will select the destination. Simplify unnecessary chains, but do not treat three hops as a universal failure threshold. Googlebot generally follows up to 10 redirect hops for general web crawling, while specific tools and products may behave differently. A 404 is not automatically an SEO defect. If a page is intentionally gone and has no relevant replacement, leaving it gone can be appropriate. Do not redirect it to an unrelated homepage by default. See Google’s canonicalization guidance and its HTTP status and redirect documentation.
Check what search engines can render
JavaScript use alone does not prove that a page is unindexable or require a framework migration. Google may queue a crawled page for rendering, and rendering can occur later. For important pages, compare the initial response HTML with the rendered DOM. Check critical headings, body copy, product details, internal links, canonical tags, and robots directives.
If a key price or product detail appears only after JavaScript runs, investigate whether the required scripts and resources are accessible, whether rendering produces the expected content, and whether JavaScript errors interfere. Use Google Search Console’s URL Inspection evidence for priority cases. Google’s JavaScript SEO guidance explains the crawl, render, and index process; its link guidance describes crawlable links. Make important content available in initial HTML when practical, but treat server-side rendering and static generation as options rather than universal requirements.
Measure mobile experience and Core Web Vitals correctly
Under mobile-first indexing, Google uses the mobile version of a site’s content for indexing and ranking. Check that important content, links, metadata, robots directives, and structured data remain available on mobile. A responsive layout is not enough if the mobile version omits a key section or link.
Google’s current good-experience thresholds are LCP under 2.5 seconds, INP under 200 milliseconds, and CLS under 0.1. These are experience thresholds, not guaranteed ranking cutoffs. Core Web Vitals are among the signals Google’s ranking systems use, but good scores do not guarantee rankings and Google says there is no single page-experience ranking signal. See its Core Web Vitals guidance and page-experience guidance.
Keep field observations separate from lab tests. Field data reflects observed user experiences over a reporting period. A Lighthouse run is a controlled diagnostic that can help investigate causes. PageSpeed Insights can show available field data and lab diagnostics, but one lab run is not equivalent to a field percentile. Store field metrics by URL or URL group, device, metric, percentile, and period. Store lab runs separately with tool, device profile, and run time. Use field data to assess user impact and lab results to investigate likely causes. Test with PageSpeed Insights.
Turn observations into implementation work
The audit becomes operational when each finding has a defined input, deterministic check, validation gate, destination, and fallback. The following patterns are proposed audit designs, not vendor-published schemas or guaranteed integrations.
| Trigger and input | AI job | Validation and action | Fallback |
|---|---|---|---|
| Scheduled crawl collects a URL from a sitemap, internal link, Search Console export, log, or manual seed. Record status, final URL, canonical, directives, source, page type, and priority. | No AI is needed. Optionally summarize the evidence packet for a reviewer. | Validate URL normalization and source response. Create a separate issue for a sitemap, noindex, redirect, or canonical conflict. Send evidence to the audit database and issue tracker. | If sources disagree, retain all discovery sources and route the URL to the technical SEO owner instead of selecting one automatically. |
| A priority page has different initial HTML and rendered DOM values, such as missing pricing details or internal links. | Summarize the difference only after deterministic comparison identifies the changed fields. | Check browser rendering, blocked resources, console errors, and URL Inspection evidence. Open an engineering investigation with both snapshots. | If the difference is non-critical or caused by an approved interaction, document the exception and monitor it rather than requiring a rendering architecture change. |
| A template or Search Console report supplies field data, while PageSpeed Insights or Lighthouse supplies a lab run. Store device, period, metric, percentile, tool, and run time. | No AI is needed to compare measurements with thresholds. It may summarize approved diagnostic evidence. | Assess user impact from field data and investigate causes with lab data. Keep LCP, INP, and CLS separate and do not combine them into an unqualified score. | If field data is unavailable or the sample is not comparable, label the result as diagnostic only and avoid broad template changes based on one lab run. |
| A separate AI visibility study runs a versioned prompt on a named engine and records the response, cited URLs, locale, and timestamp where available. | Extract candidate citations or summarize visibility observations, with provenance and reviewer status. | Store one prompt-run observation, one record per citation, and an aggregate summary at its reporting-period grain. Do not write generated interpretations to a CRM as verified facts. | If model, prompt, or citation evidence is unavailable, retain the raw response or hash and mark the observation incomplete rather than inferring a citation. |
For the URL workflow, an illustrative observation record might look like this:
{
"crawl_run_id": "run_2026_10_14_01",
"normalized_url": "https://example.com/products/widget",
"requested_url": "https://example.com/products/widget",
"final_url": "https://example.com/products/widget",
"discovered_from": ["sitemap", "internal_link"],
"http_status": 200,
"redirect_chain": [],
"canonical_target": "https://example.com/products/widget",
"robots_meta": "index,follow",
"robots_txt_allowed": true,
"in_xml_sitemap": true,
"page_type": "product",
"business_priority": "high"
}This is one URL observation within one crawl run, not a current-state record, issue record, or remediation event. A separate issue observation should identify the crawl run, normalized URL, issue type, observed-state hash, evidence fields, and recommended review. A separate remediation event should identify the approved change, owner, approval, deployment reference, and post-change crawl.
For recurring runs, use a proposed issue key such as tenant, normalized URL, issue type, and observed-state hash. Enforce uniqueness with a database constraint and use a transactional upsert when concurrent workers can write the same issue. A lookup followed by create is not race-safe. Keep immutable crawl-run observations even if a separate current-state table is maintained.
Prioritize findings and govern remediation
Rank issues using affected URL count, business priority, observed search or conversion impact, severity, and confidence in the diagnosis. A broken link on a high-priority product template may deserve attention before a low-impact issue on an obsolete page. Route approved work to the team’s issue tracker with the URL, evidence, proposed action, owner, and status.
Automate collection, deterministic checks, and low-risk ticket creation. Do not let a crawler or AI summary select or apply a canonical, redirect, noindex, robots.txt, sitemap, hreflang, or content-removal change. Require review before deployment, and escalate high-traffic, revenue, international, regulated, login-related, or conflicting-signal URLs to the relevant product or engineering owner. For teams coordinating ownership and approved work, ClickUp consulting is an optional way to organize tasks; ClickUp does not perform the SEO crawl or validate search signals.
- Confirm the exact affected URLs, page types, traffic or revenue context, and intended policy.
- Verify the destination, canonical, or indexing directive against response, header, HTML, and rendered evidence.
- Check sitemap entries, internal links, hreflang relationships, and mobile content affected by the change.
- Record the accountable owner, approval, deployment reference, and exception decision where relevant.
- Re-crawl after deployment and save before-and-after observations without overwriting the original evidence.
What technical SEO can support in AI search
Accessible HTML, reliable crawling, clear content, and accurate structured data can support machine understanding, but they cannot guarantee that an answer engine will mention or cite a brand. Google’s breadcrumb structured-data guidance is an example of support for a specific search feature, not a promise of a rich result or AI citation.
If a team separately measures AI visibility, keep the records distinct. One prompt run is an observation. Each citation in its response is a separate citation record. A visibility summary is an aggregate over a defined period and prompt set. Store the engine, model or variant where available, prompt version, run time, locale, cited URL, response hash, and source evidence. Treat generated interpretations as reviewable, provenance-backed observations. Public product pages may describe visibility and citation features, but they do not by themselves establish an external API, webhook, export format, database field, or automation workflow.
A strong audit ends with comparable evidence, clear ownership, and verified post-change observations. Its value is not a perfect score. It is knowing which URL signals agree, which conflict, what the evidence supports, and who must decide what happens next.