A usable content inventory starts with a decision about what one record represents. It then reconciles URLs from several sources, separates stable asset details from dated observations, and gives people clear responsibility for upkeep.
For example, one product guide might have a CMS record, an English URL, a French URL, and an old redirected URL. Treating each URL as a separate asset could distort ownership and reporting. A better model records the logical content asset once and retains each URL as an observation with its own state and source.
This guide shows how to define the grain and scope, build a reliable URL universe, preserve performance evidence, design safe review workflows, select tools by their actual role, and keep the inventory connected to publishing and site-change processes.
What a content inventory records, and what it does not decide
A content inventory is a structured record of content assets and useful attributes such as identity, URL, content type, owner, status, and review dates. A content audit evaluates those records for quality, usefulness, performance, and strategic fit, then recommends actions. Nielsen Norman Group describes inventories and audits as related activities, often with the inventory created first.
An inventory can store audit decisions, but the decision and its evidence should remain traceable. A crawler’s URL list is not automatically a complete inventory, and an inventory does not become an action plan merely because it contains traffic metrics. Publication age can prompt review, but it does not prove that information is obsolete. Check claims, links, purpose, audience, business value, and risk before deciding what to do.
Choose the inventory grain and scope before choosing fields
The inventory grain defines what one record represents. A normalized canonical URL plus a site ID can work for a simple site. Use a CMS asset ID when one managed asset can appear at multiple URLs. For multilingual, syndicated, or parameter-heavy sites, use one logical content item linked to multiple observed URLs.
Titles, H1s, and keywords change or repeat, so they are not dependable unique identifiers. Keep the original observed URL as well as any normalized or canonical URL. This preserves evidence of redirects, alternate renderings, and URL changes. If your CMS has distinct record types, account for them. WordPress documentation distinguishes posts, pages, attachments, revisions, and custom post types, for example.
Write a scope note before crawling or importing. Specify the domains and subdomains, content types, locales, date range, source systems, exclusions, and whether unpublished, noindex, orphaned, redirected, or retired records are included. Include those records when migration, governance, or retirement work needs them, and label both their state and their reason for inclusion. Have the content lead and technical owner approve the scope.
Choose what one record represents before choosing its columns. An unclear grain creates duplicates and unreliable joins that extra fields cannot repair.
Design records that preserve history
Separate relatively stable asset details from information that changes with each crawl, report, or review. A practical starting model has a content asset record, crawl observations, performance observations, and decision events. These can be separate database tables or linked collections. A small project can represent the same separation with distinct spreadsheet tabs.
For the asset record, consider asset_id, site_id, canonical_url, title, content_type, owner, status, business_goal, last_reviewed_at, and next_review_at. Add fields only when they support a real decision or operational responsibility.
- Asset: One logical content item or canonical page. Prefer an internal asset ID. Otherwise use a documented normalized URL with its site or property ID.
- Crawl observation: One URL observed during one crawl run. Keep the run ID, observed URL, status, canonical target, indexability, source, and retrieval time.
- Performance observation: One report row for a defined property, date range, dimension set, metric set, and report configuration.
- Decision event: One review decision with an action, evidence references, approver, rationale, and date.
The following is a proposed schema example, not a vendor template:
{
"asset_id": "asset-00418",
"site_id": "marketing-site",
"canonical_url": "https://example.com/guides/example",
"content_type": "guide",
"owner": "Content operations",
"status": "published",
"last_reviewed_at": "2026-09-15"
}For repeated imports, define a unique key for each record’s grain. Use a database-enforced uniqueness constraint with a transactional upsert where concurrent processing is possible. A lookup followed by a create can race when two workers process the same item at once. If the destination cannot enforce uniqueness, use an idempotency key, serialized queue, or durable processing log.
Store source_system, source_record_id, source_url, retrieval time, reporting dates where applicable, and transformation_version. Provenance makes it possible to explain why a value exists and to reprocess an import without silently overwriting history.
Build the URL universe by reconciling sources
Use a crawler as one discovery source, not as proof of completeness. A crawler finds pages it can reach under its settings. It may miss orphaned URLs, unpublished CMS records, or URLs excluded by crawl rules. Compare its output with XML sitemaps, CMS exports, analytics landing pages, Search Console page data, redirect files, migration lists, and manually maintained URL sources.
Google explains that sitemaps can help with URL discovery, but submitting one does not guarantee crawling or indexing. Import each source with a source label and extraction time. Preserve the source URL, then apply a documented normalization policy.
A normalization policy should define treatment of host casing, default ports, fragments, trailing slashes, query parameters, and redirects. Do not discard parameters until you know whether they identify distinct content. Keep both the original observation and the normalized value so a later reviewer can understand the transformation.
Flag sitemap-only URLs, crawl-only URLs, analytics pages missing from the crawl, non-200 responses, canonical conflicts, and redirect chains. A technical SEO or web-platform owner should resolve URL and crawl conflicts. The inventory steward should handle schema, mapping, and import problems.
Add performance evidence without mixing metric meanings
Choose measures that match the page’s purpose. Search Console can provide clicks, impressions, click-through rate, and average position. Google Analytics reports can provide page and event measures relevant to site use and selected actions. Keep each source’s property, reporting start and end dates, requested dimensions, metrics, reporting identity where relevant, and retrieval time with its observations.
The Search Analytics API groups results by requested dimensions and has documented data limitations. The Google Analytics Data API supports programmatic reports, but a production setup still needs property access, authentication, report design, and a destination schema. An absent API row is not automatically zero. It may not have been returned under the report configuration or source limits.
Join observations to an asset using a documented URL mapping or asset ID. Keep a page-level Search Console row separate from a page-query-device row. For GA4, retain the report configuration and dimension set used to produce each row. Review performance alongside accuracy, links, conversions, customer usefulness, strategic purpose, and risk.
Turn evidence into review decisions, with a human approval gate
Choose an action from the evidence, not from a page’s age or one weak traffic measure.
- Keep: The page meets its purpose, has an owner, and has no material technical or factual issue.
- Update: The page has useful audience or business value but contains fixable gaps such as stale facts, broken links, weak metadata, or incomplete coverage.
- Consolidate: Multiple pages substantially overlap and one stronger primary page can serve the need.
- Redirect: A relevant successor exists and the old URL should no longer resolve independently.
- Remove: The page has no meaningful audience, business, legal, historical, or successor value.
- Create: Evidence shows a validated gap in a customer journey, search need, product education path, or support experience.
Consolidation is an editorial and information-architecture decision. A redirect is a technical routing action that may follow consolidation. It is not a substitute for choosing an appropriate successor, and an unrelated homepage is not a safe destination merely because it avoids a 404.
Use deterministic checks for status codes, duplicate normalized URLs, missing owners, stale review dates, and redirect chains. AI may suggest a topic group or draft a rationale from supplied evidence, but a person should approve redirects, removals, consolidation, canonical changes, regulated claims, and low-confidence recommendations. For teams assessing bounded AI agent design, keep suggestions separate from permission to change source systems.
The following comparison describes proposed workflow designs, not native vendor templates:
| Workflow | Trigger and input | Bounded automation and validation | Destination and fallback |
|---|---|---|---|
| URL reconciliation | Refresh or migration; crawl, sitemap, CMS, and redirect exports. | Normalize by written rules; check duplicates, status, canonical conflicts, and chains. | Save asset and crawl records; technical owner resolves unexplained differences. |
| Metric enrichment | Scheduled report; authorized Search Console and GA4 properties, dates, and dimensions. | Import separate observations; validate access, dates, mapping, and report grain. | Save dated evidence; analytics owner investigates missing or inconsistent results. |
| Action review | Scheduled review, content change, or material technical issue. | Rules flag objective exceptions; AI may suggest an action and rationale in controlled fields. | Send a review task; named owner approves consequential changes before write-back. |
For an action proposal, store suggested_action, rationale, evidence_refs, confidence, and reviewer_required. Parse the output against a fixed schema and allow only Keep, Update, Consolidate, Redirect, Remove, or Create. Reject unknown values or broken evidence references. After approval, record a decision event and route the task to the asset owner or technical owner.
Separate the proposal from the write-back. The review queue should contain the suggested action and its evidence, while the CMS or redirect system should receive only an approved, validated task.
Select tools by the job they actually perform
No single tool in this group is established as a complete cross-platform content inventory. Product features, limits, prices, and billing terms can change by plan, region, currency, and billing period. Verify current vendor terms before purchase.
- Screaming Frog SEO Spider: A desktop crawler for technical URL discovery and extraction. Its free version has a 500-URL crawl limit. At the October 11, 2026 research date, its pricing page displayed £199 per user per year for the paid version, with advanced capabilities such as saved crawls and JavaScript rendering. Check the official pricing page for current details.
- Semrush Site Audit: A hosted technical audit option that can use a website, sitemap, or imported URL list and supports scheduled audits. Use its exported findings as technical observations to reconcile with a governed inventory, not as the inventory’s full ownership and decision history. See Semrush’s Site Audit configuration guide.
- HubSpot Content Hub: A content-management option when relevant assets already live in HubSpot. HubSpot lists content management, SEO-related guidance, reporting, AI-assisted creation, and approval capabilities across different editions. Combine HubSpot records with other source data for cross-platform coverage.
- Airtable: A configurable collaborative tracker for imported records, ownership, views, and review status. The reviewed pricing material does not establish native website crawling, so import crawler or CMS data and verify plan limits for the intended workflow.
Choose a crawler when URL discovery is the gap, a CMS when managed content records are the gap, and a governed table or database when coordination across sources is the gap. A combined stack is often necessary when content spans multiple domains or publishing systems.
Assign ownership and make maintenance part of publishing
Separate page ownership from inventory stewardship. A page owner maintains content accuracy and completes assigned reviews. An inventory steward maintains the schema, access, imports, data quality, deduplication, and review process. Name both roles, even if one person initially fills them.
Update records when content is published, revised, redirected, consolidated, or retired. Attach the update to the existing CMS, migration, or retirement process where possible. Monthly or quarterly checks can be practical starting points, but cadence should reflect publishing velocity, change frequency, regulatory risk, and capacity.
Track missing owners, overdue reviews, unresolved URL conflicts, duplicate imports, and decisions awaiting approval. When an action is approved, record its owner, rationale, date, and resulting URL or task. The number of rows is not proof of a useful inventory. Operational completeness is a better measure.
- The inventory grain, scope, and URL normalization rules are documented.
- Source URLs retain their origin and extraction time.
- Asset and observation keys prevent duplicates during concurrent re-imports.
- Performance reports retain property, dates, dimensions, configuration, and metric names.
- Every in-scope asset has an owner or a visible exception.
- One steward owns schema changes, data quality, permissions, and review cadence.
- Redirects, removals, consolidations, and canonical changes require named approval.
Questions teams ask before they start
Should noindex, redirected, unpublished, or orphaned pages be included?
Include them when the work covers governance, migration, or retirement visibility. Label each record’s state and reason for inclusion instead of silently mixing it with published, indexable pages.
Does publication date prove that content is stale?
No. Treat age as a prompt to check accuracy, links, audience need, business purpose, and risk.
Can AI decide whether a page should be deleted or redirected?
AI can suggest an action and summarize supplied evidence. Require an accountable person to approve deletion, redirect, consolidation, and canonical changes.
Does low traffic alone make a page a removal candidate?
No. Check links, assisted value, customer-support use, legal or historical obligations, and whether the page serves a smaller but important audience.
How do we avoid importing the same page repeatedly?
Use an internal asset ID when available, or a documented normalized URL and site ID for the asset record. Give observations keys that include their grain, such as crawl run, reporting window, dimension set, or prompt run. Enforce uniqueness in the database and use a transactional upsert when concurrent imports are possible.
A reliable first run is straightforward: approve the scope and grain, reconcile URL sources, define keys and owners, import dated observations, review exceptions, then assign and record human-approved actions. Keep the inventory useful by making those updates part of ordinary publishing and site-change work.