HubSpot robots.txt settings control how compliant search engine crawlers access parts of a hosted website. The file can help reduce crawling of low-value utility paths and make crawler behavior more predictable, but it is not a security control and it does not directly remove pages from search results.
The safest setup is to begin with the default HubSpot configuration, change it only when there is a clear crawl-management reason, and test the published file against important URLs. A robots.txt rule should support a known SEO or operational decision, not act as a general-purpose way to hide content.
This guide explains where to find the setting, how the main directives work, which mistakes create the most risk, and how to introduce a review process so a future edit does not accidentally block public content.
What HubSpot robots.txt controls
A robots.txt file is a publicly accessible text file at the root of a domain, normally available at /robots.txt. It provides instructions to crawlers that choose to follow the Robots Exclusion Protocol. In HubSpot, the platform supplies a default file for hosted domains and may provide account-level controls for adding custom rules, depending on the domain and subscription configuration.
The file controls crawler access to URL paths. It does not control who can view a page in a browser, and it does not replace authentication, access restrictions, password protection, or removal of content from a site. It also should not be treated as a guaranteed deindexing mechanism. A blocked URL may still appear in search results if a search engine learns about it from links or other signals.
Use robots.txt to manage crawler access to public URL paths. Use access controls and an appropriate noindex strategy when the real requirement is privacy or removal from search results.
When a custom HubSpot robots.txt file is justified
Most sites do not need a long or frequently changing robots.txt file. Custom rules become useful when the site has a specific crawl-management problem that can be expressed as a path-based rule.
- Preventing crawlers from repeatedly requesting known test, preview, or utility paths that should not contribute to organic search.
- Reducing exposure of internal search result pages or other URL patterns that create large numbers of low-value variations.
- Making crawler instructions consistent across a domain with unusual content structures or multiple publishing teams.
- Declaring the location of an XML sitemap when that information is not already clearly available to crawlers.
Do not add a rule simply because a URL looks technical. A technical-looking path may still contain a valuable landing page, resource, product page, or rendered asset required for search engines to understand the page. The decision should be based on the business and search state of the URL, not its naming style.
A robots.txt change is a crawl decision, not a content strategy. Before blocking a path, identify what the path contains, who owns it, and what outcome the rule is expected to produce.
Where to find robots.txt settings in HubSpot
HubSpot places robots.txt controls within the settings for website domains and URL management. The exact labels and available options can change as HubSpot updates its interface or as account permissions vary, so treat the following as a navigation pattern rather than a guaranteed click-by-click path.
- Sign in to the HubSpot account with permission to manage website settings.
- Open the account settings.
- Go to the area for website domains, URLs, or related website configuration.
- Select the relevant connected domain and locate its robots.txt or crawler-rules setting.
- Review the current generated configuration before enabling custom editing.
Confirm that you are editing the intended domain. A portal may have separate production, regional, campaign, or redirected domains, and a rule for one host does not automatically govern every other host. If you cannot find the setting, check your permissions and the domain’s connection status before assuming that custom editing is unavailable.
How to configure the main robots.txt directives
Robots.txt uses groups of directives. A group begins with a User-agent line and applies to one crawler or, with an asterisk, to crawlers generally. Disallow identifies a path that the crawler should not request. Allow can make an exception where supported by the crawler. Sitemap identifies an XML sitemap URL and is not limited to a particular user-agent group.
What the rule says
The directive describes which crawler may access which URL path. It is a request to compliant crawlers, not a permission system for human visitors.
Why the rule exists
The rule should map to a defined outcome, such as reducing duplicate crawl paths while keeping important pages and assets accessible.
A simple generic structure might look like this:
User-agent: *
Disallow: /internal-search/
Sitemap: https://www.example.com/sitemap.xmlDo not copy an example into production without checking that the paths match the actual site. Path matching, trailing slashes, query parameters, wildcard behavior, and crawler support can produce different results from what a team expects. When a rule affects an important directory, test specific URLs rather than relying on a quick visual review.
A practical sequence for making a HubSpot robots.txt change
Use a small change process so that a technical SEO edit has an owner, a reason, and a rollback path.
This sequence separates diagnosis from implementation. It also makes later troubleshooting easier because the team can compare the intended business outcome with the actual crawler behavior.
How to test and review the live file
After saving a change in HubSpot, open the public robots.txt URL for the exact domain being edited. Confirm that the live content contains the expected directives and that no generated or existing rules disappeared when custom mode was enabled.
Test at least one URL that should remain crawlable and one URL that should be restricted. Include a URL close to the boundary of the rule. For example, if the proposed rule targets /resources/private/, also test /resources/ and a public page under a nearby path. Review the result with a current crawler testing tool where available, but remember that tools may model crawler behavior differently.
- The file is available on the intended production domain.
- Important pages, images, scripts, and styles remain accessible when needed for rendering and discovery.
- No rule blocks the XML sitemap location or the site’s primary content paths.
- The restricted path is not being used as a substitute for security or privacy controls.
- The change has a named owner, a reason, a date, and a rollback instruction.
Review the site’s crawl and indexing signals after a meaningful change. A robots.txt file cannot tell you whether search engines have indexed a page, whether a page is canonicalized correctly, or whether a page should be removed from results. Those questions require separate checks across page directives, canonical URLs, sitemaps, redirects, and search reporting.
Common HubSpot robots.txt mistakes
Blocking an entire content directory
A broad rule can affect current pages, future pages, media files, or templates that were not considered when the change was made. Directory names such as /blog/, /resources/, or /products/ should never be blocked merely because the team wants to reduce a few unwanted URLs.
Relying on robots.txt to hide sensitive information
Because the file is public, listing a sensitive path can reveal its existence. Sensitive material should not be published openly and then hidden with Disallow. Use appropriate access controls and remove or restrict the content at its source.
Assuming Disallow means noindex
Disallow primarily asks a crawler not to fetch a URL. It does not reliably communicate that the URL should be removed from search results, especially when the URL is known through external references. Decide whether the actual requirement is crawl reduction, indexing control, or access restriction before choosing a mechanism.
Allowing rules to drift without ownership
Robots.txt is often edited once and forgotten. Website migrations, new landing page folders, campaign templates, and changes to search functionality can make old assumptions wrong. Assign an owner and review the file when the site’s URL structure changes.
A robots.txt file should be small enough for a new website owner to understand and specific enough for an SEO owner to test.
Example: separating crawl control from content governance
Imagine a HubSpot site creates internal search URLs under /search/ and uses a staging-like folder for an unfinished campaign. The team may decide to restrict those URL patterns from crawling, while keeping the site’s published resource library accessible. That decision should be tested against real URLs, documented, and reviewed when the campaign folder is converted into a public section.
If the same team discovers that a page contains confidential material, the answer is not to add its path to robots.txt. The page should be removed from public access or protected through the appropriate publishing and access controls. The two situations look similar in a spreadsheet but require different system behavior.
Operational ownership for robots.txt
SEO, marketing operations, and website teams may all depend on robots.txt, but one person or team should own the live decision. A useful ownership record includes the affected domain, the rule, the reason, the expected result, the approving stakeholder, and the date for review.
For broader HubSpot architecture work, including domain configuration, CRM data flows, automation, and reporting, see HubSpot consulting. If the robots.txt issue is part of a wider lead management or lifecycle problem, CRM consulting can help clarify ownership and system behavior. ConsultEvo’s broader systems and automation services are also relevant when website rules need to fit into a larger operating model.
The best HubSpot robots.txt setup is not the most elaborate one. It is the smallest, clearest configuration that supports a defined crawl decision, preserves access to valuable content, and can be reviewed safely as the website evolves.
Frequently asked questions
What is the purpose of robots.txt in HubSpot?
HubSpot robots.txt provides crawl instructions for compliant search engine crawlers. It can restrict access to selected URL paths and identify an XML sitemap, but it is not a security control and does not reliably remove URLs from search results.
Can robots.txt prevent a page from being indexed?
Not reliably. A Disallow rule asks crawlers not to fetch a URL, but a search engine may still know about the URL through links or other signals. Use the appropriate indexing, access-control, or content-removal method for the actual requirement.
Where can I edit robots.txt in HubSpot?
The setting is generally found in HubSpot account settings under website domain or URL management. The exact location and available controls can vary by account permissions, connected domain, subscription, and interface version.
Should I block staging or private content with robots.txt?
Robots.txt can support crawl management for public test or utility paths, but it should not protect private content. Sensitive or unfinished content should be restricted or removed from public access using proper publishing and access controls.
How should I test a HubSpot robots.txt change?
Open the live /robots.txt file on the intended domain, confirm the published directives, and test representative URLs that should be allowed, blocked, and near the rule boundary. Record the change and monitor related crawl and indexing signals afterward.
Need a safer HubSpot website configuration?
If your robots.txt question is part of a wider HubSpot, CRM, or website operations problem, ConsultEvo can help clarify the process, ownership, and system changes before automation or configuration work begins.
