Two files sit quietly at the root of almost every website and have an outsized influence on how quickly Google discovers, crawls, and indexes a site's content. The XML sitemap tells Google what pages exist and where to find them. The robots.txt file tells Google which parts of the site it should not crawl. Together they form the foundation of crawlability, which is the prerequisite for everything else in SEO. A page that Google cannot find or has been inadvertently told to ignore cannot rank regardless of its content quality, schema markup, or E-E-A-T signals.
Both files are simple in principle and frequently misconfigured in practice. This article documents what each file does, the specific mistakes that appear most consistently on new WordPress sites, and the correct configuration for a content site that wants to be indexed efficiently from day one.
What is an XML sitemap and why does it matter?
An XML sitemap is a file that lists the URLs on your website in a structured format that search engine crawlers can read. It tells Google which pages exist on your site, when they were last updated, and how frequently they are likely to change. It is not a guarantee that Google will index every URL listed, but it is a direct communication to Google about what your site considers worth indexing.
For a new domain with no backlinks and no crawl history, an XML sitemap is the most reliable way to ensure Google discovers all of your content rather than only the pages that happen to be linked from somewhere Google has already crawled. Without a sitemap, Google's crawler discovers pages by following links. On a new site with few internal links and no external links, the crawler may miss pages entirely or take significantly longer to find them than it would with a sitemap in place.
The sitemap also serves as a crawl efficiency tool. By specifying which URLs should be crawled, you help Google allocate its crawl budget, the number of pages it will crawl on your site in a given period, toward content that matters rather than toward admin pages, login screens, tag archives, and other non-content URLs that add no SEO value.
What is a robots.txt file and what does it control?
A robots.txt file is a plain text file located at the root of your domain, accessible at yourdomain.com/robots.txt, that contains instructions for search engine crawlers. It specifies which parts of your site crawlers are allowed to access and which parts they should not crawl.
The robots.txt file uses a simple syntax. A User-agent directive specifies which crawler the following rules apply to. An asterisk means all crawlers. A Disallow directive specifies a URL path that the specified crawler should not access. An Allow directive explicitly permits access to a URL within a broader disallowed path. A Sitemap directive points crawlers to the location of your XML sitemap.
It is important to understand what robots.txt does not do. It prevents crawling but it does not prevent indexing. If another page links to a URL that your robots.txt disallows, Google may still index that URL based on the link, even though it has not crawled the page. For pages you want excluded from the index entirely, a noindex meta tag in the page's HTML is the correct mechanism. The robots.txt file and the noindex tag serve different purposes and are often confused.
What are the most common XML sitemap mistakes on new WordPress sites?
The XML sitemap errors that appear most consistently on new WordPress sites fall into a small number of recognisable patterns.
Not submitting the sitemap to Google Search Console. Generating a sitemap is not sufficient. It must be submitted to Google Search Console so Google knows where to find it and begins processing it. A sitemap that exists but has never been submitted to GSC provides significantly less crawl benefit than one that has been submitted and verified. In Google Search Console, navigate to the Sitemaps section under Index and submit the sitemap URL directly.
Including non-canonical URLs. A sitemap should list only the canonical version of each URL. If your site is accessible at both www.yourdomain.com and yourdomain.com, both versions should not appear in the sitemap. Only the canonical version, the one specified in your canonical tag settings, should be included. Yoast SEO handles this automatically when canonical settings are configured correctly, but sites that have been manually adjusted or migrated often have sitemap inconsistencies worth checking.
Including low-value pages. Sitemaps on WordPress sites frequently include tag archives, category archives, author archives, search result pages, and pagination pages that add no indexable value and dilute the crawl budget Google allocates to the site. Yoast SEO allows you to control which post types and taxonomies are included in the sitemap. Tag pages and author archives should typically be excluded on a content site unless they contain unique curated content.
Not updating the sitemap after publishing new content. Yoast SEO generates a dynamic sitemap that updates automatically when new content is published, which means this is not usually a manual problem. However sites using manually generated or statically cached sitemaps can fall out of date, causing new content to go undiscovered for longer than necessary.
Sitemap returning a 404 or 500 error. A sitemap that cannot be accessed by Google's crawler is worthless. After submitting your sitemap to Google Search Console, verify that it shows as successfully processed rather than returning an error. GSC displays sitemap status in the Sitemaps report and flags errors that prevent processing.
What are the most common robots.txt mistakes on new WordPress sites?
Robots.txt mistakes range from configuration gaps that allow crawlers unnecessary access to catastrophic errors that block indexing entirely.
Blocking the entire site during development. WordPress has a setting under Settings, Reading that adds a Disallow all directive to the robots.txt file when checked, intended to prevent search engines from indexing a site while it is still being built. The most severe crawlability problem we see on new sites is this box being left checked after launch. A site with this setting active will not be indexed regardless of how much content is published. In Google Search Console this appears as pages being crawled but not indexed with the reason given as blocked by robots.txt. Verify that this setting is unchecked before publishing any content, and verify it again after launch by checking your robots.txt file directly in a browser.
Blocking CSS and JavaScript files. Older SEO advice recommended blocking CSS and JavaScript from crawlers to conserve crawl budget. Google's crawler has rendered JavaScript since 2015 and needs access to CSS and JavaScript files to fully render and evaluate pages. Blocking these resources prevents Google from seeing your pages as users see them, which can negatively affect how the content is evaluated. The default WordPress robots.txt does not block these resources, but sites that have inherited old configurations sometimes do.
No sitemap reference in the robots.txt file. Adding a Sitemap directive at the bottom of your robots.txt file pointing to your sitemap URL is a best practice that makes the sitemap discoverable by any crawler that reads the robots.txt file, not only the crawlers you have manually submitted it to in their respective search consoles. This is a minor addition that takes seconds and requires no ongoing maintenance.
Disallowing wp-admin without allowing admin-ajax.php. The WordPress admin directory should be blocked from crawlers, but the admin-ajax.php file within it needs to remain accessible because many WordPress front-end features, including some forms and interactive elements, use it to communicate with the server. A robots.txt that disallows the entire wp-admin directory without explicitly allowing admin-ajax.php can break front-end functionality that depends on it.
What should the robots.txt file look like on a WordPress content site?
The robots.txt configuration recommended for a standard WordPress content site is straightforward. The file should allow all crawlers access to the site by default, explicitly disallow the wp-admin directory while allowing admin-ajax.php, and include a reference to the sitemap location.
The correct configuration looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/wp-sitemap.xml
Replace yourdomain.com with your actual domain. This configuration is sufficient for the majority of content sites. Additional Disallow rules should only be added when there is a specific reason to prevent crawling of a particular path, such as a members-only section, a staging subdirectory, or a utility page that should not appear in search results.
Verify your robots.txt file is accessible and returning the correct content by navigating to yourdomain.com/robots.txt in a browser. Google Search Console also includes a robots.txt testing tool that allows you to test whether specific URLs would be blocked by your current configuration before making changes to the live file.
How do XML sitemaps and robots.txt work together?
XML sitemaps and robots.txt serve complementary functions and should be configured consistently with each other. A URL listed in your sitemap should never be blocked by your robots.txt file. If a URL appears in the sitemap, you are telling Google it is worth crawling. If the same URL is blocked by robots.txt, you are telling Google not to crawl it. The contradiction wastes crawl budget and produces GSC errors that are worth avoiding.
The most common version of this conflict occurs when robots.txt blocks a directory that contains pages listed in the sitemap, usually as a result of configuration changes made at different times without checking consistency between the two files. After any significant change to either file, cross-checking the two for conflicts is a simple verification step that prevents indexing problems from developing unnoticed.
Google Search Console's URL Inspection tool allows you to check individual URLs to see whether they are indexed, whether they are blocked by robots.txt, and whether the sitemap has submitted them for crawling. Using this tool on your most important pages after any configuration change confirms that the changes have had the intended effect.
How does crawlability affect indexing speed on a new domain?
Crawlability is the first variable in the indexing process. Before Google can evaluate a page's content quality, E-E-A-T signals, or Core Web Vitals performance, it must first find and crawl the page. On a new domain with no crawl history, the time between publishing a page and Google discovering, crawling, and indexing it depends heavily on how clearly the site's crawlable structure is communicated.
A new domain with a correctly configured sitemap submitted to Google Search Console, a clean robots.txt file with no accidental blocks, and a logical internal linking structure connecting all published pages will typically see first indexing events within 3 to 14 days of publishing initial content. A new domain without these foundations in place can take significantly longer, not because Google cannot eventually find the content, but because it has not been given the signals that prioritise crawling it.
This is why crawlability configuration should be completed before publishing any content, not retrofitted after the first articles are live. The indexing clock starts when Google's crawler first visits a URL. The earlier that visit happens, the earlier the page begins accumulating the ranking signals that determine its eventual position.
At KatvTech, indexing speed is one of our active experiment categories. We are tracking the specific technical factors that influence how quickly new content on new domains moves from published to indexed across different configurations. Results will be published in the Ranking Experiments category when the data reaches the measurement endpoints we have defined.
What is an XML sitemap in WordPress?
An XML sitemap is a file that lists your website's URLs in a structured format that search engine crawlers can read. In WordPress, Yoast SEO generates and maintains an XML sitemap automatically, accessible at yourdomain.com/wp-sitemap.xml. It should be submitted to Google Search Console under the Sitemaps section so Google processes it directly rather than relying on discovering it through other means.
What does a robots.txt file do?
A robots.txt file instructs search engine crawlers which parts of your site they are allowed to crawl and which parts they should not access. It does not prevent indexing of blocked pages if those pages are linked from elsewhere. For complete exclusion from Google's index, a noindex meta tag in the page HTML is required in addition to or instead of a robots.txt disallow rule.
How do you check if your WordPress site is blocking Google?
Navigate to Settings, Reading in your WordPress dashboard and verify that the option to discourage search engines from indexing the site is unchecked. Then navigate to yourdomain.com/robots.txt in a browser and confirm the file does not contain a Disallow all directive. In Google Search Console, the URL Inspection tool shows whether specific pages are blocked by robots.txt and whether they have been successfully indexed.
Should you include all pages in an XML sitemap?
No. A sitemap should include only indexable, canonical pages that represent content worth ranking. Tag archives, author archives, pagination pages, search result pages, and admin pages should typically be excluded. In Yoast SEO you can control which post types and taxonomies are included in the sitemap under the SEO settings for each content type.
How long does it take Google to index a new website after sitemap submission?
On a new domain with a correctly submitted sitemap, clean robots.txt configuration, and logical internal linking, first indexing events typically occur within 3 to 14 days of publishing initial content. The timeline varies based on domain age, crawl budget allocation, content volume, and how frequently Google's crawler has previously visited the domain. Submitting the sitemap to Google Search Console immediately after publishing first content is the single most reliable step for accelerating the initial indexing process.
