Add a Sitemap and robots.txt to an AI-Built App

To add a sitemap and robots.txt to an AI-built app, create a sitemap.xml that lists every public page you want indexed, create a robots.txt that allows crawling and points to the sitemap, and submit the sitemap in Google Search Console. The two files do different jobs, and mixing them up is the most common way an AI-built site ends up invisible.

What each file is for

A sitemap is a list of URLs you would like search engines to know about. A robots.txt file is a set of instructions about which URLs crawlers may fetch. One is an invitation, the other is a request for restraint.

Google's own introduction says robots.txt is used mainly to avoid overloading your site with requests, and that it is not a mechanism for keeping a web page out of Google. A blocked URL can still be indexed if other pages link to it. To keep a page out, Google points to noindex or password protection instead.

That one sentence prevents a lot of wrong fixes.

Add a sitemap to an AI-built app, step by step

Google's documentation on building a sitemap gives the limits that matter: a single sitemap can hold at most 50,000 URLs or 50MB uncompressed, the file must be UTF-8, and a sitemap at the site root can affect all files on the site. Past those limits, split it into several files and list them in a sitemap index.

A minimal file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.example/</loc>
    <lastmod>2026-10-11</lastmod>
  </url>
  <url>
    <loc>https://yourdomain.example/pricing</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
</urlset>

Rules worth enforcing in what your AI tool generates:

1. Use full absolute URLs with the exact domain and protocol you serve.

2. List only pages that return 200 and that you want indexed. No redirects, no 404s, no login pages.

3. Keep lastmod honest. Set it from the real edit date of each page, not the build time.

4. Regenerate it whenever pages are added. A sitemap written by hand once goes stale.

For an app with user-generated or database-driven pages, ask for a sitemap route that is generated from the database on request, with a cache.

The robots.txt file

For most small sites the file is short:

User-agent: *
Allow: /

Sitemap: https://yourdomain.example/sitemap.xml

It must be served at the root, https://yourdomain.example/robots.txt. The Sitemap: line is one of the ways Google says you can tell it about a sitemap, alongside the Search Console Sitemaps report. Submitting a sitemap is a hint, not a promise that every URL will be crawled.

The mistake that hides a whole site

A staging-friendly robots.txt can end up in production, and it looks like this:

User-agent: *
Disallow: /

That asks every crawler to stay out of the entire site. It is fine on a preview URL and disastrous in production. It is easy to ship by accident because the site looks perfect to a human. Open /robots.txt on your live domain right now and check that you do not see Disallow: / on its own line.

The reverse problem exists too. Adding Disallow rules to hide private pages tells everyone those paths exist and does not stop them from appearing in results. Protect private pages with a login, and use noindex for pages that should stay public but out of search.

Submit it and check it

Once the files are live:

• Open both URLs in a browser and confirm they load as plain text and XML.

• In Search Console, open the Sitemaps report and submit sitemap.xml. The report shows when Googlebot last read it and any processing errors.

• Spot-check three URLs from the sitemap in a private window to confirm each loads and returns the page you expect.

If your pages are not being picked up after a few weeks, the broader checklist in how to get an AI-built app indexed by Google goes through the other usual causes, and open graph link previews covers how those pages look once people share them. The overall build is laid out in how to build an app with AI.

The prompt to paste

Add a sitemap and robots.txt to this app.
- sitemap.xml at the site root, UTF-8, absolute URLs on our production domain, only public pages that return 200, lastmod from each page's real updated date. Generate it from the database for dynamic pages.
- robots.txt at the root allowing all crawlers, with a Sitemap line pointing to the sitemap.
- Make sure production does NOT contain "Disallow: /". Keep any staging block behind an environment check.
- Do not use robots.txt to hide private pages; use authentication or noindex instead.

Frequently asked questions

Does my site need a sitemap?

Small sites with good internal links can be found without one, but a sitemap helps crawlers discover new pages sooner and shows you in Search Console which URLs were read. It is cheap to add.

Does robots.txt stop a page appearing in Google?

No. Google says robots.txt is not a mechanism for keeping a page out of Google, and a disallowed URL can still be indexed if it is linked from elsewhere. Use noindex or a password.

How many URLs can a sitemap hold?

Google's limit is 50,000 URLs or 50MB uncompressed per sitemap. Use several sitemaps and an index file beyond that.

Where should the sitemap file live?

At the root of your site, so it can cover every path. Then reference it in robots.txt and submit it in Search Console.