Every Zarla site has a robots.txt file. It tells Google, Bing and AI tools like ChatGPT which pages they can read. Zarla creates it for you each time you publish, and unless you change it, they can read every page. That is what most businesses want, so most owners never need to touch it.
You can see the file at your site's address followed by /robots.txt, for example www.yourbusiness.com/robots.txt.
See or change the file
- Go to Settings in the sidebar.
- Find the Search engines and AI tools card. It shows the file your site uses and a line that says who can read your site.
- Click Edit file, change Your rules, then click Save changes.
- If your site is live, the card says your file is saved. Click Publish changes there to put it on your live site.
Search engines re-read this file about once a day, so allow a day for a change to show, and longer for search results to catch up.
What you can write
Each rule starts by saying who it is for, then what they can or can't read:
User-agent:names who the rule is for.User-agent: *means everyone.Disallow:is a page address they should not read.Allow:is one they can.Crawl-delay:asks them to wait a number of seconds between pages. Google ignores this line; Bing reads it.- A line that starts with
#is a note to yourself.
Zarla checks every line when you save. If a line isn't a rule search engines understand, the card names the line and says what to fix. Use plain letters, numbers and symbols only.
The Sitemap line is Zarla's. It sits under your rules, always points at your site's real address, and follows you if you connect your own domain. You can't change or remove it, and you don't need to add one.
Rules you can add
The card has two rules ready to add, and one that puts things back:
- Stop one page being read.
Disallow: /thank-you$underUser-agent: *stops search engines opening that one page. Swap/thank-youfor your own page's address. This does not take the page out of Google. To do that, switch off Show in Google search in the page's settings in the editor, and leave the page out of this file. If you do both, Google can't open the page to see that you switched it off. - Stop ChatGPT being trained on your site.
Disallow: /underUser-agent: GPTBotasks OpenAI not to use your pages to train its AI. It does not take you out of ChatGPT's answers, which use a different crawler. Other companies use other names, such as ClaudeBot. - Let everyone read every page. This is Zarla's default.
Hiding your whole site from search engines
A rule that blocks everyone from every page, such as Disallow: / under User-agent: *, tells Google and other search engines not to read your site. People who search for your business are unlikely to find the website, though it stays online for anyone who has the link.
This is not the same as making a site private. Search engines stop reading your pages, but Google can still list your address, without a description, if other websites link to it. To keep a page out of Google completely, leave it out of this file and switch off Show in Google search in that page's settings. To take the whole site down, use Take site offline on the Settings page.
Zarla asks you to confirm before saving a rule like that. If you go ahead, the Settings card says Your site is hidden from search engines every time you open it, and Get Found lists it as the first thing to fix.
To be found again, click Reset to default on the card (or edit the file), then publish.
Go back to Zarla's file
Click Reset to default next to Edit file. Your own rules are removed, and search engines and AI tools can read every page again once you publish.