robots.txt
Definition
robots.txt is a plain text file at the root of a website that tells crawlers which paths they may or may not request. It controls crawling, not indexing, and it is a request that well-behaved bots follow rather than a security measure.
Each group of rules names a crawler with a User-agent line and then lists paths to allow or disallow, and a Sitemap line can point to your XML sitemap. One stray line can hide a whole site, and the classic cause is 'Disallow: /' carried over from a staging copy. Open yourdomain.com/robots.txt and read every line, because each subdomain has its own file.
Blocking a page in robots.txt does not remove it from search. If other pages link to it, Google can still list the address without its content, and a noindex tag on a blocked page is never seen because the page is never fetched. To keep a page out of results, leave it crawlable and mark it noindex, or put it behind a login. The file is also public, so listing a private folder in it advertises the folder.
The same file is where a site states its position on AI crawlers, one group per crawler name. A rule nobody can explain in one sentence is a rule to delete or document. On a bilingual site, confirm that nothing in the file hides the Arabic folder.
A group reading 'User-agent: GPTBot' followed by 'Disallow: /' asks OpenAI's GPTBot to stay off the entire site and says nothing to any other crawler.
Definitions are free.So is the cost review.
A short call, then a written review inside 72 hours: what your marketing costs, what it produces, and what is worth fixing first.