Opens in a new tab
CRE Glossary

Robots.txt

A small text file on a website that tells automated crawlers which parts they are asked not to read.

In plain English

Crawlers are programs that read websites in order to build indexes and answers. A robots.txt file is a short note sitting at a fixed address on your site, telling those programs where they may go and where they should not. One line can quietly block the whole site, and one line can open it back up.

For an advisor this is a check, not a project. The thing worth confirming is that the file does not block the public pages you want read: listings, market pages, your track record. Keeping a private admin area out of it is the sensible use.

Why it matters

  • An accidental block can hide a site from the very readers it was built for.
  • A sudden drop in visitors is one of the first things to check here.
  • Blocking a private area is reasonable; blocking public content is usually a mistake.
  • Site rebuilds can restore old settings nobody chose on purpose.

What to do next

  • Open your own site address followed by robots.txt in a browser and read it.
  • Confirm it does not contain a single line that blocks the entire site.
  • Ask whoever maintains your site to explain any instruction you do not recognize.
  • Look at it again after any site rebuild or hosting change.
← The full glossary

Your next client is checking you out.

UpliftCRE puts your deals, listings and market knowledge on a website AI can read, with an AI staff that keeps it current in ten-minute meetings.

The Look-Me-Up Moment book cover
Not ready yet?

Read the book first.

Before UpliftCRE was software, it was a story. The Look-Me-Up Moment follows Jake Ellis, a broker four years in, as he builds a prospecting engine the hard way. Read it in one evening. Start the system Monday.