Opens in a new tab
CRE Glossary

Crawling

The step where automated programs called bots follow links across the web to discover which pages exist.

In plain English

Crawling is the finding step. A small program, often called a bot or a spider, starts on a page it already knows and follows the links on it, then the links on the next page. Your newest listing gets discovered only if something points to it: another page on your site, a sitemap file listing your addresses, or a link from someone else's site.

AI tools send their own bots to read the web before answering. 69% of AI crawlers can't run JavaScript (searchVIU, Nov 2025), the scripting that assembles a modern page, so a page that needs scripting to appear can look nearly blank to them.

Why it matters

  • A page that is never crawled cannot be filed, read or repeated by anyone, however good it is.
  • Broken links and settings that block bots quietly cut off the parts of your site you care most about.
  • Deep pages, such as a single market report or one building, are the ones most often missed because little points to them.
  • Plain text served in the page itself is what these bots can read, which is why a heavy scripted page can arrive empty.

What to do next

  • Confirm your site lets both search bots and AI bots read it, which is controlled by a small file called robots.txt.
  • Link from your main pages to the deeper pages that matter, so a crawler has a path to follow.
  • Repair broken links you find, since a dead end stops the trail.
  • Ask someone to check whether your pages show real text before any scripting runs.
← The full glossary

Your next client is checking you out.

UpliftCRE puts your deals, listings and market knowledge on a website AI can read, with an AI staff that keeps it current in ten-minute meetings.

The Look-Me-Up Moment book cover
Not ready yet?

Read the book first.

Before UpliftCRE was software, it was a story. The Look-Me-Up Moment follows Jake Ellis, a broker four years in, as he builds a prospecting engine the hard way. Read it in one evening. Start the system Monday.