The Staging Site You Forgot to Block and Other Noindex Accidents

There is a specific kind of marketing emergency that produces no error message, no broken page, and no angry phone call from a client. The website looks perfect. Every page loads. The forms work. And organic traffic falls off a cliff, or a brand new site simply never starts ranking at all, because somewhere in the build a single line of code is telling Google to stay away.

These are the noindex accidents. They are the most preventable technical failures in law firm marketing and, in our experience auditing sites that arrive with unexplained traffic problems, among the most common. A firm spends forty to eighty thousand dollars on a rebuild, launches on a Tuesday, and discovers in November that the developer never removed the block that kept the staging site private. Four months of content, link building, and paid traffic funneling into a site Google was politely refusing to read.

This article covers the three failure modes we see most often, the exact check that catches each one, and the launch day verification list that should be non-negotiable before any law firm site goes live.

Failure One, the Staging Copy Google Found

Every competent web build happens twice. There is the live site your clients see, and there is a staging or development copy where the agency assembles the new one. The staging copy usually sits at an address like staging.yourfirm.com, yourfirm.wpengine.com, or dev.agencyname.com/yourfirm.

That copy is a complete duplicate of your website. Same practice area pages, same attorney bios, same headlines. If it is publicly reachable and nothing stops crawlers, Google will find it and index it, because Google finds things by following links and by watching certificate transparency logs, and staging subdomains show up in both.

The damage is not theoretical. You now have two versions of every page competing in the index. Google has to pick one as canonical and sometimes picks the staging URL, which means your search result shows staging.yourfirm.com in the address line. Worse, staging sites are frequently left with placeholder content, test form submissions that go nowhere, old phone numbers, and lorem ipsum in the footer. We have seen a firm’s staging site outrank its live site for the firm’s own name, sending prospects to a form that emailed a developer’s personal inbox.

Staging copies also tend to be the least maintained software on a firm’s hosting account. Nobody patches them, nobody watches them, and they run the same plugins as production. That combination is exactly how sites get compromised, which is a separate and uglier problem covered in our piece on hacked law firm websites and spam injection.

The Check

Search Google for site:yourfirm.com and read the results, then search site:*.yourfirm.com -www to surface subdomains. Then search for a distinctive sentence from your homepage in quotation marks and see whether anything other than your live site returns it. Do this once a quarter and always two weeks after a launch.

The fix is not a noindex tag. Staging environments should be protected at the server level with HTTP authentication, an IP allowlist, or an access policy in front of the hostname. A password prompt means no crawler ever sees content in the first place, and it also means a curious opposing counsel cannot read your unpublished practice area pages. Ask your developer to confirm which of those three methods is in use and to send you the staging URL so you can verify the prompt appears in a private browsing window.

Failure Two, the Noindex Tag That Survived Launch

While a site is being built, the standard practice is to add a directive that tells search engines not to index anything. In WordPress this is a single checkbox under Settings, Reading, labeled to discourage search engines from indexing this site. It writes a meta robots noindex tag into the head of every page.

It is the right setting during a build. It is a catastrophe if it survives launch, and it survives launch constantly, because the person who checks the box during development is often not the person who pushes the site live, and nothing about the live site looks wrong when the box is still checked.

The pattern of damage is distinctive. A brand new site launches and never gains traction. A rebuilt site loses ninety percent of its organic traffic over six to ten weeks as Google gradually drops pages from the index. Nobody suspects the tag because everything else works, so the firm blames the content, or the agency, or the algorithm. We have picked up sites where this had been true for over a year.

There is a subtler version. Some SEO plugins and page builders set noindex on individual post types or templates. Attorney bios, location pages, and blog category archives are the usual casualties. The homepage indexes fine so nobody notices, but a third of the site is invisible.

The Check

On any page of the live site, view source and search the markup for the word noindex. If it appears inside a meta robots tag, that page is telling Google to drop it. Do this on the homepage, one practice area page, one attorney bio, one location page, and one blog post. Five pages, two minutes.

Then confirm at scale. Run the site through a crawler such as Screaming Frog, which is free for the first five hundred URLs and covers most law firm sites outright, and look at the Directives report. Every URL that should rank must return index and follow. In Google Search Console, open the Pages report under Indexing and look for the reason labeled excluded by noindex tag. That report is the single most useful screen in the tool for this class of problem, and it is where most of the diagnosis in when Google will not index your law firm pages starts.

One caution. The tag can also be delivered as an HTTP header called X-Robots-Tag, which does not appear in the page source at all. If your crawl says a page is indexable but Search Console says it is excluded by noindex, check the response headers. Browser developer tools show them under the Network tab on the first document request.

Failure Three, the Robots.txt Disallow Nobody Removed

The file at yourfirm.com/robots.txt tells crawlers which parts of the site they may request. During a build it often contains a blanket block, User-agent star, Disallow slash, which means stay out of everything. Sometimes that file gets deployed to production along with everything else.

Other times the damage is narrower and lives longer. A line blocking /wp-content/ breaks the rendering of your CSS and images so Google evaluates your pages as unstyled text. A line blocking /locations/ was added years ago when those pages were thin, and nobody removed it after they were rewritten. A line blocking /?s= to keep search results out of the index accidentally matches your filtered practice area URLs.

Robots.txt is also one of the files most likely to be silently replaced during a platform migration, since it lives at the domain root rather than inside the theme. Firms moving between hosts or content management systems inherit whatever the new stack ships by default, which is a good reason to treat the file as a launch checklist item rather than a set-and-forget artifact. If you are weighing that kind of move, the tradeoffs are laid out in choosing a website platform for your law firm.

The Check

Type yourfirm.com/robots.txt into a browser and read it. It is usually under twenty lines. Every Disallow line should have a reason you can state out loud. If you cannot explain a line, it does not belong. Then use the robots.txt report in Search Console, which shows the exact file Google fetched and when, so you are reading what the crawler read rather than what you think is deployed. Cloudflare’s reference on what robots.txt is and how it works is a clear primer worth sending to anyone on your team who needs to approve changes to it.

Why Robots.txt and Noindex Are Not Interchangeable

This confusion causes real damage, so it is worth being precise. Robots.txt controls crawling. Noindex controls indexing. They are different levers and they interact badly.

If a page is blocked in robots.txt, Google will not fetch it, which means Google cannot see the noindex tag on it. A page blocked in robots.txt can still appear in search results, usually as a bare URL with no description, because Google learned about it from links elsewhere. Firms routinely try to remove a page from search by adding a Disallow line, and the result is that the page stays in the index permanently in its ugliest possible form.

The correct sequence to remove a page from search is to serve noindex on the page, leave it crawlable, wait for Google to fetch it and drop the page, and only then add a robots.txt block if you also want to save crawl budget. In practice, for a law firm site of a few hundred pages, crawl budget is not a real constraint and the robots.txt block is usually unnecessary.

The Launch Day Verification List

This should take under thirty minutes and it should be run by someone other than the person who deployed the site, because the deployer verifies what they intended rather than what shipped.

1. Load yourfirm.com/robots.txt in a browser. Confirm there is no Disallow slash line and that every remaining Disallow has a stated reason.
2. View source on the homepage and search for noindex. Repeat on a practice area page, an attorney bio, a location page, and a blog post.
3. Check the response headers on those same five URLs for X-Robots-Tag.
4. Confirm the canonical tag on each of those five pages points to the live https www version of itself, not to a staging hostname.
5. Open the staging URL in a private window and confirm you get an authentication prompt or an access denied response, not the site.
6. Run a crawl of the full site and check the Directives and Canonicals reports for anything unexpected.
7. Submit the new XML sitemap in Search Console and confirm it lists the live domain, has the expected page count, and returns no errors.
8. Use the URL Inspection tool in Search Console on the homepage and two interior pages, click Test Live URL, and read the indexability verdict directly from Google.
9. Confirm the site returns a single canonical hostname, that http redirects to https, and that the non-preferred www variant redirects rather than resolving in parallel.
10. Confirm your analytics and conversion tracking fired on the live domain, since a tracking snippet still pointed at the staging property is the same class of mistake.

Then set a calendar reminder for fourteen days after launch to open Search Console and check that indexed page count is climbing rather than falling. Google is not instant. A block that survives launch usually shows up as a downward slope over a month, not a cliff on day one, which is exactly why nobody catches it in the launch week celebration.

The Ongoing Check That Catches Regressions

Launch day is not the only exposure. Plugin updates, theme changes, a new developer restoring an old backup, and staff toggling settings in an SEO plugin all reintroduce these problems months after launch. The realistic cadence for a law firm site is a five minute monthly check and a fuller quarterly pass.

Monthly, open the Pages report in Search Console and compare indexed page count to last month. A drop of more than a few percent that you did not cause deserves an explanation. Also glance at the excluded reasons for any growth in blocked by robots.txt or excluded by noindex tag.

Quarterly, rerun the ten point list above. Uptime monitors are also worth configuring to watch robots.txt specifically, alerting on any change to the file’s contents. Several inexpensive monitoring services support content-change alerts on a single URL, and the file rarely changes for legitimate reasons, so false alarms are close to zero.

An Honest Note on What This Does and Does Not Fix

Removing an accidental block does not create rankings. It removes an artificial ceiling. When we lift a noindex that has been in place for months, recovery typically starts within one to three weeks as pages return to the index, and the site lands roughly where its content, links, and authority would have put it anyway. If the underlying site was mediocre, an indexable mediocre site is what you get back. The value of this work is that it stops you from spending a year diagnosing a content problem you do not have.

It is also worth being clear that these checks are cheap and the failures are expensive. A partner can personally run the robots.txt check and the view source check in under five minutes with no technical background. There is no reason for a firm to be four months into an invisible website.

Rubiks runs this verification on every law firm site we launch and on every site we inherit, before we quote a single hour of content or link work, because there is no point optimizing pages Google has been told to ignore. Talk to Rubiks and we will check your live site, your staging environment, and your robots file, and tell you plainly whether anything is blocking you before you commit to anything else.