← Back to blog

Meta Pushed Its Website-Crawling Terms to November 20. The Clause Still Doesn't Mention robots.txt.

⭐ Featured

Meta's new business terms were due to take effect on October 30. They now start on November 20, 2026. A Meta spokesperson told Reuters the extra three weeks are "to allow more time for us to explain to our partners and stakeholders what's changing and what's not."

What is not changing, so far, is the part that matters to anyone who wants to be found in AI answers. The clause still gives Meta's crawlers permission to read the websites of every business that uses its ad and business tools. And it still says nothing about robots.txt.

What the clause does

We covered the first draft on October 16. In short: a business that uses Meta's products authorizes Meta to use "web scrapers, bots, robots, spiders, crawlers, and user agents" on the sites tied to those products, and to "periodically pull updates" from them. The version in force since March 2024 has no such permission.

The clause applies to every business with an ad account or a Meta Business Suite login, not only to publishers. For a B2B startup, that is the company that runs a retargeting pixel and a Facebook Page.

What Meta changed, and what it didn't

The first draft said Meta could use crawled content to "train" its products. Within days of the reporting, Meta removed that word. The stated purpose now reads "providing, developing, improving, researching, and maintaining advertising and business products."

That edit is smaller than it looks. Lawyers quoted by Nieman Lab said "develop" is broad enough to cover training anyway. The list of things the crawlers may do (access, index, cache, collect, crawl) stayed the same.

Meta's position is that nothing new is happening. It told Adweek that "the crawling terms we consolidated only applied to our ads and business products," and that the update "does not change how Meta uses web crawlers." It says site owners "can use robots.txt to signal their preferences."

Why the delay happened

Pressure, mostly from news publishers. The UK News Media Association asked the Competition and Markets Authority to secure a pause. Four European publisher groups (EMMA, ENPA, EPC and NME) asked the European Commission for interim measures. WAN-IFRA wrote to Meta on October 22.

The delay answered the timing complaint and nothing else. WAN-IFRA's CEO, Stig Ørskov, said the postponement "is a welcome step, but it does not address our concerns." No regulator has opened a formal investigation. Meta says it will explain how it uses crawlers before November 20, but it has not named any change to the clause.

"Signal" is the word to watch

Meta's spokespeople keep saying robots.txt lets you signal a preference. That is accurate, and it is the problem. A robots.txt file is a request that a well-behaved crawler chooses to honor. A signed commercial agreement is a contract. If the contract grants crawling rights and the robots.txt file refuses them, nothing in the terms says which one wins.

So the move from October 30 to November 20 buys three weeks, not an answer. Until the text says Meta's crawlers obey robots.txt, a business has to assume the contract could be read as overriding it.

There is a fair counterpoint. Meta has said for months that its bots respect robots.txt, and nobody has shown it ignoring a block since the draft appeared. The risk here is legal, not technical: it changes what Meta could argue later, not what its crawlers do today.

Which Meta bots this touches

Meta documents five user agents, and they do different jobs. meta-externalads serves ads and business products, which is what the new clause describes. meta-externalagent trains models and indexes content, under one token, so blocking it blocks both. meta-webindexer feeds Meta AI's search answers. facebookexternalhit builds link previews and meta-externalfetcher handles user-requested fetches; Meta says both may bypass robots.txt.

That last detail is the useful one. If you block meta-webindexer, you are likely to drop out of what Meta AI says about you. If you block meta-externalagent, you lose indexing along with training. There is still no setting that means "cite me, but don't train on me."

The bigger picture

For most B2B startups, the right response to the November 20 date is not to block Meta. It is to decide on purpose, bot by bot, and to write the decision down before November 20. Check your server logs for each of the five user agents, see which pages they fetch, and make sure the pages they fetch are the ones you want quoted when a buyer asks Meta AI who to hire.

The same exercise applies to every engine. Each one makes two decisions about your company: whether it can read your site, and whether it names you when a buyer asks. Meta has now tied the first decision to your ad account. You can still influence the second.

ClawWorld runs AI visibility work for B2B startups. Start with a free baseline report: 10 buyer questions, who ChatGPT and Claude name instead of you, and the pages they cite.

Get your free baseline report →