Meta pauses the terms that would crawl your website

Meta pauses the terms that would crawl your website

Meta has indefinitely paused the update to its commercial terms that would have required business account holders to let Meta crawl external websites linked from Facebook. Your existing terms stay in effect. So if you run a Facebook business page and your own site, nothing changes for you right now, and the November 20 deadline you may have been bracing for is off the table.

Here is what was on the table. To keep your business page, you would have had to allow Meta to crawl any external site you linked to. There was no way to opt out. For content behind a paywall, the terms went further, requiring test credentials so Meta’s bots could get past it reliably. The date moved once already, from October 30 to November 20, before the pause.

Meta says its crawling practices are unchanged and that content owners can keep signalling their preferences using robots.txt, a small file on your site that tells automated crawlers what they may and may not fetch. The catch is that the revised terms include no specific language about respecting robots.txt. Publishers also pointed out that dropping the word “train” from the terms changed little, because the terms still allow using scraped content to develop any business product.

What to do now

  1. Check whether your site has a robots.txt file and what it says. It is the only lever Meta pointed to, so make sure it reflects what you actually want crawled.
  2. List the pages you link to from Facebook. Pricing, client work, gated guides, member areas. Those are the ones a crawl would reach first.
  3. Read the next version of the terms before you accept it, rather than clicking through the notice on your business page.

One caveat, and it is the important one. This is a pause, not a cancellation. It came after court injunctions in Brazil and Norway, a formal investigation by the UK competition regulator, and letters from US senators, and Meta says it is still consulting on the policy. Treat robots.txt as a signal you send, not a promise you hold.

Written by OFFART's AI news desk

This is OFFART's own take on a story first reported elsewhere. It is drafted by AI, checked against the original report, and published by OFFART Studio.

Originally reported by
Ars Technica
Back to Articles