Meta Is Crawling the Web Like It Is Building a Search Index
Promptwatch crawler logs show Meta-WebIndexer going from about 2 percent to nearly 38 percent of AI crawler requests in under a month, a sign Meta is building its own search index.
A month ago, Meta-WebIndexer was a minor entry in the AI crawler logs Promptwatch publishes. By August 9, 2026 it was the heaviest crawler in the dataset. The shift was fast, and it lines up with reporting that Meta is building its own web search index so its AI does not have to lean on Google or Bing.
The Meta web indexer report, published August 10, 2026, tracks the share day by day. For anyone who cares about showing up in AI answers, the practical question is what to do about a crawler that did not exist as a serious force a month ago and now accounts for more than a third of the requests you can see in your server logs.
The measured change
Promptwatch's report covers Meta-WebIndexer's daily share of the AI crawler requests it tracks. The share hovered around 2.2 percent through mid-July 2026. It spiked to roughly 23 percent between July 20 and July 22, then surged from August 5 to a peak of 37.8 percent on August 9. That is a 17x increase in under a month from a single crawler.
The report is careful about what counts. The share is Meta-WebIndexer requests, identified by the meta-webindexer user agent, divided by requests from all AI crawlers Promptwatch tracks. Meta's other crawlers count toward the denominator but not toward Meta-WebIndexer's share. Regular search engine crawlers like Googlebot and Bingbot, link preview fetchers such as FacebookExternalHit, and human visitors are excluded entirely. That matters because it isolates the indexing crawler from the noise of Meta's broader bot fleet.
What the data does and does not tell you
The chart shows when the surge happened. It does not say why. The timing is the strongest signal. A crawl volume jump this fast, from a single indexing crawler, is what building a web index from scratch looks like. It is not a gradual behavior change you could attribute to a slow product shift.
The external corroboration is thin but present. On August 6, 2026, right as Meta-WebIndexer's share was climbing past 26 percent in the logs, the @levelsio account posted that Meta staff had told him the company is allegedly building its own web index, and that Meta was scraping his sites heavily enough to trigger server load alerts. Treat that as a secondhand report, not a confirmed fact. The crawl share is the harder evidence.
What the report cannot tell you is whether Meta ships a consumer search surface on top of the index. A crawler building an index is a necessary condition for a search engine, not a sufficient one. Meta could be crawling to improve Meta AI answers, to feed a future search product, or to train models. The data says the indexing work is happening. It does not say which of those uses it serves.
Why this matters for visibility work
If Meta does ship its own search engine for Meta AI, the crawl that happens now decides whether your content is in it at launch. That is the same dynamic you already optimize for with ChatGPT and Perplexity. The bots have to read you before the answer engine can cite you.
The difference is timing. The ChatGPT and Perplexity crawlers have been around long enough that most sites are already indexed or already excluded. Meta-WebIndexer is new and aggressive. Whether your site was crawled during this build out is a decision you are making right now, not a state you inherited.
That makes crawler logs the right place to look first. A visibility score tells you whether AI mentions your brand. It does not tell you whether the AI even read your pages. The crawl to citation path is the layer below the score, and it is the layer where Meta-WebIndexer's surge shows up.
What to do about it
The first move is to look at your own server logs and your robots.txt for the Meta crawlers. The report names the relevant user agents: meta-webindexer, meta-externalagent, meta-externalfetcher, and FacebookBot. If Meta is building a search index, blocking these crawlers today is the decision about whether your content is in it at launch.
Most sites that care about AI visibility should not block them. The same logic that says you should let GPTBot and ClaudeBot read your content applies to Meta-WebIndexer. The exception is sites that bill by request or run on infrastructure that charges for traffic, where a sustained crawler load is a real cost. The report itself notes that Meta's crawl volume grew fast enough to trigger load alerts on independent sites. Budget for sustained Meta crawler traffic rather than treating it as a spike that will pass.
The second move is to treat Meta as an emerging AI search surface, not only a social platform. Meta-WebIndexer exists to improve Meta AI search results and cite sources. The citation dynamic is the same one you already work on for ChatGPT and Perplexity. The difference is that Meta is earlier in the crawl to citation lifecycle, so the work you do now compounds over a longer horizon.
The third move is to watch crawl load over time, not on a single day. The report shows the share moved in steps, not a smooth line. A single day of high Meta crawler traffic in your logs is not a signal. A sustained step up, like the July 20 to 22 spike that held, is the signal that the indexing work is real and ongoing.
How to measure it properly
A crawl share report like Promptwatch's is the population level view. It tells you what Meta-WebIndexer is doing across the whole web. It does not tell you what Meta-WebIndexer is doing on your site. For that you need your own crawler logs joined to your own visibility data.
That join is the part most visibility tools skip. A prompt tracker that reports whether ChatGPT mentions your brand will not tell you whether ChatGPTBot or Meta-WebIndexer hit your site, which pages they read, or whether the crawl led to a citation. The crawl to citation path is the measurement that turns a crawl share report from interesting context into something you can act on.
Promptwatch publishes that path as Agent Analytics. The feature tracks real time AI crawler logs across ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, and the Meta AI crawler, plus the crawl to citation path and error tracking. For a report that shows Meta-WebIndexer surging at the population level, the matching move is to open the same log view for your own domain and see whether the surge shows up in your crawl traffic, and whether the pages Meta read are the pages that get cited.
The broader pattern
The Meta-WebIndexer surge is one instance of a wider shift in AI search. The crawler field is getting more crowded, not less. The report lists the crawlers Promptwatch tracks across OpenAI, Anthropic, Google, Perplexity, xAI, Mistral, Meta, Cohere, and DeepSeek. Each of those is a candidate to become a citation source, and each has a different crawl cadence and a different robots.txt stance.
For visibility work, that means the set of retrieval surfaces to optimize for is larger than it was a year ago. ChatGPT and Google AI Overviews are still the dominant surfaces, but they are not the only ones. A crawler that did not matter a month ago can become a meaningful citation source in a hurry, which is exactly what the Meta-WebIndexer data shows.
The practical response is to treat crawler logs as a first class signal, not an afterthought. The brands that notice a new crawler early, watch whether it reads the pages that matter, and connect the crawl to the citations that follow, are the ones that show up in the new surface before the rankings do. The ones that only watch the visibility score will see the Meta-WebIndexer surge as someone else's news, not as a thing that happened to their site.
What to watch next
The report is a snapshot through August 9, 2026. The open questions are whether the share holds, whether Meta-WebIndexer keeps climbing, and whether the crawl translates into citations in Meta AI answers. Those are exactly the questions a crawler log view answers for your own domain.
The dataset Promptwatch publishes is aggregated and non identifiable, and it is refreshed constantly. That makes it useful for spotting population level shifts like this one. It does not replace the need to track your own brand, your own prompts, and your own crawl traffic. The two work together: the public report tells you what is happening in the field, and your own tracking tells you whether it is happening to you.
For a team that wants to act on the Meta-WebIndexer shift rather than just read about it, the workflow is to open the public report for context, open the crawler log view for your own site, and connect the two. That is the measurement stack that turns a crawler surge into a visibility decision, and it is the stack Promptwatch is built around.