The Weekly Search Pulse: PHP Downtime Protocols, AI Crawler Governance, and Emerging URL Standards

Navigating the complexities of search engine optimization and server management requires a precise understanding of how automated systems interact with web infrastructure. Recent guidance from Google and industry developments regarding AI agents highlight a shifting landscape where technical precision—from status codes to URL encoding—directly impacts search visibility and data integrity. As website administrators grapple with emergency security patches and the rapid evolution of artificial intelligence, adherence to established protocols has become the primary defense against indexing errors and unintended content utilization.

Managing Server Downtime During Emergency PHP Patches

On October 21, Google Search Central issued a critical reminder to web administrators regarding the necessity of returning a 503 Service Unavailable status code when taking servers offline for maintenance, such as critical PHP security patches. This directive followed the release of emergency fixes for two major vulnerabilities within the PHP core, which prompted widespread server-side updates across the WordPress ecosystem and beyond.

The 503 status code is technically defined as a signal that the server is currently unable to handle the request due to temporary overloading or maintenance. From an SEO perspective, this is the only appropriate response for planned or unplanned outages. By serving a 503, a site effectively tells Googlebot: "The content is here, but the server is momentarily indisposed; please check back later." Conversely, returning a 404 Not Found error instructs the search engine to remove the page from its index, while a 200 OK status—if served during a broken state—can lead to indexing errors or the caching of incomplete, faulty pages.

John Mueller, a Search Advocate at Google, clarified that the 503 response does not need to be generated by the PHP environment itself. In fact, if the PHP environment is the source of the downtime, the status code must be triggered by an upstream entity, such as a Content Delivery Network (CDN), a load balancer, or a temporary maintenance server pointed to by DNS records. Google’s internal documentation warns that sites should limit 503 status durations to no longer than 1–2 days. Persistent 503 errors beyond this timeframe may result in the search engine dropping URLs from the index, viewing the downtime as a permanent state.

The Limits of Robots.txt and AI Crawler Governance

As AI-driven search experiences become more prevalent, the debate over content usage has shifted toward the limitations of the robots.txt protocol. During a Search Central video released on October 21, Gary Illyes of Google emphasized that the RFC 9309 standard for robots.txt is strictly an "access preference protocol." Its primary function is to govern whether a compliant crawler is permitted to access a file, not to dictate how that data is utilized after it has been retrieved.

This distinction is crucial for publishers attempting to manage AI training. Currently, the industry relies on specific user-agent tokens to manage crawling. For example, Google-Extended is the specific token used to signal that crawled content should not be used for training Gemini models or grounding applications on Vertex AI. Crucially, blocking the Google-Extended token does not prevent a site from appearing in Google Search results, as it is not a separate crawler but a policy flag for the existing infrastructure.

The industry is currently moving toward a more formal framework, with Illyes co-authoring a draft for the Internet Engineering Task Force (IETF) that proposes a new "Content-Usage" rule within robots.txt. Until such a standard is ratified and adopted by all major AI developers, publishers remain in a transition period. Blocking a crawler via robots.txt prevents the ingestion of new data, but it does not retroactively remove data already stored in an AI’s training set.

Cloudflare and OpenAI: A New Discovery Paradigm

The intersection of web infrastructure and AI search was further highlighted by a joint research pilot between Cloudflare and OpenAI. Announced on October 15, the project utilized signals from a participating subset of Cloudflare’s network—approximately 1% of its total traffic—to help the OpenAI crawler identify new or updated web pages more efficiently.

This development is significant because it shifts the discovery mechanism from traditional backlink-based crawling to real-time network signaling. By identifying non-existent pages and highlighting updates, the pilot allowed OpenAI to index hundreds of millions of new pages for ChatGPT Search. However, both companies were quick to clarify that this signal does not equate to permission. The "discovery" of a page does not override existing robots.txt directives or authorization requirements.

For site owners, this reinforces the need to manage their robots.txt files with granularity. OpenAI’s crawling infrastructure distinguishes between "OAI-SearchBot," which powers ChatGPT’s search capabilities, and "GPTBot," which is primarily used for training large language models. A site owner can theoretically permit search discovery while simultaneously opting out of training by using specific disallow directives for the respective bots.

URL Structure and the Technicality of Brackets

Technical precision extends to the formatting of URLs, a topic John Mueller addressed on October 19 regarding the treatment of square brackets. Mueller acknowledged that he had previously provided inaccurate guidance, confirming that raw square brackets and their percent-encoded counterparts—%5B and %5D—are treated as distinct URLs by search engines.

Under the RFC 3986 standard, reserved characters must be percent-encoded. When a website inconsistently uses both raw and encoded brackets across links, canonical tags, and sitemaps, it forces Google to crawl multiple versions of the same page. While Google’s algorithms are generally capable of identifying these as duplicates and selecting a canonical version, this process wastes "crawl budget"—the finite amount of time and resources Google allocates to crawling a specific domain.

Mueller’s advice to webmasters is simple: "Pick one." Consistency is the key to efficient indexing. For existing sites that have historically used raw brackets, he noted that the engine would likely continue to index them, but he strongly advised against the practice for new implementations.

Limitations of Image Formats: The JPEG XL Case

Despite the rapid adoption of new image formats in browser technology, support in search indexing remains selective. On October 15, John Mueller confirmed that JPEG XL is not currently supported in Google Images or for Discover previews, even as major browsers like Chrome and Firefox have begun integrating support for the format.

The implications for developers are clear: while JPEG XL offers superior compression and quality, it cannot currently be the sole format served to search engines. To ensure images remain discoverable in Google Images and appear in Discover, developers must use the <picture> element. This allows for a "fallback" strategy: providing a JPEG XL source for supported browsers while keeping a standard JPEG, WebP, or AVIF file in the <img> tag. This ensures that Google’s crawlers, which read the src attribute of the <img> tag, can still parse and display the image correctly.

The Future of AI Agents: Browsing vs. Protocols

Looking toward the broader future of the internet, Google CEO Sundar Pichai discussed the evolution of AI agents during an interview on the "Sources" podcast. Pichai posited that AI agents will access the web through a dual-track system: browsing pages much like human users, and interacting through machine-readable protocols.

While "agentic" browsing—where bots navigate UI elements and click through pages—is currently the most prevalent method, it is inherently inefficient compared to structured protocols. Protocols like the Universal Commerce Protocol represent a more scalable future, allowing agents to interact directly with backend services rather than scraping front-end interfaces.

However, this transition is in its infancy. For now, the web remains a landscape dominated by human-centric design, which AI agents are forced to interpret. As Google continues to integrate its own "user-triggered fetchers" into the search experience, the distinction between a traditional crawler and an AI agent will likely blur, forcing web administrators to adopt a more robust approach to server security and content management.

In summary, the events of the past week emphasize that technical compliance—whether in responding to server errors, correctly encoding URLs, or managing bot access—is no longer merely an optimization task. It is a fundamental requirement for maintaining digital presence in an era where machines, not just humans, are the primary audience for the web.