IndexNow Protocol for instant information of search engines about changes on the site

2021: Creation of a protocol IndexNow

On October 20, 2021, it became known that Microsoft and Yandex created a IndexNow protocol to independently alert search engine webmasters about changes on the site. This allows you not to wait for the site to visit the robot crawler, and reflect all updates in the search results. IndexNow works in Bing and Yandex search, other search engines are not supported.

IndexNow. Photo: searchengines.guru.

IndexNow is an initiative aimed at optimizing the efficiency of the Internet. By telling search engines whether a URL has been changed, website owners give a clear signal to help searchers prioritize scans for these URLs, thereby limiting the need for research scans to verify whether content has been changed. In the future, search engines intend to limit scanning websites that use IndexNow. says on the Microsoft website

As reported, the essence of the IndexNow is precisely to accelerate the provision of data on site changes to search engines. According to Yandex, without it, the process of detecting changes by search engines on the site can take quite a long time - from several days to several weeks.

The distribution of the IndexNow began on October 18, 2021 under the terms of the Attribution-ShareAlike Creative Commons License. However, this protocol is not available to all search engines. According to representatives of Yandex, the protocol works exclusively in Bing (owned by Microsoft) and in the search engine of Yandex itself. At first, the company reported that the IndexNow is also supported, Chinese Baidu but later added that it does not have access to this service. At the same time, the mention of Baidu on the Yandex page with a description of the service is present.

At the time of publication of the material, representatives of Yandex could not answer the request for the timing of launching support IndexNow other search engines.

Note that as of September 2021, Yandex was the second most popular search engine in the world with a share of 41.69%. It was ahead of only Google (56.07%, statistics StatCounter). Bing was on the fourth line with a tiny fraction of 0.42%.

Also, the initiative to implement IndexNow was supported by a company CloudFlare providing Content Delivery Network (CDN) services, protection against DDoS-, attacks secure access to resources servers domain and names (DNS). Support for the updated protocol is implemented as part of the Crawler Hints project launched in the summer of 2021 ("tips for crawlers"). This is a separate service that optimizes the efficiency of work traffic coming from web crawlers and. boats

The protocol is IndexNow running in version 1.0. the site owner must generate and place a special key in the site directory to verify the user before using it.

Search engines will check the key whenever you use the IndexNow protocol. It is recommended that you place it in the site root directory. In addition, if there are subdomains on the site, then you need to create a separate key for each of them.

Having fulfilled this requirement, website owners can begin to send URLs of changed or deleted pages to Bing and Yandex. To send one address, use the GET method, and if a "wholesale" send is required, use the POST method.

It is important to note that you only need to send the addresses of those pages that received changes after starting protocol support through the IndexNow. If, for example, the page was updated a month or a year ago, you do not need to notify search engines about this.

According to the MSPowerUser portal, notifying one search engine that supports IndexNow about changes on the site, the web wizard will automatically notify another. Yandex does not confirm this information, but does not refute^[1].

Notes