Robots.txt

lastmod 2025-08-26created 2021-06-029 edits

Setting up

Setting up robots.txt - Naver Search Advisor

robots.txt is an international recommendation that allows or restricts search robots from crawling a site and its web pages. The robots.txt file must always be located in the site's root directory and must be written as a plain text file that follows the Robots Exclusion Standard. The Naver search robot follows the rules written in robots.txt, and if there is no robots.txt file in the site's root directory, it assumes that all content may be crawled.

Some incomplete search robots, including web scrapers developed for specific purposes, may not follow the rules in robots.txt. So content that must not be exposed externally, including personal information, should be protected with a login feature or another blocking method.

robots.txt location

The robots.txt file must be located in the site's root directory and must be accessible as a text file (text/plain).

Example)http://www.example.com/robots.txt

Handling by HTTP response code

When the Naver search robot accesses your site's robots.txt, please return a normal 2xx response code. The search robot behaves as follows depending on the HTTP response code.

-	Response code group	Description
2xx	Successful	Interprets and uses the rules that follow the Robots Exclusion Standard. If robots.txt is returned as an HTML document, it may be interpreted as having no robots.txt (allow all) even if it contains valid rules. So it is recommended to write it as a plain text file (text/plain) that follows the robots exclusion rules.
3xx	Redirection	Allows up to 5 HTTP redirects; if more redirections occur, it stops and interprets this as 'allow all'. Redirection via HTML and JavaScript is not interpreted.
4xx	Client Error	Interpreted as 'allow all'.
5xx	Server Error	Interpreted as 'disallow all'. However, if there are robots.txt rules that were previously crawled successfully, they may be used temporarily.

robots.txt rule examples

Rules written in a robots.txt file apply only to pages under the same host, protocol, and port number. The contents of http://www.example.com/robots.txt do not apply to http://example.com/ or https://example.com/.

Typical rules are shown below; adjust them to suit the nature of your site's content.

Disallows crawling by other search engines' robots and allows crawling only by the Naver search robot.

User-agent: *
Disallow: /
User-agent: Yeti
Allow: /

Allows crawling by all search engines' robots.

User-agent: *
Allow: /

Allows crawling of only the site's root page.

User-agent: *
Disallow: /
Allow: /$

Please disallow crawling of web pages that search robots must not visit, such as administrator pages and personal information pages.

The example below tells the Naver search robot not to crawl /private-image, /private-video, and so on.

User-agent: Yeti
Disallow: /private*/

Tells all search robots that crawling of all pages of the site is not allowed.

This example is not recommended, because no page of the site will be included in crawling.

User-agent: *
Disallow: /

Please allow crawling of JavaScript and CSS files

Some sites disallow crawling of resource URLs such as JavaScript and CSS files in their robots.txt rules. In such cases, the Naver search robot may have difficulty interpreting the main areas of a page. Please allow search robots to crawl resource files, including JavaScript. For details, see the JavaScript search optimization documentation.

Please allow crawling of the favicon

If the favicon of a web page that is "shown in search results" is blocked by robots.txt, the favicon may exceptionally be crawled without following the robots.txt rules. This is because the favicon is treated as an essential component resource of a web page shown in search. Please set the internal components (css, javascript, image..) of a page shown in search with the same robots.txt rules as the page itself. For more details on favicons, see the favicon markup guide.

Specifying sitemap.xml

You can record the location of the sitemap.xml that lists the pages on your site in robots.txt, helping search robots crawl your site's content better.

User-agent: *
Allow: /
Sitemap: http://www.example.com/sitemap.xml

Block at the IP level if there are problems

See Thinkbot#Block at the IP level if there are problems.