Web crawler
lastmod 2025-10-10created 2015-10-1422 edits
A web crawler is a computer program that browses the World Wide Web in a methodical, automated manner. Other terms for web crawlers include ants, automatic indexers, bots, worms, web spider, and web robot.
The work a web crawler does is called web crawling or spidering. Many sites, such as search engines, crawl the web to keep their data up to date. Web crawlers are mostly used to create copies of all pages of the sites they visit, and search engines index the pages created this way for faster searching. Crawlers are also used for automatic website maintenance tasks such as link checking and HTML code validation, and for collecting specific types of information from web pages, such as automatic email harvesting.
A web crawler is a form of bot or software agent. A web crawler usually starts from a list of URLs called seeds, and recognizes all the hyperlinks on a page to update the URL list. The updated URL list is visited again recursively.
Categories
- Web Scraping (Korean: 웹 스크래핑, Korean: 웹 스크래퍼)
- Selenium WebDriver
- WebDriver
- Beautiful Soup - Python package that makes it easy to build various scraping apps
- PhantomJS
- Web archiver
How to avoid Bot checks when web crawling
- 7 Ways to hide your Bot Automation from Detection | How to make Selenium undetectable and stealth
- Remove the Navigator.Webdriver flag
- Remove specific strings from ChromeDriver / GeckoDriver
- Change resolution, User-Agent, etc.
- Use real URL flows. Avoid visiting invisible links
- Change IP using a Proxy
- Use random delay values
- Do not use a headless browser (detected by checking Notification Permission)