Skip to content
Search

Web Scraping

lastmod 2025-10-10created 2017-01-2412 edits

Web scraping, web harvesting, or web data extraction is data scraping used to extract data from websites. Web scraping software may directly access the World Wide Web using HTTP or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.

Scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when a user views a page). Therefore, web crawling is a main component of web scraping, to fetch pages for later processing. Once fetched, extraction can take place. The content of a page may be parsed, searched, and reformatted, and its data copied into a spreadsheet or loaded into a database. Web scrapers typically take something out of a page to make use of it for another purpose somewhere else. An example would be finding and copying names and telephone numbers, companies and their URLs, or e-mail addresses to a list (contact scraping).

As well as contact scraping, web scraping is used as a component of applications used for web indexing, web mining and data mining, online price change monitoring and price comparison, product review scraping (to watch the competition), gathering real estate listings, weather data monitoring, website change detection, research, tracking online presence and reputation, web mashups, and web data integration. Popular web scraping tools on the market include Octoparse, listly, and hashscraper.

Web pages are built using text-based markup languages (HTML and XHTML), and frequently contain a wealth of useful data in text form. However, most web pages are designed for human end-users and not for ease of automated use. As a result, specialized tools and software have been developed to facilitate the scraping of web pages.

Newer forms of web scraping involve monitoring data feeds from web servers. For example, JSON is commonly used as a transport mechanism between the client and the web server.

Some websites use methods to prevent web scraping, such as detecting and disallowing bots from crawling (viewing) their pages. In response, there are web scraping systems that rely on techniques in DOM parsing, computer vision, and natural language processing to simulate human browsing and collect web page content for offline parsing.

Cutting-edge web scraping techniques

Git scraping

  • What is a Git scraper?
    • Uses GitHub Actions to regularly scrape a website or resource and record changes
    • Easy to set up by cloning from a template repo on GitHub
    • Enter a specific website URL or JSON file URL and it runs automatically every day
  • Appending .atom to a repository's commits page URL produces an RSS feed
    • Website changes can be followed in real time through an RSS reader

In-browser JavaScript scraping

  • Extracting data from complex websites is difficult with simple HTML parsing alone
  • Pages can be scraped with JavaScript after loading them in a real browser
  • Extracting table data: Table data can be converted to JSON using JavaScript in the browser developer tools
  • Handling infinite-scroll pages: JavaScript can automatically collect content that keeps being added to the page
  • Automation with Shot-scraper:
    • Capture full-page screenshots with specific commands
    • Run JavaScript code and output the data as JSON

Structured data extraction with LLMs

  • LLMs (Large Language Models) can convert unstructured data into structured data
  • Requires OpenAI and Google Gemini API keys
  • Can run in Codespaces or a local Python environment
  • Data extraction with schemas
    • A schema defines the format of the data the LLM should return
    • For example, defining title, URL, date, etc. in a schema allows data to be extracted in a consistent format
    • Output can be schema-based when scraping data from web pages
  • Cost calculation and model choice
    • Each model charges based on the number of input and output tokens
    • The GPT-4o mini model is cheap, but other models can be expensive
    • For example, the GPT-4.5 model can cost far more for the same task
    • Gemini models are relatively cheap and offer various options
  • Extracting data from PDFs
    • Data can be extracted from PDF documents such as the FEMA Daily Operations Briefing
    • Gemini models can take PDF files as input and output structured JSON
    • Specific tables or text can be extracted from PDFs
  • Extracting data from images
    • GPT-4o can extract data from images
    • Feed a screenshot to the model to output structured data
  • Model selection tips
    • The Gemini 2.0 Pro model is free but has strict rate limits
    • Test several models and choose based on cost efficiency and performance
    • Depending on the web page, shot-scraper may be more useful than an LLM
  • Challenges with complex data
    • Model performance can vary when extracting data from complex infographics or maps
    • Try various models and choose the one that performs best

Video scraping with Google AI Studio

  • Video scraping is a powerful way to extract data from websites that are hard to scrape normally
  • Google Gemini models can take video input and convert it into structured JSON data
  • Record the screen, then feed the video to an AI model to extract data
  • Video scraping process
    • Navigate to the section of the website containing the desired data
    • Start a screen recording tool (e.g., QuickTime Player)
    • Browse the website → pause briefly at important parts
    • Upload the recorded video to Google AI Studio
    • Write a model prompt in AI Studio and extract the data
    • Use the schema extraction tool to organize the data as JSON
  • Advantages of video scraping
    • Can bypass complex website structures
    • Useful for sites that require JavaScript rendering
    • Can extract a variety of data on the page (text, tables, images, etc.)
  • Caveats
    • Most features in AI Studio are free, but the input data may be used for training
    • Avoid entering security-sensitive data
    • For complex websites, video scraping can be more efficient than other scraping techniques
  • Real-time streaming option
    • The Stream Realtime option in AI Studio enables real-time data extraction
    • Can be effective even for dynamic content that is hard to scrape
  • Use in newsrooms
    • Useful for complex data extraction and automation in newsrooms
    • Test the performance and accuracy of Gemini models to find the best approach
    • Collaboration and feedback are possible after the workshop

Additional tools and resources

List of local files

ZetCode, tutorials for programmers
http://zetcode.com/
Zetcode.com-170124.zip
wxPython, wxWidgets
Commonly used features in a side-by-side format
http://hyperpolyglot.org/
Hyperpolyglot.org.zip
Programming language

See also

Favorite site

author Zer0 sitemap