Pipet

lastmod 2024-10-17created 2024-10-171 edits

Swiss-army tool for scraping and extracting data from online assets, made for hackers

A CLI tool for scraping/extracting data from online sources

Features

• Supports 3 modes of operation: HTML parsing, JSON parsing, and client-side JavaScript execution
• Relies on existing tools like curl and works using Unix pipes
• Commands are composed in a .pipet file, which consists of multiple blocks
  • Each block has 3 sections
    • The first line is the resource URL
    • The following lines are queries specifying the data to fetch
    • A line starting with > is a selector for fetching the next page (optional)

Usage

Create a hackernews.pipet file containing this:

curl https://news.ycombinator.com/
.title .titleline
  span > a
  .sitebit a

See also

• curl

Favorite site

• Github - bjesus/pipet