Skip to content
Search

Wget

lastmod 2024-07-29created 2014-10-1725 edits

GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

How to use 

$ wget [OPTION]... [URL]...

Commandline options

Startup

  • -V, --version: Version information.
  • -h, --help: Show help.
  • -b, --background: Go to background after startup.
  • -e, --execute=COMMAND: Execute a wgetrc-style command.

Logging and input file

  • -o, --output-file=FILE: Log messages to a file.
  • -a, --append-output=FILE: Append messages to a file.
  • -d, --debug: Print debug output.
  • -q, --quiet: Quit without output.
  • -v, --verbose: Print verbose messages. (default option)
  • -nv, --non-verbose: Do not print verbose messages.
  • -i, --input-file=FILE: Read the URLs to download from a file.
  • -F, --force-html: Treat input files as HTML.
  • -B, --base=URL, --sslcertfile=FILE, --sslcertkey=KEYFILE: Authenticate ssl using a URL relative to the file or links, and a keyfile.

Download

  • --bind-address=ADDRESS: Set the address of the local host.
  • -t, --tries=NUMBER: Retry the given number of times. 0 means unlimited.
  • -O, --output-document=FILE: Write documents to a file.
  • -nc, --no-clobber: Do not copy files with the same file name.
    • When a file is downloaded more than once into the same directory, Wget's behavior depends on a few options, including -nc.
    • In some cases, the local file is clobbered or overwritten on repeated downloads. In other cases, it is preserved.
  • -c, --continue: Allow resuming a partially downloaded file.
  • --dot-style=STYLE: Set the retrieval style.
  • -N, --timestamping: Do not re-download files unless they are newer than the local ones.
  • -S, --server-response: Print the server response.
  • --spider: Do not download anything.
  • -T, --timeout=SECONDS: Set the timeout in seconds.
  • -w, --wait=SECONDS, --waitretry=SECONDS: Set the wait time between retrievals in seconds.
  • --random-wait: wait from 0.5WAIT...1.5WAIT secs between retrievals
  • -Y, --proxy=on/off: Turn proxy mode on or off.
  • -Q, --quota=NUMBER: Set the quota to the given number.

Directories

  • -nd, --no-directories: Do not create directories.
  • -x, --force-directories: Force creation of directories.
  • -nH, --no-host-directories: Do not create host-name directories.
  • -P, --directory-prefix=PREFIX: Save files to the given location.
  • --cut-dirs=NUMBER: Remove and ignore the given number of directory components.

Http options

  • --http-user=USER: Set the http user ID.
  • --http-passwd=PASS: Set the http password.
  • -C, --cache=on/off: Server-side cached data is normally allowed, but can be disallowed.
  • -E, --html-extension: Save all text/html documents with the .html extension.
  • --ignore-length: Ignore the Content-Length header field.
  • --header=STRING: Insert a string among the headers.
  • --proxy-user=USER: Set the proxy user name.
  • --proxy-passwd=PASS: Set the proxy password.
  • --referer=URL: Add a URL referrer to the HTTP request header.
  • -s, --save-headers: Save HTTP headers to the file.
  • -U, --user-agent=AGENT: Identify as an agent instead of Wget/Version.
  • --no-http-keep-alive: Disable HTTP Keep-alive (persistent connections).
  • --cookies=off: Do not use cookies.
  • --load-cookies=FILE: Load cookies from a file before the session.
  • --save-cookies=FILE: Save cookies to a file after the session.

Ftp options

  • -nr, --dont-remove-listing: Do not remove the .listing files.
  • -g, --glob=on/off: Turn file name globbing on or off.
  • --passive-ftp: Use the passive transfer mode.
  • --retr-symlinks: When recursing, fetch linked files (not directories).

Recursive retrieval

  • -r, --recursive: Recursively suck in the web. Use with caution!
  • -l, --level=NUMBER: Maximum recursion depth. A depth of 0 or inf recurses without limit.
  • --delete-after: Delete the local files after downloading them.
  • -k, --convert-links: Convert non-relative links to relative ones.
  • -K, --backup-converted: Before converting file X, back it up as X.orig.
  • -m, --mirror: Turn on options suitable for mirroring.
    • This option turns on recursion (-r) and time-stamping (-N), sets infinite recursion depth (-l inf), and keeps FTP directory listings (--no-remove-listing).
    • Equivalent to the options -r -N -l inf --no-remove-listing.
  • -p, --page-requisites: Get all images and everything else needed to display the HTML page.

Recursive accept/reject

  • -A, --accept=LIST: Comma-separated list of accepted extensions.
  • -R, --reject=LIST: Comma-separated list of rejected extensions.
  • -D, --domains=LIST: Comma-separated list of accepted domains.
  • --exclude-domains=LIST: Comma-separated list of rejected domains.
  • --follow-ftp: Follow FTP links from HTML documents.
  • --follow-tags=LIST: Comma-separated list of HTML tags to follow.
  • -G, --ignore-tags=LIST: Comma-separated list of HTML tags to ignore.
  • -H, --span-hosts: Go to other hosts when recursing.
  • -L, --relative: Follow relative links only.
  • -I, --include-directories=LIST: List of allowed directories.
  • -X, --exclude-directories=LIST: List of excluded directories.
  • -nh, --no-host-lookup: Do not use DNS host lookup.
  • -np, --no-parent: Do not ascend to the parent directory.

Example

FTP Download

To download a specific file through an FTP address: 

$ wget --ftp-user=[ID] --ftp-password=[PASSWORD] -r [URL] -O [OUTPUT]

Downloading just one page

Downloading just one page (and all the images, fonts, JS, CSS) 

$ wget -E -H -k -K -p http://<site>/<document>

Downloading an entire site

To download an entire site: 

$ wget -U -nc -np -r -k -l 0 [URL]
HTTP Error 403 – Forbidden.
If you get a 403 error, add the -U option (user agent).
Restricting external hosts.
The -L option (leLative) fetches only links that use relative addresses.
Conversely, the -H option (span-hosts) recursively traverses other hosts.
Downloading subdirectories.
The -r option downloads all files in subdirectories.
Converting to relative links.
The -k option converts absolute links to relative links.
Preventing duplicate file copies.
The -nc option (not-clobber) does not copy files with the same file name.
Preventing parent directory traversal.
The -np option (no-parent) does not traverse parent folders.
Limiting directory depth.
The -l [DEPTH] option limits how many levels of subdirectories are entered (DEPTH).
If [DEPTH] is 0, it follows links to the end and downloads every file in all subdirectories.
Fetching images.
The -p option gets all images and everything else needed to display the HTML page.

Downloading an entire site (HTTPS)

To download without certificate verification: 

$ wget --no-check-certificate -c -m -U -nc -np -r -k -l 0 [URL]

Web crawling

The command for crawling is as follows. 

$ wget -U -H -r -nc -p -l 0 [URL]

Ignore robots.txt 

$ wget -e robots=off [URL]

Random wait 

$ wget --wait=1 --random-wait [URL]

Disable HTTP redirect 

$ wget --max-redirect=0 [URL]

Healthcheck

When you need to use it in place of curl in a Dockerfile: 

HEALTHCHECK  --interval=5m --timeout=3s \
  CMD wget --no-verbose --tries=1 --spider http://localhost/ || exit 1
  • --no-verbose - Turn off verbose without being completely quiet (use -q for that), which means that error messages and basic information still get printed.
  • --tries=1 - If not set, some wget implementations will retry indefinitely when HTTP 200 response is not returned.
  • --spider - Behave as a Web spider, which means that it will not download the pages, just check that they are there.
    • exit 1 - Ensures exit code 1 on failure. Heathcheck only expects the following:
      • 0: success - the container is healthy and ready for use
      • 1: unhealthy - the container is not working correctly
      • 2: reserved - do not use this exit code

For reference, the curl version is: 

HEALTHCHECK --interval=5m --timeout=3s \
  CMD curl -f http://localhost/ || exit 1

See also

Favorite site

References

  1. Gnu-software-wget.zip ↩

author Zer0 sitemap