Wget & curl
Wget
Command syntax
wget [option] [URL]
Download sequence of files using Bash range syntax
wget http://example.com/file_{1..4}.webp
Useful options
| Option | Description |
|---|---|
| --continue (-c for short) | Resume a previously interrupted download. |
| --debug | Display request and response headers. |
| --header="name: value" | Add or update a request header. |
| -l depth | Maximum depth level for recursion, inf can be used for unlimited. |
| -m | Turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. |
| --no-parent | Do not ascend to the parent directory when retrieving recursively. Only the files below a certain hierarchy will be downloaded. |
| --no-proxy | Don't use proxies, even if the appropriate *_proxy environment variable is defined. |
| -O fileName | Save target as fileName. If - is used as argument, documents will be printed to standard output. |
| -r | Turn on recursive retrieving. The default maximum depth is 5. |
| --show-progress | Display the progress bar (in any verbosity). |
| --user=usr --password=pwd | Specify the username and password to be used for both FTP and HTTP authentication. |
curl
Basic usage of the command is the same as wget.
Useful options
| Option | Description |
|---|---|
| -A 'Mozilla/5.0' | Set user-agent |
| -x proxy:port | Use HTTP proxy |
| -H 'Authorization: bearer-token' | Use custom header |
| -u username:password | HTTP Basic Authentication |
| -L | Follow any redirects until final destination is reached |
| -X PUT | Set request method; if not passed GET is used by default |
| -X POST -d 'key1=val1&key2=val2' | Specify key-value pairs in request body |
| -X POST -d 'key1=val1' -d 'key2=val2' | Alternative way of sending multiple keys and values |
| -X POST -d '{"k1":v1}' -H 'Content-Type: application/json' | Use JSON data |
| -o output.txt | Save response to a file |
| -O | Download a file and save it as the same name |
| -O -C | Continue a partial download |
| -O --limit-rate 1M | Transfer rate limited to 1MB/s |