Linux: Download Website: wget, curl

By Xah Lee. Date: . Last updated: .

Here's how to download websites, 1 page or entire site.

wget

Download 1 Web Page

# download a file
wget http://example.org/somedir/largeMovie.mov

Download Entire Website

# download website, 2 levels deep, wait 9 sec per page
wget --wait=9 --recursive --level=2 http://example.org/

Some sites check on user agent. (user agent basically means browser). so you might add this option “--user-agent=”.

wget http://example.org/ --user-agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36'

What's My User Agent String

Your user agent string is:

curl

Download Image Sequence

# download all jpg files named cat01.jpg to cat20.jpg
curl -O http://example.org/xyz/cat[01-20].jpg
# download all jpg files named cat1.jpg to cat20.jpg
curl -O http://example.org/xyz/cat[1-20].jpg

Other useful options are:

Note: curl cannot be used to download entire website recursively. Use wget for that.

Linux Shell Basics

  1. Get System Info
  2. Shell Basics
  3. grep, cat, awk, uniq
  4. sort
  5. find, xargs
  6. diff Files/Dir
  7. dir size: du
  8. dir tree
  9. tar gzip bzip2 xz 7zip rar zip
  10. wget, curl, GET, HEAD
  11. rsync
  12. Install Packages

Sys Admin

  1. Job Control
  2. ps
  3. top
  4. htop
  5. RAM stat
  6. Users and Groups
  7. File Permission
  8. Opened Files: lsof
  9. shutdown, sleep

Bash/Terminal

  1. Bash Keys, Terminal Keys, Man Page Keys
  2. Bash Prompt Setup
  3. Bash Color Prompt
  4. .bashrc, .profile, .bash_profile
  5. Virtual Console
  6. Terminal Control Sequence Keys
  7. Reset Terminal
  8. tmux
  9. man page
  10. Bash Manual in Chapters
  11. BASH Shell Misc Tips
  12. Log Terminal Session

Linux Desktop

  1. Most Useful GUI Apps
  2. Set Default App
  3. Shell Commands for GUI Apps
  4. Image Viewers
  5. Music Players
  6. Move File to Trash by Command
  7. X11 Selection and Clipboard
  8. How to Switch to LXDE, Xfce
  9. LXDE Keyboard Shortcuts
  10. LXDE/Openbox, Change Keyboard Shortcuts
  11. LXDE Set Key Repeat Rate
  12. LXDE/OpenBox, Disable Mouse Scroll Wheel Hide Window
  13. Xfce Keyboard Shortcuts
  14. Xfce Good Themes
  15. xmonad Keybinding
  16. How to Restart X11
  17. Why Tiling Window Manager Sucks
  18. Standard Fonts
  19. How to Install Font

If you have a question, put $5 at patreon and message me.