Wget Command in Linux/Unix

Wget is the non-interactive network downloader which is used to download files from the server even when the user has not logged on to the system and it can work in the background without hindering the current process. 

• GNU wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies. 
  
• wget is non-interactive, meaning that it can work in the background, while the user is not logged on. This allows you to start a retrieval and disconnect from the system, letting wget finish the work. By contrast, most of the Web browsers require constant user’s presence, which can be a great hindrance when transferring a lot of data. 
  
• wget can follow links in HTML and XHTML pages and create local versions of remote web sites, fully recreating the directory structure of the original site. This is sometimes referred to as recursive downloading. While doing that, wget respects the Robot Exclusion Standard (/robots.txt). wget can be instructed to convert the links in downloaded HTML files to the local files for offline viewing. 
  
• wget has been designed for robustness over slow or unstable network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved. If the server supports resuming, it will instruct the server to continue the download from where it left off.  

Basic Syntax : 

The basic syntax of the Wget command is as follows:

wget [option] [URL]

Here

[options] represents various command-line options that modify the behavior of Wget

[URL] is the address of the file or website to be downloaded.

Options:

Here are the options available in wget command in linux.

Option	Description	Syntax
-v / –version	Display the version of Wget installed on your system.	$ wget -v
-h / –help	Print a help message displaying all available command-line options for Wget.	$ wget -h [URL]
-o logfile	Direct all system-generated messages to the specified logfile. If no logfile is specified, messages are redirected to the default logfile (‘wget-log’).	$ wget -o logfile [URL]
-b / –background	Send the process to the background as soon as it starts, allowing other processes to continue. If no output file is specified, output is redirected to ‘wget-log’ by default.	$ wget -b [URL]
-a	Append output messages to the current output logfile without overwriting it. This preserves the log of previous commands, with the current log appended after them.	$ wget -a logfile [URL]
-i	Read URLs from a file. If specified as the file, URLs are read from standard input. If URLs are present both in the command line and input file, those on the command line take precedence. The file need not be an HTML document.	$ wget -i inputfile<br>$ wget -i inputfile [URL]
-t number / –tries=number	Set the number of retry attempts. Specify ‘0’ or ‘inf’ for infinite retrying. The default is 20 retries, with exceptions for fatal errors like connection refusal or link not found.	$ wget -t number [URL]
-c	Resume a partially downloaded file if the file supports resuming. If resuming is not supported, the download cannot be resumed.	$ wget -c [URL]
-w	Set the system to wait for the specified number of seconds between retrievals. This option helps reduce server load by spacing out requests. Time can be specified in seconds, minutes (m), hours (h), or days (d).	$ wget -w number_in_seconds [URL]
-r	Enable recursive retrieval of specified links, even in the event of fatal errors. This option recursively follows links within the given URL.	$ wget -r [URL]

Example :

1. To simply download a webpage: 

To download a single file from a URL, simply specify the URL as follows:

wget http://example.com/sample.php

2. To download the file in background 

Run the download process in the background using the -b or --background option:

wget -b http://www.example.com/samplepage.php

3. To overwrite the log while of the wget command 

Redirect the output messages to a log file using the -o option: 

wget http://www.example.com/filename.txt -o /path/filename.txt

4. To resume a partially downloaded file 

If a download is interrupted, resume it using the -c option:

wget -c http://example.com/samplefile.tar.gz

5. To try a given number of times 

Specify the number of retry attempts using the --tries option: 

wget --tries=10 http://example.com/samplefile.tar.gz

6. Set Wait Time Between Retrievals:

Set the wait time between retrievals using the -w option (in seconds, minutes, hours, or days):

wget -w 10 http://example.com/large_file.zip

7. Enable Recursive Retrieval:

Enable recursive retrieval to download an entire website using the -r option:

wget -r http://example.com/

8. Read URLs from File:

Read URLs from a file and download them using the -i option:

wget -i urls.txt

Wget Command – FAQs

What is the wget command used for?

The wget command is a powerful command-line tool used in Linux and Unix-based systems to retrieve files from the internet via HTTP, HTTPS, and FTP protocols. It is particularly useful for non-interactive downloads, mirroring websites, and recursive downloads.

How do you download a file using wget?

To download a single file using wget, you can simply run the command followed by the URL of the file you want to download.

For example:

wget https://example.com/file.zip

How can I download an entire website using wget?

To download an entire website recursively, use the -m or --mirror option along with the -p or --page-requisites option to download all necessary files.

For example:

wget -m -p https://example.com

How do I limit the download speed or bandwidth when using wget?

You can use the --limit-rate option to limit the download speed or bandwidth usage.

For example: To limit the download speed to 500KB/s, you can run:

wget --limit-rate=500k https://example.com/file.zip

How can I continue a interrupted download using wget?

If a download is interrupted, you can use the -c or --continue option to resume the download from where it left off.

For example:

wget -c https://example.com/file.zip

Conclusion

In this article we discussed Wget command which is a handy tool in Linux for downloading files from the internet without needing user interaction. It works quietly in the background, which means you can start a download and do other things while it works. Wget can handle various types of web addresses and can even copy entire websites. It’s helpful for slow or unreliable internet connections because it keeps trying to download until it succeeds. Plus, it offers useful features like resuming interrupted downloads and setting wait times between retrievals. By learning its simple commands and options, users can efficiently manage their downloads and save time.