Nikhil Bhaskar- May 22, 2021
What is curl utility and how its work in Linux
curl is a command line utility for transferring data from to a server designed to work without user interaction. With the use of curl, you can download or upload data using one of the support protocols including HTTP, HTTPS, SCP, FTP. curl provides a number of options which allows you to resume transfer, proxy support, user authentication, limit of bandwidth and many more.
Install curl utility
curl package is pre-installed on most linux distributions. To check the curl package is installed in system then open terminal and type curl in the terminal and press enter. If curl installed it will print curl: try ‘curl –help’ or ‘curl –manual’ for more information. If in case will see something like
curl not command not found
Install Curl on Ubuntu and Debian
sudo apt update
sudo apt install curl
Install Curl on CentOS and Fedora
sudo yum install curl
How to use curl utility
The syntax for the curl command is as follows:
curl [options] [URL...]
Curl will displays the specified resources to the standard outputs.
For example:- to retrieve sample.com homepage then run below command in terminal:
curl sample.com
The command will print the source code of the sample.com homepage in your terminal.
Save the Output to a File
To save the result of curl command use -o or -O option.
Lowercase -o saves the file with a predefined files as shown below:
curl -o vue-v2.6.10.js https://cdn.jsdelivr.net/npm/vue/dist/vue.js
Uppercase -O saves the file with its original file:
curl -O https://cdn.jsdelivr.net/npm/vue/dist/vue.js
Download Multiple files
To download multiple files at once use multiple -O options followed by the URL.
In the following example we are downloading the Arch Linux and Debian iso files:
curl -O http://mirrors.edge.kernel.org/archlinux/iso/2018.06.01/archlinux-2018.06.01-x86_64.iso \ -O https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
Resume Download
To resume a download by using the -C option. It is very useful if your connection drops during download of a large file and instead of starting the download from scratch, you can continue the previous one.
For example, if you are downloading the Ubuntu 18.04 iso file using the command:
curl -O http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso
and suddenly your connection drops you can resume the download with:
curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso
Get the HTTP Headers of a URL
HTTP headers are colon-separated key-value pairs containing information such as user agent, content type, and encoding. Headers are passed between the client and the server with the request or the response.
Use -I option to fetch only the HTTP headers of the specified resource:
curl -I --http2 https://www.ubuntu.com/
Test if a Website Supports HTTP/2
To check whether a particular URL supports HTTP2 protocols fetch the HTTP Headers with -I along with the --http2 option:
curl -I --http2 -s https://linux.com/ | grep HTTP
The -s option tells curl to run in a silent (quiet) and hide the progress meter and error messages.
If the remote server supports HTTP/2, curl prints HTTP/2.0 200:
HTTP/2 200
Otherwise, the response is HTTP/1.1 200:
HTTP/1.1 200 OK
If curl version 7.47.0 or newer then need to use --http2 option because HTTP/2 its enabled by default for all HTTPS connections.
Follow Redirects
By default, curl doesn’t follow the HTTP Location headers.
If you try to retrieve the non-www version of google.com, you will notice that instead of getting the source of the page you’ll be redirected to the www version:
curl google.com
The -L option instructs curl to follow any redirect until it reaches the final destination:
curl -L google.com
Change the User-Agent
Sometimes when downloading a file, the remote server may be set to block the Curl User-Agent or to return different contents depending on the visitor device and browser.
In this condition use the -A option.
For example to emulates Firefox 60 you would use:
curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" https://getfedora.org/
Specify a Maximum Transfer Rate
The --limit-rate option allows you to limit the data transfer rate. The value can be expressed in bytes, kilobytes with k suffix, megabytes with the m suffix, and gigabytes with the g suffix.
In the following example curl will download Go binary and limit the download speed to 1 mb:
curl --limit-rate 1m -O https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz
This option is useful to prevent curl consuming all the available bandwidth.
Transfer Files via FTP
To access a protected FTP server with curl, use the -u option and specify the username and password as shown below:
curl -u FTP_USERNAME:FTP_PASSWORD ftp://ftp.example.com/
Once logged in the command lists all files and directories in the user’s home directory.
You can download a single file from the FTP server using the following syntax:
curl -u FTP_USERNAME:FTP_PASSWORD ftp://ftp.example.com/file.tar.gz
To upload a file to the FTP server use -T followed by the name of the file you want to upload:
curl -T newfile.tar.gz -u FTP_USERNAME:FTP_PASSWORD ftp://ftp.example.com/
Send Cookies
Sometimes you may need to make an HTTP request with specific cookies to access a remote resource or to debug an issue.
By default when requesting a resource with curl, no cookies are sent or stored.
To send cookies to the server, use -b switch followed by a filename containing the cookies or a string.
For example, to download the Oracle Java JDK, rpm files jdk-10.0.2_linux-x64_bin.rpm you’ll need to pass a cookie named oracle license with value a:
curl -L -b "oraclelicense=a" -O http://download.oracle.com/otnpub/java/jdk/10.0.2+13/19aef61b38124481863b1413dce1855f/jdk-10.0.2_linux-x64_bin.rpm
Using Proxies
curl supports different types of proxies, including HTTP, HTTPS and SOCKS. To transfer data through a proxy server, use the -x (--proxy) option, followed by the proxy URL.
The following command downloads the specified resource using a proxy on 192.168.44.1 port 8888:
curl -x 192.168.44.1:8888 http://amazon.com/
If the proxy server requires authentication, use the -U (--proxy-user) option followed by the user name and password separated by a colon (user:password):
curl -U username:password -x 192.168.44.1:8888 http:/sample.com/
curl is a command-line tool that allows you to transfer data from or to remote host. It is useful for troubleshooting issues, downloading files, and more.