How To Quickly Transfer Large Files Over Network In Linux And Unix
Today, I had to reinstall my Ubuntu server that I use often to test different applications. My Ubuntu server has over 200 GB data, and I don’t want to lose it. I can transfer those data via scp, or setup NFS, FTP to copy files, but I am pretty sure it would take hours to transfer such big files. While searching for an alternative method, I came across the following method. We can quickly transfer large files between between two systems over Network using netcat, tar, and pv commands in any GNU/Linux and Unix-like operating systems. Unlike other methods, I find it very fast and quick. For those who don’t know, Netcat is a simple Unix utility which reads and writes data across network connections using TCP or UDP protocol. tar is a commandline archiving tool, and pv, stands for pipe viewer, is used to monitor the progress of data.
Now, allow me to show you how to transfer large files quickly between two Linux systems. It’s not as difficult as you think. Read on.
Quickly Transfer Large Files Over Network Between Two Systems In GNU/Linux
Make sure you have installed “netcat” and “pv” utilities on your systems. If they are not installed already, you can install them as shown below. The “tar” package is available by default on most Linux systems, so you don’t have to install it.
On Arch Linux and its derivatives:
$ sudo pacman -S netcat pv
On RHEL, CentOS, Fedora:
$ sudo yum install epel-release
$ sudo yum install nc pv
$ sudo dnf install nc pv
On Debian, Ubuntu, Linux Mint:
$ sudo apt-get install netcat pv
Now let us see how to quickly copy the large file(s) between two systems.
To do so, run the following command as root user on the receiving node (destination system):
# netcat -l -p 7000 | pv | tar x
On the sending node (source system), run this command as root user:
# tar cf - * | pv | netcat 192.168.1.105 7000
Here, 192.168.1.105 is my destination system. tar cf – * will copy everything in the current working directory to the destination system, and the files will be extracted at the other end.
Note: On RHEL, CentOS systems, use “nc” instead of “netcat” as shown below. And, you need to add the port “7000” to the iptables / firewall-cmd on the target system.
After adding the port on target system, you can transfer the larger files as shown below.
On destination system:
# nc -l -p 7000 | pv | tar x
On source system:
# tar cf - * | pv | nc 192.168.1.105 7000
Also, You can specify a particular file like below.
# tar cf - /home/sk/test.file | pv | netcat 192.168.1.105 7000
Please be mindful that both system should have netcat installed. Now, grab a cup of coffee. You’ll see that the files will be copied very quickly than the traditional methods like scp.
Also, you will not see any sign of the file transfer completion on both sides. These commands will keep running until you manually stop them. You need to manually check the file sizes on both systems using “du -h <filename>” command. If the file size in destination system is same as in source system, then you can assume the file transfer process is completed and quit the command by pressing CTRL+C.
Quickly Transfer Large Files Between Two Systems In Unix
On Unix operating systems, netcat is called as nc. So, to copy large files between systems over network, the command would be:
On destination system:
# nc -l 7000 | pv | tar -xpf -
On source system:
# tar -cf - * | pv | nc 192.168.1.105 7000
Again, these commands should be run as root user. And, both source and destination systems should have netcat and pv installed. Transferring large files over LAN using netcat and tar can indeed will save you a lot of time.
Disclaimer: Please be mindful that there is no security in this method. Because, as you see in the above examples, there is no authentication on either side. All you need to know is the destination system’s IP address. Transferring files using netcat is recommended only inside protected networks. If you are paranoid about security, I strongly suggest you to use scp command.
That’s it. Do you know any other way to copy large files quickly? Please share it in the comment section below. I am all ears!
I will be soon here with another interesting article. Until then, stay tuned with OSTechNix!