Pigz – Compress And Decompress Files In Parallel In Linux
Pigz, short for parallel implementation of gzip, is a free, open source multi-threaded compression utility to compress and uncompress files in Linux. Pigz, pronounced as pig-zee, uses the zlib and pthread libraries and takes full advantage of multiple processors and multiple cores when compressing data. Since pigz compresses using threads to make use of multiple processors and cores, It can be able to archive larger files much faster than with gzip. To put this simply, pigz does what gzip does, but it allocates the work to multiple processors and cores when compressing and speed up the compression/decompression process significantly. In this guide, let us learn to compress and decompress files in parallel using Pigz in Linux.
Install Pigz On Linux
Pigz is available in the official repositories of popular Linux operating systems.
On Arch Linux and its variants like Manjaro Linux, enable [Community] repository and install Pigz using command:
$ sudo pacman -S pigz
On CentOS 8/7/6:
$ sudo yum install pigz
$ sudo apt-get install pigz
$ sudo dnf install pigz
On Ubuntu, Linux Mint:
$ sudo add-apt-repository universe
$ sudo apt install pigz
$ sudo zypper install pigz
On other Linux distributions, compile and install from source as shown below.
First, make sure you have installed the following prerequisites:
For example, on Debian-based systems you can install the above packages using command:
$ sudo apt install make gcc zlib-devel
On RPM-based systems:
$ sudo yum install make gcc zlib-devel
Then download the latest Pigz version from the official website (Link is given at the end):
$ wget https://zlib.net/pigz/pigz-2.4.tar.gz
Extract the pigz tar file:
$ tar -xzvf pigz-2.4.tar.gz
This command will extract the contents in a folder named pigz-2.4 in the current working directory.
CD into “pigz-2.4” directory:
$ cd pigz-2.4
Compile and install it using the following command:
Copy the pigz binary to your $PATH, for example /usr/local/bin:
$ sudo cp pigz /usr/local/bin/
Log out and log back in to start using pigz.
Compress And Decompress Files In Parallel Using Pigz
Pigz divides the given input into 128 KB chunks and compresses each one of them in parallel. The compressed data format generated is in the gzip, zlib, or single-entry zip format using the deflate compression method. By default, it compresses the file gzip (.gz) format.
Let me show you some practical examples. For the purpose of this guide, I will be using archlinux.iso file (size 677.4 MB) for testing.
To compress a file with default options, simply run:
$ pigz archlinux.iso
The above command will compress the given file i.e. archlinux.iso and save it as archlinux.iso.gz in the current working directory.
Please note that the above command will delete the original file i.e. archlinux.iso after compressing it. If you don’t want to delete the original file, use -k (keep) to instruct Pigz not to delete the original file after processing it.
$ pigz -k archlinux.iso
List the contents of an archive file
To list the contents of the above archived file, without extracting it, use -l flag.
$ pigz -l archlinux.iso.gz compressed original reduced name 654901763 677380096 3.3% archlinux.iso
As you can see in the above output, the input file has been reduced by 3.3% from its original size.
Specify compression methods
Pigz supports the following compression methods:
- Slowest and best compression (-9)
- Fastest and less compression (-1)
- No compression (-0)
- Default compression (-6)
For example, to compress a file using the best compression method (slow), run:
$ pigz -9 archlinux.iso
Compress a file with less compression (fast):
$ pigz -1 archlinux.iso
$ pigz -0 archlinux.iso
Change compression format
By default, it saves the output file in gzip format. You can, however, change it to different format.
The following command compresses the given file to zlib (.zz) instead of gzip format:
$ pigz -k -z archlinux.iso
Here, we used the -k (lower case) to keep the original file after compressing it.
Similarly, to compress the file in zip format, use -K (upper case) flag:
$ pigz -k -K archlinux.iso
Pigz doesn’t have any option to directly compress folders. However, we can achieve this by combining tar command with pigz like below.
$ tar cf - Pictures/ | pigz > pictures.tar.gz
The above command will compress Pictures folder and saves it as pictures.tar.gz file.
As one of our reader Wazz said in the comment section below, we can also use the ‘use-compress-program’ parameter within tar to run pigz instead of piping it out.
$ tar --use-compress-program=pigz -cf yourfile.tar.gz folder1 folder2
Limit number of processors while compressing
Like I already said, Pigz exploits multiple processors and cores to the hilt when compressing files. You can change this behaviour using -p flag.
For example the following command will compress a file using best compression method and 4 processors and kep the original file:
$ pigz -9 -k -p4 archlinux.iso
To find the number of processor cores in Linux, refer the following guide:
We can decompress the compressed files to their original form using pigz -d or unpigz commands.
$ pigz -d archlinux.iso.gz
$ unpigz archlinux.iso.gz
For more details, refer the help section and man pages:
$ pigz --help
$ man pigz
Gzip vs Pigz – Performance comparison
I tested the archlinux.iso (size 677.4 MB) file with Pigz And Gzip utilities. Here are the results:
Compression using Gzip:
$ time gzip archlinux.iso real 0m35.086s user 0m32.719s sys 0m1.333s
Compression using Pigz:
$ time pigz archlinux.iso real 0m12.545s user 0m46.376s sys 0m1.520s
Decompression with Gzip:
$ time gzip -d archlinux.iso.gz real 0m6.847s user 0m5.621s sys 0m0.783s
Decompression with Pigz:
$ time unpigz archlinux.iso.gz real 0m2.398s user 0m2.370s sys 0m1.931s
As you can see in the above outputs, Pigz can compress and decompress a single file of 677.4 MB two times faster than with gzip.
So, if you have a modern multi-processor, multi-core system, and want to compress larger files as fast as possible, by using all of your available CPU cores, pigz will be a good choice! Give it a try and let us know your thoughts about Pigz compression utility in comment section below.