A new open source compression method developed by Facebook has lead to a significant reduction in storage and compute requirements for Facebook data centers.
Replacing zlib--the prior method--with Zstandard 1.0 resulted in six percent storage reduction in Facebook data warehouses, 19 percent reduction in CPU requirements for compression and 40 percent reduction in CPU requirements for decompression, said Yann Collet, Facebook compression expert.
Zstandard 1.0 compresses data about three times to five times faster than the previous method used by Facebook. Zstandard 1.0 also takes up 10 percent to 15 percent less storage space.
Compression speed also is about two times faster than zlib, the previous method. “Command line tooling numbers” are more than three times faster, as well.
No comments:
Post a Comment