Thursday, September 1, 2016

New Facebook Compression Method Works 3-5 Times Faster Than Older Method

A new open source compression method developed by Facebook has lead to a significant reduction in  storage and compute requirements for Facebook data centers.

Replacing zlib--the prior method--with Zstandard 1.0 resulted in six percent storage reduction in Facebook data warehouses, 19 percent reduction in CPU requirements for compression and 40 percent reduction in CPU requirements for decompression, said Yann Collet, Facebook compression expert.

Zstandard 1.0 compresses data about three times to five times faster than the previous method used by Facebook. Zstandard 1.0 also takes up 10 percent to 15 percent less storage space.

Compression speed also is about two times faster than zlib, the previous method. “Command line tooling numbers” are more than three times faster, as well.

No comments:

Many Winners and Losers from Generative AI

Perhaps there is no contradiction between low historical total factor annual productivity gains and high expected generative artificial inte...