Repository navigation
add compression #20
Description
Snappy or LZ4 comes to mind: https://github.com/BurntSushi/rust-snappy, https://github.com/bozaro/lz4-rs
Snappy or LZ4 comes to mind: https://github.com/BurntSushi/rust-snappy, https://github.com/bozaro/lz4-rs
Activity
lunemec commentedon Aug 3, 2016
Ideal would be something that can compress the stream in multiple threads to achieve highest throughput. This could be tested for both libraries though.
rmmh commentedon Aug 6, 2016
Note that the best compressor to use depends on the particular combination of compression speed, bandwidth, and decompression speed. Single-threaded zlib is better than LZ4 at <~5MB/s, ZSTD is the best at <32MB/s (but doesn't have a pure-rust implementation), etc.
Some more details and theory here: http://fastcompression.blogspot.com/2015/01/zstd-stronger-compression-algorithm.html
mcginty commentedon Aug 8, 2016
The crypto is currently not much of a bottleneck, so I think it's also wondering how much we want to burn the laps of our users :).