Skip to content
Snippets Groups Projects
  1. Nov 15, 2018
    • Nick Terrell's avatar
      Merge pull request #1416 from terrelln/rsync · f15f1bfe
      Nick Terrell authored
      - Factor out LDM's hash function for reuse
      - Add rsyncable mode to zstdmt and expose it via the advanced API
      - Fix `-B`/`--block-size` to actually set the job size
      - Add rsyncable tests to `zstreamtest` and `playTests.sh`
      
      Tested by:
      
      ```
      > cat A.100MB B.100MB C.100MB D.100MB E.100MB | zstd --rsyncable -fo src/file.zst
      /*stdin*\            : 48.22%   (524288000 => 252837782 bytes, src/file.zst)
      > rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
      Total bytes sent: 252868779
      total size is 252837782  speedup is 1.00
      > echo test > test
      > cat test A.100MB test B.100MB test C.100MB test D.100MB test E.100MB | zstd --rsyncable -fo src/file.zst
      /*stdin*\            : 48.23%   (524288025 => 252838025 bytes, src/unicorn.tar.zst)
      > rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
      Total bytes sent: 4605696
      total size is 252838025  speedup is 53.60
      ```
      
      Close #1155.
      f15f1bfe
    • Nick Terrell's avatar
      [tests] Add rsyncable tests · 71fe78cb
      Nick Terrell authored
      71fe78cb
    • Nick Terrell's avatar
      [cli] Set the job size · 43164094
      Nick Terrell authored
      43164094
    • Nick Terrell's avatar
      [cli] Add rsyncable · f9a671ac
      Nick Terrell authored
      f9a671ac
    • Nick Terrell's avatar
      [lib] Add rsyncable mode · b9693d3a
      Nick Terrell authored
      - Add rsyncable mode to multithreaded mode
      - Factor out LDM's hash function for reuse
      b9693d3a
  2. Nov 14, 2018
  3. Nov 13, 2018
  4. Nov 12, 2018
  5. Nov 09, 2018
    • Yann Collet's avatar
      Merge pull request #1411 from facebook/prefetch_dict · 7b0c551b
      Yann Collet authored
      Improves decompression speed when using cold dictionary
      7b0c551b
    • Yann Collet's avatar
      Merge pull request #1410 from facebook/prefetch_dec · 1b4a9c51
      Yann Collet authored
       improve long-range decoder speed
      1b4a9c51
    • Yann Collet's avatar
      Improves decompression speed when using cold dictionary · 483759a3
      Yann Collet authored
      by triggering the prefetching decoder path
      (which used to be dedicated to long-range offsets only).
      
      Figures on my laptop :
      no content prefetch : ~300 MB/s (for reference)
      full content prefetch : ~325 MB/s (before this patch)
      new prefetch path : ~375 MB/s (after this patch)
      
      The benchmark speed is already significant,
      but another side-effect is that this version
      prefetch less data into memory,
      since it only prefetches what's needed, instead of the full dictionary.
      
      This is supposed to help highly active environments
      such as active databases,
      that can't be properly measured in benchmark environment (too clean).
      
      Also :
      fixed the largeNbDict test program
      which was working improperly when setting nbBlocks > nbFiles.
      483759a3
  6. Nov 08, 2018
  7. Nov 06, 2018
Loading