Skip to content
Snippets Groups Projects
  1. Dec 01, 2018
    • Nick Terrell's avatar
      [regression] Add dictionary support · e8598623
      Nick Terrell authored
      Dictionaries are prebuilt and saved as part of the data object.
      The config decides whether or not to use the dictionary if it is
      available. Configs that require dictionaries are only run with
      data that have dictionaries. The method will skip configs that are
      irrelevant, so for example ZSTD_compress() will skip configs with
      dictionaries.
      
      I've also trimmed the silesia source to 1MB per file (12 MB total),
      and added 500 samples from the github data set with a dictionary.
      
      I've intentionally added an extra line to the `results.csv` to make
      the nightly build fail, so that we can see how CircleCI reports it.
      
      Full list of changes:
      
      * Add pre-built dictionaries to the data.
      * Add `use_dictionary` and `no_pledged_src_size` flags to the config.
      * Add a config using a dictionary for every level.
      * Add a config that specifies no pledged source size.
      * Support dictionaries and streaming in the `zstdcli` method.
      * Add a context-reuse method using `ZSTD_compressCCtx()`.
      * Clean up the formatting of the `results.csv` file to align columns.
      * Add `--data`, `--config`, and `--method` flags to constrain each
        to a particular value. This is useful for debugging a failure
        or debugging a particular config/method/data.
      e8598623
  2. Nov 30, 2018
  3. Nov 29, 2018
    • Nick Terrell's avatar
      [regression] Add initial regression test framework · 4aaa36f7
      Nick Terrell authored
      The regression tests run nightly or on the `regression`
      branch for convenience. The results get uploaded as the
      artifacts of the job. If they change, check the diff
      printed in the job. If all is well, download the new
      results and commit them to the repo.
      
      This code will only run on a UNIX like platform. It
      could be made to run on Windows, but I don't think that
      it is necessary. It also uses C99.
      
      * data: This module defines the data to run tests on.
        It downloads data from a URL into a cache directory,
        checks it against a checksum, and unpacks it. It also
        provides helpers for accessing the data.
      * config: This module defines the configs to run tests
        with. A config is a set of API parameters and a set of
        CLI flags.
      * result: This module is a helper for method that defines
        the result type.
      * method: This module defines the compression methods
        to test. It is what runs the regression test using the
        data and the config. It reports the total compressed
        size, or an error/skip.
      * test: This is the test binary that runs the tests for
        every (data, config, method) tuple, and prints the
        results to the output file and stderr.
      * results.csv: The results that the current commit is
        expected to produce.
      4aaa36f7
  4. Nov 28, 2018
  5. Nov 27, 2018
    • Yann Collet's avatar
      Merge pull request #1426 from lzutao/upgrade-mesonbuild · 74f8a5c5
      Yann Collet authored
      Update meson build system
      74f8a5c5
    • Lzu Tao's avatar
      Update meson build system · 9a721e52
      Lzu Tao authored
      NOTE: This commit only tested on Linux (Ubuntu 18.04). Windows
      build may not work as expected.
      
      * Use meson >= 0.47.0 cause we use install_man function
      * Add three helper Python script:
        * CopyFile.py: To copy file
        * CreateSymlink.py: To make symlink (both Windows and Unix)
        * GetZstdLibraryVersion.py: Parse lib/zstd.h to get zstd version
        These help emulating equivalent functions in CMake and Makefile.
      * Use subdir from meson to split meson.build
        * Add contrib build
        * Fix other build
      * Add new build options
        * build_programs: Enable programs build
        * build_contrib: Enable contrib build
        * build_tests: Enable tests build
        * use_static_runtime: Link to static run-time libraries on MSVC
        * zlib_support: Enable zlib support
        * lzma_support: Enable lzma support
      9a721e52
    • Lzu Tao's avatar
    • Lzu Tao's avatar
      Add meson build guide · 2abd5139
      Lzu Tao authored
      2abd5139
    • Yann Collet's avatar
      Merge pull request #1419 from ryandesign/universal · 64569a18
      Yann Collet authored
      Fix feature detection with multiple -arch flags
      64569a18
  6. Nov 16, 2018
  7. Nov 15, 2018
    • Nick Terrell's avatar
      Merge pull request #1416 from terrelln/rsync · f15f1bfe
      Nick Terrell authored
      - Factor out LDM's hash function for reuse
      - Add rsyncable mode to zstdmt and expose it via the advanced API
      - Fix `-B`/`--block-size` to actually set the job size
      - Add rsyncable tests to `zstreamtest` and `playTests.sh`
      
      Tested by:
      
      ```
      > cat A.100MB B.100MB C.100MB D.100MB E.100MB | zstd --rsyncable -fo src/file.zst
      /*stdin*\            : 48.22%   (524288000 => 252837782 bytes, src/file.zst)
      > rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
      Total bytes sent: 252868779
      total size is 252837782  speedup is 1.00
      > echo test > test
      > cat test A.100MB test B.100MB test C.100MB test D.100MB test E.100MB | zstd --rsyncable -fo src/file.zst
      /*stdin*\            : 48.23%   (524288025 => 252838025 bytes, src/unicorn.tar.zst)
      > rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
      Total bytes sent: 4605696
      total size is 252838025  speedup is 53.60
      ```
      
      Close #1155.
      f15f1bfe
    • Nick Terrell's avatar
      [tests] Add rsyncable tests · 71fe78cb
      Nick Terrell authored
      71fe78cb
    • Nick Terrell's avatar
      [cli] Set the job size · 43164094
      Nick Terrell authored
      43164094
    • Nick Terrell's avatar
      [cli] Add rsyncable · f9a671ac
      Nick Terrell authored
      f9a671ac
    • Nick Terrell's avatar
      [lib] Add rsyncable mode · b9693d3a
      Nick Terrell authored
      - Add rsyncable mode to multithreaded mode
      - Factor out LDM's hash function for reuse
      b9693d3a
  8. Nov 14, 2018
  9. Nov 13, 2018
  10. Nov 12, 2018
Loading