Skip to content
Snippets Groups Projects
  • Nick Terrell's avatar
    [regression] Add dictionary support · e8598623
    Nick Terrell authored
    Dictionaries are prebuilt and saved as part of the data object.
    The config decides whether or not to use the dictionary if it is
    available. Configs that require dictionaries are only run with
    data that have dictionaries. The method will skip configs that are
    irrelevant, so for example ZSTD_compress() will skip configs with
    dictionaries.
    
    I've also trimmed the silesia source to 1MB per file (12 MB total),
    and added 500 samples from the github data set with a dictionary.
    
    I've intentionally added an extra line to the `results.csv` to make
    the nightly build fail, so that we can see how CircleCI reports it.
    
    Full list of changes:
    
    * Add pre-built dictionaries to the data.
    * Add `use_dictionary` and `no_pledged_src_size` flags to the config.
    * Add a config using a dictionary for every level.
    * Add a config that specifies no pledged source size.
    * Support dictionaries and streaming in the `zstdcli` method.
    * Add a context-reuse method using `ZSTD_compressCCtx()`.
    * Clean up the formatting of the `results.csv` file to align columns.
    * Add `--data`, `--config`, and `--method` flags to constrain each
      to a particular value. This is useful for debugging a failure
      or debugging a particular config/method/data.
    e8598623