Merge fastCover into DictBuilder (#1274)
* Minor fix * Run non-optimize FASTCOVER 5 times in benchmark * Merge fastCover into dictBuilder * Fix mixed declaration issue * Add fastcover to symbol.c * Add fastCover.c and cover.h to build * Change fastCover.c to fastcover.c * Update benchmark to run FASTCOVER in dictBuilder * Undo spliting fastcover_param into cover_param and f * Remove convert param functions * Assign f to parameter * Add zdict.h to Makefile in lib * Add cover.h to BUCK * Cast 1 to U64 before shifting * Remove trimming of zero freq head and tail in selectSegment and rebenchmark * Remove f as a separate parameter of tryParam * Read 8 bytes when d is 6 * Add trimming off zero frequency head and tail * Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed) * Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary * Change nbDmer to always read 8 bytes even when d=6 * Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER * Update comments and benchmarking result * Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover * Add dictType enum and fix bug about passing zParam when converting to coverParam * Combine finalize and skip into a single parameter * Update acceleration parameters and benchmark on 3 sample sets * Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets * Initialize variables outside of for loop in benchmark.c * Update benchmark result for hg-manifest * Remove cover.h from install-includes * Add explanation of f * Set default compression level for trainFromBuffer to 3 * Add assertion of fastCoverParams in DiB_trainFromFiles * Add checkTotalCompressedSize function + some minor fixes * Add test for multithreading fastCovr * Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish * Free segmentFreqs * Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment * Add FASTCOVER_MEMMULT * Minor fix * Update benchmarking result
Showing
- build/VS2008/fuzzer/fuzzer.vcproj 8 additions, 0 deletionsbuild/VS2008/fuzzer/fuzzer.vcproj
- build/VS2008/zstd/zstd.vcproj 8 additions, 0 deletionsbuild/VS2008/zstd/zstd.vcproj
- build/VS2008/zstdlib/zstdlib.vcproj 8 additions, 0 deletionsbuild/VS2008/zstdlib/zstdlib.vcproj
- build/VS2010/fuzzer/fuzzer.vcxproj 2 additions, 0 deletionsbuild/VS2010/fuzzer/fuzzer.vcxproj
- build/VS2010/libzstd-dll/libzstd-dll.vcxproj 1 addition, 0 deletionsbuild/VS2010/libzstd-dll/libzstd-dll.vcxproj
- build/VS2010/libzstd/libzstd.vcxproj 1 addition, 0 deletionsbuild/VS2010/libzstd/libzstd.vcxproj
- build/VS2010/zstd/zstd.vcxproj 2 additions, 0 deletionsbuild/VS2010/zstd/zstd.vcxproj
- build/cmake/lib/CMakeLists.txt 3 additions, 0 deletionsbuild/cmake/lib/CMakeLists.txt
- contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile 3 additions, 7 deletions.../experimental_dict_builders/benchmarkDictBuilder/Makefile
- contrib/experimental_dict_builders/benchmarkDictBuilder/README.md 824 additions, 101 deletions...experimental_dict_builders/benchmarkDictBuilder/README.md
- contrib/experimental_dict_builders/benchmarkDictBuilder/benchmark.c 34 additions, 25 deletions...perimental_dict_builders/benchmarkDictBuilder/benchmark.c
- contrib/experimental_dict_builders/fastCover/fastCover.c 2 additions, 2 deletionscontrib/experimental_dict_builders/fastCover/fastCover.c
- lib/BUCK 1 addition, 0 deletionslib/BUCK
- lib/dictBuilder/cover.c 68 additions, 79 deletionslib/dictBuilder/cover.c
- lib/dictBuilder/cover.h 83 additions, 0 deletionslib/dictBuilder/cover.h
- lib/dictBuilder/fastcover.c 701 additions, 0 deletionslib/dictBuilder/fastcover.c
- lib/dictBuilder/zdict.c 5 additions, 5 deletionslib/dictBuilder/zdict.c
- lib/dictBuilder/zdict.h 56 additions, 3 deletionslib/dictBuilder/zdict.h
- programs/README.md 1 addition, 0 deletionsprograms/README.md
- programs/dibio.c 36 additions, 14 deletionsprograms/dibio.c
Loading
Please register or sign in to comment