Skip to content
  • Joe Thornber's avatar
    [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. · 61de3f92
    Joe Thornber authored
    The Rust implementation is multithreaded, performs better in general and
    does custom compression of btree nodes to achieve much better compression
    ratios.  unpack also checksums expanded metadata to validate it.
    
    Format version has jumped to 3, no backwards compatibility, but I think
    that's ok since we never made a release that contained the C++ version
    of these tools.
    
    Benchmarks
    ==========
    
    On an 8 core, 16 hyperthread machine.
    
    metadata 1G, full:
    
          Pack size    pack time     unpack time
    ------------------------------------------------------
    C++      193M        50.3s          6.9s (no verify)
    Rust      70M         1.4s          1.8s (verify)
    
    metadata 16G, sparse:
    
           Pack size    pack time     unpack time
    ------------------------------------------------------
    C++      21M          68s           1s   (no verify)
    Rust      4M           8.6s         0.5s (verify)
    61de3f92