CDC File Transfer
lastmod 2024-05-21created 2023-01-133 edits
A Windows-to-Linux file sync and streaming tool
Features
- Works quickly and efficiently using FastCDC (Content Defined Chunking)
- A tool used in Google Stadia
- Game developers usually develop on Windows and need to run on a Linux machine
- cdc_rsync and cdc_stream are open-source tools built for fast copying in this case
- Similar to the existing rsync, but syncs about 3 times faster
- The CDC-based remote diff algorithm is up to about 30 times faster
- Skips files if timestamp/file size are the same
- Uses fast compression for all data transfers
- If a file has changed, it checks which parts changed and transfers only the changed parts
- Conceptually similar to sshfs, but optimized for read speed. About 2 to 5 times faster
- Caches streamed data on the Linux device
- When a file is changed on Windows and then read again on Linux, only the changed parts are streamed and the rest is read from the cache
- Directory metadata works in a streaming-friendly way, so Stat operations are very fast