Skip to content
Search

Data Version Control

lastmod 2026-09-30created 2024-07-0925 edits

🦉 ML Experiments and Data Management with Git

DVC alone cannot version data. DVC generates metadata and configuration files used to track data, and git tracks the files generated by DVC to handle versioning.

Install

Using pip: 

pip install dvc

Depending on the type of the remote storage you plan to use, you might need to install optional dependencies: [s3], [gdrive], [gs], [azure], [ssh], [hdfs], [webdav], [oss]. Use [all] to include them all.

Using snap: 

snap install --classic dvc

Remote Storage : Supported storage types

Cloud providers

Self-hosted / On-premises

Initializing a repository

Move to the folder containing .git and run 

dvc init

and the following files are added.

  • .dvc/.gitignore
  • .dvc/config
  • .dvcignore

Adding a remote

-d is the same as the --default option. 

## SSH/SFTP
dvc remote add -d yournas ssh://yournas/volume5/20TB_DATA/DVC/ddrm

## WebDAV Over TLS
dvc remote add -d yournas webdavs://yournas:5006/20TB_DATA/DVC/ddrm

Configuring a remote

Using modify saves to the .dvc/config file.

Using the --local option uses a Git-ignored local configuration file (located in .dvc/config.local).

Adding connection information (user settings, not repository settings)

  • dvc remote modify --local {리모트이름} user {사용자명} - set the username
  • dvc remote modify --local {리모트이름} password {비밀번호} - set the password
  • dvc remote modify --local {리모트이름} port {포트번호} - change the port number
  • dvc remote modify {리모트이름} ask_password true - whether to prompt for the password when connecting, true or false

When connecting with an SSH KEY:

  • dvc remote modify --local {리모트이름} keyfile {/path/to/keyfile}
  • dvc remote modify {리모트이름} ask_passphrase true
  • dvc remote modify --local {리모트이름} passphrase mypassphrase

SSL verification:

  • dvc remote modify {리모트이름} ssl_verify false - SKIP verification

Quick start

Not really sure; after dvc init, proceed as follows.

INFORMATION

The directory must exist at the upload path in the remote repository URL written below.

.dvc/config file: 

[core]
    remote = yournas
    autostage = true
['remote "yournas"']
    url = webdavs://yournas:5006/20TB_DATA/DVC/업로드할경로
    ask_password = true
    ssl_verify = false

Update the hosts file and add user information: 

sudo echo "{IP} yournas" >> /etc/hosts
dvc remote modify --local yournas user ${USER}

.dvc/config.local file: 

['remote "yournas"']
    user = yourid

.dvc/.gitignore file: 

/config.local
/tmp
/cache

Then add the files you want to add. For example, for cvp/assets/*.sqlite files: 

dvc add cvp/assets/*.sqlite

Add the related files with git or not, as you see fit, then 

dvc commit
dvc push

and you're done.

Adding tracking

Add the file or directory you want. 

dvc add ./ddrm/assets/checkpoints/

The output looks like this: 

100% Adding...|██████|1/1 [00:04,  4.35s/file]

To track the changes with git, run:

        git add ddrm/assets/checkpoints.dvc

To enable auto staging, run:

        dvc config core.autostage true

DVC itself has no history management, so add it with git: 

git add ddrm/assets/checkpoints.dvc

Removing tracking 

dvc remove {삭제할데이터파일명.dvc}

Removing files from the remote repository

WARNING

Needs verification

Delete files in the remote repository that are not tracked in the dvc workspace 

dvc gc -w -c -r {원격저장소명} -f

Checking status

Output is based on *.dvc files. 

dvc status

But viewed this way, it's hard to tell what it means.

Adding "data" gives somewhat git status-like output. 

dvc data status

For example, the output looks like this: 

Not in cache:
  (use "dvc fetch <file>..." to download files)
        ddrm/assets/checkpoints/mobile_sam.pt
        ddrm/assets/checkpoints/rtmdet-ins_x_8xb16-300e_coco-cam1-epoch_300.pth
        ddrm/assets/checkpoints/rtmdet-ins_x_8xb16-300e_coco_2nd-epoch_300.pth

DVC uncommitted changes:
  (use "dvc commit <file>..." to track changes)
  (use "dvc checkout <file>..." to discard changes)
        deleted: ddrm/assets/checkpoints/mobile_sam.pt
        deleted: ddrm/assets/checkpoints/rtmdet-ins_x_8xb16-300e_coco-cam1-epoch_300.pth
        deleted: ddrm/assets/checkpoints/rtmdet-ins_x_8xb16-300e_coco_2nd-epoch_300.pth
(there are other changes not tracked by dvc, use "git status" to see)

There is also a diff command: 

dvc diff

Updating tracked files (commit)

dvc add adds untracked files so they are tracked. dvc commit "finalizes" the changes and updates the cache when files that are already tracked have been modified. (That is, it does not upload to the server.) 

dvc commit

It goes in the order git add -> git commit, but should be thought of separately from that.

Example

Right after the data has been modified, let's check the data status: 

$ opy-dvc data status
DVC uncommitted changes:
  (use "dvc commit <file>..." to track changes)
  (use "dvc checkout <file>..." to discard changes)
        modified: demos/
(there are other changes not tracked by dvc, use "git status" to see)

The "modified:" part is printed in <span style="color: yellow;">yellow. Just follow the comments.

  • To apply the changes: "dvc commit <file>..."
  • To discard the changes: "dvc checkout <file>..."

Here we commit. 

$ opy-dvc commit
outputs ['demos'] of stage: 'demos.dvc' changed. Are you sure you want to commit it? [y/n] y

If there are changes, it asks whether to commit them. Enter y. 

$ opy-dvc data status
DVC committed changes:
  (git commit the corresponding dvc files to update the repo)
        modified: demos/
(there are other changes not tracked by dvc, use "git status" to see)

The "modified:" part is printed in <span style="color: green;">green. If you want to upload to the server, push. 

$ opy-dvc push
Collecting                                                                                                                       |267 [00:00,  976entry/s]
Pushing
183 files pushed

Syncing with the remote

Download from the DVC remote: 

dvc pull

Upload to the DVC remote: 

dvc push

Reverting to a previous version 

dvc checkout

dvc is used to change the version of data/models. Let's look at how to change versions!

Suppose you change some hyperparameters in train.py and train a new model.

Suppose that after training, model_output contains the new model file. 

dvc add model_output 
git commit model_output.dvc -m "dvc: Model Updata vx.x" 
dvc push 
git push origin master

Push the new version of the data/model to dvc as above. 

git checkout <commit>
dvc checkout

To switch to a previous version of the data, go back to the commit you want to use, as above, and return to the dvc contents at that time!

Cache

Is A TTY 

DVC_IGNORE_ISATTY=1 dvc push

Data Management

Remote Storage

DVC remotes provide access to external storage locations to track and share data and ML models.

Usually, these are shared between the devices or team members working on a project.

For example, you can download data artifacts created by colleagues without spending time and resources to regenerate them locally.

ddrm repository initialization example

Using sftp on a Synology DiskStation (NAS) kept causing errors during mkdirs, so I installed the WebDAV package and used the TLS connection port (5006). 

dvc init
dvc remote add -d yournas webdavs://yournas:5006/20TB_DATA/DVC/ddrm
dvc config core.autostage true
dvc add ./ddrm/assets/checkpoints/
git rm -r --cached ddrm/assets/checkpoints
git add ddrm/assets/checkpoints.dvc

Troubleshooting

Client error '401 Unauthorized' for url '..' 

Collecting
Fetching
ERROR: unexpected error - received 401 (Unauthorized): Client error '401 Unauthorized' for url 'https://yournas:5006/20TB_DATA/DVC/ddrm/files/md5/00'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

You need to add at least the user ID. 

dvc remote modify --local yournas user ${USER}

If needed, have it prompt for the password too. 

dvc remote modify yournas ask_password true

unknown module name (_ssl.c:2633)

When using the webdavs protocol and running dvc push: 

ERROR: unexpected error - [CONF: UNKNOWN_MODULE_NAME] unknown module name (_ssl.c:2633): [CONF: UNKNOWN_MODULE_NAME] unknown module name (_ssl.c:2633)

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

Ugh.. -_-; Just install it with pip and use that.

output '...' is already tracked by SCM (e.g. Git)

Running the dvc add ddrm/assets/checkpoints/ command gave: 

Computing md5 for a large file '/home/yourid/Project/ddrm/ddrm/assets/checkpoints/rtmdet-ins_x_8xb16-300e_coco-cam1-epoch_300.pth'. This is only done once.
Computing md5 for a large file '/home/yourid/Project/ddrm/ddrm/assets/checkpoints/rtmdet-ins_x_8xb16-300e_coco_2nd-epoch_300.pth'. This is only done once.
Adding...
ERROR:  output 'ddrm/assets/checkpoints' is already tracked by SCM (e.g. Git).
    You can remove it from Git, then add to DVC.
        To stop tracking from Git:
            git rm -r --cached 'ddrm/assets/checkpoints'
            git commit -m "stop tracking ddrm/assets/checkpoints"

You need to remove the specified files/folders from SCM version control. 

git rm -r --cached 'ddrm/assets/checkpoints'
git commit -m "stop tracking ddrm/assets/checkpoints"

See aslo

  • git
  • MLOps
  • Lnk - Git-based single-binary dotfiles manager

Favorite site

author Zer0 sitemap