lfc: A git extension for large files

lfc, which stands for “Large File Control,” is an add-on for git that provides a method of tracking large files in git repositories. It is inspired by dvc but is simpler and has a different feature set.

The way it works is that users declare individual files “large,” which means that when users clone, pull, or fetch from a repo, those files are not automatically downloaded but instead can be retrieved on-demand. This has two main advantages:

• you can clone a repo with large files quickly, but without the large files (for example to work on an input file); and

• in cases where the large file(s) is (are) needed, you only need the most recent version instead of the whole history of that file.

In general, LFC is for any project that uses git but also generates large files or even small files that happen to be binary. Here are some specific example use cases where LFC might be useful.

• You’re working in computational fluid dynamics (CFD) that requires very large inputs and produces even larger outputs, and you’d like to make the whole process reproducible without resorting to ad hoc scripts.

• You’re a 21st-century developer using modern programming tools and git version control, but your colleagues keep sending you Word and Excel files that are the inputs to your process. You can use LFC to track those documents in a traceable and reliable way.

• You have a process that generates binary output files and you’d like to keep track of all the different versions of those files without having to store all of them locally.

• You have a repo that generates some binary files (which may or may not be “large”), for example a picture, every day. You’d like users to be able to clone the repo that makes and tracks those pictures but not have to download the entire picture-of-the-day archive when they clone the repo.

• And many others.

• 1. Getting started with LFC
• 2. The story of lfc
• 3. User Manual: LFC Command Line Interface
  • 3.1. lfc-init: Initialize LFC repo
  • 3.2. lfc-remote: Show or set URL to an LFC remote cache
  • 3.3. lfc-add: Add or update a large file
  • 3.4. lfc-pull: Retrieve and checkout one or more large files
  • 3.5. lfc-push: Push one or more large files to remote cache
  • 3.6. lfc-checkout: Check out a large file from cache
  • 3.7. lfc-config: View or set LFC config variables
  • 3.8. lfc-show: Print contents of a large file to STDOUT
  • 3.9. lfc-ls-files: List large files
  • 3.10. lfc-set-mode: Set the mode of one or more LFC files
  • 3.11. lfc-clone: Clone a repo (using git) and pull all mode-2 LFC files
  • 3.12. lfc-auto-pull: Pull all mode-2 files
  • 3.13. lfc-auto-push: Push all mode-2 files
  • 3.14. lfc-install-hooks: Install LFC git-hooks
  • 3.15. lfc-replace-dvc: Replace any DVC settings and file names

lfc Python package:

• lfc: A git extension to track large files
  • cli: Command-line interface to lfc
  • lfcrepo: Interface to git repos with large-file control
  • lfcclone: Command-line interface for lfc-clone
  • lfcerror: Errors for lfc modules
• argread: Parse command-line arguments and options
• argread.clitext: Process help messages for console output
• kwparse: Tools to map, parse, and validate keyword arguments
• gitutils: Python interface to git repositories
• gitutils.gitrepo: Interact with git repos using system interface
• gitutils.giterror: Errors for gitutils Git repo tools
• shellutils: System calls with STDOUT capture, SFTP, and more
• Links to built-in Python features
• Links to built-in Python standard library features
• Links to third-party modules
  • numpy: The fundamental package for scientific computing with Python

Indices and tables

• Index

• Module Index

• Search Page

Notices

Copyright © 2024 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.

Disclaimers

No Warranty: THE SUBJECT SOFTWARE IS PROVIDED “AS IS” WITHOUT ANY WARRANTY OF ANY KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR FREEDOM FROM INFRINGEMENT, ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL BE ERROR FREE, OR ANY WARRANTY THAT DOCUMENTATION, IF PROVIDED, WILL CONFORM TO THE SUBJECT SOFTWARE. THIS AGREEMENT DOES NOT, IN ANY MANNER, CONSTITUTE AN ENDORSEMENT BY GOVERNMENT AGENCY OR ANY PRIOR RECIPIENT OF ANY RESULTS, RESULTING DESIGNS, HARDWARE, SOFTWARE PRODUCTS OR ANY OTHER APPLICATIONS RESULTING FROM USE OF THE SUBJECT SOFTWARE. FURTHER, GOVERNMENT AGENCY DISCLAIMS ALL WARRANTIES AND LIABILITIES REGARDING THIRD-PARTY SOFTWARE, IF PRESENT IN THE ORIGINAL SOFTWARE, AND DISTRIBUTES IT “AS IS.”

Waiver and Indemnity: RECIPIENT AGREES TO WAIVE ANY AND ALL CLAIMS AGAINST THE UNITED STATES GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT. IF RECIPIENT’S USE OF THE SUBJECT SOFTWARE RESULTS IN ANY LIABILITIES, DEMANDS, DAMAGES, EXPENSES OR LOSSES ARISING FROM SUCH USE, INCLUDING ANY DAMAGES FROM PRODUCTS BASED ON, OR RESULTING FROM, RECIPIENT’S USE OF THE SUBJECT SOFTWARE, RECIPIENT SHALL INDEMNIFY AND HOLD HARMLESS THE UNITED STATES GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT, TO THE EXTENT PERMITTED BY LAW. RECIPIENT’S SOLE REMEDY FOR ANY SUCH MATTER SHALL BE THE IMMEDIATE, UNILATERAL TERMINATION OF THIS AGREEMENT.

• NASA OPEN SOURCE AGREEMENT VERSION 1.3