cape.manage: Manage CFD case folders

This module provides methods to manage and archive files for run folders. It provides extensive tools for archiving results to other locations either as a tar ball, tar bomb, or zip archive of the entire folder or a folder of several smaller zip files.

It also provides tools for deleting files either before or after creating the archive. In addition, there is an easy interface for keeping only the most recent n files of a certain glob. For example, if there are solution files flow.01, flow.02, flow.03, and flow.04, setting the PostUpdateFiles parameter to

{"flow.??": 2}

will delete only flow.01 and flow.02.

The module provides methods to perform deletions, conditional deletions, and grouping files into tar or zip archives at multiple stages. For example, PreDeleteFiles() deletes files after a case has been completed and before generating the archive of the case. On the other hand, PostDeleteFiles() deletes files after creating the archive; the difference is whether or not the file in question is included in the archive before it is deleted.

A case cannot be archived until it has been granted the status of PASS, by meeting the requested number of iterations and phases and by getting marked with a p in the run matrix file. However, less aggressive pruning via the --clean command can always be performed, even if the case is currently running.

The archiving process can be “reversed” (although this does not delete the archived copy) using UnarchiveFolder(), which copies files from the archive to the working folder. This can be useful if you determine that a necessary file for post-processing was cleaned out or if the case needs to be extended (run for more iterations).

Functions such as ProgressDeleteFiles() deletes files between phases, and as such it is much more dangerous. Other methods will only delete or archive once the case has been marked as PASS by the user.

An even more aggressive action can be taken using the --skeleton command-line option. After creating or updating the archive, this deletes even more files from the working copy than --archive using the SkeletonFolder() function. A common use of this dichotomy is to set up --archive so that all post-processing can still be done, and once the post-processing --skeleton deletes everything but a minimal set of information about the case. The --skeleton also has an extra capability to replace the working copy with the last few lines of that file, which might contain information on the most recent iteration run, for example. In order to avoid catastrophe, the --skeleton command effectively calls --archive as part of its normal procedure.

This module contains many supporting methods to perform specific actions such as comparing modification dates on the working and archived copies. The main functions that are called from the command line are:

  • ManageFilesProgress(): delete files progressively, even if the case is still running (can be used for example to keep only the two most recent check point files); does not archive

  • ManageFilesPre(): delete or tar files/folders immediately prior to creating archive; reduces size or archive but not performed until case has PASS status

  • ManageFilesPost(): deletes or tar files/folders after creating archive

Each of the four phases also has several contributing functions that perform a specific task at a specific phase, which are outlined below.

The difference between DeleteFiles and UpdateFiles is merely on what action is taken by default. The user can manually specify how many most recent files matching a certain glob to keep using a dict such as

{"flow[0-9][0-9]": 2}

but if a glob is given without a dict, such as simply

"flow[0-9][0-9]"

by default DeleteFiles() will delete all of them while UpdateFiles() will keep the most recent.

Finally, the main command-line folder management calls each interface directly with one function from this module.

cape.manage.ArchiveCaseWhole(opts)

Archive an entire run folder

This function must be run from the case folder to be archived

Call:
>>> ArchiveCaseWhole(opts)
Inputs:
opts: cape.cfdx.options.Options

Options interface

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ArchiveFiles(opts, fsub=None, phantom=False)

Delete files that match a list of glob

The function also searches in any folder matching the directory glob or list of directory globs fsub.

Call:
>>> manage.ArchiveFiles(opts, fsub=None, phantom=False)
Inputs:
opts: cape.cfdx.options.Options

Options interface

fsub: str | list[str]

Folder, list of folders, or glob of folders to also search

phantom: True | {False}

Only copy files if True

Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2016-12-09 @ddalle: v1.1, use ArchiveFiles option

  • 2017-03-06 @ddalle: v1.2, add phantom option

cape.manage.ArchiveFolder(opts, fsub=[], phantom=False)

Archive a folder and clean up nonessential files

Call:
>>> cape.manage.ArchiveFolder(opts, fsub=[], phantom=False)
Inputs:
opts: cape.cfdx.options.Options

Options interface including management/archive interface

fsub: list[str]

Globs of subdirectories that are adaptive run folders

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-12-09 @ddalle: v1.0

  • 2017-12-15 @ddalle: v1.1, add phantom option

cape.manage.CleanFolder(opts, fsub=[], phantom=False)

Delete files before archiving and regardless of status

Call:
>>> cape.manage.CleanFolder(opts, fsub=[], phantom=False)
Inputs:
opts: cape.cfdx.options.Options

Options interface including management/archive interface

fsub: list[str]

Globs of subdirectories that are adaptive run folders

phantom: True | {False}

Only delete files if False

Versions:

  • 2017-03-10 @ddalle: v1.0

  • 2017-12-15 @ddalle: Added phantom option

cape.manage.CreateArchiveCaseFolder(opts)

Create the group and run folders in the archive, as appropriate

This function must be run within the folder that is to be archived.

Call:
>>> CreateArchiveCaseFolder(opts)
Inputs:
opts: cape.cfdx.options.Options

Options interface

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.CreateArchiveFolder(opts)

Create the folder that will contain the archive, if necessary

Call:
>>> CreateArchiveFolder(opts)
Inputs:
opts: cape.cfdx.options.Options

Options interface

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.CreateArchiveGroupFolder(opts)

Create the group folder in the archive, as appropriate

This function must be run from within the folder getting archived.

Call:
>>> CreateArchiveGroupFolder(opts)
Inputs:
opts: cape.cfdx.options.Options

Options interface

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.DeleteDirs(fdel, fsub=None, n=1, phantom=False)

Delete folders that match a glob

The function also searches in any folder matching the directory glob or list of directory globs fsub.

Call:
>>> DeleteDirs_SubDir(fdel, n=1, fsub=None, phantom=False)
Inputs:
fdel: str

File name or glob of files to delete

fsub: str | list[str]

Folder, list of folders, or glob of folders to also search

n: int

Number of folders to keep at end of glob

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.manage.DeleteFiles(fdel, fsub=None, n=1, phantom=False)

Delete files that match a list of globs

The function also searches in any folder matching the directory glob or list of directory globs fsub.

Call:
>>> cape.manage.DeleteFiles(fdel, fsub=None, n=1, phantom=False)
Inputs:
fdel: str

File name or glob of files to delete

fsub: str | list[str]

Folder, list of folders, or glob of folders to also search

n: int

Number of files to keep

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add phantom option

cape.manage.DeleteFilesExcept(fskel, dskel=[], fsub=None, n=0, phantom=False)

Delete all files except those that match a list of globs

Call:
>>> cape.manage.DeleteFilesExcept(fskel, **kw)
Inputs:
fskel: list[str | dict]

List of file names or globs of files to delete

dskel: {[]} | list

List of folder names of globs of folder names to delete

fsub: str | list[str]

Folder, list of folders, or glob of folders to also search

n: {0} | int

Number of files to keep if not set by dictionary options for each file; if 0, keep all by default

phantom: True | {False}

Only delete files if False

Versions:

Expand any links in a full glob if linked to relative file

If the link points to an absolute path, the entry is not replaced with the target of the link.

Call:
>>> flst = ExpandLinks(fglob)
Inputs:
fglob: list[str]

List of file names, possibly including links

Outputs:
flst: list[str]

List of file names plus expanded versions of any links

Versions:

  • 2017-12-13 @ddalle: v1.0

cape.manage.GetDirMatches(fname, fsub=None, n=0, qdel=False)

Get list of all folders matching a list of patterns

Call:
>>> fglob = GetDirMatches(fname, fsub=None, n=0, qdel=False)
>>> fglob = GetDirMatches(fname, fsubs)
Inputs:
fname: list[dict | str]

List of file name patterns

fsub: str | list[str]

Folder name of folder name pattern

n: int

Default number of files to ignore at end of glob

qdel: True | {False}

Interpret n as how many files to keep if True; as how many files to copy if False

Outputs:
fglob: list[str]

List of files matching input pattern

See also:
Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add qdel kwarg

cape.manage.GetFileMatches(fname, fsub=None, n=0, qdel=False)

Get list of all files or links matching a list of patterns

Call:
>>> fglob = GetFileMatches(fname, fsub=None, n=0, qdel=False)
>>> fglob = GetFileMatches(fname, fsubs)
Inputs:
fname: list[dict | str]

List of file name patterns

fsub: str | list[str]

Folder name of folder name pattern

n: int

Default number of files to ignore at end of glob

qdel: True | {False}

Interpret n as how many files to keep if True; as how many files to copy if False

Outputs:
fglob: list[str]

List of files matching input pattern

See also:
Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add qdel kwarg

cape.manage.GetImpliedFolders(fglob, fdirs=[])

Check a list of files to get implicit matching folders

For example, ["case.json", "adapt00/input.nml"] implies the folder adapt00/

Call:
>>> fsubs = GetImpliedFolders(fglob, fdirs=[])
Inputs:
fglob: list[str]

List of file names including any folder names

fdirs: {[]} | list[str]

List of folders to append to

Outputs:
fsubs: list[str]

Unique list of folders including entries from fdirs

Versions:

  • 2017-12-13 @ddalle: v1.0

cape.manage.GetLinkMatches(fname, fsub=None, n=0, qdel=False)

Get list of all links matching a list of patterns

Call:
>>> fglob = GetLinkMatches(fname, fsub=None, n=0, qdel=False)
>>> fglob = GetLinkMatches(fname, fsubs)
Inputs:
fname: list[dict | str]

List of file name patterns

fsub: str | list[str]

Folder name of folder name pattern

n: int

Default number of files to ignore at end of glob

qdel: True | {False}

Interpret n as how many files to keep if True; as how many files to copy if False

Outputs:
fglob: list[str]

List of files matching input pattern

See also:
Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add qdel

cape.manage.GetMatches(fname, fsub=None, fkeep=None, ftest=None, n=0, fsort=None, qdel=False)

Get matches based on arbitrary rules

Call:
>>> fglob = GetMatches(fname, **kw)
Inputs:
fname: str

File name or file name pattern

fsub: str | list[str]

Subfolder name of folder name patterns in which to search

fkeep: list[str]

List of file names matching a negative file glob

ftest: func

Function to test file type, e.g. os.path.isdir()

n: int

Default number of files to ignore from the end of the glob or number of files to ignore from beginning of glob if negative

fsort: callable

Non-default sorting function

qdel: True | {False}

Interpret n as how many files to keep if True; as how many files to copy if False

Outputs:
fglob: list[str]

List of files matching input pattern

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add qdel kwarg

cape.manage.GetMatchesList(flist, fsub=None, ftest=None, n=0, qdel=False)

Get matches from a list of file glob descriptors

Call:
>>> fglob = GetMatchesList(flist, **kw)
Inputs:
flist: list[str | dict]

List of file names or file name patterns

fsub: str | list[str]

Subfolder name of folder name patterns in which to search

ftest: func

Function to test file type, e.g. os.path.isdir()

n: int

Default number of files to ignore from the end of the glob or number of files to ignore from beginning of glob if negative

qdel: True | {False}

Interpret n as how many files to keep if True; as how many files to copy if False

Outputs:
fglob: list[str]

List of files matching input pattern

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2017-03-06 @ddalle: Added qdel option

cape.manage.GetSearchDirs(fsub=None, fsort=None)

Get list of current folder and folders matching pattern

Call:
>>> fdirs = GetSearchDirs()
>>> fdirs = GetSearchDirs(fsub)
>>> fdirs = GetSearchDirs(fsubs)
Inputs:
fsub: str

Folder name of folder name pattern

fsubs: list[str]

List of folder names or folder name patterns

fsort: callable

Non-default sorting function

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.manage.ManageFilesPost(opts=None, fsub=None, phantom=False)

Delete or group files and folders after creating archive

Call:
>>> cape.manage.ManageFilesPost(opts=None, **kw)
Inputs:
opts: cape.cfdx.options.Options | dict

Options interface for archiving

fsub: list[str]

Globs of subdirectories that are adaptive run folders

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add phantom option

cape.manage.ManageFilesPre(opts=None, fsub=None, phantom=False)

Delete or group files and folders before creating archive

Call:
>>> cape.manage.ManageFilesPre(opts=None, fsub=None)
Inputs:
opts: cape.cfdx.options.Options | dict

Options interface for archiving

fsub: list[str]

Number of files to keep

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add phantom option

cape.manage.ManageFilesProgress(opts=None, fsub=None, phantom=False)

Delete or group files and folders at end of each run

Call:
>>> cape.manage.ManageFilesProgress(opts=None, **kw)
Inputs:
opts: cape.cfdx.options.Options | dict

Options interface for archiving

fsub: list[str]

List of globs of subdirectories that are adaptive run folders

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add phantom option

cape.manage.PostDeleteDirs(opts, fsub=None, aa=None, phantom=False)

Delete appropriate folders after completing archive

Call:
>>> PostDeleteDirs(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PostDeleteFiles(opts, fsub=None, aa=None, phantom=False)

Delete appropriate files after completing archive

Call:
>>> PostDeleteFiles(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PostTarDirs(opts, fsub=None, aa=None, frun=None)

Tar folders after archiving

The folders are not deleted after the tar ball is created

Call:
>>> PostTarDirs(opts, fsub=None, aa=None, frun=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

frun: str

Name of case folder

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PostTarGroups(opts, fsub=None, aa=None, frun=None)

Tar file/folder groups

The files are not deleted after the tar ball is created

Call:
>>> PostTarGroups(opts, fsub=None, aa=None, frun=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

frun: str

Case folder name

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2023-10-18 @ddalle: v2.0; allow multi-key dicts

cape.manage.PostUpdateFiles(opts, fsub=None, aa=None, phantom=False)

Delete files after archiving, by default keeping most recent

Call:
>>> PreUpdateFiles(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PreDeleteDirs(opts, fsub=None, aa=None, phantom=False)

Delete folders that match a list of patterns before archiving

Call:
>>> PreDeleteDirs(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PreDeleteFiles(opts, fsub=None, aa=None, phantom=False)

Delete files that match a list of file name patterns

Call:
>>> PreDeleteFiles(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PreTarDirs(opts, fsub=None, aa=None)

Tar folders before archiving

The folders are deleted after the tar ball is created

Call:
>>> PreTarDirs(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.PreTarGroups(opts, fsub=None, aa=None)

Tar file/folder groups

The files are deleted after the tar ball is created

Call:
>>> PreTarGroups(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

Versions:

  • 2016-03-14 @ddalle: v1.0

  • 2023-10-18 @ddalle: v2.0; allow multi-key dicts

cape.manage.PreUpdateFiles(opts, fsub=None, aa=None, phantom=False)

Delete files that match list, keeping the most recent by default

Call:
>>> PreUpdateFiles(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ProgressArchiveFiles(opts, fsub=None, aa=None, phantom=False)

Archive files of active (in progress) case folder

Call:
>>> ProgressArchiveFiles(opts, fsub=None, **kw)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ProgressDeleteDirs(opts, fsub=None, aa=None, phantom=False)

Delete appropriate folders of active (in progress) case

Call:
>>> ProgressDeleteDirs(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ProgressDeleteFiles(opts, fsub=None, aa=None, phantom=False)

Delete appropriate files from active (in progress) case folder

Call:
>>> ProgressDeleteFiles(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ProgressTarDirs(opts, fsub=None, aa=None)

Tar folders of active (in progress) case

The folders are deleted after the tar ball is created

Call:
>>> ProgressTarDirs(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ProgressTarGroups(opts, fsub=None, aa=None)

Tar file/folder groups after each run

The files are deleted after the tar ball is created

Call:
>>> ProgressTarGroups(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.ProgressUpdateFiles(opts, fsub=None, aa=None, phantom=False)

Delete files in active folder, by default keeping most recent

Call:
>>> ProgressUpdateFiles(opts, fsub=None, aa=None)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.SkeletonDeleteFiles(opts, fsub=None, aa=None, phantom=False)

Delete all files except those matching file name patterns

Call:
>>> SkeletonDeleteFiles(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2017-12-14 @ddalle: v1.0

cape.manage.SkeletonFolder(opts, fsub=[], phantom=False)

Perform post-archiving clean-out actions; create a “skeleton”

Call:
>>> cape.manage.SkeletonFolder(opts, fsub=[], phantom=False)
Inputs:
opts: cape.cfdx.options.Options

Options interface including management/archive

fsub: list[str]

Globs of subdirectories that are adaptive run folders

phantom: True | {False}

Only delete files if False

Versions:

  • 2017-12-13 @ddalle: v1.0

  • 2017-12-15 @ddalle: v1.1, add phantom option

cape.manage.SkeletonTailFiles(opts, fsub=None, aa=None, phantom=False)

Replace files with their last few lines, possibly in a new file

Call:
>>> SkeletonTailFiles(opts, fsub=None, aa=None, phantom=False)
Inputs:
opts: None | Archive | dict

Options dictionary or options interface

fsub: list[str] | str

List of sub-directory globs in which to search

aa: callable

Conversion function applied to opts

phantom: True | {False}

Only delete files if False

Versions:

  • 2017-12-14 @ddalle: v1.0

cape.manage.TailFiles(ftail, fsub=None, n=1, phantom=False)

Tail a list of files

An example input is [{"run.resid": [2, "run.tail.resid"]}]; this tells the function to tail the last 2 files from run.resid and put them in a file called run.tail.resid.

Call:
>>> cape.manage.TailFiles(ftail, fsub=None, n=1, phantom=False)
Inputs:
ftail: list[dict]

List of dictionaries of files to tail

fsub: str | list[str]

Folder, list of folders, or glob of folders to also search

n: int

Number of files to keep

phantom: True | {False}

Only delete files if False

Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2017-03-06 @ddalle: v1.1, add phantom option

cape.manage.TarDir(cmd, ftar, fdir, clean=True)

Archive a folder and delete the folder

Call:
>>> TarDir(cmd, ftar, fdir)
Inputs:
cmd: list[str]

List of archiving/compression commands

ftar: str

Name of file to create

fdir: str

Name of folder

clean: bool

Whether or not to delete folder afterwards

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.manage.TarGroup(cmd, ftar, fname, n=0, clean=False)

Archive a group of files and delete the files

Only the present folder will searched for file name matches

Call:
>>> TarGroup(cmd, ftar, fname, clean=False)
Inputs:
cmd: list[str]

List of archiving/compression commands

ftar: str

Name of file to create

fname: list[dict | str]

File name pattern or list thereof to combine into archive

clean: bool

Whether or not to clean up after archiving

Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2016-03-14 @ddalle: v2.0; generalized

Tar all links existing in the current folder

Call:
>>> TarLinks(cmd, ext, clean=True)
Inputs:
cmd: list[str]

List of archiving/compression commands

ftar: str

Name of file to create

clean: bool

Whether or not to clean up after archiving

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.manage.UnarchiveCaseWhole(opts)

Unarchive a tar ball that stores results for an entire folder

Call:
>>> UnarchiveCaseWhole(opts)
Inputs:
opts: cape.cfdx.options.Options

Options interface

Versions:

  • 2017-03-13 @ddalle: v1.0

cape.manage.UnarchiveFolder(opts)

Unarchive a case archived as one or more compressed files

Call:
>>> cape.manage.UnarchiveFolder(opts)
Inputs:
opts: cape.cfdx.options.Options

Options interface

Versions:

  • 2017-03-13 @ddalle: v1.0

cape.manage.Untar(cmd, ftar)

Unarchive a tar ball and then delete it

Call:
>>> cape.manage.Untar(cmd, ftar)
Inputs:
cmd: list[str]

List of archiving/compression commands

ftar: str

Name of file to create

fname: str

File name pattern or list thereof to combine into archive

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.manage.getmtime(fname)

Get the modification time of a file, using ssh if necessary

For local files, this function uses os.path.getmtime(). If fname contains a :, this function issues an ssh command via subprocess.Popen().

Call:
>>> t = getmtime(fname)
Inputs:
fname: str

Name of file

Outputs:
t: float | None

Modification time; None if file does not exist

Versions:

  • 2017-03-17 @ddalle: v1.0

cape.manage.getmtime_glob(fglob)

Get the modification time of a glob, using ssh if necessary

Call:
>>> t = getmtime_glob(fglob)
Inputs:
fglob: list[str]

List if names of files

Outputs:
t: float | None

Modification time of most recently modified file

Versions:

  • 2017-03-13 @ddalle: v1.0

Handle to test if a file is a broken link

Call:
>>> q = cape.manage.isbrokenlink(fname)
Inputs:
fname: str

Name of link

Outputs:
q: bool

Whether or not it is a link that is broken

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.isfile(fname)

Handle to test if file or link

Call:
>>> q = cape.manage.isfile(fname)
Inputs:
fname: str

Name of file

Outputs:
q: bool

Whether or not file both exists and is either a file or link

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.process_ArchiveFile(f, n=1)

Process a file glob description

This can be either a file name, file name pattern, or dictionary of file name pattern and number of files to keep

Call:
>>> fname, nkeep = process_ArchiveFile(fname, n=1)
>>> fname, nkeep = process_ArchiveFile(fdict, n=1)
>>> fname, nkeep = process_ArchiveFile({fname: nkeep}, n=1)
Inputs:
fname: str

File name or file name pattern to delete/group/etc.

nkeep: int

Number of matching files to keep

n: int

Default number of files to keep

Outputs:
fname: str

File name or file name pattern to delete/group/etc.

nkeep: int

Number of matching files to keep

Versions:

  • 2016-03-01 @ddalle: v1.0

  • 2022-02-02 @ddalle: v1.1; python 3 fixes

cape.manage.process_ArchiveGroup(grp)

Process an archive group, which has a precise format

It must be a dictionary with one entry

Call:
>>> ftar, fpat = cape.manage.process_ArchiveGroup(grp)
>>> ftar, fpat = cape.manage.process_Archivegroup({ftar: fpat})
Inputs:
grp: dict, len: 1

Single dictionary with name and pattern or list of patterns

Outputs:
ftar: str

Name of archive to create without extension

fpat: str | dict | list

File name pattern or list of file name patterns

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.manage.sortfiles(fglob)

Sort a glob of files based on the time of their last edit

Call:
>>> fsort = sortfiles(fglob)
Inputs:
fglob: list[str]

List of file names

Outputs:
fsort: list[str]

Above listed by os.path.getmtime()

Versions:

  • 2016-03-14 @ddalle: v1.0

cape.manage.validate_targroups(grpopts, name: str) → dict

Process and validate an archive group

It can be either a single dict or list thereof. It returns a single combined dict. The values of the dict must each be a list of strings.

Call:
>>> opts = validate_targroups(grpopts, name)
Inputs:
grpopts: dict | list[dict]

Unprocessed definition of tar groups

name: str

Section name to use in error messages

Outputs:
opts: dict[list[str]]

Validated options as a single combined dict

Versions:

  • 2023-10-18 @ddalle: v1.0

cape.manage.write_log(txt, fname='archive.log')

Write the date to the archive log

Call:
>>> cape.manage.write_log(txt, fname='archive.log')
Inputs:
fname: {"archive.log"} | str

Name of acrhive log file

Versions:

  • 2016-07-08 @ddalle: v1.0

cape.manage.write_log_date(fname='archive.log')

Write the date to the archive log

Call:
>>> cape.manage.write_log_date(fname='archive.log')
Inputs:
fname: {"archive.log"} | str

Name of acrhive log file

Versions:

  • 2016-07-08 @ddalle: v1.0