cape.cfdx.options.archiveopts: Case archiving options

This module provides a class to access options relating to archiving folders that were used to run CFD simulations.

The class provided in this module, ArchiveOpts, is loaded into the "RunControl" section of the main options interface.

class cape.cfdx.options.archiveopts.ArchiveArchiveOpts(*args, **kw)

Options for the --archive archiving action

The intent of the --archive phase is that it archives the critical contents of the case to long-term storage (of some kind) and only runs after a case has been marked PASS or ERROR. In most cases, the --archive action will also delete many files to save space in the run folder. Ideally, post-processing should still be possible after archiving, but restarting the case may not be.

class cape.cfdx.options.archiveopts.ArchiveCleanOpts(*args, **kw)

Options for the --clean archiving action

The intent of the --clean phase is that it deletes files (and optionally archives files) in the case folder that don’t prevent the continued running of the case or interfere with post-processing.

class cape.cfdx.options.archiveopts.ArchiveOpts(*args, **kw)

Archiving options

In addition to defining several generic properties of the archiving approach (format, full vs partial, glob vs regex, and location of archive), it defines three sections:

  • clean: archiving and folder clean-up while still running

  • archive: post-completion archiving, reports still generate

  • skeleton: post-archiving clean-up with no checks

See also:
get_ArchiveExtension() → str

Get file extension for [compressed] archive groups

Call:
>>> ext = opts.get_ArchiveExtension()
Inputs:
opts: cape.options.Options

Options interface

Outputs:
ext: str

File extension based on compression type

Versions:

  • 2024-09-14 @ddalle: v1.0

  • 2024-09-17 @ddalle: v1.1; fewer lines to test

get_ArchiveFolder(j=None, i=None, **kw)

Get path to the archive root

Call:
>>> ArchiveFolder = opts.get_ArchiveFolder(j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

j: {None} | int

Phase index; use None to just return v

i: {None} | int | np.ndarray

opts.x index(es) to use with @expr, @map, etc.

vdef: {None} | object

Manual default

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

ring: {opts._optring[key]} | True | False

Override option to loop through phase inputs

listdepth: {0} | int > 0

Depth of list to treat as a scalar

x: {None} | dict

Ref conditions to use with @expr, @map, etc.; often a run matrix; used in combination with i

sample: {True} | False

Apply j, i, and other settings recursively if output is a list or dict

Outputs:
ArchiveFolder: {''} | str

path to the archive root

get_ArchiveFormat(j=None, i=None, **kw)

Get format for case archives

Call:
>>> ArchiveFormat = opts.get_ArchiveFormat(j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

j: {None} | int

Phase index; use None to just return v

i: {None} | int | np.ndarray

opts.x index(es) to use with @expr, @map, etc.

vdef: {None} | object

Manual default

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

ring: {opts._optring[key]} | True | False

Override option to loop through phase inputs

listdepth: {0} | int > 0

Depth of list to treat as a scalar

x: {None} | dict

Ref conditions to use with @expr, @map, etc.; often a run matrix; used in combination with i

sample: {True} | False

Apply j, i, and other settings recursively if output is a list or dict

Outputs:
ArchiveFormat: '' | 'bz2' | 'gz' | 'lzma' | {'tar'} | 'xz' | 'zip' | 'zst'

format for case archives

get_ArchiveType(j=None, i=None, **kw)

Get flag for single (full) or multi (sub) archive files

Call:
>>> ArchiveType = opts.get_ArchiveType(j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

j: {None} | int

Phase index; use None to just return v

i: {None} | int | np.ndarray

opts.x index(es) to use with @expr, @map, etc.

vdef: {None} | object

Manual default

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

ring: {opts._optring[key]} | True | False

Override option to loop through phase inputs

listdepth: {0} | int > 0

Depth of list to treat as a scalar

x: {None} | dict

Ref conditions to use with @expr, @map, etc.; often a run matrix; used in combination with i

sample: {True} | False

Apply j, i, and other settings recursively if output is a list or dict

Outputs:
ArchiveType: {'full'} | 'partial'

flag for single (full) or multi (sub) archive files

get_SearchMethod(j=None, i=None, **kw)

Get method for declaring multiple files with one string

Call:
>>> SearchMethod = opts.get_SearchMethod(j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

j: {None} | int

Phase index; use None to just return v

i: {None} | int | np.ndarray

opts.x index(es) to use with @expr, @map, etc.

vdef: {None} | object

Manual default

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

ring: {opts._optring[key]} | True | False

Override option to loop through phase inputs

listdepth: {0} | int > 0

Depth of list to treat as a scalar

x: {None} | dict

Ref conditions to use with @expr, @map, etc.; often a run matrix; used in combination with i

sample: {True} | False

Apply j, i, and other settings recursively if output is a list or dict

Outputs:
SearchMethod: {'glob'} | 'regex'

method for declaring multiple files with one string

read_old(**kw)

Convert CAPE 1.x ArchiveOptions to newer standard

Call:
>>> kw_new = opts.read_old(**kw)
Inputs:
opts: ArchiveOpts

Archiving options instance

kw: dict

Options suitable for previous archiving options class

Outputs:
kw_new: dict

Filtered and mapped version of kw

Versions:

  • 2024-09-17 @ddalle: v1.0

set_ArchiveFolder(v, j=None, mode=None)

Get path to the archive root

Call:
>>> opts.set_ArchiveFolder(ArchiveFolder, j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

ArchiveFolder: {''} | str

path to the archive root

j: {None} | int

Phase index; use None to just return v

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

listdepth: {0} | int > 0

Depth of list to treat as a scalar

set_ArchiveFormat(v, j=None, mode=None)

Get format for case archives

Call:
>>> opts.set_ArchiveFormat(ArchiveFormat, j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

ArchiveFormat: '' | 'bz2' | 'gz' | 'lzma' | {'tar'} | 'xz' | 'zip' | 'zst'

format for case archives

j: {None} | int

Phase index; use None to just return v

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

listdepth: {0} | int > 0

Depth of list to treat as a scalar

set_ArchiveType(v, j=None, mode=None)

Get flag for single (full) or multi (sub) archive files

Call:
>>> opts.set_ArchiveType(ArchiveType, j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

ArchiveType: {'full'} | 'partial'

flag for single (full) or multi (sub) archive files

j: {None} | int

Phase index; use None to just return v

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

listdepth: {0} | int > 0

Depth of list to treat as a scalar

set_SearchMethod(v, j=None, mode=None)

Get method for declaring multiple files with one string

Call:
>>> opts.set_SearchMethod(SearchMethod, j=None, i=None, **kw)
Inputs:
opts: ArchiveOpts

options interface

SearchMethod: {'glob'} | 'regex'

method for declaring multiple files with one string

j: {None} | int

Phase index; use None to just return v

mode: {None} | 0 | 1 | 2 | 3

Warning mode code

0:

no checks

1:

validate silently

2:

validate and show warnings

3:

raise an exception if invalid

listdepth: {0} | int > 0

Depth of list to treat as a scalar

class cape.cfdx.options.archiveopts.ArchivePhaseOpts(*args, **kw)

Options for a single archive action

Actions include

  • --clean

  • --archive

  • --skeleton

The options in this class each describe a single action. The order of those sub-actions is

  1. PreDeleteDirs

  2. PreDeleteFiles

  3. ArchiveFiles

  4. ArchiveTarGroups

  5. ArchiveTarDirs

  6. PostTarGroups

  7. PostTarDirs

  8. PostDeleteDirs

  9. PostDeleteFiles

  10. PostTailFiles

class cape.cfdx.options.archiveopts.ArchiveSkeletonOpts(*args, **kw)

Options for the --skeleton archiving action

The --skeleton action first runs --archive, and therefore it can only be done on a case marked PASS or FAIL. This is a final clean-up action that deletes many files and is one step short of deleting the entire case.

The idea behind this capability is that it might preserve minimal information about each case such as how many iterations it ran, how many CPU hours it used, or just the residual history.

There are no safety checks performed, so users may delete critical files.

class cape.cfdx.options.archiveopts.OldArchiveOpts(*args, **kw)

Archive manangement options interface

Call:
>>> opts = ArchiveOpts(**kw)
Inputs:
kw: dict

Dictionary of archive options

Outputs:
opts: ArchiveOpts

Archive options interface

Versions:

  • 2016-30-02 @ddalle: v1.0 (Archive)

  • 2022-10-14 @ddalle: v2.0; OptionsDict

get_ArchiveCmd()

Get archiving command

Call:
>>> cmd = opts.get_ArchiveCmd()
Inputs:
opts: cape.options.Options

Options interface

Outputs:
cmd: list[str]

Tar command and appropriate flags

Versions:

  • 2016-03-01 @ddalle: v1.0

get_ArchiveExtension() → str

Get file extension for [compressed] archive groups

Call:
>>> ext = opts.get_ArchiveExtension()
Inputs:
opts: cape.options.Options

Options interface

Outputs:
ext: str

File extension based on compression type

Versions:

  • 2024-09-14 @ddalle: v1.0

get_UnarchiveCmd()

Get command to unarchive

Call:
>>> cmd = opts.get_UnarchiveCmd()
Inputs:
opts: cape.options.Options

Options interface

Outputs:
cmd: list[str]

Untar command and appropriate flags

Versions:

  • 2016-03-01 @ddalle: v1.0

cape.cfdx.options.archiveopts.auto_Archive(opts, cls=<class 'cape.cfdx.options.archiveopts.ArchiveOpts'>)

Automatically convert dict to ArchiveOpts

Call:
>>> opts = auto_Archive(opts)
Inputs:
opts: dict

Dict of either global, “RunControl” or “Archive” options

Outputs:
opts: ArchiveOpts

Instance of archiving options

Versions:

  • 2016-02-29 @ddalle: v1.0

  • 2022-10-21 @ddalle: v2.0, add cls input

cape.cfdx.options.archiveopts.expand_fileopt(rawval: list | dict | str, vdef: int = 0) → dict

Expand Archive file name/list/dict to common format

The output is a dict where the key is the pattern of file names to process and the value is an int that represents the number of most recent files matching that pattern to keep.

Call:
>>> patdict = expand_fileopt(rawstr, vdef=0)
>>> patdict = expand_fileopt(rawlist, vdef=0)
>>> patdict = expand_fileopt(rawdict, vdef=0)
Inputs:
rawstr: str

Pattern of file names to process

rawlist: list[str]

List of filee name patterns to process

rawdict: dict[int]

Dictionary of file name patterns to process and n to keep

Outputs:
patdict: dict[int]

File name patterns as desribed above

Versions:

  • 2024-09-02 @ddalle: v1.0