cape.io: Binary file input/output tools

This is a module to provide fast and convenient utilities for reading and writing binary data in Cape. The module relies heavily on the NumPy functions fromfile() and tofile(), but it also performs checks and conversions for big-ending vs little-endian data in a manner that can be mostly hidden from the user. It also handles Fortran start-of-record and end-of-record markers and performs the available checks without the need for users to write extra code every time a Fortran file is accessed.

In most cases classes built upon this library are responsible for detecting the presence or lack of Fortran record markers and detecting the endianness of the file automatically.

In addition, users can write to either endianness regardless of the system byte order. Although defaulting to the system file format is recommended for most modern applications, for some software this is not an available option.

This module frequently utilizes the following naming conventions for file formats, which are semi-standard but not necessarily universally recognizable. They take the form of one or two letters for the endianness of the file and one integer for the number of bytes used to represent a real-valued float.

Code

Description

b4

Single-precision big-endian C format

b8

Double-precision big-endian C format

lb4

Single-precision little-endian C format

lb8

Double-precision little-endian C format

ne4

Single-precision native-endian C format

r4

Single-precision big-endian Fortran format

r8

Double-precision big-endian Fortran format

lr4

Single-precision little-endian Fortran format

lr8

Double-precision little-endian Fortran format

These codes are used frequently in the names of functions within this module. In addition, the functions in this module usually contain a suffix of i (integer), f (float), or s (string). For example read_record_lr4_i() reads a little-endian int record, and read_record_r8_f() reads a double-precision float record.

By convention, Fortran double-precision files often use single-precision integers, so functions like read_record_r8_i() are unlikely to be utilized. To add further confusion, Fortran record markers are almost (?) always 4-byte integers even for double-precision float records. Methods such as read_record_r8_f2() are provided for the theoretical case in which the record marker is a long (8-byte integer). The full table of record-type suffixes for big-endian files is below. Just prepend the suffix with an l for the little-endian versions.

Suffix

Class

Description

r4_i

int32

Common integer record

r8_i

int32

Long integer record

r8_i2

int64

Long integer with long record markers

r4_f

float32

Common single-precision float

r8_f

float32

Common double-precision float

r8_f2

float64

Double float with long record markers

r4_u

uint32

Common unsigned integer record

r8_u

uint32

Long uint record

r8_u2

uint64

Long uint with long record markers

b4_s

str

String from 4-byte char codes
cape.io.check_record(fp, dtype)

Check for consistent record based on record markers

Call:
>>> q = check_record(fp, dtype)
Inputs:
fp: file

File handle, open ‘rb’ or similar

dtype: str

Data type for np.fromfile()

Outputs:
q: True | False

Whether or not fp has a valid record in the next position

Version:

  • 2018-01-11 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 2.0; fork check_record_lr4()

cape.io.fromfile_b4_f(fp, n)

Read n 4-byte float big-endian

Call:
>>> x = fromfile_b4_f(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[float]

Array of n floats if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_b4_i(fp, n)

Read n 4-byte int big-endian

Call:
>>> x = fromfile_b4_i(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[int]

Array of n integers if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_b8_f(fp, n)

Read n 8-byte float big-endian

Call:
>>> x = fromfile_b8_f(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[float]

Array of n floats if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_b8_i(fp, n)

Read n 8-byte int big-endian

Call:
>>> x = fromfile_b8_i(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[int]

Array of n integers if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_lb4_f(fp, n)

Read n 4-byte float little-endian

Call:
>>> x = fromfile_lb4_f(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[float]

Array of n floats if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_lb4_i(fp, n)

Read n 4-byte int little-endian

Call:
>>> x = fromfile_lb4_i(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[int]

Array of n integers if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_lb8_f(fp, n)

Read n 8-byte float little-endian

Call:
>>> x = fromfile_lb8_f(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[float]

Array of n floats if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.fromfile_lb8_i(fp, n)

Read n 8-byte int little-endian

Call:
>>> x = fromfile_lb8_i(fp, n)
Inputs:
fp: file

File handle, open ‘wb’ or similar

n: int

Number of integers to read

Outputs:
x: np.ndarray[int]

Array of n integers if possible

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.get_env_byte_order()

Determine byte order from system and environment variables

This checks the following environment variables to override the system byte order if found. (Listed in order of precedence)

  1. F_UFMTENDIAN (flag for ifort)

    1. "little" for little-endian

    2. "big" for big-endian

  2. GFORTRAN_CONVERT_UNIT (flag for gfortran and related)

    1. "little_endian" for little-endian

    2. "big_endian" for big-endian

Call:
>>> ebo = get_env_byte_order()
Outputs:
ebo: "big" | "little"

Implied default byte order

Versions:

  • 2021-12-29 @ddalle: Version 1.0

cape.io.get_filenametype(fname)

Get the file type by trying to read first line

Call:
>>> ft = get_filenametype(fname)
Inputs:
fname: str

File name

Outputs:
ft: str
File type code:

  • "": empty file

  • "|": ASCII

  • "<4": little-endian single-precision (32-bit)

  • "<8": little-endian double-precision (64-bit)

  • ">4": big-endian single-precision (32-bit)

  • ">8": big-endian double-precision (64-bit)

  • "?": not ASCII and no Fortran records

Versions:

  • 2016-09-04 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 2.0; use _get_filetype()

cape.io.get_filetype(fp)

Get the file type by trying to read first line

Call:
>>> ft = get_filetype(fp)
Inputs:
fp: file

File handle open for reading

Outputs:
ft: str
File type code:

  • "": empty file

  • "|": ASCII

  • "<4": little-endian single-precision (32-bit)

  • "<8": little-endian double-precision (64-bit)

  • ">4": big-endian single-precision (32-bit)

  • ">8": big-endian double-precision (64-bit)

  • "?": not ASCII and no Fortran records

Versions:

  • 2016-09-04 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 2.0; allow fp

cape.io.read_b4_s(fp)

Read C-style string assuming 4 big-endian bytes per char

Call:
>>> s = read_b4_s(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
s: str

String read from file

Versions:

  • 2016-11-14 @ddalle: Version 1.0

cape.io.read_c_bytes(fp, nmax=1000)

Read bytes of a C-style string from a binary file

String is terminated with a null \0 character

Call:
>>> s = read_c_str(fp, nmax=1000)
Inputs:
fp: file

File handle, open ‘wb’ or similar

nmax: {1000} | int

Maximum number of characters, to avoid infinite loops

Outputs:
s: bytes

String read from file until \0 character

Versions:

  • 2016-11-14 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; from read_c_str()

cape.io.read_c_str(fp, encoding='utf-8', nmax=1000)

Read a C-style string from a binary file

String is terminated with a null \0 character

Call:
>>> s = read_c_str(fp, encoding="utf-8", nmax=1000)
Inputs:
fp: file

File handle, open ‘wb’ or similar

encoding: {"utf-8"} | "ascii" | str

Valid encoding name

nmax: {1000} | int

Maximum number of characters, to avoid infinite loops

Outputs:
s: str

String read from file until \0 character

Versions:

  • 2016-11-14 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; fix data types

cape.io.read_lb4_s(fp)

Read C-style string assuming 4 little-endian bytes per char

Call:
>>> s = read_lb4_s(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
s: str

String read from file

Versions:

  • 2016-11-14 @ddalle: Version 1.0

cape.io.read_record_end(fp, dtype, r1)

Read and check Fortran-style end-of-record marker

Call:
>>> r1 = read_record_end(fp, dtype, r1)
Inputs:
fp: file

File handle, open ‘rb’ or similar

dtype: str

Data type for np.fromfile()

r1: np.int

Start-of-record, usually number of bytes in record

Outputs:
r2: r1.__class__

End-of-record, matches r1

Raises:

IOError: if r1 and r2 do not match

Versions:

  • 2021-12-29 @ddalle: Version 1.0

cape.io.read_record_lr4_f(fp)

Read 4-byte little-endian float record

Call:
>>> x = read_record_lr4_f(fp)
Inputs:
fp: file

File handle, open ‘rb’ or similar

Outputs:
x: np.ndarray[float]

Array of floats

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_lr4_i(fp)

Read 4-byte little-endian int record

Call:
>>> x = read_record_lr4_i(fp)
Inputs:
fp: file

File handle, open ‘rb’ or similar

Outputs:
x: np.ndarray[int]

Array of integers

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_lr8_f(fp)

Read 8-byte little-endian float record

Call:
>>> x = read_record_lr8_f(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[float]

Array of floats

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_lr8_f2(fp)

Read 8-byte little-endian float record

with 8-byte int record markers

Call:
>>> x = read_record_lr8_f2(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[float]

Array of floats

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_lr8_i(fp)

Read 8-byte little-endian int record

Call:
>>> x = read_record_lr8_i(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[int]

Array of integers

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_lr8_i2(fp)

Read 8-byte little-endian int record

with 8-byte int record markers

Call:
>>> x = read_record_lr8_i2(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[int]

Array of integers

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_r4_f(fp)

Read 4-byte big-endian float record

Call:
>>> x = read_record_r4_f(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[float]

Array of floats

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_r4_i(fp)

Read 4-byte big-endian int record

Call:
>>> x = read_record_r4_i(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[int]

Array of integers

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_r8_f(fp)

Read 8-byte big-endian float record

Call:
>>> x = read_record_r8_f(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[float]

Array of floats

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_r8_f2(fp)

Read 8-byte big-endian float record

using 8-byte int record markers

Call:
>>> x = read_record_r8_f2(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[float]

Array of floats

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_r8_i(fp)

Read 8-byte big-endian int record

Call:
>>> x = read_record_r8_i(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[int]

Array of integers

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_r8_i2(fp)

Read 8-byte big-endian int record

using 8-byte int record markers

Call:
>>> x = read_record_r8_i2(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

Outputs:
x: np.ndarray[int]

Array of integers

Version:

  • 2016-09-05 @ddalle: Version 1.0

  • 2021-12-29 @ddalle: Version 1.1; read_record_start()

cape.io.read_record_start(fp, dtype)

Read Fortran-style start-of-record marker

Call:
>>> r1 = read_record_start(fp, dtype)
Inputs:
fp: file

File handle, open ‘rb’ or similar

dtype: str

Data type for np.fromfile()

Outputs:
r1: dtype

Start-of-record, usually number of bytes in record

Versions:

  • 2021-12-29 @ddalle: Version 1.0

cape.io.tofile_b4_f(fp, x)

Write a float or array to single-precision big-endian file

Call:
>>> tofile_b4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

Float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_b4_i(fp, x)

Write an integer or array to single-precision big-endian file

Call:
>>> tofile_b4_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_b4_s(fp, s)

Write C-style string assuming 4 big-endian bytes per char

Call:
>>> tofile_b4_s(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

s: str

String to write to binary file

Versions:

  • 2017-03-29 @ddalle: Version 1.0

cape.io.tofile_b8_f(fp, x)

Write a float or array to double-precision big-endian file

Call:
>>> tofile_b4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

Float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_b8_i(fp, x)

Write an integer or array to double-precision big-endian file

Call:
>>> tofile_b8_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_lb4_f(fp, x)

Write a float or array to single-precision little-endian file

Call:
>>> tofile_lb4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

Float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_lb4_i(fp, x)

Write an integer or array to single-precision little-endian file

Call:
>>> tofile_lb4_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_lb4_s(fp, s)

Write C-style string assuming 4 little-endian bytes per char

Call:
>>> tofile_lb4_s(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

s: str

String to write to binary file

Versions:

  • 2017-03-29 @ddalle: Version 1.0

cape.io.tofile_lb8_f(fp, x)

Write a float [array] to double-precision little-endian file

Call:
>>> tofile_lb4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

Float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_lb8_i(fp, x)

Write an integer [array] to double-precision little-endian file

Call:
>>> tofile_lb8_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_ne4_f(fp, x)

Write a float or array to single-precision native-endian file

Call:
>>> tofile_ne4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

Float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_ne4_i(fp, x)

Write an integer or array to single-precision native-endian file

Call:
>>> tofile_ne4_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_ne4_s(fp, s)

Write C-style string assuming 4 native-endian bytes per char

Call:
>>> tofile_lb4_s(fp)
Inputs:
fp: file

File handle, open ‘wb’ or similar

s: str

String to write to binary file

Versions:

  • 2017-03-29 @ddalle: Version 1.0

cape.io.tofile_ne8_f(fp, x)

Write a float or array to double-precision native-endian file

Call:
>>> tofile_ne8_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

Float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.tofile_ne8_i(fp, x)

Write an integer or array to double-precision native-endian file

Call:
>>> tofile_ne8_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_lr4_f(fp, x)

Write Fortran float32 record to little-endian file

The record markers are 4-byte int32.

Call:
>>> write_record_lr4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_lr4_i(fp, x)

Write Fortran int32 record to little-endian file

The record markers are int32.

Call:
>>> write_record_lr4_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_lr8_f(fp, x)

Write Fortran float64 record to little-endian file

The record markers are 4-byte int32.

Call:
>>> write_record_lr8_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_lr8_f2(fp, x)

Write special Fortran float64 record little-endian

The record markers from this function are int64.

Call:
>>> write_record_lr8_f2(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_lr8_i(fp, x)

Write Fortran int64 record to little-endian file

The record markers are 4-byte int32.

Call:
>>> write_record_lr8_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_lr8_i2(fp, x)

Write special Fortran int64 record little-endian file

The record markers are int64.

Call:
>>> write_record_lr8_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_r4_f(fp, x)

Write Fortran float32 record to big-endian file

The record markers are 4-byte int32.

Call:
>>> write_record_r4_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_r4_i(fp, x)

Write Fortran int32 record to big-endian file

The record markers are 4-byte int32.

Call:
>>> write_record_r4_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_r8_f(fp, x)

Write Fortran float64 record big-endian

The record markers are 4-byte int32.

Call:
>>> write_record_r8_f(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_r8_f2(fp, x)

Write special Fortran float64 record big-endian

The record markers are 8-byte int64.

Call:
>>> write_record_r8_f2(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: float | np.ndarray

float or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_r8_i(fp, x)

Write Fortran int64 record to big-endian file

The record markers are 4-byte int32.

Call:
>>> write_record_r8_i(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0

cape.io.write_record_r8_i2(fp, x)

Write special Fortran int64 record big-endian

The record markers are 8-byte int64.

Call:
>>> write_record_r8_i2(fp, x)
Inputs:
fp: file

File handle, open ‘wb’ or similar

x: int | np.ndarray

Integer or array to write to file

Versions:

  • 2016-09-05 @ddalle: Version 1.0