shellutils: System calls with STDOUT capture, SFTP, and more¶
This module mainly provides two types of tools:
The convenience functions include
Essentially each of these functions create an instance of
subprocess.Popen and then call
subprocess.Popen.communicate(). They are all calls to
_call() but with different defaults and outputs for each case. The
o stands for “output,” as in STDOUT will be captured. The e is
for “error,” referring to STDERR capture, and q is for “quiet,”
meaning that by default both STDOUT and STDERR will be suppressed.
Finally, check_o() is very similar to
subprocess.check_output(); it will capture STDOUT by default, but
it will also raise an exception if tne command’s return code is nonzero.
All of these functions can also be used to execute a command on a remote
host using the host keyword argument.
The second category is collection of persistent-process classes:
The first works by logging into a remote host and creating a process so that you can run a sequence of commands there. It starts with a
ssh -q {host} {shell}
command, for example
ssh -q pfe bash
It then sets STDIN, STDOUT, and STDERR of this process to be non-blocking so that you can run multiple commands and query STDOUT and the others more than one time.
The second class, Shell, works in a similar way but on the
local host. It can be convenient because a Shell instance has
its own current working directory. It also enables writing code for
which remote hosts and local hosts can share the vast majority of their
code.
SFTP is a similar class that starts an SFTP session to a remote
host and behaves much like the system sftp itself. However, users
will generally prefer the fourth class to using this one directly.
The SSHPortal is simply a combination of SSH and
SFTP. More specifically, it’s a new class that have an instance
of both SSH and SFTP as attributes. Although this
requires two logins, it is usually preferable because it allows the
SSH instance to do detailed checks and can create folders with
complex file permissions, while the SFTP is used for the actual
transfer of data.
You can instantiate any of these classes (except Shell, which
usually takes no arguments) by just giving it the name of the remote
host as a single positional parameter.
proc = SSH("pfe")
These classes provide several common functions, such as
SSH.listdir(), SSH.chdir(), SFTP.cd_local(), and
SSHPortal.get() and SSHPortal.put(). But they (except for
SFTP) also have generic run() commands that allow for
running arbitrary code by writing the command to STDIN.
- class lfc._vendor.gitutils._vendor.shellutils.SFTP(host: str, **kw)¶
Interface to an SFTP process
- Call:
>>> sftp = SFTP(host, **kw)
- Inputs:
- Outputs:
- sftp:
SFTP SFTP file transfer instance
- sftp:
- Attributes:
- proc¶
subprocess.Popen– Subprocess used to interfacesftpexecutable
- class lfc._vendor.gitutils._vendor.shellutils.SSH(host=None, **kw)¶
Class for persistent SSH subprocess
This class opens
subprocess.Popeninstance that logs into a remote host of your choosing and then waits for you to issue commands. It can also collect and return STDOUT and STDERR.- Call:
>>> ssh = SSH(host, **kw)
- Inputs:
- Outputs:
- ssh:
SSH Persistent SSH subprocess
- ssh:
- Attributes:
- assert_isdir(fdir: str)¶
Assert that a folder exists on remote host
- Call:
>>> ssh.assert_isdir(fdir)
- Inputs:
- Raises:
ShellutilsFileNotFoundErrorif fdir is not a dir- Versions:
2022-12-19
@ddalle: v1.0
- assert_isfile(fname: str)¶
Assert that a file exists on remote host
- Call:
>>> ssh.assert_isfile(fdir)
- Inputs:
- Raises:
ShellutilsFileNotFoundErrorif fname is not file- Versions:
2022-12-19
@ddalle: v1.0
- chdir(wd: str)¶
Change the current working directory on the remote host
- Call:
>>> ssh.chdir(wd)
- Inputs:
- ssh:
SSH Persistent SSH subprocess
- ssh:
- Versions:
2022-12-19
@ddalle: v1.0
- newfile(fname: str)¶
Create an empty file w/ correct ACLs based on parent folder
The purpose of this command is to create an empty file with the correct name but obeying the default ACLs of the parent folder. SFTP does not upload files with the correct access control lists, so this file creates a new file using
touch. When the empty file already exists, SFTP does not change the ACLs, so the result is a file with the expected permissions. If the file already exists, this function deletes it.
- proc¶
subprocess.Popen– Subprocess interface forsshfor this instance
- read_stdout()¶
Read remote process’s current STDOUT buffer
- wait(timeout=None, dt=0.02)¶
Wait for any running commands to terminate
The method for this is to echo a random string and wait for it to show up in STDOUT. The random string is then removed from the STDOUT buffer so that other methods can access the intended STDOUT text.
- Call:
>>> ierr = ssh.wait()
- Inputs:
- Outputs:
- ierr:
0|1|2 Status indicator:
0: success1: wait() call timed out2: keyboard interrupt
- ierr:
- Versions:
2022-12-19
@ddalle: v1.0
- class lfc._vendor.gitutils._vendor.shellutils.SSHBase¶
Base class for both
SSHandSFTPThis class does not have an
__init__()method- close()¶
Close SSH process
- Call:
>>> ssh.close()
- Inputs:
- ssh:
SSH Persistent SSH subprocess
- ssh:
- Versions:
2022-12-19
@ddalle: v1.0
- read_stderr()¶
Read remote process’s current STDERR buffer
- read_stdout()¶
Read remote process’s current STDOUT buffer
- class lfc._vendor.gitutils._vendor.shellutils.SSHPortal(host: str, cwd=None, encoding='utf-8')¶
Combined SSH and SFTP interface
This class works by logging into a remote host twice, once for a shell (SSH) and another for SFTP.
- Call:
>>> portal = SSHPortal(host, cwd=None, encoding="utf-8")
- Inputs:
- Outputs:
- portal:
SSHPortal Combined SSH and SFTP interface
- portal:
- Attributes:
- assert_isfile_local(fname: str)¶
Assert that a file exists locally
- Call:
>>> portal.assert_isfile_local(fname)
- Inputs:
- Raises:
ShellutilsFileNotFoundErrorif fname does not exist relative to portal.cwd
- close()¶
Close both SSH and SFTP shells, waiting for running commands
- Call:
>>> portal.close()
- Inputs:
- portal:
SSHPortal Combined remote shell and file transfer portal
- portal:
- Versions:
2022-12-19
@ddalle: v1.0
- get(fremote: str, flocal=None, wait=True, **kw)¶
Transfer a file from remote to local host
- Call:
>>> portal.get(fremote, flocal=None, wait=True, **kw)
- Inputs:
- portal:
SSHPortal Combined SSH and SFTP interface
- fremote:
str Name of remote file to send
- flocal: {
None} |str Name of destination file on local host; if
None, thenos.path.basename(fremote)- wait: {
True} |False Option to block (wait) until transfer completes
- progress: {wait} |
True|False Option to display transfer progress percentage
- fprog: {
None} |str File name to show during transfer; default is fremote truncated to fit in current terminal width
- portal:
- put(flocal: str, fremote=None, wait=True, **kw)¶
Transfer a file from local to remote host
- Call:
>>> portal.put(flocal, fremote=None, wait=True, **kw)
- Inputs:
- portal:
SSHPortal Combined SSH and SFTP interface
- flocal:
str Name of local file to send
- fremote: {
None} |str Name of destination file on remote host; if
None, thenos.path.basename(flocal)- wait: {
True} |False Option to block (wait) until transfer completes
- progress: {wait} |
True|False Option to display transfer progress percentage
- fprog: {
None} |str File name to show during transfer; default is flocal truncated to fit in current terminal width
- portal:
- class lfc._vendor.gitutils._vendor.shellutils.Shell(**kw)¶
Class for persistent local subprocess
This class opens
subprocess.Popeninstance that starts a shell process (usually BASH) on your local host and then waits for you to issue commands. It can also collect and return STDOUT and STDERR.- Call:
>>> shell = Shell(**kw)
- Inputs:
- Outputs:
- shell:
Shell Persistent local subprocess
- shell:
- Attributes:
- proc¶
subprocess.Popen– Subprocess interface forsshfor this instance
- exception lfc._vendor.gitutils._vendor.shellutils.ShellutilsError¶
- exception lfc._vendor.gitutils._vendor.shellutils.ShellutilsFileExistsError¶
Error class for already existing file
- exception lfc._vendor.gitutils._vendor.shellutils.ShellutilsFileNotFoundError¶
Error class for file/folder not found
- exception lfc._vendor.gitutils._vendor.shellutils.ShellutilsFilenameError¶
Error class for invalid file names
- exception lfc._vendor.gitutils._vendor.shellutils.ShellutilsIsADirectoryError¶
Error class for attempting to access folder as if a file
- lfc._vendor.gitutils._vendor.shellutils._call(cmd, **kw)¶
Generic system interface
This command can either capture STDOUT [and/or STDERR], print it to the terminal, or pipe it to a file. It more or less encapsulates the capabilities of
subprocess.call()andsubprocess.check_output()
although it works by using
subprocess.Popendirectly.- Call:
>>> out, err, ierr = _call(cmd, **kw)
- Inputs:
- cmd:
list[str] System command to run, broken into parts as for
subprocess.call()- stdout: {
None} | PIPE |file Destination for standard output
- stderr: {
None} | PIPE |file Destination for standard error messages
- encoding: {
"utf-8"} |str Name of encoding to use for converting strings to bytes
- host: {
None} |str Name of remote host (if not
None) on which to run- cwd: {
None} |str Folder in which to run command
- executable: {
"sh"} |str Name of shell to use if on remote host
- cmd:
- Outputs:
- Versions:
2020-12-24
@ddalle: v1.02021-01-27
@ddalle: v1.1; fix default cwd w/ host
- lfc._vendor.gitutils._vendor.shellutils.call(cmd, **kw)¶
Run a system command and ignore STDOUT and STDERR
Setting stdout and/or stderr to PIPE will suppress them. Setting them to
None(the default) will cause them to display normally and not be captured.- Call:
>>> ierr = call(cmd, **kw)
- Inputs:
- cmd:
list[str] System command to run, broken into parts as for
subprocess.call()- stdout: {
None} | PIPE |file Destination for standard output
- stderr: {
None} | PIPE |file Destination for standard error messages
- encoding: {
"utf-8"} |str Name of encoding to use for converting strings to bytes
- host: {
None} |str Name of remote host (if not
None) on which to run- cwd: {
os.getcwd()} |str Folder in which to run command
- executable: {
"sh"} |str Name of shell to use if on remote host
- cmd:
- Outputs:
- ierr:
int Return code from executing command
- ierr:
- Versions:
2020-12-24
@ddalle: v1.0
- lfc._vendor.gitutils._vendor.shellutils.call_o(cmd, **kw)¶
Run a system command and capture STDOUT
- Call:
>>> out, err, ierr = call_o(cmd, **kw)
- Inputs:
- cmd:
list[str] System command to run, broken into parts as for
subprocess.call()- stdout:
None| {PIPE} |file Destination for standard output
- stderr: {
None} | PIPE |file Destination for standard error messages
- encoding: {
"utf-8"} |str Name of encoding to use for converting strings to bytes
- host: {
None} |str Name of remote host (if not
None) on which to run- cwd: {
os.getcwd()} |str Folder in which to run command
- executable: {
"sh"} |str Name of shell to use if on remote host
- cmd:
- Outputs:
- Versions:
2020-12-24
@ddalle: v1.0
- lfc._vendor.gitutils._vendor.shellutils.call_oe(cmd, **kw)¶
Run a system command and capture STDOUT
- Call:
>>> out, err, ierr = call_oe(cmd, **kw)
- Inputs:
- cmd:
list[str] System command to run, broken into parts as for
subprocess.call()- stdout:
None| {PIPE} |file Destination for standard output
- stderr:
None| {PIPE} |file Destination for standard error messages
- encoding: {
"utf-8"} |str Name of encoding to use for converting strings to bytes
- host: {
None} |str Name of remote host (if not
None) on which to run- cwd: {
os.getcwd()} |str Folder in which to run command
- executable: {
"sh"} |str Name of shell to use if on remote host
- cmd:
- Outputs:
- Versions:
2021-07-19
@ddalle: v1.0
- lfc._vendor.gitutils._vendor.shellutils.call_q(cmd, **kw)¶
Run a system command, suppressing STDOUT and STDERR
- Call:
>>> ierr = call_q(cmd, **kw)
- Inputs:
- cmd:
list[str] System command to run, broken into parts as for
subprocess.call()- stdout:
None| {PIPE} |file Destination for standard output
- stderr:
None| {PIPE} |file Destination for standard error messages
- encoding: {
"utf-8"} |str Name of encoding to use for converting strings to bytes
- host: {
None} |str Name of remote host (if not
None) on which to run- cwd: {
os.getcwd()} |str Folder in which to run command
- executable: {
"sh"} |str Name of shell to use if on remote host
- cmd:
- Outputs:
- ierr:
int Return code from executing command
- ierr:
- Versions:
2020-12-24
@ddalle: v1.0
- lfc._vendor.gitutils._vendor.shellutils.check_o(cmd, **kw)¶
Run a system command and capture STDOUT
- Call:
>>> out, err = check_o(cmd, **kw)
- Inputs:
- cmd:
list[str] System command to run, broken into parts as for
subprocess.call()- stdout:
None| {PIPE} |file Destination for standard output
- stderr: {
None} | PIPE |file Destination for standard error messages
- encoding: {
"utf-8"} |str Name of encoding to use for converting strings to bytes
- host: {
None} |str Name of remote host (if not
None) on which to run- cwd: {
os.getcwd()} |str Folder in which to run command
- executable: {
"sh"} |str Name of shell to use if on remote host
- cmd:
- Outputs:
- Versions:
2020-12-24
@ddalle: v1.0
- lfc._vendor.gitutils._vendor.shellutils.identify_host(where=None)¶
Identify possible remote host/local path breakdown
- lfc._vendor.gitutils._vendor.shellutils.validate_absfilename(fname: str, sep='/')¶
Check if a file name is valid, allowing for folder names
This version only disallows punctuation characters that cause problems on at least one common file system. Single quotes and foreign language characters are allowed.
It also restricts the file name length to 255 characters, even though this is not strictly speaking a hard limit on Windows systems.
- Call:
>>> validate_absfilename(fname, sep='/')
- Inputs:
- fname:
str Name of file
- sep: {
'/'} |"\" Path separator
- fname:
- Versions:
2022-12-19
@ddalle: v1.02023-10-25
@ddalle: v1.1; add sep
- lfc._vendor.gitutils._vendor.shellutils.validate_dirname(fdir: str, sep='/')¶
Check if a folder name is valid
This version only disallows punctuation characters that cause problems on at least one common file system. Single quotes and foreign language characters are allowed.
It also restricts the file name length to 255 characters, even though this is not strictly speaking a hard limit on Windows systems.
- Call:
>>> validate_dirname(fdir, sep='/')
- Inputs:
- fdir:
str Name of file
- fdir:
- Versions:
2022-12-19
@ddalle: v1.02023-10-25
@ddalle: v1.1; add sep
- lfc._vendor.gitutils._vendor.shellutils.validate_filename(fname: str)¶
Check if a file name is valid
This version only disallows punctuation characters that cause problems on at least one common file system. Single quotes and foreign language characters are allowed.
It also restricts the file name length to 255 characters, even though this is not strictly speaking a hard limit on Windows systems.
- Call:
>>> validate_filename(fname)
- Inputs:
- fname:
str Name of file
- fname:
- Versions:
2022-12-19
@ddalle: v1.0
- lfc._vendor.gitutils._vendor.shellutils.validate_globname(pattern: str)¶
Check if a glob pattern is valid
This version only disallows punctuation characters that cause problems on at least one common file system. Single quotes and foreign language characters are allowed.
It also restricts the file name length to 255 characters, even though this is not strictly speaking a hard limit on Windows systems.
- Call:
>>> validate_globname(pattern)
- Inputs:
- fname:
str Name of file
- fname:
- Versions:
2022-12-19
@ddalle: v1.0
