Contributions

Contributors

The fmdtools library is developed by small team of developers at NASA Ames Research Center, along with external open-source contributors.

NASA Contributors

NASA Alumni

Interns

  • Inga Girshfeld, contributed to examples/navigating_rover fault creation using coevolutionary algorithm

  • Cole Jetton, contributed to examples library

  • Zain Khatri, contributed to fmdtools graph functionality

  • Cody Wang, contributor to examples/airspacelib/contingency_management path planning

  • [Dhruv Kothari](https://github.com/kotharidhruv, contributor to examples/airspacelib/water_rescue models

  • Iris Chen, contributor to examples/airspacelib/water_rescue models

  • Isabel Agostino, contributor defone/pathplan path planning functionality (as well as associated tests and examples)

External Contributors

  • Johan Louwers

  • Sylvester Kaczmarek

fmdtools at OSU

Prior to fmdtools being developed at NASA, it was a project in the Design Engineering Lab at Oregon State University (available here), which was conceptualized in this initial paper by Daniel Hulse, Hannah Walsh, Andy Dong, Christopher Hoyle and Irem Tumer (at OSU) and Chetan Kulkarni and Kai Goebel (at NASA). Special thanks to Arpan Biswas and Hongyang Zhang also for working on some case studies in the repository.

fmdtools Publication List

The list below is a chronological accounting of relevant publications from the fmdtools team:

How to Contribute

Development of fmdtools is coordinated by the fmdtools team at NASA Ames Research Center. As an NASA-developed and maintained open-source tool, outside contributions are welcomed. To be able to submit contributions (e.g., pull requests), external contributors should first submit a contributors license agreement (Individual CLA , Corporate CLA).

Development Installation

To develop fmdtools, use an editable install with all optional dependencies. In the terminal:

cd "/path/to/fmdtools_folder"
git clone https://github.com/nasa/fmdtools.git  #replace with bitbucket url if developing internally
uv venv --python 3.14       # Set up project virtual environment. Use the primarily-supported python version
.venv/Scripts/activate      # Activate the virtual environment. On mac and linux: source .venv/bin/activate
uv pip install -e .[all] --group dev  # Installs both examples and all testing/documentation/dev workflow dependencies

If developing with Spyder, you will want to set this Python venv as your default interpreter.

You may do this by navigating to: Tools > Python interpreter > Selected interpreter

And then selecting the interpreter from the file menu. The path should look something like: path/to/fmdtools/.venv/Scripts/python.exe

After clicking “Apply”, close the ipython console panel so that the console will restart.

At this point, there may be an error message in the ipython console if spyder-kernels is not installed or the wrong version. To fix this, go back to the terminal and install using

uv pip install spyder-kernels==<VERSION>

where <VERSION> is the version recommended in the error message.

To edit certain markdown files as presentations, it may be helpful to download quarto: https://quarto.org/

If developing with VSCode, make sure to install the standard Python, Jupyter, and Quarto extensions.

Repository Structure

../_images/repo_organization.svg

Getting started with development first requires some basic familiarity with the repo structure, shown above. As shown, the repo contains:

  • /fmdtools, which is where the toolkit sub-packages and modules are held,

  • /tests, which has tests of the modules,

  • /examples, which are different case study examples,

  • /docs-source, which contains the sphinx documentation source files, and

  • /docs, which contains documentation (built from source).

There are additionally a few scripts/config files with specific purposes to serve the development process:

  • tests/conftest.py which defines configuration options for pytest (our testing suite),

  • pyproject.toml which defines all python project and build configuration information,

  • conf.py which defines sphinx documentation settings, and

  • MAKE, which is used to build the sphinx documentation.

Git Structure and Setup

../_images/git_structure.svg

Development of fmdtools uses a two-track development model, in which contributions are provided within NASA as well as by external collaborators. To support this, there are multiple repositories which must be managed in the development process, as shown above. Essentially, there is:

  • An internal bitbucket, origin, where NASA coordination, development, and continuous integration takes place,

  • A public GitHub, public, where collaboration with outside developers takes place (and where documentation is hosted), and

  • A PyPI repository, which contains stable versions of fmdtools which can be readily installed via pip. This repository is automatically updated when a new version is released on GitHub.

  • The fmdtools GitHub Pages site, which updates from the gh-pages branch.

The fmdtools team is responsible for coordinating the development between the internal and external git repositories. Managing multiple repositories can best be coordinated by:

  • setting up the public and origin remotes on a single git repo on your machine

  • using the dev branch and feature branches on origin for development and integration

  • releasing to the main branch on origin and public

To assist with this, the custom git alias below can be helpful:

[alias]
        up = "!git merge dev main"
        pp = "!f() { git push public tag main --follow-tags; }; f"
        po = "!f() { git push origin tag main --follow-tags; }; f"
        release = "!f() { git checkout main && git up && git pp && git po; }; f"
        fb = "!f() { git fetch origin && git fetch public; }; f"
        mm = "!git merge main dev"
        sync-into-dev = "!f() { git checkout dev && git fb && git pull public dev && git pull origin dev && git merge main dev && git merge test dev; }; f"

Adding this block to your repository’s git config file (e.g., .git/config) adds custom git commands which can be used to simplify the release process. Specifically:

  • git sync-into-dev will merge all main and dev branches (local and remote) into your local dev branch

  • git release will merge dev into main and upload it to public and origin.

Git Tags for versions (as well as other indicators of version) are controlled with bump-my-version and applies the format Major.Minor.Patch.StageNum where Stage is the stage of development (dev or rc) and num is the iteration (e.g., a second round of testing would be test2).

To ensure that tags make it to the servers, make sure to set git config –global push.followTags true.

Git Development Workflow

../_images/dev_process.svg

To encourage code quality we follow the general process above to manage contributions:

  1. Development begins on a /dev branch for a given version of fmdtools. This branch is used to test and integrate contributions from multiple sources.

  2. The first step in making a contribution is then to create an issue, which describes the work to be performed and/or problem to be solved.

  3. This issue can then be taken on in an issue branch (or repo for external contributions) and fixed.

  4. When the contributor is done with a given fix, they can submit a pull request, which enables the review of the fix as a whole.

  5. This fix is reviewed by a member of the fmdtools team, who may suggest changes.

  6. When the review is accepted, it is merged into the dev branch.

  7. When all the issues for a given version are complete (this may also happen concurrently with development), tests and documentation are updated for the branch. If tests do not pass (or are obsolete), contributions may be made directly on the dev branch to fix it, or further issues may be generated based on the impact of the change.

  8. When the software team deems the release process to be complete, the dev branch may be merged into the main branch. These branches are then used to create releases.

The major exceptions to this process are:

  • bug fixes, which, if minor, may occur on main/dev branches (or may be given their own branches off of main),

  • external contributions, which are managed via pull request off of main (or some external dev branch), and

  • minor documentation changes.

Release Process

Releases are made to fmdtools to push new features and bugfixes to the open-source community as they are developed. Some important things to remember about the release process are:

  • It’s important to test prior to release to ensure (1) bugs aren’t being released that could have been caught easily with a test (2) test results are accurate to the current version of the code and (3) the documentation stays up to date with the release. Currently, this is managed via Bamboo CI, which automatically builds releases on the /dev branch.

  • Releases are made to the fmdtools GitHub repository using the Draft a new release button on the Releases page. Once this release is generated, GitHub Actions uploads it to the fmdtools PyPi repo automatically.

To ensure all of the steps of the release process are performed, follow the Release Checklist, see below:

Release Checklist

Step

Description

Complete?

Comment

1

Sync appropriate branches into release branch using git sync-into-dev

2

Test

2a

Bump version: if version not initialized: “run bump-my-version stage” else “run bump-my-version patch” or “rum bump-my-bersion num” so version is V2.X.X.testnum

2b

Sync to bamboo to run tests (git push origin dev). If tests fail, fix tests and go to 2.

3

Docs

3a

Update project plan and docs with changed dependencies, version, new activities, contributors, etc.

3b

Bump version: if version not initialized: “run bump-my-version stage” else “run bump-my-version patch” so version is V2.X.X.docs (or V2.X.X)

3c

Sync to bamboo to build docs (git push origin dev).

4

Release

4a

Release to GitHub/remotes using ``bump-my-version stage’ ‘ (so version is V2.X.X) and ``git release’’ ``

4b

“Toggle CodeFactor so it updates”

4c

Create a release in GitHub with narrative summary of features and changes and auto-generated bitbucket and github release notes (and this checklist!)

Roles

  • team lead: coordinates all activities and has technical authority over project direction

  • full developer: can make changes off of version and main branches and has full ability to perform the release process

  • contributor: creates issues and develops off of issue branches

Documentation

Documentation is generated using Sphinx, which generates html from rst files. This is performed automatically on the Bamboo server. The process for generating documentation locally is to open powershell and run:

cd path/to/fmdtools
./make clean
./make html

Style/Formatting

Generally, we try to follow PEP8 style conventions. To catch these errors, it is best to turn on PEP8 style linting in your IDE of choice.

Style conventions can additionally be followed/enforced automatically using the Black code formatter. See resources:

Headers

All source code files to be released under fmdtools should have the Apache-2.0 license applied to them.

In modules and scripts, it is best practice to use the following format for the header:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
<One-line software description here>

<Further module-level docstring information, if needed>

Copyright © 2024, United States Government, as represented by the Administrator
of the National Aeronautics and Space Administration. All rights reserved.

The “"Fault Model Design tools - fmdtools version 2"” software is licensed
under the Apache License, Version 2.0 (the "License"); you may not use this
file except in compliance with the License. You may obtain a copy of the
License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
"""

<local/project imports>

<fmdtools imports (examples, then tests, then modules)>

<external imports>

For jupyter notebooks, the following block should be inserted at the end of the first markdown cell, after the title/description:

```
Copyright © 2024, United States Government, as represented by the Administrator of the National Aeronautics and Space Administration. All rights reserved.

The “"Fault Model Design tools - fmdtools version 2"” software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
```

Testing

There are two major types of tests:

  • quantitative tests, which are tests that pytest can return, and

  • qualitative tests, which plots/figures that are to be verified by checking the outputs the example notebooks

Why was fmdtools developed?

The fmdtools library was developed to study resilience, which is an important consideration in designing safe, low-risk systems. Resilience is the ability of a system to mitigate hazardous scenarios as they arise. As shown below, the key defining aspect of resilience is the dynamics of failure events, which may lead to recovery (or, a safe outcome) or failure (or, an unsafe outcome).

importance of considering resilience

Resilience is important to consider when dynamics of system behavior can lead to hazardous or unsafe outcomes.

The impetus for developing fmdtools was a lack of existing open-source tools to model these dynamics at a high level (i.e., functions and flows) in the design process. Thus, researchers in this area had to re-implement modeling, simulation, and analysis approaches for each new case study or methodological improvement. The fmdtools package resolved this problem by separating resilience modeling, simulation, and analysis constructs from the model under study, enabling reuse of methodologies between case studies. The goals of the fmdtools project have since shifted to the more general goal of improving the hazard assessment process by better representing systems resilience. Towards this end, fmdtools provides the following capabilities:

  • Representing system dynamics to enable the quantification of resilience properties. Typically, hazard assessment processes neglect the consideration of resilience because they focus on the immediate effects of faults on the function of the system, rather than an assessment of how these effects play out over time. The fmdtools library enables this consideration by providing a behavioral view of hazardous scenarios. This is important both for understanding hazardous behaviors, but also how they can be mitigated as they arise.

  • Representing operational behaviors and actions to enable the assessment the contributions of human operators and autonomous/AI-enabled systems to overall risk and resilience. Traditional hazard assessment approaches do not consider the feedback between operators, the system, and the environment, instead leaving them as “accidents” or “mistakes” to be blamed on the operator. With fmdtools, these hazards can be considered directly by modelling potential operator behaviors and how they support or degrade overall systems resilience. These approaches can also be used to better understand the risks posed by AI/autonomous systems.

  • Enabling a Model/Simulation-based hazard analysis paradigm by allowing the iterative, consistent analysis of resilience through the design, implementation, and V&V processes. The traditional hazard assessment process is a manual, expert-driven approach that is inefficient to iterate on or change as a the design changes or assumptions are validated (or invalidated). In contrast, because all assumptions in fmdtools are represented as code, they can easily be modified as assumptions change while maintaining the overall integrity of the analysis. Furthermore, simulations in fmdtools can be efficiently and consistently be varied to analyze a system in more detail or in different configurations.

While this library primarily provides code structures, a major objective of this library is further to enable these techniques to be used in a graphical simulation tool for hazard assessment.