Spack-Manager

Spack-Manager is a light-weight extension to Spack that is intended to streamline the software development and deployment cycle for software projects on specific machines. A given software project typically has multiple configurations across many machines. Spack-Manager is quite literal in its name, in that it provides a way to manage and organize these configurations across multiple machines, and multiple projets.

The intent of this project is to maintain as thin of a buffer as possible between software packages and Spack, and to be pushing ideas and workflow improvements back to Spack on a regular basis to reduce the code that is maintained here. As such we do not try to fully mask Spack’s workflow or commands from users, but rather expose them at a level appropriate for the user type.

Additional details about the philosophy, and general knowledge can be found in the following documentation section.

• General Information
  • Spack-Manager Structure
  • Important Spack Features
    • Spack Specs
    • Spack Commands
    • Spack Knowledgebase
  • History and inspirations for Spack-Manager

User Profiles

Spack-Manager’s design is intended to serve three separate user types in a software project, and the level of expected spack exposure/knowledge required decreases as the user becomes further removed from the build process. The three user profiles are:

• System administrators (Spack is heavily exposed and relied on)

• Code developers (some basics about Spack are required, but much of the workflow can be scripted away)

• Analysts (zero exposure to Spack)

Separate documentation exists for each of these user profiles.

• User Profiles
  • Analysts Documentation
  • Developer Documentation
    • Developers: What you need to know about Spack
      • Querying the Spack commands
      • Reading and writing Spack specs
      • Major steps of the Spack build process
    • Quick-Start: Developer Workflow
      • Setup Spack-Manager
      • Creating an Environment
      • Building and Making Code Changes
      • Running Tests and Coming Back
      • Quick Start
    • Useful Commands for Development
      • Environment setup process
      • Environment loading process
      • Environment setup commands
      • Environment re-use commands
      • Other Commands
    • Advanced Topics for Developers
      • Handling Multiple Environments
      • Cleaning Up Old Environments
      • Modifying Specs in an Environment
      • Debugging Spack Errors
      • When and How to Use Multiple Instances of Spack-Manager
      • Strategies for Conserving Disk Space
      • Constructing and Utilizing Multiple builds
    • Snapshot Developer Workflow Example
      • Setup
      • Building
      • Editing Code
      • Running
      • Iterating
    • Externals and snapshots
      • How is an external defined
      • CMD: spack manager external
      • How to use spack manager external to share binares with others
      • Relationship with Spack-Manager snapshots
  • System Administrator Documentation
    • Creating and configuring a Spack-Manager Project
      • Creating a Project from Scratch
      • Populating Machine Specific Configurations
      • Configuring Machine Auto-detection
      • Setting up Spack Package Repositories
      • Adding Version Control
• Frequently Asked Questions:
  • How do I setup and build custom branches of amr-wind and openfast together?
  • What should I do if Spack Keeps failing to clone code for me on Eagle?
  • How do I build for GPU’s?
  • I’m getting lots of weird Spack errors, what should I do?
  • What is Clingo and why is Spack saying it is bootstrapping it?
  • What are the ^ in the specs?
  • How do I turn on an optional dependency for my software?
  • Where is my build directory/where is my install directory?
  • How do I run tests after building the software?
  • Should I worry about Warning: included configuration files should be updated manually?
  • Should I worry abour Warning: the original concretizer is currently being used.?
  • How do I use the executables I built in my development environment?
  • What to do about Error: Couldn’t find patch for package exawind.hdf5?
  • I’m getting an error saying there is a missing variant? Error: variant [foo] not found …
  • Permissions issues on NREL’s Eagle machine: [Errno 13] Permission denied:
  • How do I compile faster using a parallel DAG?

Benefits of Spack-Manager

• Spack-Manager allows additional agility and coordination at the project level that can not be maintained for a larger tool/database such as Spack.

  This is done through heavy use of custom Spack package repositories and Spack’s custom extensions to allow for prototyping at the project level, as well as the option to maintain features that are only intended for the project’s development team.

• Spack-Manager maximizes overlap between the workflows of the three user profiles.

  The main thought behind Spack-Manager is the exact same infrastructure that the system administrators use to deploy binaries and modules on various machines can be recycled to run nightly tests, generate time stamped snapshots of binaries for developers to link against, and supply end users with production executables. This framework is stitched together through Spack, but the end products (binaries and modules) are designed to be as independent from Spack as possible.

• Spack-Manager provides the benefits of Spack. Some of these benefits include:

  • Build reproducibiilty, and scalability

  • Buy-in and feedback from HPC vendors and software developers over thousands of projects

  • Support options, extended documentation and testing through the much larger Spack project