ASPIRE Configuration

This tutorial reviews the default ASPIRE configuration and common patterns for overriding.

Default Configuration

ASPIRE uses the confuse library for managing configuration. While this document should cover common uses, advanced users and developers should consider reviewing their documentation.

The ASPIRE package ships with config_default.yaml. This represents a base configuration. The shipped configuration for this version of ASPIRE is:

version: 0.12.2
common:
    # numeric module to use - one of numpy/cupy
    numeric: numpy
    # fft backend to use - one of pyfftw/scipy/cupy/mkl
    fft: pyfftw

    # Set cache directory for ASPIRE example data.
    # By default the cache location will be set by pooch.os_cache(),
    # which sets cache based on operating system as follows:
    # Mac: ~/Library/Caches/<AppName>
    # Unix: ~/.cache/<AppName> or the value of the XDG_CACHE_HOME environment variable, if defined.
    # Windows: C:\Users\<user>\AppData\Local\<AppAuthor>\<AppName>\Cache
    cache_dir: ""

logging:
    # Set log_dir to a relative or absolute directory
    # Default is a subfolder `logs` in your current working directory.
    log_dir: logs
    log_exceptions: 1
    # tqdm_disable=True disables progress bars.
    tqdm_disable: False

    # set Python logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL
    # logging level for console output:
    console_level: INFO
    # logging level for log file output:
    log_file_level: DEBUG

    # custom log file name prefix can be set using 'log_prefix'
    # will appear as '[log_dir]/[log_prefix]-{timestamp}.log'
    log_prefix: aspire

nufft:
    # NUFFT backends should be one of finufft, cufinufft, pynfft.
    # They will be attempted in order from left to right.
    backends: [cufinufft, finufft, pynfft]

ray:
    # Ray will default to a OS specific tmp dir.
    #   By default on linux this is `/tmp/ray`.
    # If you find your machine has a very small /tmp,
    #   try setting `temp_dir` to `/dev/shm`
    #   or some other fast scratch dir.
    temp_dir: /tmp/ray

System Overrides

From here we can override with a custom config file in your home dir, specifically $HOME/.config/ASPIRE/config.yaml on most Linux platforms. Items in this file will take precedence over the default configuration. For other platforms, refer to the confuse documentation.

As an example, suppose you want to change the ray temp_dir variable when working on a specific machine. By creating $HOME/.config/ASPIRE/config.yaml with the following contents on that machine, ASPIRE’s configuration utility will overload the temp_dir directory from a /tmp/ray folder to /scratch/tmp/ray.

ray:
  temp_dir: /scratch/tmp/ray

Override Configuration Directory

Users may specify a directory containing the configuration file. This is done by using the enviornment variable ASPIREDIR If you wanted a file in your working directory to take precedence over system-overrides, we can create a local config.yaml.

Suppose you want to store ASPIRE logs at /tmp/my_proj/aspire_logs when working on a specific project. Create the following config.yaml.

logging:
  log_dir: /tmp/my_proj/aspire_logs

This directory must then be set before invoking any code.

export ASPIREDIR=$PWD

Similarly, you could specify any directory you like that might contain a configuration. This allows you to store configurations for reuse.

In-Code Overrides

You can also specify your own file from an arbitrary location from within Python code. For precise behavior refer to the confuse documentation.

aspire.config.set_file('/path/to/some_experimental_config.yaml')

Or simply set specific variables as needed. Here we will disable progress bars displayed by aspire.utils.trange and aspire.utils.tqdm.

import time

from aspire import config
from aspire.utils import trange

# Progress bars are displayed by default.

print("Disabling progress bars")
config["logging"]["tqdm_disable"] = True

for _ in trange(3):
    time.sleep(1)
print("Done Loop 1\n")

print("Re-enabling progress bars")
config["logging"]["tqdm_disable"] = False

for _ in trange(3):
    time.sleep(1)
print("Done Loop 2\n")

Disabling progress bars
Done Loop 1

Re-enabling progress bars

  0%|          | 0/3 [00:00<?, ?it/s]
 33%|███▎      | 1/3 [00:01<00:02,  1.00s/it]
 67%|██████▋   | 2/3 [00:02<00:01,  1.00s/it]
100%|██████████| 3/3 [00:03<00:00,  1.00s/it]
100%|██████████| 3/3 [00:03<00:00,  1.00s/it]
Done Loop 2

Resolution

ASPIRE logs the config_dir() for your system at startup, along with the configuration sources and resolved configuration at import time. This should give an accurate snapshot of the configuration before any in-code overrides. To view these as saved in your log, you will need to locate your log_dir. If you are not sure where it is, we can ask the config:

import aspire

print(aspire.config["logging"]["log_dir"].as_filename())

/home/runner/work/ASPIRE-Python/ASPIRE-Python/gallery/tutorials/logs

You can also resolve the config in code

print(aspire.config.dump())

logging:
    tqdm_disable: no
    log_dir: /home/runner/work/ASPIRE-Python/ASPIRE-Python/gallery/tutorials/logs
    log_exceptions: 1
    console_level: INFO
    log_file_level: DEBUG
    log_prefix: aspire
common:
    cache_dir: /home/runner/.cache/ASPIRE-data
    numeric: numpy
    fft: pyfftw
version: 0.12.2

nufft:
    backends: [cufinufft, finufft, pynfft]

ray:
    temp_dir: /tmp/ray

Logging preferences

The “Logging” section contains options for controlling the verbosity and destination of log messages generated by ASPIRE.

By default, the log_dir is a directory called logs within the current working directory from which ASPIRE was invoked, but it can be changed to an absolute path (for instance, to go to the same folder as a custom configuration file stored elsewhere, such as in the example above):

logging:
    log_dir=/tmp/my_proj/aspire_logs

ASPIRE directs output streams to both the console and to a log file during each session. The logging verbosity for both of these can be specified individually via console_level and log_file_level. These levels are passed through to Python’s logging system, so the labels are the same: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL". In general, it may make sense to have the log file store more verbose output than the console. By default, only INFO and above level messages are printed to the terminal, while detailed DEBUG messages are saved to disk:

logging:
    console_level: INFO
    log_file_level: DEBUG

The following modification would save all DEBUG messages to the log file, but only print ERROR and CRITICAL messages in the terminal:

logging:
    console_level: ERROR
    log_file_level: DEBUG

By default, the filename of the log file is aspire-{%Y-%m-%dT%H-%M-%S.%f}.log. The prefix before the timestamp can be customized via the log_prefix option.

More details on logging behavior in Python can be found in the Python logging HOWTO.

Advanced logging options

Users with sophisticated knowledge of Python’s logging library are able to directly work with the logging.conf file stored in src/aspire in the source.

[loggers]
keys = root

[handlers]
keys = consoleHandler, fileHandler

[formatters]
keys = simpleFormatter

[logger_root]
level = DEBUG
handlers = consoleHandler, fileHandler

[handler_consoleHandler]
class = StreamHandler
level = %(console_level)s
formatter = simpleFormatter
args = (sys.stdout,)

[handler_fileHandler]
class = FileHandler
level = %(log_file_level)s
formatter = simpleFormatter
# You may set a custom filename here:
args =(os.path.join('%(log_dir)s', '%(log_prefix)s-%(dt_stamp)s.log'),)

[formatter_simpleFormatter]
format = %(asctime)s %(levelname)s [%(name)s] %(message)s

This file exposes options that ASPIRE’s config.yaml does not, for example the precise formatting options for log messages. However, values from ASPIRE’s configuration are passed through this file, so caution is needed when modifying it.