PyAutoNervesConfiguration

Signal. Connect. Understand.

← all sources · library · 17 files · 572 lines · GitHub

(top level) (4)

general.yaml

40 lines · 40 keys · settings · 23 used · 16 section-read · 1 unused · GitHub

updateshpcinversionoutputparallelprofilingtest
source (40 lines)
1updates:2  iterations_per_quick_update: 1e99 # Non-linear search iterations between every quick update, which just displays the maximum likelihood model fit.3  iterations_per_full_update: 1e99  # Non-linear search iterations between every full update, which outputs all visuals and result fits (e.g. model.result, search.summary), this exits the search and can be slow.4  quick_update_background: false    # If True, quick-update visualization runs on a background thread so sampling is not blocked.5  live_visual_update: false         # If True, quick-update visuals are pushed to a live surface (a Jupyter cell when in a kernel, a matplotlib viewer subprocess otherwise) in addition to being written to disk.6hpc:7  hpc_mode: false                   # If True, use HPC mode, which disables GUI visualization, logging to screen and other settings which are not suited to running on a super computer.8  iterations_per_quick_update: 1e99 #  Non-linear search iterations between every quick update, which just displays the maximum likelihood model fit.9  iterations_per_full_update: 1e99  # Non-linear search iterations between every full update, which outputs all visuals and result fits (e.g. model.result, search.summary), this exits the search and can be slow.10  status: true                      # If True, every fit writes a status record beside its output folder (<output_path>.status.json, at most 2 KB; .status.jsonl attempt history; .traceback.txt on failure) and flushes its checkpoint on SIGUSR1 / SIGTERM. If False, nothing is written and no signal handlers are installed.11  status_interval_s: 60             # Seconds between the status record's cadence writes while a fit runs (it is also written at start, at every full update, on a signal, on completion and on failure).12inversion:13  check_reconstruction: true        # If True, the inversion's reconstruction is checked to ensure the solution of a meshs's mapper is not an invalid solution where the values are all the same.14  reconstruction_vmax_factor: 0.5   # Plots of an Inversion's reconstruction use the reconstructed data's bright value multiplied by this factor.15output:16  dataset_identity: true            # If True, every fit writes files/dataset_identity.json at start: its object id, the paths of its inputs (from Analysis.dataset_identity) and, with dataset_identity_sha256, their SHA256, so the results index can find duplicate and wrong-input fits.17  dataset_identity_sha256: true     # If True, dataset_identity.json records the SHA256 of each input file (reads every input once at fit start). If False, only paths and sizes are recorded.18  force_pickle_overwrite: false     # If True pickle files output by a search (e.g. samples.pickle) are recreated when a new model-fit is performed.19  force_visualize_overwrite: false  # If True, visualization images output by a search (e.g. subplots of the fit) are recreated when a new model-fit is performed.20  info_whitespace_length: 80        # Length of whitespace between the parameter names and values in the model.info / result.info21  log_level: INFO                   # The level of information output by logging.22  log_to_file: false                # If True, outputs the non-linear search log to a file (and not printed to screen).23  log_file: output.log              # The name of the file the logged output is written to (in the non-linear search output folder)24  model_results_decimal_places: 3   # Number of decimal places estimated parameter values / errors are output in model.results.25  remove_files: false               # If True, all output files of a non-linear search (e.g. samples, visualization, etc.) are deleted once the model-fit has completed, such that only the .zip file remains.26  search_internal : false    # If True, the search internal folder which contains a saved state of the non-linear search is delete, saving on hard-disk space.27  samples_to_csv: true              # If True, non-linear search samples are written to a .csv file.28  unconverged_sample_size : 100     # If outputting results of an unconverged search, the number of samples used to estimate the median PDF values and errors.29  visualize_ep_factor_searches: false  # Expectation propagation only. If True, every EP factor search draws its own per-search visuals (e.g. corner plots, model-fit images) and redraws its before-fit visuals. If False, only the first search of each factor draws before-fit visuals and no factor search draws per-search visuals (samples and summaries are still written; the EP optimiser's own graph.png / ep_history output is unaffected).30parallel:31  warn_environment_variables: true  # If True, a warning is displayed when the search's number of CPU > 1 and enviromment variables related to threading are also > 1.32profiling:33  parallel_profile: false           # If True, the parallelization of the fit is profiled outputting a cPython graph.34  repeats: 1                        # The number of repeat function calls used to measure run-times when profiling. unused35test:36  check_likelihood_function: true   # if True, when a search is resumed the likelihood of a previous sample is recalculated to ensure it is consistent with the previous run.37  exception_override: false # If True, disable model assertions during likelihood evaluation.38  lh_timeout_seconds:               # If a float is input, the log_likelihood_function call is timed out after this many seconds, to diagnose infinite loops. Default is None, meaning no timeout.39  log_likelihood_ceiling:           # If a float is input, log likelihoods whose magnitude exceeds it are treated as numerical garbage and replaced by the resample figure of merit. Blank (null) or inf disables the guard, which is the default: a log likelihood scales with the noise-map units, so a fixed magnitude can reject a legitimate fit on a badly-scaled dataset. Opt in where the units are known (e.g. autolens_profiling sets 1.0e+20).40  parallel_profile: false # If True, save worker profiling statistics; also enable profiling.parallel_profile.

logging.yaml

19 lines · 15 keys · settings · 1 used · 14 section-read · 0 unused · GitHub

versiondisable_existing_loggershandlersrootformatterstotal_files_open
source (19 lines)
1version: 1 # Python logging dictionary schema version (1).2disable_existing_loggers: false # If True, disable existing non-root loggers when configuring logging.34handlers:5  console:6    class: logging.StreamHandler # Python handler class used to emit log records.7    level: INFO # Minimum logging severity emitted by this handler or logger.8    stream: ext://sys.stdout # Output stream receiving console log records.9    formatter: formatter # Name of the formatter applied by this handler.1011root:12  level: INFO # Minimum logging severity emitted by this handler or logger.13  handlers: [ console ] # Names of handlers receiving root logger records.1415formatters:16  formatter:17    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s' # Python logging format string applied to each log record.1819total_files_open : false # Whether to output info on the total number of open files in the system, used for debugging parallelism issues.

notation.yaml

41 lines · 23 keys · settings · 5 used · 18 section-read · 0 unused · GitHub

labellabel_format
source (41 lines)
1# The notation configs define the labels of every model parameter which are used when2# visualizing results (for example labeling the axis of the PDF triangle plots output by a non-linear search).345# label: The label given to the each parameter, for plots like PDF corner plots.67# For example, if `centre=x`, the plot axis will be labeled 'x'.8910# superscript: the superscript used on certain plots that show the results of different model-components.1112# For example, if `Gaussian=g`, plots where the parameters of the Gaussian model-component have superscript `g`.1314label:15  label:16    centre: x # Plot label used for the centre parameter.17    normalization: norm # Plot label used for the normalization parameter.18    parameter0: a # Plot label used for the parameter0 parameter.19    parameter1: b # Plot label used for the parameter1 parameter.20    parameter2: c # Plot label used for the parameter2 parameter.21    rate: \lambda # Plot label used for the rate parameter.22    sigma: \sigma # Plot label used for the sigma parameter.23  superscript:24    Exponential: e # Class suffix for an unnamed Exponential Model; named components use component paths.25    Gaussian: g # Class suffix for an unnamed Gaussian Model; named components use component paths.26    ModelComponent0: M0 # Class suffix for an unnamed ModelComponent0 Model; named components use component paths.27    ModelComponent1: M1 # Class suffix for an unnamed ModelComponent1 Model; named components use component paths.2829# label_format: The format certain parameters are output as in output files like the `model.results` file.3031# For example, if `centre={:.2d}`, the format of the centre parameter in results files will use this Python format.3233label_format:34  format:35    centre: '{:.2f}' # Python format string used for centre in result summaries.36    normalization: '{:.2f}' # Python format string used for normalization in result summaries.37    parameter0: '{:.2f}' # Python format string used for parameter0 in result summaries.38    parameter1: '{:.2f}' # Python format string used for parameter1 in result summaries.39    parameter2: '{:.2f}' # Python format string used for parameter2 in result summaries.40    rate: '{:.2f}' # Python format string used for rate in result summaries.41    sigma: '{:.2f}' # Python format string used for sigma in result summaries.

output.yaml

111 lines · 18 keys · settings · 13 used · 5 section-read · 0 unused · GitHub

defaultsamplessamples_weight_thresholdsearch_internalstart_pointlatent_during_fitlatent_after_fitlatent_draw_via_pdflatent_draw_via_pdf_sizelatent_csvlatent_resultslatent_model_parameterscovariancedatanoise_mapsearch_logmodel_graphmodel_figure
source (111 lines)
1# Determines whether files saved by the search are output to the hard-disk. This is true both when saving to the2# directory structure and when saving to database.34default: true # If true then files which are not explicitly listed here are output anyway. If false then they are not.56### Samples ###78# The `samples.csv`file contains every sampled value of every free parameter with its log likelihood and weight.910# This file is often large, therefore disabling it can significantly reduce hard-disk space use.1112# `samples.csv` is used to perform marginalization, infer model parameter errors and do other analysis of the search13# chains. Even if output of `samples.csv` is disabled, these tasks are still performed by the fit and output to14# the `samples_summary.json` file. However, without a `samples.csv` file these types of tasks cannot be performed15# after the fit is complete, for example via the database.1617samples: true # If True, retain the sampled parameter values, likelihoods and weights in samples.csv.1819# The `samples.csv` file contains every accepted sampled value of every free parameter with its log likelihood and20# weight. For certain searches, the majority of samples have a very low weight and have no numerical impact on the21# results of the model-fit. However, these samples are still output to the `samples.csv` file, taking up hard-disk22# space and slowing down analysis of the samples (e.g. via the database).2324# The `samples_weight_threshold` below specifies the threshold value of the weight such that samples with a weight25# below this value are not output to the `samples.csv` file. This can be used to reduce the size of the `samples.csv`26# file and speed up analysis of the samples.2728# For many searches (e.g. MCMC) all samples have an equal weight of 1.0, and this threshold therefore has no impact.29# For these searches, there is no simple way to save hard-disk space. This input is more suited to nested sampling,30# where the majority of samples have a very low weight..3132# Set value to empty (e.g. delete 1.0e-10 below) to disable this feature.3334samples_weight_threshold: 1.0e-10 # Exclude samples with weights below this threshold from samples.csv; null disables filtering.3536### Search Internal ###3738# The search internal folder which contains a saved state of the non-linear search in its internal reprsenetation,39# as a .pickle or .dill file.4041# For example, for the nested sampling dynesty, this .dill file is the `DynestySampler` object which is used to42# perform sampling, and it therefore contains all internal dynesty representations of the results, samples, weights, etc.4344# If the entry below is false, the folder is still output during the model-fit, as it is required to resume the fit45# from where it left off. Therefore, settings `false` below does not impact model-fitting checkpointing and resumption.46# Instead, the search internal folder is deleted once the fit is completed.4748# The search internal folder file is often large, therefore deleting it after a fit is complete can significantly49# reduce hard-disk space use.5051# The search internal representation that can be loaded from the .dill file has many additional quantities specific to52# the non-linear search that the standardized autofit forms do not. For example, for emcee, it contains information on53# every walker. This information is required to do certain analyes and make certain plots, therefore deleting the54# folder means this information is list.5556search_internal: false # If False, delete the search internal checkpoint directory after the fit completes.5758### Start Point ###5960# If an Initalizer is used to provide a start point for the non-linear search, visualization of that start point can be61# output to hard-disk to show the user the initial model-fit that is used to start the search. This visualization is62# the visualizer wrapped in the Analysis class, and therefore should show things like the quality of the fit63# to the data and the residuals at the start point.6465start_point: true # If True, output visualization of the initializer start-point fit.6667### Latent Variables ###6869# A latent variable is not a model parameter but can be derived from the model. Its value and errors may be of interest70# and aid in the interpretation of a model-fit.7172# For example, for the simple 1D Gaussian example, it could be the full-width half maximum (FWHM) of the Gaussian. This73# is not included in the model but can be easily derived from the Gaussian's sigma value.7475# By overwriting an Analysis class's `compute_latent_variables` method we can manually specify latent variables that76# are calculated and output to a `latent.csv` file, which mirrors the `samples.csv` file. The `latent.csv` file has77# the same weight resampling performed on the `samples.csv` file, controlled via the `samples_weight_threshold` above.7879# There may also be a `latent.results` and `latent_summary.json` files output, which the inputs below control whether80# they are output and how often.8182# Outputting latent variables manually after a fit is complete is simple, just call83# the `analysis.compute_latent_variables()` function.8485# For many use cases, the best set up may be to disable autofit latent variable output during the fit and perform it86# manually after completing a successful model-fit. This will save computational run time by not computing latent87# variables during a any model-fit which is unsuccessful.8889latent_during_fit: false # Whether to output the `latent.csv`, `latent.results` and `latent_summary.json` files during the fit when it performs on-the-fly output. Also whether the latent corner plot (`visualize/plots_search.yaml` -> `corner_anesthetic_latent`) is made during the fit; it is always made once the fit completes.90latent_after_fit: true # If `latent_during_fit` is False, whether to output the `latent.csv`, `latent.results` and `latent_summary.json` files after the fit is complete.91latent_draw_via_pdf : true # Whether to draw latent variable values via the PDF of every sample, which uses fewer samples to estimate latent variable errors. If False, latent variable values are drawn from every sample.92latent_draw_via_pdf_size : 100 # The number of samples drawn to estimate latent variable errors if `latent_draw_via_pdf` is True.93latent_csv: true # Whether to ouptut the `latent.csv` file.94latent_results: true # Whether to output the `latent.results` file.95latent_model_parameters: true # Whether the latent parameters a model component declares (e.g. a shear's magnitude and angle for its gamma_1 / gamma_2) join the latent output (`latent.csv`, `latent.results`, `latent_summary.json`). An angle keeps the range its conversion gives it (the same in every fit); only the corner plots recentre it on the posterior.9697# Other Files:9899covariance: true # `covariance.csv`: The [free parameters x free parameters] covariance matrix.100data: true # `data.json`: The value of every data point in the data.101noise_map: true # `noise_map.json`: The value of every RMS noise map value.102103search_log: true # `search.log`: logging produced whilst running the fit method104105model_graph: false # `model.graph`: graphical-model factor graph summary. Only meaningful for graphical models (autofit.graphical); empty for ordinary PriorModel/Collection fits, so disabled by default.106107# `model.png`: the model figure drawn beside `model.info` (autofit.model_figure / af.ModelPlotter).108# Opt-in until the lens acceptance renders pass (model-figures epic phase 3). Unlike other keys this109# one does NOT fall back to `default` when absent -- an absent key means off.110111model_figure: false # If True, output model.png beside model.info.

non_linear (1)

non_linear/GridSearch.yaml

5 lines · 3 keys · settings · 0 used · 0 section-read · 3 unused · GitHub

parallel
source (5 lines)
1# The settings of a parallelized grid search of non-linear searches.23parallel: unused4  number_of_cores: 3 # The number of cores the search is parallelized over by default, using Python multiprocessing. unused5  step_size: 0.1     # The default step size of each grid search parameter, in terms of unit values of the priors. unused

priors (9)

priors/Exponential.yaml

30 lines · 30 keys · prior file · GitHub

centrenormalizationrate
priors (6 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
centrelimits
centrewidth_modifierAbsolutevalue 20.0
normalizationlimits
normalizationwidth_modifierRelativevalue 0.5
ratelimits
ratewidth_modifierRelativevalue 0.5

priors/Gaussian.yaml

30 lines · 30 keys · prior file · GitHub

centrenormalizationsigma
priors (6 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
centrelimits
centrewidth_modifierAbsolutevalue 20.0
normalizationlimits
normalizationwidth_modifierRelativevalue 0.5
sigmalimits
sigmawidth_modifierRelativevalue 0.5

priors/Gaussian2D.yaml

40 lines · 40 keys · prior file · GitHub

centre_0centre_1normalizationsigma
priors (8 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
centre_0limits
centre_0width_modifierAbsolutevalue 20.0
centre_1limits
centre_1width_modifierAbsolutevalue 20.0
normalizationlimits
normalizationwidth_modifierRelativevalue 0.5
sigmalimits
sigmawidth_modifierRelativevalue 0.5

priors/GaussianKurtosis.yaml

40 lines · 40 keys · prior file · GitHub

centrekurtosisnormalizationsigma
priors (8 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
centrelimits
centrewidth_modifierAbsolutevalue 20.0
kurtosislimits
kurtosiswidth_modifierAbsolutevalue 20.0
normalizationlimits
normalizationwidth_modifierRelativevalue 0.5
sigmalimits
sigmawidth_modifierRelativevalue 0.5

priors/MultiLevelGaussians.yaml

10 lines · 10 keys · prior file · GitHub

higher_level_centre
priors (2 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
higher_level_centrelimits
higher_level_centrewidth_modifierAbsolutevalue 20.0

priors/model.yaml

62 lines · 62 keys · prior file · GitHub

ExponentialGaussian
priors (6 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
ExponentialcentreUniformlower 0.0upper 100.0Absolute 20.0[-inf, inf]
ExponentialnormalizationLogUniformlower 1e-06upper 1000000.0Relative 0.5[0.0, inf]
ExponentialrateUniformlower 0.0upper 10.0Relative 0.5[0.0, inf]
GaussiancentreUniformlower 0.0upper 100.0Absolute 20.0[-inf, inf]
GaussiannormalizationLogUniformlower 1e-06upper 1000000.0Relative 0.5[0.0, inf]
GaussiansigmaUniformlower 0.0upper 25.0Relative 0.5[0.0, inf]

priors/prior.yaml

7 lines · 7 keys · prior file · GitHub

gaussian.GaussianPrior
priors (2 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
gaussian.GaussianPriorlower_limitConstantvalue -inf
gaussian.GaussianPriorupper_limitConstantvalue inf

priors/profiles.yaml

62 lines · 62 keys · prior file · GitHub

ExponentialGaussian
priors (6 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
ExponentialcentreUniformlower 0.0upper 100.0Absolute 20.0[-inf, inf]
ExponentialnormalizationLogUniformlower 1e-06upper 1000000.0Relative 0.5[0.0, inf]
ExponentialrateUniformlower 0.0upper 10.0Relative 0.5[0.0, inf]
GaussiancentreUniformlower 0.0upper 100.0Absolute 20.0[-inf, inf]
GaussiannormalizationLogUniformlower 1e-06upper 1000000.0Relative 0.5[0.0, inf]
GaussiansigmaUniformlower 0.0upper 25.0Relative 0.5[0.0, inf]

priors/template.yaml

53 lines · 53 keys · prior file · GitHub

ModelComponent0ModelComponent1
priors (6 params)
ClassParamTypeCentre / lowerWidth / upperWidth modifierLimits
ModelComponent0parameter0Uniformlower 0.0upper 1.0Absolute 20.0
ModelComponent0parameter1LogUniformlower 1e-06upper 1000000.0Relative 0.5
ModelComponent0parameter2Uniformlower 0.0upper 25.0Relative 0.5
ModelComponent1parameter0Uniformlower 0.0upper 1.0Absolute 20.0[-inf, inf]
ModelComponent1parameter1LogUniformlower 1e-06upper 1000000.0Relative 0.5[0.0, inf]
ModelComponent1parameter2Uniformlower 0.0upper 1.0Relative 0.5[0.0, inf]

visualize (3)

visualize/general.yaml

2 lines · 2 keys · settings · 1 used · 1 section-read · 0 unused · GitHub

general
source (2 lines)
1general:2  backend: default         # The matploblib backend used for visualization. `default` uses the system default, can specifiy specific backend (e.g. TKAgg, Qt5Agg, WXAgg).

visualize/plots_search.yaml

11 lines · 11 keys · settings · 0 used · 11 section-read · 0 unused · GitHub

nestmcmcmle
source (11 lines)
1nest:2  corner_anesthetic: true   # Output corner figure (using anestetic) during a non-linear search fit?3  corner_anesthetic_components: true   # Also output one corner figure per top-level model component (e.g. per galaxy), each named corner_anesthetic_<component>? Follows corner_anesthetic when absent.4  corner_anesthetic_latent: false   # Output a corner figure of the latent parameters (e.g. a shear's magnitude and angle rather than gamma_1 / gamma_2), named corner_anesthetic_latent? Independent of the two above. Made after the fit like latent.csv (during it only if output.yaml's latent_during_fit is on).5  corner_anesthetic_latent_components: true   # With corner_anesthetic_latent, also output one latent corner figure per top-level component that has a latent parameter, each named corner_anesthetic_latent_<component>? Follows corner_anesthetic_latent when absent.6mcmc:7  corner_cornerpy: true     # Output corner figure (using corner.py) during a non-linear search fit?8mle:9  subplot_parameters: true   # Output a subplot of the best-fit parameters of the model?10  log_likelihood_vs_iteration: true  # Output a plot of the log likelihood versus iteration number?11  figure_of_merit_vs_iteration: true  # Output the global-best figure-of-merit trace (auto-convergence gradient searches)?

visualize/plots_settings.yaml

9 lines · 9 keys · settings · 7 used · 2 section-read · 0 unused · GitHub

corner_anestheticcorner_cornerpy
source (9 lines)
1corner_anesthetic:2  figsize_per_parammeter: 4   # The figsize of the matplotlib figure is given as the number of parameters times this value, for exmaple with 3 free parameters and a value 4 the figsize will be 3*4 = 12.3  fontsize: 20               # The size of the font of the tick values and parameter labels on the x and y axis of the corner plot.4  facecolor: white           # The facecolor of the corner plot.5  alpha: 0.9                 # The alpha value of the corner plot.6  max_detailed_parameters: 10  # Above this many plotted parameters the corner plot is compact: 2D histograms instead of KDE contours (about a second per panel) and no upper triangle.7  max_parameters_search: 30  # A search skips its full corner plot of a model with more free parameters than this (the per-component plots are still made).8corner_cornerpy:9  fontsize: 14               # The size of the font of the tick values and parameter labels on the x and y axis of the corner plot.