exarl.candlelib.default_utils

Module Contents

Classes

ArgumentStruct	Class that converts a python dictionary into an object with
ListOfListsAction	This class extends the argparse.Action class by instantiating
Benchmark	Class that implements an interface to handle configuration options for the

Functions

fetch_file(link, subdir, untar=False, md5_hash=None)	Convert URL to file path and download the file
verify_path(path)	Verify if a directory path exists locally. If the path
set_up_logger(logfile, logger, verbose)	Set up the event logging system. Two handlers are created.
eval_string_as_list(str_read, separator, dtype)	Parse a string and convert it into a list of lists.
eval_string_as_list_of_lists(str_read, separator_out, separator_in, dtype)	Parse a string and convert it into a list of lists.
str2bool(v)	This is taken from:
set_seed(seed)	Set the seed of the pseudo-random generator to the specified value.
finalize_parameters(bmk)	Utility to parse parameters in common as well as parameters
get_default_neon_parser(parser)	Parse command-line arguments that are default in neon parser (and are common to all frameworks).
get_common_parser(parser)	Parse command-line arguments. Ignore if not present.
args_overwrite_config(args, config)	Overwrite configuration parameters with
get_choice(name)	Maps name string to the right type of argument
directory_from_parameters(params, commonroot='Output')	Construct output directory path with unique IDs from parameters
keras_default_config()	Defines parameters that intervine in different functions using the keras defaults.

Attributes

file_path
lib_path
work_path
DEFAULT_SEED
DEFAULT_TIMEOUT
DEFAULT_DATATYPE

exarl.candlelib.default_utils.file_path

exarl.candlelib.default_utils.lib_path

exarl.candlelib.default_utils.work_path

exarl.candlelib.default_utils.DEFAULT_SEED = 7102

exarl.candlelib.default_utils.DEFAULT_TIMEOUT

exarl.candlelib.default_utils.DEFAULT_DATATYPE

exarl.candlelib.default_utils.fetch_file(link, subdir, untar=False, md5_hash=None)

Convert URL to file path and download the file if it is not already present in spedified cache.

Parameters

• link (link path) – URL of the file to download

• subdir (directory path) – Local path to check for cached file.

• untar (boolean) – Flag to specify if the file to download should be decompressed too. (default: False, no decompression)

• md5_hash (MD5 hash) – Hash used as a checksum to verify data integrity. Verification is carried out if a hash is provided. (default: None, no verification)

Returns

local path to the downloaded, or cached, file.

exarl.candlelib.default_utils.verify_path(path)

Verify if a directory path exists locally. If the path does not exist, but is a valid path, it recursivelly creates the specified directory path structure.

Parameters

path (directory path) – Description of local directory path

exarl.candlelib.default_utils.set_up_logger(logfile, logger, verbose)

Set up the event logging system. Two handlers are created. One to send log records to a specified file and one to send log records to the (defaulf) sys.stderr stream. The logger and the file handler are set to DEBUG logging level. The stream handler is set to INFO logging level, or to DEBUG logging level if the verbose flag is specified. Logging messages which are less severe than the level set will be ignored.

Parameters

• logfile (filename) – File to store the log records

• logger (logger object) – Python object for the logging interface

• verbose (boolean) – Flag to increase the logging level from INFO to DEBUG. It only applies to the stream handler.

exarl.candlelib.default_utils.eval_string_as_list(str_read, separator, dtype)

Parse a string and convert it into a list of lists.

Parameters

• str_read (string) – String read (from configuration file or command line, for example)

• separator (character) – Character that specifies the separation between the lists

• dtype (data type) – Data type to decode the elements of the list

Returns

decoded_list (list) – List extracted from string and with elements of the specified type.

exarl.candlelib.default_utils.eval_string_as_list_of_lists(str_read, separator_out, separator_in, dtype)

Parse a string and convert it into a list of lists.

Parameters

• str_read (string) – String read (from configuration file or command line, for example)

• separator_out (character) – Character that specifies the separation between the outer level lists

• separator_in (character) – Character that specifies the separation between the inner level lists

• dtype (data type) – Data type to decode the elements of the lists

Returns

decoded_list (list) – List of lists extracted from string and with elements of the specified type.

exarl.candlelib.default_utils.str2bool(v)

This is taken from: https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse Because type=bool is not interpreted as a bool and action=’store_true’ cannot be undone.

Parameters

v (string) – String to interpret

Returns

• Boolean value. It raises and exception if the provided string cannot be interpreted as a boolean type.

• Strings recognized as boolean True – ‘yes’, ‘true’, ‘t’, ‘y’, ‘1’ and uppercase versions (where applicable).

• Strings recognized as boolean False – ‘no’, ‘false’, ‘f’, ‘n’, ‘0’ and uppercase versions (where applicable).

class exarl.candlelib.default_utils.ArgumentStruct(**entries)

Class that converts a python dictionary into an object with named entries given by the dictionary keys. This structure simplifies the calling convention for accessing the dictionary values (corresponding to problem parameters). After the object instantiation both modes of access (dictionary or object entries) can be used.

class exarl.candlelib.default_utils.ListOfListsAction(option_strings, dest, type, **kwargs)

Bases: argparse.Action

This class extends the argparse.Action class by instantiating an argparser that constructs a list-of-lists from an input (command-line option or argument) given as a string.

Initialize a ListOfListsAction object. If no type is specified, an integer is assumed by default as the type for the elements of the list-of-lists.

Parameters

• option_strings (string) – String to parse

• dest (object) – Object to store the output (in this case the parsed list-of-lists).

• type (data type) – Data type to decode the elements of the lists. Defaults to np.int32.

• kwargs (object) – Python object containing other argparse.Action parameters.

__call__(self, parser, namespace, values, option_string=None)

This function overrides the __call__ method of the base argparse.Action class.

This function implements the action of the ListOfListAction class by parsing an input string (command-line option or argument) and maping it into a list-of-lists. The resulting list-of-lists is added to the namespace of parsed arguments. The parsing assumes that the separator between lists is a colon ‘:’ and the separator inside the list is a comma ‘,’. The values of the list are casted to the type specified at the object initialization.

Parameters

• parser (ArgumentParser object) – Object that contains this action

• namespace (Namespace object) – Namespace object that will be returned by the parse_args() function.

• values (string) – The associated command-line arguments converted to string type (i.e. input).

• option_string (string) – The option string that was used to invoke this action. (optional)

exarl.candlelib.default_utils.set_seed(seed)

Set the seed of the pseudo-random generator to the specified value.

Parameters

seed (int) – Value to intialize or re-seed the generator.

exarl.candlelib.default_utils.finalize_parameters(bmk)

Utility to parse parameters in common as well as parameters particular to each benchmark.

Parameters

bmk (benchmark object) – Object that has benchmark filepaths and specifications

Returns

gParameters (python dictionary) – Dictionary with all the parameters necessary to run the benchmark. Command line overwrites config file specifications

exarl.candlelib.default_utils.get_default_neon_parser(parser)

Parse command-line arguments that are default in neon parser (and are common to all frameworks). Ignore if not present.

Parameters

parser (ArgumentParser object) – Parser for neon default command-line options

exarl.candlelib.default_utils.get_common_parser(parser)

Parse command-line arguments. Ignore if not present.

Parameters

parser (ArgumentParser object) – Parser for command-line options

exarl.candlelib.default_utils.args_overwrite_config(args, config)

Overwrite configuration parameters with parameters specified via command-line.

Parameters

• args (ArgumentParser object) – Parameters specified via command-line

• config (python dictionary) – Parameters read from configuration file

exarl.candlelib.default_utils.get_choice(name)

Maps name string to the right type of argument

exarl.candlelib.default_utils.directory_from_parameters(params, commonroot='Output')

Construct output directory path with unique IDs from parameters

Parameters

• params (python dictionary) – Dictionary of parameters read

• commonroot (string) – String to specify the common folder to store results.

class exarl.candlelib.default_utils.Benchmark(filepath, defmodel, framework, prog=None, desc=None, parser=None)

Class that implements an interface to handle configuration options for the different CANDLE benchmarks. It provides access to all the common configuration options and configuration options particular to each individual benchmark. It describes what minimum requirements should be specified to instantiate the corresponding benchmark. It interacts with the argparser to extract command-line options and arguments from the benchmark’s configuration files.

Initialize Benchmark object.

Parameters

• filepath (./) – os.path.dirname where the benchmark is located. Necessary to locate utils and establish input/ouput paths

• defmodel ('p*b*_default_model.txt') – string corresponding to the default model of the benchmark

• framework ('keras', 'neon', 'mxnet', 'pytorch') – framework used to run the benchmark

• prog ('p*b*_baseline_*') – string for program name (usually associated to benchmark and framework)

• desc (' ') – string describing benchmark (usually a description of the neural network model built)

• parser (argparser (default None)) – if ‘neon’ framework a NeonArgparser is passed. Otherwise an argparser is constructed.

parse_from_common(self)

Functionality to parse options common for all benchmarks. This functionality is based on methods ‘get_default_neon_parser’ and ‘get_common_parser’ which are defined previously(above). If the order changes or they are moved, the calling has to be updated.

parse_from_benchmark(self)

Functionality to parse options specific specific for each benchmark.

format_benchmark_config_arguments(self, dictfileparam)

Functionality to format the particular parameters of the benchmark.

Parameters

• dictfileparam (python dictionary) – parameters read from configuration file

• args (python dictionary) – parameters read from command-line Most of the time command-line overwrites configuration file except when the command-line is using default values and config file defines those values

read_config_file(self, file)

Functionality to read the configue file specific for each benchmark.

set_locals(self)

Functionality to set variables specific for the benchmark - required: set of required parameters for the benchmark. - additional_definitions: list of dictionaries describing the additional parameters for the benchmark.

check_required_exists(self, gparam)

Functionality to verify that the required model parameters have been specified.

exarl.candlelib.default_utils.keras_default_config()

Defines parameters that intervine in different functions using the keras defaults. This helps to keep consistency in parameters between frameworks.