experiment.experiment module

Configuration environment for experiments.

This module introduces a framework for configuring experiments. It is based on the jupyter configuration framework (traitlets and application).

from experiment import Experiment
from traitlets import Float


class Main(Experiment):

    lr = Float(0.1, config=True, help="Learning rate of training")

    def run(self):
        <Run the experiment>

if __name__ == "__main__":
    main = Main()
    main.initialize()
    main.start()

For more examples see the examples folder.

class experiment.experiment.Experiment(**kwargs)

Bases: traitlets.config.application.Application

A singleton experiment with full configuration support.

cache

An instance of a Python dict.

class_config_section(cls)

Get the config class config section

config_file

A trait for unicode strings.

config_file_name

A trait for unicode strings.

create_aliases()

Flatten all class traits using aliasses.

custom_log_handlers

An instance of a Python list.

exit(exit_status=0)

generate_config

A boolean (True, False) trait.

generate_config_file()

generate default config file from Configurables

initialize(argv=None)

Do the basic steps to configure me.

Override in subclasses.

load_config_file(suppress_errors=True)

Load the config file.

By default, errors in loading config are handled, and a warning printed on screen. For testing, the suppress_errors option is set to False, so errors will make tests fail.

log_level

An enum whose value must be in a given sequence.

name

A trait for unicode strings.

results_path

A trait for unicode strings.

run()

The logic of the experiment.

Should be implemented by the subclass.

start()

Start the whole thing

strict_git

A boolean (True, False) trait.

write_config()

Write our config to a .py config file

class experiment.experiment.MLflowExperiment(**kwargs)

Bases: experiment.experiment.Experiment

A singleton experiment with support for mlflow logging.

Note

This object will setup a connection with the mlflow server. To successfully do so it is recommended to setup the following environment variable:

• MLFLOW_SERVER - URL:PORT of mlflow server.

mlflow_server

A trait for unicode strings.

start()

Start the whole thing

class experiment.experiment.TensorboardXExperiment(**kwargs)

Bases: experiment.experiment.Experiment

A singleton experiment with support of TensorBoard logging.

Note

This object requires tensorBoardX to produce log summaries. The log summaries are stored in the path set by self.tb_log_dir. By default a unique folder will be created by script_name/jobid_date/ To successfully do so it is recommended to setup the following environment variable:

• TENSORBOARD_BASE_DIR - Base path for storing log summaries (optional). Defaults to /tmp/tensorboard.

summary_writer

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

Subclasses can declare default classes by overriding the klass attribute

tb_log_dir

A trait for unicode strings.

class experiment.experiment.VisdomExperiment(**kwargs)

Bases: experiment.experiment.Experiment

A singleton experiment with support of Visdom logging.

Note

This object will setup a connection with the visdom server. To successfully do so it is recommended to setup the following environment variables:

• VISDOM_SERVER_URL - URL of visdom server.
• VISDOM_USERNAME - User name to use for logging to the visdom server (optional).
• VISDOM_PASSWORD - Password to use for loffing to the visdom server (optional).

vis

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

Subclasses can declare default classes by overriding the klass attribute

visdom_env

A trait for unicode strings.

visdom_server

A trait for unicode strings.

experiment.experiment.ensure_dir_exists(path, mode=511)

ensure that a directory exists

If it doesn’t exist, try to create it, protecting against a race condition if another process is doing the same.

The default permissions are determined by the current umask.