Experiment

Framework for running experiments.

The experiment package is meant for simplifying conducting experiments by hiding most of the “boring” boiler plate code, e.g. experiment configuration and logging. It is based on the Traitlets package.

Note

The experiment package is still in beta state and the API might change.

• Free software: MIT license

TL;DR

Copy the following example to a python file hello_experiment.py:

from experiment import Experiment
import logging
import time
from traitlets import Int, Unicode


class Main(Experiment):
    description = Unicode("My hellow world experiment.")
    epochs = Int(10, config=True, help="Number of epochs")

    def run(self):
        """Running the experiment"""

        logging.info("Starting experiment")

        loss = 100
        for i in range(self.epochs):
            logging.info("Running epoch [{}/[]]".format(i, self.epochs))
            time.sleep(.5)

        logging.info("Experiment finished")


if __name__ == "__main__":
    main = Main()
    main.initialize()
    main.start()

Run the script from the command line like:

$ python hello_experiment.py --epochs 15

The configuration, logs and results of the script will be stored in a unique folder under /tmp/results/....

To check the script documentation, run the following from the command line:

$ python hello_experiment.py --help

See the documentation for more advanced usage.

Features

• Clean and versatile configuration system based on the Traitlets package.
• Automatic logging setup.
• Configuration and logging are automatically saved in a unique results folder.
• Run parameters are stored in a configuraiton file to allow for replaying the same experiment.
• Support for multiple logging frameworks: mlflow, visdom, tensorboard
• Automatic monitoring of GPU usage.

The examples folder contains multiple examples showcasing the package features.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.