Machine Learning Experiments Done Right

A detailed guideline for designing machine learning experiments that produce reliable, reproducible results.

Machine learning (ML) practitioners run experiments to compare the effectiveness of methods for both specific applications and for general types of problems. The validity of experimental results hinges on how practitioners design, run, and analyze their experiments. Unfortunately, many ML papers lack valid results. Recent studies [] reveal a lack of reproducibility in published experiments, attributing this to practices such as:

• Data contamination: engineering training datasets to include data that is semantically similar to, or directly from, the test dataset
• Cherrypicking: selectively choosing an experimental setup or results that favorably present a method
• Misreporting: including “the improper use of statistics to analyze results, such as claiming significance without proper statistical testing or using the wrong statistic test []

  The goal of an experiment is to find the best configuration of controllable factors that maximizes the response function while minimizing the impact of uncontrollable factors. A well-designed experiment needs two key elements: a systematic way to test different combinations of controllable factors, and a method to account for randomness from uncontrollable factors.

  Building on these principles, a clear and organized framework is crucial for effectively designing and conducting experiments. Below, we present a checklist that guides a practitioner through the planning and execution of an ML experiment.

  A Machine Learning Experiment Checklist

  To plan and perform a rigorous ML experiment:

  1. State the objective of your experiment
  2. Select the response function, or what you want to measure
  3. Decide what factors vary, and what remains the same
  4. Describe one run of the experiment, which should define:
     (a) a single configuration of the experiment
     (b) the datasets used
  5. Choose an experimental design, which should define:
     (a) how we explore the factor space and
     (b) how we repeat our measurements (cross validation)
  6. Perform the experiment
  7. Analyze the data
  8. Draw conclusions and recommendations

  1. State the objective of the experiment

  The objective should state clearly why is the experiment to be performed. It is also important to specify a meaningful effect size. For example, if the goal of an experiment is “to determine the if using a data augmentation technique improves my model’s accuracy”, then we must add, “a significant improvement is greater than or equal to 5%.”

  2. Select the response function, or what you want to measure

  The response function of a Machine Learning experiment is typically an accuracy metric relative to the task of the learning function, such as classification accuracy, mean average precision, or mean squared error. It could also be a measure of interpretability, robustness or complexity — so long as the metric is be well-defined.

  3. Decide what factors vary, and what remains the same

  A machine learning system has several controllable factors, such as model design, data pre-processing, training strategy, and feature selection. In this step, we decide what factors remain static, and what can vary across runs. For example, if the objective is “to determine the if using a data augmentation technique improves my model’s accuracy”, we could choose to vary the data augmentation strategies and their parameters, but keep the model the same across all runs.

  4. Describe one run of the experiment

  A run is a single instance of the experiment, where a process is applied to a single configuration of factors. In our example experiment with the aim “to determine the if using a data augmentation technique improves my model’s accuracy”, a single run would be: “to train a model on a training dataset using one data augmentation technique and measure its accuracy on a held-out test set.”

  In this step, we also select the data for our experiment. When choosing datasets, we must consider whether our experiment a domain-specific application or for generic use. A domain-specific experiment typically requires a single dataset that is representative of the domain, while experiments that aim to show a generic result should evaluate methods across multiple datasets with diverse data types [] for more information).

  Cross validation is essential for ML experiments, as this reduces the variance of your results which come from the choice of dataset split. To determine the number of cross-validation samples needed, we return to our objective statement in Step 1. If we plan to perform a statistical analysis, we need to ensure that we generate enough data for our specific statistical test.

  A final part of this step is to think about resource constraints. How much time and compute does one run take? Do we have enough resources to run this experiment as we designed it? Perhaps the design must be altered to meet resource constraints.

  6. Perform the experiment

  To ensure that the experiment runs smoothly, It is important to have a rigorous system in place to organize data, track experiment runs, and analyze resource allocation. Several open-source tools are available for this purpose (see , or checking out my .

  [2] Design & Analysis of Machine Learning Experiments — Machine Learning — Spring 2016 — Professor Kogan. .

  [4] Questionable Practices in Machine Learning. .

  [6] A Step Toward Quantifying Independently Reproducible Machine Learning Research. was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf towardsdatascience.com.

  ↗ Original-Artikel auf towardsdatascience.com lesen