We explain why benchmarking is needed beyond theoretical analysis, present the steps of proper benchmarking, and introduce OpenML.