Title:Large-scale Open Dataset, Pipeline, and Benchmark for Bandit Algorithms

Abstract:We build and publicize the Open Bandit Dataset to facilitate scalable and reproducible research on bandit algorithms. It is especially suitable for off-policy evaluation (OPE), which attempts to estimate the performance of hypothetical policies using data generated by a different this http URL construct the dataset based on experiments and implementations on a large-scale fashion e-commerce platform, ZOZOTOWN. The data contain the ground-truth about the performance of several bandit policies and enable fair comparisons of different OPE estimators. We also build a Python package called the Open Bandit Pipeline to streamline implementations of bandit algorithms and OPE estimators. Our open data and pipeline will allow researchers and practitioners to easily evaluate and compare their bandit algorithms and OPE estimators with others in a large, real-world setting. Using our data and pipeline, we provide extensive benchmark experiments of existing OPE estimators. Our experiments open up essential challenges and new avenues for future OPE research. Our pipeline and example data are available at this https URL. You can follow the updates of the whole project at this https URL.