DAVID E. JONES
Department of Statistical Science
Designing Test Information and Test Information in Design
DeGroot (1962) developed a general framework for constructing Bayesian measures of the expected information that an experiment will provide for estimation. These and subsequent measures are often optimized in experimental design for estimation, but the literature has comparatively overlooked design for decision problems. To address this, we propose an analogous framework for constructing information measures when the goal is hypothesis testing, classification, or model selection. We demonstrate how our “test information” measures can be used in experimental design, and show that the resulting designs differ from designs based on estimation information, e.g., Fisher information or the measures proposed by Lindley (1956) and DeGroot (1962). The underlying intuition of our design proposals is straightforward: to distinguish between two or more models we should collect data from regions of the covariate space for which the models differ most. Nonetheless, the decision problem context presents some unique challenges and we identify a fundamental coherence identity which, when satisfied, ensures that the optimal design does not depend on which hypothesis is true. We additionally establish an asymptotic equivalence between our test information measures and Fisher information, thereby offering insight for contexts where both testing and estimation are of interest. Lastly, we illustrate our design ideas by applying them to a times series classification problem in astronomy.
Monday, 1/29/2018, 11:30 AM, BLOC 457