Topic: Testing the Prediction Profiler with Disallowed Combinations: A Statistical Engineering Case Study

Speaker: Yeng Saanchi

Date: December 1st at 10:00 a.m. ET 

Abstract

Statistical software is increasingly used to support complex analytical decisions, yet validating that such software behaves correctly across the wide range of problems encountered by users presents a substantial challenge. This presentation describes a statistical engineering approach to validating a new capability in the JMP prediction profiler for handling design spaces constrained by disallowed combinations.
The team adopted an experimental design approach to software testing, namely combinatorial testing. Strength-2 covering arrays were used to construct an efficient test suite that covered all two-way interactions among factors while keeping the number of test cases manageable. A novel aspect of the approach was combining the generation of synthetic data sets and the selection of profiler test cases as a single combinatorial testing problem. This allowed the team to systematically explore a diverse space of possible data characteristics while controlling the number of test cases to consider.  Equivalence partitioning was used to determine the levels of inputs with a wide range of possible values. The result was, in effect, a dataset of datasets designed to exercise the profiler enhancement across a wide variety of conditions.
Particular attention was given to determining appropriate test oracles for the selected test cases to ensure that the software is working as intended. The presentation will show how design of experiments techniques can be adapted to software testing and other problems in which exhaustive testing is impractical. It will also demonstrate how statistical engineering can provide a framework for bringing statistical knowledge, software engineering, optimization, and other domain expertise together to solve complex validation problems. 

Speaker Bio

Yeng Saanchi is a Research Statistician Tester at JMP Statistical Discovery, LLC. She received her M.S. in Statistics from the University of Michigan, Ann Arbor, and her Ph.D. in Statistics from North Carolina State University. At JMP, her work focuses on the numerical verification and validation of statistical routines and methodologies implemented in software. Her research interests include stochastic optimization, applications of optimal experimental design to precision medicine, and statistical software validation. She also enjoys exploring new areas and finding opportunities to apply statistical thinking to challenging problems.

Register Now