A Precision-Based and Adaptive Approach to Number of Replications for Monte Carlo Studies of Robustness and Power

Abstract

Monte Carlo (MC) researchers must determine how many replications or repeated samples to draw for each condition under investigation. MC experiments performed with too few replications may produce idiosyncratic results, but too many replications may be inefficient. The purpose of this paper is to examine the number of replications needed in MC experiments designed to investigate robustness and statistical power. A precision-based method for determining an appropriate number of replications, uniquely combined here with robustness criteria, is recommended. Using analytical and meta-Monte Carlo methods, implications of this precision-based method are considered and tables for the recommended number of replications based on the method are provided. Further, recommendations are made to enhance both the accuracy and consistency of MC studies of robustness and power using an adaptive, continuous criterion comparison method of programming combined with the precision-based approach. Ultimately, we show that tables provided here can also be used when MC researchers desire an appropriate number of replications for estimating both proportion parameters (e.g., Type I error, statistical power) and non-proportion parameters (e.g., means, regression coefficients).