Blair, Graeme, Jasper Cooper, Alexander Coppock, and Macartan Humphreys. “Declaring and Diagnosing Research Designs.” Working Paper.
The evaluation of research depends on assessments of the quality of underlying research designs. Surprisingly, however, there is no standard definition for what a design is. We provide a framework for formally characterizing the analytically relevant features of a research design. The approach to design declaration we describe requires defining population structures, a potential outcomes function, a sampling strategy, an assignment strategy, estimands, and an estimation strategy. Given a formal declaration of a design in code, Monte Carlo techniques can the be easily applied to a design in order to diagnose properties, such as power, bias, expected mean squared error, external validity with respect to some population, and other “diagnosands.” Declaring a design in computer code lays researchers’ assumptions bare and allows for clear communication with funders, journal editors, reviewers, and readers. Ex ante design declarations can be used to improve designs and facilitate preregistration, analysis, and ex post reconciliation of intended and actual analyses. Design declaration is also useful ex post however and can be used to describe and share designs as well as to facilitate reanalysis and critique. We provide an open-source software package, DeclareDesign, to implement the proposed approach.
Preliminary draft here.