Exploring Threats to Internal Validity of Direct Assessment in Single-Case Design Research

Single-case design research studies have historically used external observers to collect time series data that may be used to evaluate intervention effectiveness; however, single-case interventions implemented in educational settings may use the person implementing the intervention (e.g., teacher) to collect data in order to maximize feasibility. The implementer’s knowledge of intervention goals and phase has the potential to influence assessment of dependent variables, particularly when ratings involving some degree of judgment (e.g., Direct Behavior Rating-Single Item Scales [DBR-SIS]) are used. Given the potential for rater effects and expectancy to influence data collection, this study sought to determine whether DBR-SIS measuring social skills collected in vivo by interventionists with full knowledge of intervention goals and phase were equivalent to data collected by external raters masked to intervention phase. Results indicated in vivo DBR-SIS differed from those completed by masked external raters, which has the potential to result in different conclusions regarding intervention effectiveness. The potential for negative effects resulting from sole reliance on in vivo ratings conducted by an interventionist may be mitigated by including additional data streams collected by external personnel masked to intervention phase or by using effect sizes that account for baseline trends. Implications for training and practice are discussed.