Assessment of the multivariate outlier approach for differential item functioning detection : a Monte Carlo simulation study

Abstract

Differential item functioning (DIF) is an important statistical tool that is used to detect potential bias between groups (e.g., males vs. females) in assessment items. The research pertaining to DIF, and the methods that assess DIF have predominantly focused on when only two groups are present (e.g., males vs. females). However, there are times that a psychometrician may want to detect potential bias in items when there are more than two groups. This can occur for large scale assessments when taking into account grouping variables such as ethnicity, SES, and location of the test taker. Due to the need of methods that can estimate DIF when more than two groups are present, generalized Mantel-Haenszel, generalized logistic regression, Lord’s chi-square, and the multivariate outlier approach were created. However, the multivariate outlier approach has not been assessed in a simulation study to determine how well this method controls for Type I error, and power. Because of this, the current study conducted a simulation study to compare the multivariate outlier approach to generalized Mantel-Haenszel, generalized logistic regression, and Lord’s chi-square in regard to Type I error rates and power. The current simulation study used a variety of simulation conditions. Specifically, various group sizes, percent contamination, impact, group size ratio, level of DIF, and sample size were used to generate the data. From these conditions, the results provided three main suggestions. First, if power is the only concern, then generalized Mantel-Haenszel should be used, as this method had the highest power, but did not control for Type I error. Second, if power and Type I error are both important to the researcher, then the multivariate outlier approach Orthogonalized Gnanadesikan-Ketternring beta with generalized logistic regression should be used. This is because this method had comparable power to generalized Mantel-Haenszel, but controlled for Type I error better than generalized Mantel- Haenszel. Last, if the concern is only Type I error then the multivariate outlier approach chi-square methods should be used, as Type I error was controlled for in most of the conditions.