In household telephone surveys, a long field period may be required to maximize the response rate and achieve adequate sample sizes. However, long field periods can be problematic when measures of seasonally affected behavior are sought. Surveys of child care use are one example because child care arrangements vary by season. Options include varying the questions posed about school-year and sum…
Although experiments are viewed as the gold standard for evaluation, some of their benefits may be lost when, as is common, outcomes are not defined for some sample members. In evaluations of marriage interventions, for example, a key outcome—relationship quality—is undefined when a couple splits up. This article shows how treatment-control differences in mean outcomes can be misleadi…
Disciplinary alternative schools have a reputation as gateways to the juvenile and criminal justice systems. The authors conducted an evaluation of an intervention (Strategies for Success) designed to divert seventh-, eighth-, and ninth-grade alternative school students from this gateway. They used propensity score matching and a multivariate random effects model to estimate program impacts and…
This article explores the statistical methodologies used in demonstration and effectiveness studies when the treatments are applied across multiple settings. The importance of evaluating and how to evaluate these types of studies are discussed. As an alternative to standard methodology, the authors of this article offer an empirical binomial hierarchical Bayesian model as a way to effectively e…
Suicide rates are higher among those who own or live in a household with a hand gun. This article examines the association between hand gun ownership and mental health, another risk factor for suicide. Data from the General Social Survey, a series of surveys of U.S. adults, are analyzed to compare general emotional and mental health, sadness and depression, functional mental health, and mental …
Previous studies have shown that active consent procedures result in sampling bias in surveys dealing with adolescent risk behaviors such as cigarette smoking and illicit drug use. To examine sampling bias from active consent procedures when the survey topic pertains to childhood obesity and associated health behaviors, the authors pair data obtained from both active and passive consent p…
This article describes rates of missing item responses in personal digital assistant (PDA) assessments as compared to paper assessments. Data come from the evaluation of a classroom-based leisure, life skills, and sexuality education program delivered to high school students in Cape Town, South Africa. Analyses show that the paper assessments had much higher rates of missing-ness than PDA a…
Programs using means tests to identify low-income households face a trade-off between promoting access and ensuring program integrity. The authors use a comparison-district design to estimate the effects of a pilot program to improve the accuracy of the process of certifying students for free or reduced-price meals in the National School Lunch Program. This pilot program required households to …
Active parental consent policies have been blamed for low participation rates and selection bias (i.e., loss of "high-risk" youths) in school-based studies. In this article, the authors describe active consent procedures that produced an overall active consent rate of 79% in a sample of more than 4,500 middle school students attending 29 schools in seven cities across the United States. Consent…
Regressions can be weighted by propensity scores in order to reduce bias. However, weighting is likely to increase random error in the estimates, and to bias the estimated standard errors downward, even when selection mechanisms are well understood. Moreover, in some cases, weighting will increase the bias in estimated causal parameters. If investigators have a good causal model, it seems bette…
Social science research is increasingly considering place when examining social programs and policies with a spatial component. A specific research challenge involving spatial policies is how to select a sample of individuals based on their geographic locations. This article illustrates the use of geographic information systems, tax parcels, and mail surveys to target residents in varied geogra…
This article addresses the issue of falling response rates in telephone surveys. To better understand and maintain respondent goodwill, concepts of psychological contract and respondent expectations are introduced and explored. Results of the qualitative study show that respondent expectations are not only socially contingent but also ego-expressive, utilitarian, pleasurable, and epistemic by n…
The development of European Community administrative authority has greatly influenced the development of an evaluation culture among the southern and central member states of the European Union. The present case study from Spain provides an example of this diffusion through the use of an empowerment evaluation approach to build evaluation capacity within the context of rural development. The st…
Though collaboration is often required in community initiatives, little evidence documents relationships between collaboration and program success. The authors contend that clarification of the construct collaboration is necessary for investigating its contribution to the success of community initiatives. After respecifying collaboration, they present a study of a multisite program that involve…
Both true experiments and regression discontinuity (RD) designs produce unbiased estimates of effects. However, true experiments are, of course, often criticized on equity grounds, whereas RD designs entail sacrifices in terms of statistical precision. In this article, a hybrid of true experiments and RD designs is considered. This hybrid entails smaller sacrifices in statistical efficiency tha…
Longitudinal research designs involve data collection at multiple time points to measure change over time. Therefore, identification of the same respondents is essential at each time point so that data from the same respondents can be matched for comparison over time. Subject-generated identification codes permit an anonymous means to track respondents over multiple data collection points. This…
The Collegiate Learning Assessment (CLA) program measures value added in colleges and universities, by testing the ability of freshmen and seniors to think logically and write clearly. The program is popular enough that it has attracted critics. In this paper, we outline the methods used by the CLA to determine value added. We summarize the criticisms, which revolve around the question of which…
We conduct an evaluation of MedShare, a program designed to enhance access to prescription drugs for indigent patients in the Greater Cincinnati area. The program expands access to drugs by providing subsidies to reduce the costs paid by patients for their prescriptions. The assumption is that by expanding access to prescription drugs, participant health outcomes as measured by quality of life …
The chronic and relapsing nature of substance abuse points to the need for continuing care after a primary phase of treatment. This article reviews the economic studies of continuing care, discusses research gaps, highlights some of the challenges of conducting rigorous economic evaluations of continuing care, and offers research guidelines and recommendations for future economic studies in thi…
Several school performance indices were evaluated using achievement data from multiple longitudinally matched middle school student cohorts in a large district in the southwestern United States. Measures designed to index the within-cohort achievement status and growth of students as well as the status and growth-based changes that occur over years between different student cohorts were studied…