Childhood Obesity Research Demonstration Project: Cross-Site Evaluation Methods

Introduction

The Childhood Obesity Research Demonstration (CORD) project supports the development, delivery, and evaluation of three demonstration projects that use an integrated approach to implement multilevel, multisector environmental systems and policy changes that are intended to impact childhood obesity. All three projects link services across settings for at-risk families using community health workers (CHWs). The ultimate goal is to provide evidence-based recommendations for applying the CORD model in broader populations and contexts. The three demonstration projects and respective interventions are described in detail in accompanying articles in this issue of Childhood Obesity and include: the Mass in Motion Kids project in Massachusetts (MA-CORD)¹; Our Choice/Nuestra Opción project in California (CA-CORD)²; and the Texas CORD project (TX-CORD).³ The CORD Evaluation Center (EC-CORD) will provide a comprehensive evaluation of the pooled data across the three CORD demonstration projects. This article describes the general framework and plan for the EC-CORD comprehensive evaluation.

Methods

Evaluation Framework

Although the three projects all utilize the CORD model framework,^4–8 the resulting project-specific designs, populations, interventions, planned measures, implementation schedules, and assessment plans vary considerably. Unlike a multicenter trial that has a uniform intervention protocol, each project includes activities and interventions tailored to the specific communities in which the project is implemented. To address these challenges, EC-CORD will use a general logic model (Fig. 1) and an extensive set of common measures for outcomes, key variables, and processes at multiple levels.

The logic model includes the inputs, activities, and short-term, medium-term, and long-term outcomes used to describe the CORD project communities, interventions, and results across multiple sectors in a consistent way. Consequently, it identifies and organizes the essential elements of the projects that require measurement and evaluation. In addition, the model provides the structure to explain how the integrated intervention components produce changes in behavior and health.

The CORD consortium, made up of EC-CORD investigators, the demonstration project investigators, and CDC advisors, identified a set of common measures using a collaborative process during the first year of the CORD project.^8,9 Common measures from previously validated instruments were selected for the CORD target outcomes: fruit and vegetable consumption; sugar-sweetened beverage and water consumption; physical activity (PA); screen time; and sleep duration. In addition, the set of common measures includes indicators that capture important characteristics of communities (e.g., sociodemographics), facilities (e.g., healthcare centers and schools), providers (e.g., physicians and teachers), families and children (e.g., education, income, weight status, and health-related behaviors), as well as aspects of process, sustainability, and costs. Each project collects and reports all of these common data as well as additional project-specific data to EC-CORD for warehousing and use in the comprehensive evaluation. Developing and implementing a consistent data collection protocol with an extensive and comprehensive set of common measures at multiple levels for multiple outcomes and covariates has been described as “prospective harmonization”¹⁰ or “prospectively planned pooled analysis”¹¹ and represents the optimal situation for pooled data analysis.^12,13

EC-CORD also developed a taxonomy for coding the different interventions in a comparable way across the three projects. Table 1 shows how the intervention components across the three CORD projects will be coded into one of three broad categories that reflect components of the CORD model: Public Health; Primary Care; and Primary Care Plus.⁸ Public health (PH) includes activities and programs in multiple community settings, such as schools, early child care education centers, and recreation centers, that are intended to improve the health of the general public. For example, each CORD project is using evidence-based health promotion programs in elementary schools, such as Sports, Play, and Active Recreation for Kids in CA-CORD; Eat Well Keep Moving in MA-CORD; and Coordinated Approach to Child Health in TX-CORD.^1–3 Similar evidence-based programs and associated integrated activities are delivered in early child care education centers and various community venues as described in greater detail in the accompanying articles in this issue.^1–3 Primary care (PC) involves obesity primary prevention efforts through changes to clinic policies, systems, and environments and the practices of medical professionals that are intended to affect all families at the participating clinics. A PC activity implemented by all three CORD projects is the changes to electronic health record systems to provide prompts related to diagnosis of obesity and links to resources the primary care physician may use to counsel the family.^1–3 Primary Care Plus (PC Plus) augments PC with additional, secondary prevention care and activities provided only to families with children who are overweight or obese (i.e., BMI ≥85th percentile for age and sex). Examples of PC Plus include programs to which families with an obese child are referred by the PC providers, as described in the accompanying articles from each project in this issue of Childhood Obesity: the intensive community health worker-based program in CA-CORD²; the healthy weight clinics in MA-CORD¹; and the Mind, Exercise, Nutrition, Do It program¹⁴ in TX-CORD.³

Across the three CORD project designs, children and families will be exposed to various combinations of these three components, as shown in Table 1, including PH only, PC Plus only, PH with PC, PH with PC Plus, or none (comparison cohorts). Using this taxonomy will allow for describing the interventions in a consistent way and coding the combinations of interventions to which participants were assigned by design.¹⁵ This taxonomy also will allow coding of observed combinations outside of the initial project designs, such as “PC only” if a CORD study participant attends a participating PC clinic, but does not live in the area or attend one of the local schools where the CORD PH intervention is delivered. This allows analyses to be conducted on an intention-to-treat basis to evaluate results that account for some participants not receiving or completing the intended interventions, and to be conducted on a “per protocol” basis to evaluate results from a known exposure to the respective intervention activities.

Process Evaluation

To conduct a thorough process evaluation across the three CORD demonstration projects, three common process evaluation components will be assessed: reach,¹⁶ dose delivered,^16,17 and fidelity,^16,17 each defined in Table 2. The use of these constructs in process evaluation has been well documented in similar multiproject studies, such as the HEALTHY study for the prevention of type 2 diabetes in middle school students.¹⁸

During the CORD planning year, each project provided information regarding the process data they were planning to collect and their methods of data collection (e.g., audits, observations, surveys, and interviews). This information was then compared across projects, and only data that are consistently and uniformly collected across communities and settings will be used in analyses and reported in the evaluation. The projects use standardized process evaluation data collection forms, definitions, and procedures developed and distributed to the projects by the EC-CORD to facilitate data reporting and ensure consistency for EC-CORD's pooled evaluation.

Following the CORD Model's framework for implementing changes in practices, systems, and environments that influence obesity-related behaviors and health of children and families,⁸ the CORD projects deliver their interventions through existing providers (e.g. doctors and teachers) in specific settings and institutions (e.g. clinics and schools). Program implementation and delivery to families depends on the training and support that existing providers received from the study. The intent of training is to create prepared and engaged providers who, in turn, deliver the intervention activities designed to prepare and motivate families to engage in healthy living behaviors. Therefore, process evaluation components will be assessed at two levels: researcher- (i.e., CORD investigators and project teams) to-provider (i.e., individuals and organizations delivering CORD activities in each setting) and provider-to-family (i.e., parents and children). With this approach, both providers and families will be defined as participants, which allows for a more thorough process evaluation.

The process evaluation components are measured in each of the settings where interventions are delivered: primary care clinics; early child care/education (child care) centers, public schools, and community institutions. Three process measures will be assessed for each component: provider training; family and patient education; and policy, systems, or environmental changes. An “other” measure category will also be included to capture additional information related to program implementation that did not fit into the other measures. This complex approach to process evaluation will provide a comprehensive and consistent view of CORD implementation across the three projects.

The total number of providers trained to deliver the CORD interventions in each setting and the total number of children and families participating in one or more CORD activities will be counted in each community at the end of each project year to calculate reach. The data from the project will be combined with local, state, and US Census data and information from publicly available sources (National Center for Education Statistics [NCES] for public school demographics, Environmental Systems Research Institute [Esri] ArcGIS Business Analyst to identify healthcare and early child care and education facilities) to determine the number of eligible institutions and families in each community. These data will then be compared to the counts of participating CORD providers, children, and families to describe the reach of the CORD project into each community.

The dose delivered for each intervention will be assessed by aggregating data from activity logs and checklists maintained by the projects as they deliver the interventions. This analysis will describe what the project investigators delivered to their providers as well as what those providers delivered to families and children. To assess fidelity, a priori lists of planned activities within each setting will be compared to activities actually delivered.

Impact Evaluation

The impact evaluation will estimate the longitudinal effects of the CORD interventions on child behaviors (consumption of fruits and vegetables, water, and sweetened beverages from School Physical Activity and Nutrition [SPAN] survey¹⁹; “past 7 days” PA from the National Health and Nutrition Examination Survey [NHANES]; and select items from the Pediatric Sleep Questionnaire²⁰), body weight (measured by trained examiners), and quality of life (Pediatric Quality of Life Inventory²¹). The longitudinal data collected at two of the three projects will include only children at or above the 85th BMI age- and sex-specific percentile, which will limit the cross-site impact evaluation to the effects among overweight and obese children. In addition, given that the interventions and settings differ for preschool (ages 2–5 years) and school-age (ages 5–12 years) children, the analyses will be conducted separately for each age group. Mixed-models analyses of covariance and other methods will be used to estimate the 12-month change in outcomes for children and families with the respective baseline measure as a covariate and random effects to account for clustering within projects and settings (e.g., children attending the same schools). Such models can be estimated by maximum likelihood methods when only a few time observations are available on the subjects.^22,23 Coefficients in the models will quantify how the interventions are associated with 12-month changes in the measured behavioral and health outcomes.

Various combinations of the intervention components (PH, PC, and PC Plus) are implemented within and across the eight CORD communities. For the purposes of the cross-site evaluation, the unit of intervention assignment will be defined by the combinations of community and designed intervention groups, as represented by check marks in Table 1. Because the communities and catchment areas were selected by the demonstration project investigators using population-level matching on a wide range of demographic and socioeconomic variables, the initial design groups are adequately sized, matched, and balanced to facilitate within-project comparisons.^2,3,9 Consequently, including categorical variables representing the units of assignment in the analytical model will provide a method for incorporating the heterogeneity among projects and communities into the analysis of the pooled data. This statistical approach is consistent with existing recommendations for pooled analysis and meta-analysis of individual-level data from different studies.^{11–13,24,25} Including the intervention components and units of assignment as factors in the statistical model will allow for estimating effect sizes of the respective intervention combinations relative to one another while accounting for heterogeneity among the communities. Failing to account for the heterogeneity among the projects and communities would produce inflated type I and inferential errors.^24,26 Likelihood ratio tests will be applied to assess the appropriateness of pooling the data.²²

The target enrollment of the longitudinal cohorts is over 2000 families, with an expectation of at least 20 observations per intervention cell in Table 1. Consequently, given that each row representing a particular intervention contains multiple communities and each project was powered to detect small-to-medium differences among intervention arms, there should be adequate sample size to include a number of select covariates in the model and evaluate the effect sizes with reasonable precision.

As an example, the first row in Table 1 represents the implementation of the full CORD model with integrated PH and PC Plus interventions. The average outcome across the respective groups in this row may be compared to the average outcome of one of the other rows or combinations of rows to answer important questions about the impact of the CORD project in these communities. The CORD model posits that the fully integrated interventions in row 1 (PH and PC Plus) should result in greater improvements in behaviors and weight over time. Thus, comparing row 1 to the average observed in rows 3 through 5 (PH only, PC Plus only, and PC only) will quantify the difference in response between the full CORD model and responses in groups exposed to only one of the intervention components.

Selection bias and clustering are key issues with nonrandom assignment to intervention conditions, as is the case for two of the three CORD projects. The CORD communities and respective families differ considerably on a number of important factors that may be related to their response to the various CORD interventions. Table 3 shows the measures for families^27–30 and Table 4 shows the measures for children^{27,29,31–34} that will be used to assess the multidimensional differences in the CORD samples among projects. Similarly, variability in program implementation and existing practices in the 90 child care centers and 103 schools participating across all three CORD projects will likely be associated with the measured outcomes, and the enrolled children will be clustered within these facilities. Thus, random effects representing the child care centers and schools will be included in the analytical model. Tables 5 and 6 show the facility-level covariates for child care centers^35,36 and schools,^37–40 respectively, that will be used to describe and assess heterogeneity in these settings. Finally, measures of policy, systems, practices, and process differences across the 17 participating clinics, as shown in Table 7,⁴¹ and eight communities, as shown in Table 8,⁴² will provide further contextual information for interpreting results.⁴³

Sustainability Evaluation

Sustainability determines whether proposed project activities can be sustained by the community beyond the end of the funding period.⁴⁴ Evaluating sustainability can help to determine how programs based on the CORD model might be maintained, enhanced, or changed over time. Sustainable programs demonstrate replicability and ongoing continuation of benefits through institutionalization of efforts supported by community capacity.^45,46 Example measures and data sources are presented in Table 9.

Replicability indicates the likelihood that a program can be repeated successfully and is defined, in part, by the complexity and adaptability of program elements that are necessary to reproduce the program. As suggested by the examples in Table 9, each demonstration project includes numerous components that will be counted and categorized using the Program Components Checklist. The Program Components Checklist describes which aspects of interventions were planned and implemented with fidelity in each project. Comparisons among projects will determine relative levels of complexity (i.e., the number of intervention aspects implemented), and by extension, the degree of adaptability of each program. Information on complexity will be triangulated with information gleaned from interviews of participating key informants to measure the extent to which new and existing activities, resources, and materials have been incorporated into the programs.⁴⁴ Key informants include leaders among local community partners, healthcare clinics, demonstration project staff, and school administrators who have been involved in delivering and supporting the multicomponent programs. At least one individual from each setting in each community (3–5 per CORD community) will provide information through interviews as sites approach the end of project implementation.

The potential for continuation of benefits is the extent to which initial program effects reach beyond the scope of the project and continue to have impact after the project funding period concludes. The potential for continuation of benefits will be operationalized by the following factors that suggest ease and reach of implementation: provider and organization attrition; adherence to CORD activities; reach beyond the original intended sample; and long-term population behavioral and health outcomes associated with the presence of the program in a community. For example, as demonstrated in Table 9, interviews from principal investigators at each project will be used to assess extension of program components beyond the original sample. Publicly available geographically linked data (e.g., from the CDC's Behavioral Risk Factor Surveillance System and the USDA's Food Access Research Atlas) on estimates of community health indicators (e.g., adult obesity rates) and obesity-related health behaviors (e.g., PA and dietary habits) will be accessed to provide insight on the potential reach and impact beyond the project communities and samples. Although it will be difficult to evaluate the direct effects of the program on these macro-level indicators, stronger population-level improvements in health may indicate a higher chance of program sustainability.⁴⁷

Institutionalization is defined by the degree of integration of a program into existing organizational policies and practices that might include changes in organizations, policies, and practices that support or enforce adoption of CORD program activities or systems.⁴⁷ Programs that become well integrated, with high levels of support and enforcement, are more likely to succeed and be sustained. For example, as presented in Table 9, the level of support for implementation from participating organizations can be gleaned from in-depth interviews of key informants at each project. These interviews may also indicate the degree of diffusion of program functions and activities or policy changes that may have occurred, such as modifications to an institution's systems (e.g., electronic health record [EHR] changes), continuing education efforts, and changes in institutional policies. Another example of institutionalization is the cross-institutional partnerships or collaborations that will be measured using the Wilder Collaboration Factors Inventory.⁴⁸ This measure is to be completed by members of the CORD community coalitions and measures the degree of institutionalization of the program in the community as evidenced by partnerships and collaborations resulting from the program.

Community capacity is the potential for the community to support CORD projects and activities through availability and involvement of organizations and structures. Strong community capacity can be operationalized by supportive polices and standard practices, strategic planning, funding priorities consistent with program components and goals, and stability of these elements through systems, organizations, and policies.⁴⁴ We developed the Policy Indicator Checklist (PIC) to measure the presence and intensity of policies in communities, schools, and child care sectors that support PA and nutrition, and discourage consumption of sugar-sweetened beverages and calorie-dense foods.⁴⁹ The PIC will be completed by the EC-CORD team and involves the auditing of public records of local and state policies to evaluate the need for establishment, strengthening, or maintenance of strategies to prevent obesity and promote health among partners and communities. Geographically linked and publicly available data from the US Census Bureau (ethnic density, neighborhood household income, educational attainment, and age characteristics), community websites and geographic information systems (GIS) departments (availability of parks and bicycle trails), the USDA Economic Research Service (access to supermarkets), and Esri ArcGIS Business Analyst (density of fast food restaurants) will provide information about the existence and stability of their goods, services, and resources. These contextual data will support in-depth comparisons of intervention sustainability across communities, based on all other qualitative and quantitative measures (Table 9).

Cost Analyses

Detailed information on the costs for training providers, delivering interventions in settings, and supply of equipment and other resources will be collected in a consistent manner by all projects. EC-CORD will compile and code these data for each activity using standard cost categories (labor, purchases, and in-kind). EC-CORD will facilitate retrieval of information at the three projects by abstracting financial reports, budget sheets, and ledgers from each project to create a standard format for the cost data. We will gauge the level of resolution that may be present in the cost data for each CORD project. For example, the projects were asked to track data on total costs per category, expenditures on the specific project activities (training staff, educating families, and providing equipment and supplies) and costs for categories and activities within each setting (totals by person/service/line item) will be used to assess the costs associated with replication, scalability, and sustainability of programs. The costs will be assessed longitudinally; costs at the start of an intervention are typically higher than those for maintaining activities in interventions and can change as programs are scaled up.

Furthermore, the benefits of CORD projects to the participating children will be assessed using comprehensive longitudinal models that take into account heterogeneity among populations.^22,23 The expenditures on different components of interventions are likely to affect outcomes such as reductions in the speed of weight gain among children and improvements in their quality-of-life indicators. In addition, parents of children, especially in the intervention groups, will acquire useful information by participating in their children's activities. For example, if overweight parents of children in the intervention group lose weight, then we can use the results in the literature to argue that “positive externalities” (e.g., spinoffs or “ripple effects”) owing to childhood obesity interventions are likely to reduce chronic disease burden among adults. This will be a direct and measurable public health impact. In addition, we will investigate the interventions that will have maximum benefits for specific communities. For example, if most parents are overweight and have low education levels in CA-CORD, then programs aimed at the family unit, including both parents and their young children, may be most effective. Such issues will be investigated using models estimated in the longitudinal data analyses.

The final part of the cost analyses will assess the benefits that can be gained per dollar invested in each setting and across the communities. We will investigate the extent to which interventions can be regarded as being cost-effective in terms of the costs of attaining changes in the child activity and diet behaviors, weight, and quality-of-life measures. Community characteristics, such as levels of parental education and incomes, will be taken into account in assessing cost-effectiveness of interventions. Moreover, there are methodological issues arising in the definitions of cost-effectiveness that need to be addressed. For example, assessment of whether an intervention is cost-effective depends on there being statistically significant effects of the intervention on child activity and diet behaviors, weight, and quality-of-life measures. Only when there are significant benefits of interventions can one address the cost-effectiveness aspects. The estimated parameters from comprehensive dynamic random-effects models using the longitudinal data from CORD projects will shed light on the cost-effectiveness issues. In particular, we will appraise the relative magnitudes of estimated short- and long-run effects of variables on the outcomes and compute confidence intervals for the benefits accrued from the interventions.²³ By taking into account the various effects of interventions on child and family well-being and the costs of intervention components, the research will provide useful insights into cost-effectiveness of interventions for preventing childhood obesity. Such findings will also be useful for assessing the feasibility and scalability of interventions for preventing childhood obesity in future programs.

Discussion

Planning and conducting a comprehensive evaluation of three independently designed projects implemented in eight different communities presents substantial challenges, including heterogeneity in populations, interventions, geographic locations, sociopolitical environments, measurement methods, and timelines.⁸ This article describes the evaluation plan for analyzing the data from the three independently designed and implemented CORD projects. A unifying framework identifies commonalities among the three projects to inform and structure each component of the evaluation. The resulting plan includes evaluation of process, impact, and sustainability. In addition, the research will develop a framework for assessing the short- and long-term costs and benefits for these childhood obesity prevention interventions. This plan describes our general approach to evaluating the pooled data from all three projects. The plan outlines the objectives and methods for each component of the evaluation while allowing flexibility to adjust, as needed, to issues that may arise once the full data are reported. The results of the evaluation will be used to determine whether multilevel, multisector programs integrating public health and primary care interventions are appropriate for national implementation in diverse communities.

Conclusions

EC-CORD will analyze the pooled data from three independent multilevel, multisector projects designed to induce environmental, systems, and policy changes that impact childhood obesity. The key to pooling and analyzing data from multiple projects is the CORD model framework used by all projects. The overarching logic model allows for categorizing the independent project components in comparable terms, common measures for the behavioral and health outcomes and important covariates, and a standardized approach to collecting and managing large amounts of data. The CORD project's cross-site evaluation approach includes plans for comprehensive assessments of processes, impact, sustainability, and costs.

Acknowledgments

This research was supported, in part, by cooperative agreement RFA-DP-11-007 (grant U18DP003350) from the CDC. The content is solely the responsibility of the authors and does not necessarily represent the official views of the CDC.

Author Disclosure Statement

No competing financial interests exist.