U.S. Department of Education Evaluation Policy (PDF)

ED Evaluation Policy

August 12, 2020

Page 1 of 9

U.S. Department of Education Evaluation Policy

Introduction

The Foundations for Evidence-Based Policymaking Act of 2018 (the Evidence Act) and

subsequent guidance from the Office of Management and Budget (OMB) require

agencies to establish and implement an agency-wide policy governing the design and

conduct of evaluations.

1,2

At the Department of Education (ED), evaluations include, but

are not limited to, efforts that are designed to rigorously test the efficacy or effectiveness

of education policies, programs, or practices (impact evaluations) and those that describe

how those policies, programs, or practices operate in classrooms, schools, and

communities (implementation evaluations).

Consistent with the Evidence Act and OMB guidance, this policy outlines the key

principles that guide ED’s planning and conduct of its own evaluations. These principles

should inform, but do not govern, evaluations undertaken by ED grantees of their own

projects.

ED undertakes evaluations so that it can better meet its mission of promoting student

achievement and preparation for global competitiveness by fostering educational

excellence and ensuring equal access. Evaluation findings build knowledge critical to the

development of education as a field and serve as an important source of information for

ED’s own organizational learning and improvement. Taken together, these activities

contribute to a larger goal of improved outcomes for all learners.

Program evaluations are only one source of evidence used by ED to support decision-

making. Thoughtful and rigorous policy analysis, statistical data collections, foundational

fact finding, and programmatic oversight activities such as performance measurement

and monitoring can also yield high-quality evidence that may be used to promote

mission-strategic and agency-operational improvement.

Evidence-building and its use at ED is a shared responsibility. Every ED principal

operating component (POC) is expected to use the best-available evidence in the conduct

See 5 USC § 312.

See OMB Memorandum M-20-12, available at https://www.whitehouse.gov/wp-

content/uploads/2020/03/M-20-12.pdf

ED Evaluation Policy

August 12, 2020

Page 2 of 9

of its work. Each POC should consider how its programs, policies, and practices—

including those supported through grants and contracts—could be structured to build

evidence to strengthen those activities in the future. However, no office is expected to

undertake evidence-building and use alone. ED leverages both POC-specific expertise as

well as agency-wide collaborations to develop and support the application and use of

evidence in implementing its mission of educational excellence and equity.

The Institute of Education Sciences’ (IES’) National Center for Education Evaluation and

Regional Assistance (NCEE) contributes to these efforts through the design and

execution of rigorous program evaluations and the dissemination of evaluation findings.

Congress established IES and NCEE as part of the Education Sciences Reform Act of

2002 (ESRA; Pub. L. 107-279), directing NCEE to “conduct evaluations of Federal

education programs administered by the Secretary … to determine the impact of such

programs.”

When NCEE is not directly sponsoring an evaluation, it fulfills its statutory

mission by examining “evaluations conducted or supported by others in order to

determine the quality and relevance of the evidence of effectiveness generated by those

evaluations.”

Through ESRA, Congress set high standards for the independence,

relevance, quality, transparency, and ethics of IES evaluation activities—standards that

are mirrored in OMB’s guidance on program evaluation standards and practices.

This policy affirms ED’s commitment to the foundational principles identified in OMB

guidance on program evaluation: independence and objectivity, relevance and utility,

rigor and quality, transparency, and ethics. The policy also describes how those principles

are implemented in NCEE evaluation activities.

Independence and Objectivity

If stakeholders are to use evaluations to inform their decision-making, they must view

evaluation findings as trustworthy. IES’s autonomy confers significant benefit to the

Department, helping to earn and maintain public trust in the credibility of ED evaluation

See 20 USC § 9561(b)(2).

See 20 USC § 9563(a)(1)(E).

See OMB Memorandum M-20-12, available at https://www.whitehouse.gov/wp-

content/uploads/2020/03/M-20-12.pdf

ED Evaluation Policy

August 12, 2020

Page 3 of 9

findings.

NCEE makes full and routine use of the autonomy granted IES and takes added steps to

protect independence and objectivity in the design, conduct, analysis, and reporting of its

evaluations by:

• Competitively awarding evaluation contracts to experts external to ED who are

free from conflicts of interest, financial or otherwise.

• Consistent with ESRA, conditioning the release of evaluation reports only on

approval from the IES Standards and Review Office (SRO), rather than the

approval of the Secretary or any other ED office.

ESRA requires that IES

evaluation reports are subjected to rigorous peer review before being released to

the public.

SRO, which is independent of NCEE, oversees the scientific peer-

review process. It seeks to ensure that IES reports are not only of high scientific

merit

but also objective, secular, neutral, and non-ideological, as well as free

of partisan political influence and racial, cultural, gender, or regional bias.

• Releasing all reports to the public as soon as practical once approved by the

SRO, irrespective of other ED or Executive Branch activities.

Relevance and Utility

In order to be useful, evaluation findings must be relevant to the interests and needs of

ED and program leadership and staff; agency partners such as States, territories, tribes,

and grantees; Federal, State, and school district policymakers; legislators and regulators;

school boards and community organizations; parents and families; and other interested

audiences.

While remaining fully committed to the autonomy granted to it by ESRA, NCEE seeks to

ensure the relevance of ED evaluations by:

• Building strong partnerships with policy and program leaders to: create

See 20 USC § 9576(a)

See 20 USC § 9576(c)

See 20 USC § 9563

See 20 USC § 9514

ED Evaluation Policy

August 12, 2020

Page 4 of 9

opportunities to identify evaluation topics and questions of importance; provide

periodic progress updates while the evaluation is underway and opportunities to

participate in Technical Working Groups (TWGs); share insights and interim

findings from evolving research as it is undertaken, subject to the

considerations below; inform policy and program leaders about final findings

once the evaluation is completed; and understand effective mechanisms for

sharing findings with interested stakeholders outside of ED.

• Ensuring that evaluation activities are prioritized in a manner consistent with

the ED’s Learning Agenda and considering: congressional requirements; the

President’s and Secretary’s budget and policy priorities; State, district, and

local needs as identified by ED staff; the evaluability of a program or overall

feasibility of the evaluation effort; and how recently a program’s last evaluation

occurred.

• Reviewing the literature, consulting with experts, and seeking the input of key

education stakeholders so that evaluations can generate relevant evidence for

audiences both inside and outside of the Department.

• Ensuring that all evaluations are conducted in a timely manner and findings are

reported promptly, without sacrificing quality.

• Ensuring that the products developed as part of an evaluation activity are

responsive to the information needs and communication preferences of key

stakeholders.

Considerations for Sharing Interim Findings from Evaluation Studies with ED Staff

Consistent with its goal to ensure evaluation findings are both relevant and timely, NCEE

will share selected interim findings from evaluation activities when doing so could

inform specific deliberations (e.g., grant competition design, budget policy, regulatory

development, legislative proposals). To ensure standards of quality and rigor (below) are

upheld, NCEE will only share interim findings based on completed data collections and

analyses that have been subject to internal quality assurance procedures.

To maximize efficiency, NCEE will share such interim findings during regularly

scheduled program briefings with policy and program leaders and may also be limited by

ED Evaluation Policy

August 12, 2020

Page 5 of 9

available staff and contractor resources. Interim findings will also be shared at regularly

scheduled TWGs, to which program leaders are typically invited. All interim findings

must be accompanied by a statement from NCEE indicating their scientifically

appropriate and inappropriate uses, including the risk that findings may materially change

prior to publication.

ESRA requires peer-review of evaluation studies and Secretarial notification before

findings are shared with the public. ED staff may not redistribute any interim findings

received from NCEE for any purpose.

Rigor and Quality

NCEE adheres to the highest possible standards of rigor and quality for conducting

evaluations so that stakeholders can rely upon their findings. Because quality standards

change over time as evaluation science and statistical practice evolves, IES promulgates

guidelines and other standard operating procedures to ensure NCEE’s evaluation work

consistently reflects best practice among Federal evaluators.

ED expects all impact evaluations will be designed to meet What Works Clearinghouse™

standards, either with or without reservations. NCEE may also require evaluations to

meet additional ED guidelines related to information quality, such as those required by

the Information Quality Act.

The standards below apply to all types of evaluation. They are applicable to all stages of

an evaluation, including design, data collection, data analysis, and reporting of results.

• When an evaluation seeks to answer questions about the effectiveness of a

policy, program, or practice, NCEE strongly prefers experimental designs.

The design of both impact and implementation evaluations and planned

analyses must be aligned with the research questions of interest to yield

credible findings.

• NCEE recruits and maintains an evaluation workforce with training and

See the Treasury and General Government Appropriations Act of 2001 (Pub. L. 106-554, § 515(a)), as

codified at 44 USC § §3504 (d) and 3516.

See 20 USC § 9563(a)(2)(A)

ED Evaluation Policy

August 12, 2020

Page 6 of 9

experience appropriate for planning and overseeing a portfolio of high-quality

evaluations. NCEE staff have relevant advanced degrees and experience in a

range of relevant disciplines and content areas, and NCEE leadership assigns

staff to evaluations that best fit their areas of expertise. NCEE provides

professional development opportunities so that staff are current with

methodological advances and research in their content areas.

• NCEE ensures that contractors who conduct evaluations have the expertise

needed to perform high-quality work. As part of the procurement process,

NCEE first specifies requirements for contractor expertise designed to ensure a

rigorous evaluation and then assesses the technical merit of proposals to ensure

that well-qualified contractors are selected. NCEE staff closely oversee the

entirety of the evaluation and supply technical direction to contractors on a

frequent basis.

• NCEE convenes a TWG for each evaluation it conducts. Members include

leading experts from both inside and outside government with relevant subject-

matter, methodological, and practice expertise. TWGs supply periodic guidance

to NCEE staff and the contractor during the design, analysis, and reporting

phases of the evaluation.

• As required by ESRA, evaluation reports are subjected to a rigorous peer-

review process before being published or otherwise released to the public. The

peer-review process ensures that all reports are thoroughly vetted by scientific

experts for scientific merit before publication. Procedures for peer review of

reports are approved by IES’s board, the National Board for Education

Sciences.

Considerations for Evidence-Building Conducted by ED Program Participants

When possible, ED will require grantees (including through cooperative agreements) and

contractors to conduct or commission rigorous evaluations of their activities and report

their findings to ED and the public. A recent example is the Office of Elementary and

Secondary Education’s Education Innovation and Research (EIR) program and the

development of “exit evidence” at the conclusion of the grant period.

Where appropriate, ED expects these evaluations to also meet What Works

ED Evaluation Policy

August 12, 2020

Page 7 of 9

Clearinghouse™ standards, with or without reservations. Inclusion in the Clearinghouse

helps ensure that what is learned in an evaluation can benefit the field. However, program

offices must consider their capacity to provide or support grantee technical assistance

before requiring evaluation activities.

When the capacity to support rigorous evaluation does not exist, program offices should

consider requiring grantees to conduct other evidence-building activities. These could

include, but are not limited to, participation in rigorous performance monitoring and

improvement activities or the collection of institution-level or student-level data to

demonstrate the association between program participation and important program

outcomes.

Transparency

ED promotes transparency of its evaluation work throughout a project’s lifecycle.

First, consistent with the Paperwork Reduction Act, NCEE files Information Collection

Requests with OMB and publishes the same in the Federal Register. These filings include

information about an evaluation’s purpose and benefits to stakeholders, as well as key

statistical and operational details concerning how ED will conduct the evaluation.

Second, NCEE pre-registers analysis plans for all impact evaluations. Examples of study

registries include the Registry of Efficacy and Effectiveness Studies, maintained by the

University of Michigan’s Inter-university Consortium for Political and Social Research.

Finally, NCEE releases all evaluation reports as soon as practical to ensure that

government leaders and the public have access to completed evaluations. Each report

includes a full accounting of data collection and analysis activities. NCEE posts final

evaluation reports to the IES website and deposits them in the Education Resources

Information Center, IES’s online database of education research, to ensure that they are

permanently available to the public.

NCEE also posts brief profiles of ongoing and recently completed evaluations on its

website. The profiles provide pertinent information about each evaluation, such as its

cost, anticipated timeline, background and purpose, evaluation questions, methods, data

collection, and key findings. ED includes similar information in its Annual Performance

Report. Findings from NCEE evaluations are also regularly shared via social media,

ED Evaluation Policy

August 12, 2020

Page 8 of 9

presented at conferences, and otherwise made available to relevant stakeholders.

Consistent with ESRA’s requirement for data availability,

NCEE makes the data from

its evaluations available to qualified individuals for the purpose of reproducibility and

secondary analysis via the IES Restricted Use Data Licensing Program, maintained by

the National Center for Education Statistics. NCEE adheres to all applicable laws that

protect the confidentiality and privacy of education data, including the requirements

explicitly set forth in ESRA.

ED is also transparent about this evaluation policy and its efforts to collect and use high-

quality data and evidence of all kinds. This and other relevant policies, including ED’s

Information Quality Act Guidelines, are made available on ED’s websites. As part of the

assessment required by the Evidence Act, ED staff regularly review their consistent

adherence to ED’s evaluation policy and other guidelines on evidence-building and use.

Ethics

ED and contractor staff who oversee scientific activities have a responsibility to behave

in an ethical manner and to safeguard the dignity, rights, safety, and privacy of study

participants. These responsibilities are outlined in ED’s Scientific Integrity Policy, to

which all ED staff are expected to adhere.

An important component of those safeguards

is the development and use of high-quality data management practices that ensure the

collection, processing, storage, and dissemination of data and data products are

appropriate. NCEE routinely collaborates with colleagues across ED, including those

with responsibilities for information security, student privacy, and data governance, to

design work that complies with all applicable law and policy.

ESRA places special emphasis on protecting the confidentiality of individually

identifiable information about students, families, and schools (Sec. 182-183). Other

Federal laws, such as the Family Educational Rights and Privacy Act and the Individuals

with Disabilities Education Act, similarly protect the privacy of student education

See 20 USC § 9574.

See 20 USC § 9573.

For more information on the Department’s policy, see https://ies.ed.gov/aboutus/scientific_integrity.asp.

ED Evaluation Policy

August 12, 2020

Page 9 of 9

records.

IES evaluations comply with these laws, as well as other relevant

requirements, such as regulations governing the Protection of Human Subjects (34 C.F.R.

Part 97) and the Protection of Pupil Rights Amendment (PPRA; 34 C.F.R. Part 98).

For more information on FERPA, see https://www2.ed.gov/ferpa.