Principles for
Effective Use of
Systems Thinking
in Evaluation
Systems in
Evaluation TIG
A Topical Interest Group of the
American Evaluation Association
Revised September 9, 2018
2 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
This publication was developed by the Systems in Evaluation TIG, a topical interest group of the
American Evaluation Association. Its contents are the sole responsibility of the Systems in Evaluation TIG
and do not necessarily represent the official views of the American Evaluation Association.
© 2018 Systems in Evaluation TIG
Suggested Citation:
Systems in Evaluation Topical Interest Group of the American Evaluation Association (2018). Principles
for Effective Use of Systems Thinking in Evaluation Practice.
3 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
TABLE OF CONTENTS
Foreword ............................................................................................................................ 4
How to Read this Document ........................................................................................... 5
Preamble ............................................................................................................................ 6
Summary of Principles for Effective Use of Systems Thinking in Evaluation ............ 10
Systems-in-Evaluation Principle ..................................................................................... 12
Interrelationships Principle ............................................................................................. 14
Perspectives Principle ..................................................................................................... 17
Boundaries Principle ....................................................................................................... 19
Dynamics Principle .......................................................................................................... 22
Future Directions .............................................................................................................. 25
Appendix 1: The GUIDE Framework ............................................................................. 26
4 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
FOREWORD
The Systems in Evaluation Topical Interest Group (SETIG) of the American Evaluation Association (AEA)
has developed a set of principles to guide the use of four core systems concepts in the design and
implementation of an evaluation. The primary purpose of these principles is to support evaluators and
evaluation stakeholders in the use of systems concepts in evaluation.
These principles are the result of a multi-year dialogue among SETIG members about the practice of
systems-informed evaluation. SETIG members are committed to cultivating this dialogue and deepening
our understanding through study, practice, and reflection. The SETIG may refine the current version of
the principles to reflect new knowledge and experience. The principles will play an important role in
core SETIG activities such as communications, education, and conference proposal review as
appropriate. For example, the principles can be used to explain concepts central to systems in
evaluation, and to provide a starting point for those new to systems who want to learn more about how
to use them in evaluation.
The principles were developed through a collaborative process led by the SETIG Leadership Team and
the Principles Project Coordinating Team (Meg Hargreaves, Jan Noga, and Emily Gates) that engaged the
entire TIG membership to provide input and a dedicated team of volunteers to create this document.
Many thanks to the team of volunteers who helped produce these principles (in alphabetical order):
Heather Britt Laurie Neighbors
Brandon Coffee-Borden Jan Noga
Kimberly Edmunds Kimberly Norris
Cheryl Endres Beverly Parsons
Ginger Fitzhugh Michael Quinn Patton
Jeneen Reyes García Robert Petrulis
Emily Gates Maggie Schuppert
Meg Hargreaves Bob Williams
Pat Jessup Edward Wilson
Marah Moore Pablo Vidueira
Nora F. Murphy
Thanks also to the SETIG Leadership Team for coordinating the process:
Heather Britt and Marah Moore, Co-Chairs
Brandon Coffee-Borden and Pablo Vidueira, Program Co-Chairs
5 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
HOW TO READ THIS DOCUMENT
The primary purpose of this document is to support evaluators and evaluation stakeholders in the use of
systems concepts in evaluation. The principles are the heart of this document. Each principle is further
explained in a section that outlines operating principles, guidance on what not to do, and helpful
references.
This document is developmental.
It represents the collective efforts of a specific group of people at a specific moment in time. The
foreword introduces the group, and the preamble describes the dialogue that inspired this document,
the history that informed it, and ideas for future development of the principles. Read this document and
join the conversation.
This document is practical.
The principles provide guidance for evaluators and evaluation stakeholders for designing and
implementing evaluations with a systems lens. The principles are not a recipe and this document is not a
methodological brief. Read this document to enrich your practice.
This document is grounded in the literature and intended to inspire further study.
It embraces the breadth and diversity of both systems and evaluation but does not aspire to a
comprehensive overview of either field or their multiple intersections. This document is not a treatise or
a journal article. Read this document to learn more about the exciting ongoing exchanges between
evaluation and systems.
6 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
PREAMBLE
The primary purpose of these principles is to support evaluators and evaluation stakeholders in the use
of systems concepts in evaluation. This preamble provides some high-level definitions and general
guidance to inform the use of these principles in evaluation.
The SETIG has elected to use systems concepts to develop principles of systems-informed evaluation.
However, it recognizes that other paths may also be used for that purpose, including: identifying the
core activities of designing and conducting an evaluation and then identifying principles for taking a
systems approach to each activity; identifying a particular systems approach or theory (e.g., system
dynamics, agent-based modeling, human system dynamics, complexity theory, critical systems
heuristics) (see Capra & Luisi, 2014; Williams & Hummelbrunner, 2011; and Reynolds & Holwell, 2010
for systems approaches) and developing principles for how it informs evaluation; and studying empirical
examples of systems-informed evaluation to identify principles.
Systems concepts are those that have come to define the systems field, an expansive and cross-
disciplinary area of inquiry including numerous subfields such as cybernetics, complexity theory, and
systems science (Hieronymi, 2013). Since systems concepts are those that focus on conceptualizing
systems, it is worth defining what we mean by a system. There is no single, agreed-on definition of
system in the systems field. Looking across the field, the term “system” typically refers to a set of
interrelated elements that interact to achieve an inherent or ascribed purpose (Ackoff, 1971; Meadows,
2008). Systems can be conceived as ontological realities existing out-there in the world or as
epistemological constructs used to understand the world (Reynolds, 2008). Most situations involve
multiple and intertwined systems. For practical purposes, it may be helpful to consider one system at a
time. References to “the” system in this document do not imply that only one system is relevant to a
situation or to an evaluation.
Systems thinking, in the evaluation field, often refers to a way of thinking based on core systems
concepts. To date, three distinct orientations to systems thinking have informed SETIG discussions on
the use of systems concepts in evaluation. One orientation draws from historical review and identifies
interrelationships, perspectives, and boundaries as core concepts of focus present in much of system
theory (Williams and Imam, 2007). A second orientation draws from the field of cognitive science to
identify processes for thinking that focus on distinctions, relationships, perspectives, and boundaries
(Cabrera and Cabrera, 2015). The third orientation draws on human systems dynamics theory and
focuses on concepts of containers, differences, and exchanges (Eoyang, 2007). Common across the three
orientations is the use of specific systems concepts to think about a particular situation, system,
problem, intervention, or evaluation.
“Taking a systems approach” generally refers to using systems concepts or methods. Methods such as
system dynamics, social network analysis, soft systems methodology, and critical systems heuristics are
especially associated with the systems field, however, the application of systems concepts to evaluation
is not limited to these methods.
7 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Several considerations are important to keep in mind about the principles described in this document
for effective use of systems thinking in evaluation:
First, the principles were developed based on four core systems concepts -
interrelationships, perspectives, boundaries, and dynamics.
The interrelationships between elements of a system; perspectives from which a situation or system
can be understood; boundaries between the system and its environment; and the dynamics that
influence and are influenced by the behavior of the system or situation over time and at different
scales. With a history that extends over a century and origins in multiple disciplines including the
natural and social sciences, the systems field offers many systems concepts that can be applied to
evaluation theory and practice. The SETIG drew primarily on the three orientations to systems
thinking when identifying these four concepts as particularly relevant and useful for evaluation.
Second, the systems field is highly diverse and constantly evolving.
As a consequence, key concepts do not have agreed upon definitions (Reynolds & Holwell, 2010;
Ison, 2010). The principles presented here will inevitably be interpreted differently when
understood using different definitions of the core concepts.
Third, the principles are intended to be used in ways that acknowledge both
ontological and epistemological meanings of systems concepts.
Within the systems field and among evaluators using systems concepts, there can be a tension
between what can loosely be called epistemological versus ontological uses of systems concepts.
This tension came about within the historical developments of the systems field (see Midgley, 2007
for overview) as different systems traditions emphasized systems (and associated concepts) as out-
there realities in the world that could be observed and empirically examined versus social and
epistemological constructs for understanding the world.
Fourth, the principles are intended to work together in an iterative and
dynamic process.
While these four principles and the subsequent operating principles are discussed here separately,
they are intended to be used together in a mutually influential way. The use of one principle will in
turn affect the use of the other principles, which will in turn affect that initial principle. “Systemic
triangulation” is one inspiration for the interactive application of the principles. (Ulrich, 2017)
Fifth, principles are intended to apply throughout an evaluation and are not
limited to any single point in an evaluation process.
Conceptualizing, designing and conducting evaluations include a range of iterative activities
including, but not limited to, managing the evaluation; defining what is to be evaluated; determining
8 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
the key evaluation questions; framing the boundaries for an evaluation; describing activities,
outcomes, impacts, and context; understanding causes of outcomes and impacts; gathering and
managing data; synthesizing data from one or more evaluations; and reporting and supporting use
of findings (BetterEvaluation’s Rainbow Framework). The principles are intended to be used
throughout the entire evaluation process, and therefore should be used to think through and carry
out any one or more of these activities.
Equally importantly, the principles apply to monitoring. Monitoring and evaluation are closely
related. For example, evaluations often rely on monitoring data. Monitoring and evaluation often
work together in an integrated system to provide useful information over the life of an intervention.
Applying systems concepts in monitoring furthers the effective practice of system-informed
evaluation (Britt, 2013; Williams & Britt, 2014).
Finally, principles apply to both the intervention being evaluated and the
evaluation itself.
It will perhaps be most common for evaluators and stakeholders to use these principles to examine
the policy, program, or initiative being evaluated and the broader need, problem, or situation it
addresses. Additionally, they can be used to reflect on, discuss, and make changes to the evaluation
itself.
THE GUIDE FRAMEWORK
The principles were developed using the GUIDE Framework which outlines five criteria for a high-quality
principle: 1) guiding, 2) useful, 3) inspiring, 4) developmental, and 4) evaluable (Patton, p. 43). Please
see Appendix 1 for a description on how the Systems in Evaluation principles meet the five criteria.
REFERENCES
Ackoff, R. (1971). Towards a system of systems concepts. Management Science, 7 (11), pp. 661-671.
BetterEvaluation. Framework Overview. Retrieved from https://www.betterevaluation.org/plan
Britt, H. (2013). Complexity-Aware Monitoring. Washington, DC: United States Agency for International
Development. Retrieved from:
https://usaidlearninglab.org/sites/default/files/resource/files/201sad_complexity_aware_monitoring_di
scussion_note.pdf
Cabrera, D., & Cabrera, L. (2015). Systems Thinking Made Simple: New Hope for Solving Wicked
Problems. Odyssean Press.
Capra, F., & Luisi, P. (2014). The Systems View of Life: A Unifying Vision. Cambridge: Cambridge
University Press.
9 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Eoyang, G.H. (2007). Human systems dynamics: Complexity-based approach to a complex evaluation. In:
Williams, B and Imam, I (eds). Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA:
EdgePress of Inverness, 123–40.
Hieronymi, A. (2013). Understanding systems science: A visual and integrative approach. Systems
Research and Behavioral Science 30: 580–95.
Ison, R. (2010). How to Act in a Climate Change World. Milton Keynes, UK: Springer.
Meadows, D. H., & Wright, D. (2008). Thinking in systems: A primer. White River Junction, VT: Chelsea
Green Publishing.
Midgley, G. (2007). Systems thinking for evaluation. Systems Concepts in Evaluation: An Expert
Anthology. Point Reyes, CA: EdgePress, p. 11-34.
Patton, M. Q. (2018). Principles-Focused Evaluation: The GUIDE. New York: Guilford Press.
Reynolds, M. (2008). Response to paper 'Systems Thinking' by D. Cabrera et al.: Systems thinking from a
critical systems perspective. Journal of Evaluation and Program Planning, 31(3) pp. 323–325.
Reynolds, M. & Holwell, S. (2010). Systems Approaches to Managing Change: A Practical Guide. Milton
Keynes, United Kingdom: Springer.
Ulrich, W. (2017). Systemic triangulation. Retrieved from:
http://www.wulrich.com/downloads/bimonthly_march2017.pdf
Williams, B. and Britt, H. (2014). Systemic Thinking for Monitoring: Attending to Interrelationships,
Perspectives and Boundaries. Washington, DC: United States Agency for International Development.
Retrieved from: https://usaidlearninglab.org/library/attending-interrelationships-perspectives-and-
boundaries-complexity-aware-monitoring
Williams, B. and Hummelbrunner, R. (2011). Systems Concepts in Action: A Practitioner’s Toolkit.
Stanford, CA: Stanford University Press.
Williams, B. & Imam, I. (2007) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA:
EdgePress.
10 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
SUMMARY OF PRINCIPLES FOR EFFECTIVE USE OF
SYSTEMS THINKING IN EVALUATION
SYSTEMS-IN-EVALUATION PRINCIPLE
View the evaluation situation through the lens of systems thinking.
Operating Principle S1: Throughout the evaluation process, critically deliberate on and apply
the principles of interrelationships, perspectives, boundaries, and dynamics in integrated ways.
Operating Principle S2: Understand and describe the relationship between the various
concepts, methods, and tools which inform the evaluation.
Operating Principle S3: Ensure that key evaluation documents (plans, reports, messages)
identify and describe the systems thinking concepts, conceptual frameworks, methods, and tools
which inform the evaluation.
INTERRELATIONSHIPS PRINCIPLE
Critically deliberate on, work to examine, understand and to appropriately address interrelationships
regarding both the evaluand and the evaluation itself.
Operating Principle I1: Identify, capture, map, and track key interrelationships that influence,
could influence, and/or should influence the evaluand and the evaluation itself.
Operating Principle I2: Identify key interrelationships that result from, could result from,
and/or should result from the evaluand and the evaluation itself.
Operating Principle I3: Consider alternative interrelationships within and beyond the
boundaries of the system as currently defined, along with potential consequences of including them
in the evaluation.
Operating Principle I4: Make transparent and support interrelationships that positively
influence or are positive results from the evaluation while remaining open to revision.
PERSPECTIVES PRINCIPLE
Capture, critically deliberate on, work to understand, represent, and appropriately address diverse
perspectives.
Operating Principle P1: Identify and represent diverse perspectives and the values on which
they are based. Seek dissent as well as consensus.
11 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Operating Principle P2: Attend to the types of power associated with each perspective and
consider the consequences.
BOUNDARIES PRINCIPLE
Critically deliberate on, set, and explain the boundaries and boundary decisions that relate to the
situation being evaluated and the evaluation itself.
Operating Principle B1: Identify key boundaries that influence, and should influence, the
situation being evaluated and the evaluation itself.
Operating Principle B2: Deliberate on a range of critical boundary choices along with potential
consequences.
Operating Principle B3: Make transparent and justify the boundaries used in an intervention
and the evaluation while remaining open to revision.
DYNAMICS PRINCIPLE
Focus on the patterns of change that emerge within the system to understand their influence and
significance for the evaluation.
Operating Principle D1: Consider how dynamics related to time, location, anticipated and
unanticipated reactions, and current states and rates of change interact to create patterns that are
nonlinear and multi-directional to help understand how dynamics shape the systems relevant to the
evaluation.
Operating Principle D2: Design an evaluation plan that is responsive to emergent
developments; collects information about what, when, how, and why change occurs; and
incorporates learning as it is received to document and respond to dynamics in the evaluation.
Operating Principle D3: Investigate how observers’ worldviews and their judgements about
useful and convenient representations of system behavior influence the conceptualizations of
dynamics.
Operating Principle D4: Consider the interactions and influence of the evaluator and
evaluation with and within systems relevant to the evaluation to document the role of evaluation in
shaping system behavior.
12 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
SYSTEMS-IN-EVALUATION PRINCIPLE
View the evaluation situation through the lens of systems thinking.
OPERATING PRINCIPLES
GUIDANCE ON WHAT NOT TO DO
Do not use the principles (interrelationships, perspectives, boundaries and dynamics) in isolation from
one another.
Do not assume that all systems definitions, concepts, principles, methods, and tools are compatible and
consistent with one another.
Do not assume that the relationship between systems concepts, definitions, principles, methods, and
tools is self-evident. In reporting, do not simply list the systems concepts, definitions, principles,
methods, and tools which informed the evaluation.
Do not rely on methods and tools alone to inform an evaluation. There is general agreement among
those using systems thinking in evaluation that systems methods and tools alone (such as a system map
or frameworks) are not sufficient for informing an evaluation. Ideally, core systems concepts should be
integrated throughout the evaluation.
DISCUSSION
To apply the “overarching” systems-in-evaluation principle means to systemically apply the four
constituent principles – interrelationships, perspectives, boundaries and dynamics. The four constituent
Operating Principle 1:
Throughout the evaluation process, critically deliberate on and apply the principles of
interrelationships, perspectives, boundaries, and dynamics in integrated ways.
Operating Principle 2:
Understand and describe the relationship between the various concepts, methods, and tools which
inform the evaluation.
Operating Principle 3:
Ensure that key evaluation documents (plans, reports, messages) identify and describe the systems
thinking concepts, conceptual frameworks, methods, and tools which inform the evaluation.
13 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
principles inform one another in essential ways. No one principle is complete without the others.
Together, they form a system for guiding the systemic practice of evaluation.
These four core concepts manifest themselves in each situation in unique, interdependent, and
continually evolving ways. Because the systems thinking field is broad and diverse, concepts, methods,
and tools selected to inform an evaluation may align closely with each other, complement one another,
or exist in creative tension with one another. It is important to explain how the complementarity or
contradictions between these concepts influence the evaluation.
The systems-in-evaluation principles inspire evaluators to contribute to responsible and meaningful
change in people’s lives. “Seeing” the systems we live in equips us to act in deliberate and intentional
ways. Using a systems thinking lens, we become more aware of the consequences of our actions and
those of others, as well as the possibilities and limitations of actions within and among specific systems.
Consequently, we are better equipped to undertake evaluation as a co-creative process.
PROMPTING QUESTIONS: ARE WE USING THE PRINCIPLES IN AN
INTEGRATED WAY?
The following questions may be useful for considering the ways these principles work together.
• What new interrelationships become relevant when I alter the size or nature of the boundaries
of my system, modify my perspectives, or accommodate the perspectives of other stakeholders,
or observe new or altered dynamics within the situation?
• How are my perspectives influenced or transformed when I alter my boundary decisions and/or
observe new interrelationships and/or observe new or altered dynamics within the situation?
• What boundary choices are affected when I observe new interrelationships and/or
accommodate perspectives of different stakeholder groups and/or observe new or altered
dynamics within the situation?
• How is my assessment of the dynamics of a situation affected when I alter the size or nature of
the boundaries of my system and/or modify my perspectives or accommodate the perspectives
of other stakeholders and/or observe new interrelationships?
14 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
INTERRELATIONSHIPS PRINCIPLE
Critically deliberate on, work to examine, understand and to appropriately address interrelationships
regarding both the evaluand and the evaluation itself.
OPERATING PRINCIPLES
GUIDANCE ON WHAT NOT TO DO
• Do not underestimate the role or influence of interrelationships on the evaluand and in the
evaluation.
• Do not think that interrelationships are something new or additional or optional.
• Do not assume that interrelationships in an evaluation or the situation being evaluated are
given, universal, or static.
• Do not assume that there are “right” or “wrong” interrelationships.
Operating Principle I1:
Identify, capture, map, and track key interrelationships that influence, could influence, and/or should
influence the evaluand and the evaluation itself.
Operating Principle I2:
Identify key interrelationships that result from, could result from, and/or should result from the
evaluand and the evaluation itself.
Operating Principle I3:
Consider alternative interrelationships within and beyond the boundaries of the system as currently
defined, along with potential consequences of including them in the evaluation.
Operating Principle I4:
Make transparent and support interrelationships that positively influence or are positive results from
the evaluation while remaining open to revision.
15 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
DISCUSSION
Interrelationships constitute the earliest concept of interest to the systems field (Williams & Britt, 2014).
Interrelationships are ways in which each of two or more things relate or connect to each other and
produce an effect. They delineate the physical, temporal, political, social, cultural, ideological, technical,
and ethical linkages associated with an evaluand and its evaluation. Interrelationships are constructions
which depend on the perspective(s) taken and the information under consideration. Interrelationships
can be dynamic, evolving with an evaluand and its evaluation.
The purpose of the interrelationships principle is to generate awareness about the larger system in
which the evaluand and the evaluation exist. The interrelationships principle undergirds learning about
cause-and-effect relationships and the theories of change which inform the design and implementation
of social programming.
The interrelationship principle applies during all phases and in all aspects of the evaluation.
The principle underscores evaluation’s role in:
1. Capturing the key interrelations at work in a situation
2. Describing and measuring those interrelationships
3. Ensuring that the conceptual models of interrelationships match the changing dynamics of the
situation. Understanding interrelationships may be particularly helpful during the design phase
of a program or initiative to ensure that the inputs and activities are well aligned to intended
outputs and outcomes.
The interrelationships principle inspires evaluators to think critically about the ethics of their
engagement with an evaluand and its evaluation. Interrelationships, and the way they are represented,
have consequences for both the system and the evaluation
1
. Evaluators inevitably influence
interrelationships. For instance, every time evaluators use a criterion to judge an intervention, every
time evaluators choose ways of measuring, and every time evaluators choose intended uses for
intended users, they effect the interrelationships of a situation. Systemic evaluators take responsibility
for their influence and consider whether they should mitigate any potential or actual consequences.
Careful consideration and representation of interrelationships is essential to ethical evaluation practice
while also keeping in mind that there are no “right” answers.
1
Readers may wish to consult AEA’s statement on cultural competency as well as the growing body of work on
culturally responsive evaluation. American Evaluation Association (2011). Public Statement on Cultural
Competence in Evaluation. Washington DC: Author. (http://www.eval.org/p/cm/ld/fid=92)
16 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Application of the interrelationships principle can be evaluated by iteratively designing, implementing,
and testing theories of change. Evaluation can assess whether identified interrelationships were
thoroughly captured, deliberated on, understood, and addressed. Because interrelationships are
complex and dynamic, exhaustive accounting of them may not be possible within the scope of the
evaluation. Frequent reflection, dialogue, and revision of documentation about the interrelationships
will enhance the understanding about their roles vis-à-vis the evaluand and the evaluation.
REFERENCES
Williams, B. and Britt, H. (2014). Systemic Thinking for Monitoring: Attending to Interrelationships,
Perspectives and Boundaries. Washington, DC: United States Agency for International Development.
Retrieved from: https://usaidlearninglab.org/library/attending-interrelationships-perspectives-and-
boundaries-complexity-aware-monitoring
17 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
PERSPECTIVES PRINCIPLE
Capture, critically deliberate on, work to understand, represent, and appropriately address diverse
perspectives.
OPERATING PRINCIPLES
GUIDANCE ON WHAT NOT TO DO
• Do not ignore the implications of diverse perspectives for an evaluation.
• Do not assume that there is a single relevant perspective, or that your perspective is universal.
• Do not assume individuals have a single perspective.
• Do not assume that those that share characteristics or roles also share similar perspectives.
DISCUSSION
Perspectives began to enrich the systems discourse in the 1970s by humanizing the members of systems
and by opening the goals and objectives of systems to critique (Midgley, 2007). The consideration of
perspectives gave rise to participatory systems approaches which attempt to uncover and include
diverse perspectives and broaden those conducting analytical work beyond narrowly defined “experts.”
The perspectives concept prompted a shift in understanding systems as tangible entities to
understanding them as mental models.
An individual’s perspective encompasses how they see, understand, value, and are motivated to act in a
situation (Williams & Britt, 2014). An individual’s perspective is not an opinion; rather, it represents the
understanding which undergirds an opinion. Every perspective implies a specific relationship to the
system. Every perspective draws the boundaries of the system in a specific way.
Operating Principle P1:
Identify and represent diverse perspectives and the values on which they are based. Seek dissent as
well as consensus.
Operating Principle P2:
Attend to the types of power associated with each perspective and consider the consequences.¥
18 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
It is also possible that individuals may share a similar perspective on a situation. A framing represents an
understanding of the situation that is shared by two or more individuals.
The perspectives principle prescribes learning about and taking appropriate action to deal with diverse
perspectives throughout an evaluation. The purpose of the perspectives principle is to provide an
accurate description of how to interpret the evaluation by pointing to the specific perspectives it
represents and the perspectives that might have been left out or privileged. The quality of evaluation
practice may be enhanced by including multiple perspectives.
The perspectives principles values diverse perspectives – that is, the different ways that individuals see,
understand, value, and are motivated to act in a situation. Valuing perspectives is both an ethical and
pragmatic stance. It implies that evaluation should acknowledge the multiplicity of perspectives on a
situation and evaluand and that each perspective implies a specific relationship to the system. Ideally,
an evaluation should include multiple perspectives. Evaluations that include multiple perspectives may
represent more nuanced representations of complex challenges.
Application of the perspectives principle can be evaluated by conducting an analysis of stakeholder and
stakes in the evaluation and situation, and assessing whether these perspectives were captured,
critically deliberated on, represented, and appropriately addressed. This stakeholder analysis should
include individuals and groups who have different relationships to what is being evaluated, such as
intended beneficiaries, allies, opponents, those excluded, and affected and influential bystanders.
REFERENCES
Midgley, Gerald. (2007). Systems thinking for evaluation. Systems Concepts in Evaluation: An Expert
Anthology. Point Reyes, CA: EdgePress, p. 11-34.
Williams, B. and Britt, H. (2014). Systemic Thinking for Monitoring: Attending to Interrelationships,
Perspectives and Boundaries. Washington, DC: United States Agency for International Development.
Retrieved from: https://usaidlearninglab.org/library/attending-interrelationships-perspectives-and-
boundaries-complexity-aware-monitoring
19 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
BOUNDARIES PRINCIPLE
Critically deliberate on, set, and explain the boundaries and boundary decisions that relate to the
situation being evaluated and the evaluation itself.
OPERATING PRINCIPLES
GUIDANCE ON WHAT NOT TO DO
• Do not think that boundary choices are something new, additional, or optional.
• Do not assume that the boundaries used in an evaluation or the situation being evaluated are
given, universal, or static.
• Do not assume that there are “right” boundaries.
DISCUSSION
It is not possible to evaluate everything about an intervention or situation. Boundaries delineate what is
and/or should be ‘in’ and ‘out’ of an evaluation’s focus. Choices must be made about what aspects of
the intervention or situation ought to be evaluated. Choices must be made about how an evaluation
should assess an intervention or situation. Furthermore, choices must be made about the nature of the
evaluation activity itself. These boundary choices delineate the physical, temporal, political, social,
cultural, ideological, technical, and ethical spaces occupied by the evaluation and the situation being
evaluated. Boundaries are social and cognitive constructions; they depend on the perspective(s) taken;
and boundaries may change as more information and perspectives are considered.
Operating Principle B1:
Identify key boundaries that influence and should influence the situation being evaluated and the
evaluation itself.
Operating Principle B2:
Deliberate on a range of critical boundary choices along with potential consequences.
Operating Principle B3:
Make transparent and justify the boundaries used in an intervention and the evaluation while
remaining open to revision.
20 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Boundary choices are made throughout the evaluation process regardless of evaluation purpose, model,
intervention type (e.g., change initiative, policy, program) or point in the program cycle (e.g., needs
assessment, design, implementation, closeout). Examples of boundary choices in evaluation include, but
are not limited to: defining the evaluand and key terms; defining the scope and focus of the evaluation;
framing questions; determining the methodologies, methods, techniques, and measurement systems;
selecting analytic approaches; developing a sample frame; selecting members of an evaluation advisory
group; and drafting a dissemination plan. Boundary choices determine who ought to be involved in the
design, implementation, and results of an evaluation. They influence the use, usefulness, and
consequences of an evaluation.
Evaluators also make boundary decisions related to the other systems thinking concepts –
interrelationships (e.g., who is in and who is out?), perspectives (e.g., whose perspectives are
considered?), and dynamics (e.g., which dynamics are relevant to this evaluation?).
The purpose of the boundary principle is to help evaluators and others working with evaluators be
intentional, clear, and transparent about boundary choices involved in planning and conducting the
evaluation. The boundary principle prescribes identifying boundary decisions and identifying options,
choices made, and the implications of choices throughout the evaluation process. The principle values
transparency and responsibility for boundary choices and potential consequences. It implies that
evaluation should not be done unreflectively or covertly.
The boundary principle inspires evaluators to recognize the consequences of boundary choices and
make those choices in an ethical manner. Boundaries inevitably influence who or what is included,
excluded, and marginalized in a situation and in an evaluation. Applying this principle is a matter of
professional ethics, however, it is not possible to perfectly apply this principle. Evaluators who apply the
boundary principle do not aspire to a single, all-encompassing Truth with a capital “T.” Instead they
aspire to be transparent about which evidence and values are considered relevant to the evaluation
process and conclusions. The desired result is an evaluation that clearly identifies its strengths and
limitations and is carried out with thoughtful attention to and mitigation of potential negative
consequences.
Application of the boundary principle can be evaluated by reviewing evaluation products for
identification and explanation of boundary choices and evidence of consideration of alternatives,
rationale of choices made, and consideration of consequences. Evaluation criteria identify and delineate
boundaries. Evaluations are largely based on judgments made against these criteria. Since criteria or at
least the judgments ought to be explicit, their worth can be easily assessed.
REFERENCES
Ulrich, W., & Reynolds, M. (2010). Critical systems heuristics. In M. Reynolds & S. Holwell (Eds.), Systems
Approaches to Managing Change: A Practical Guide (pp. 243–292). Milton Keynes, England: Springer.
21 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Ulrich, W. (2002a). Boundary critique. In H. G. Daellenbach & R. L. Flood (eds.), The Informed Student
Guide to Management Science. London: Thomson Learning.
Ulrich, W. (2002b). Critical systems heuristics. In H. G. Daellenbach & R. L. Flood (eds.), The Informed
Student Guide to Management Science. (p. 72). London: Thomson Learning.
Williams, B. (2015). Prosaic or profound? The adoption of systems ideas by impact evaluation. IDS
Bulletin, 46, 7–16.
Dynamics Principle: Focus on the patterns of change that emerge within the system to understand their
influence and significance for the evaluation.
22 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
DYNAMICS PRINCIPLE
Focus on the patterns of change that emerge within the system to understand their influence and
significance for the evaluation.
GUIDANCE ON WHAT NOT TO DO
Do not ignore or fail to illuminate the effects of the dynamics operating within and around the system
by looking only at the evaluand and not the context surrounding it.
Do not fail to consider multiple types of dynamic complexity present in the systems relevant to the
evaluation, including the degree of change at different timescales, feedback loops, emergence,
nonlinearity, historical dependence, self-organization, coevolution, and adaptivity.
DISCUSSION
‘Dynamics’ refers to the emergent and changing interactions between and among the parts and agents
within system(s) (Eoyang and Holladay, 2013). Bringing attention to dynamics in our systems approach
focuses us first on how a phenomenon changes over time, identifying the patterns of change that may
Operating Principle D1:
Consider how dynamics related to time, location, anticipated and unanticipated reactions, and
current states and rates of change interact to create patterns that are nonlinear and multi-directional
to help understand how dynamics shape the systems relevant to the evaluation.
Operating Principle D2:
Design an evaluation plan that is responsive to emergent developments; collects information about
what, when, how, and why change occurs; and incorporates learning as it is received to document
and respond to dynamics in the evaluation.
Operating Principle D3:
Investigate how observers’ worldviews and their judgements about useful and convenient
representations of system behavior influence the conceptualizations of dynamics.
Operating Principle D4:
Consider the interactions and influence of the evaluator and evaluation with and within systems
relevant to the evaluation to document the role of evaluation in shaping system behavior.
23 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
be recurrent, constant, or in an ongoing state of evolution. Second, it means using the evaluation to
bring understanding to how these patterns are generated by the many, multidirectional interactions
among units (i.e., organizations, social groups, individuals, even individual behavior traits or
perspectives, etc.), the parts of the system, and the larger environment.
Paying attention to dynamics involves acknowledging the history of the patterns active in a system and
incorporating a longer-term perspective often absent in evaluations (Meadows 2008, p. 170). Paying
attention to dynamics also involves acknowledging that the presence of an evaluator influences the
dynamics. Our evaluation approach should be responsive to the complex ever-changing environment in
which we are operating. Complex systems dynamics often give rise to unpredictable, counter-intuitive
outcomes (Forrester, 1995). The dynamics principle encourages us to be in a constant state of inquiry, to
learn as we go, and to keep adapting our evaluation design in response (Forss, Marra, and Schwartz,
2011, p. 331).
The purpose of the dynamics principle is to provide evaluators with a lens through which to consider
and capture the interactions of context, stakeholder views, time, place, and other elements that may be
relevant in the evaluation. Paying attention to dynamics supports more holistic decision-making in
planning and implementing the evaluation and can provide better evaluation results. The dynamics
principle encourages evaluators to consider simple, complicated and complex patterns of change,
including the effect of context, stakeholder views, time, place, and other elements that interact and may
affect results during the course of the evaluation (Glouberman and Zimmerman, 2002). The dynamics
principle encourages evaluators to be responsive to change, including the way that we (as evaluators)
change and impact the evaluation and how the evaluation acts as an intervention within the broader
system. Evaluators should continuously consider and respond to the complex ever-changing
environment in which we operate.
Application of the dynamics principle can be evaluated through consultations with stakeholders to
ensure that the evaluation has captured the various types of dynamics at work in the systems relevant
to the evaluation. Evaluation of this principle should include dynamics across time, feedback loops,
emergence, nonlinearity, historical dependence, coevolution, self-organization, and adaptivity (Patton,
2011, pp. 150-151).
REFERENCES
Eoyang, G. H., & Holladay, R. J. (2013). Adaptive action: Leveraging Uncertainty in your Organization.
Palo Alto, CA: Stanford University Press
Forrester, Jay W. (1995). Counterintuitive Behavior of Social Systems. CiteSeerX 10.1.1.35.4776
Forss, K., Marra, M., and R. Schwartz. (2011). Evaluating the Complex: Attribution, Contribution, and
Beyond. London, UK: Transaction Publishers.
24 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
Glouberman, S. and B. Zimmerman (2002). Complicated and Complex Systems: What Would Successful
Reform of Medicare Look Like? Ottawa: Commission on the Future of Health Care in Canada.
Meadows, D. (2008). Thinking in Systems: A Primer. Hartford, VT: Chelsea Green Publishing.
Patton, M. Q. (2011). Systems thinking and complexity concepts. In Developmental Evaluation: Applying
Complexity Concepts to Enhance Innovation and Use. New York: Guildford Press.
25 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
FUTURE DIRECTIONS
During the process of developing these principles, SETIG members suggested several areas for future
exploration. A few are noted below.
TEST THE PRINCIPLES
The principles are meant to be applied. The SETIG encourages its members to use the principles for
planning and guiding evaluations and assessing evaluation products, such as reports. The SETIG
encourages members to reflect on their experience of using the principles and contribute observations,
questions, lessons learned, and new areas for exploration to the ongoing dialogue on strengthening
systems informed evaluation.
EXCAVATE ASSUMPTIONS UNDERLYING THE PRINCIPLES
The principles unavoidably rest on the authors’ assumptions about the nature of social programming,
how knowledge is constructed and used, and the practice of evaluation (Shadish et al, 1991, p. 35). For
example, several operating principles may suggest that an evaluator observe system behavior over an
extended time period. SETIG members may find it useful to identify and examine the assumptions
underlying the principles, especially as the principles are tested through application.
DEVELOP A GLOSSARY
This document describes, rather than defines, the core concepts that serve as the principles’ foundation.
Some reviewers have suggested that definitions of these and other key terms would enhance the
usefulness of the document. Others underscore the point, stated in the preamble, that systems
concepts do not have agreed upon definitions (Reynolds & Holwell, 2010; Ison, 2010). SETIG members
may wish to revisit this issue after further study and discussion.
REFERENCES
Ison, R. (2010). How to Act in a Climate Change World. Milton Keynes, UK: Springer.
Reynolds, M. & Holwell, S. (2010). Systems Approaches to Managing Change: A Practical Guide. Milton
Keynes, United Kingdom: Springer.
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of Program Evaluation: Theory of
Practice. San Francisco: Sage Publications.
26 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
APPENDIX 1: THE GUIDE FRAMEWORK
The Systems in Evaluation principles were developed using the GUIDE Framework which outlines five
criteria for a high-quality principle: 1) guiding, 2) useful, 3) inspiring, 4) developmental, and 4) evaluable
(Patton, p. 43). The table below describes each criterion and how the Systems in Evaluation principles
meet them.
Criterion
Definition and the Systems In Evaluation Principles Meet This
Criterion by …
GUIDING
Definition: A principle is prescriptive. It provides advice and guidance on what to
do, how to think, what to value, and how to act to be effective. It offers direction.
The wording is imperative: Do this…to be effective. The guidance is sufficiently
distinct that it can be distinguished from contrary or alternative guidance.
Meets this criterion by: Providing clear operating principles and guidance on what
not to do.
USEFUL
Definition: A high-quality principle is useful in making choices and decisions. Its
utility resides in being actionable, interpretable, feasible, and pointing the way
toward desired results for any relevant situation.
Meets this criterion by: Providing plain language guidance useful for the practice of
evaluation.
INSPIRATIONAL
Definition: Principles are values-based, incorporating, and expressing ethical
premises, which is what makes them meaningful. They articulate what matters,
both in how to proceed and the desired result.
Meets this criterion by: Inspiring evaluators to conduct evaluations to contribute to
responsible and meaningful change in people’s lives. The principles make explicit
the ethical premises underlying the choices made in evaluation.
DEVELOPMENTAL
Definition: The developmental nature of a high-quality principle refers to its
adaptability and applicability to diverse contexts and over time. A principle is thus
both context sensitive and adaptable to real-world dynamics, providing a way to
navigate the turbulence of complexity and uncertainty. In being applicable over
time, it is enduring (not time-bound), in support of ongoing development and
adaptation in an ever-changing world.
Meets this criterion by: Providing guidance for any situation in which evaluation is
conducted and exhorting evaluators to continuously consider and respond to the
complex ever-changing environment in which we operate. The principles apply
across various boundaries including, but not limited to, geographical, social,
economic, programmatic, and jurisdictional boundaries. They apply to an
intervention, change initiative, policy, or program develops and throughout its
27 Principles for Effective Use of Systems Thinking in Evaluation: Systems in Evaluation TIG (9/9/2018)
implementation. They provide guidance for any number of intended uses, and
applies to different purposes for evaluation (accountability, program improvement,
strategy analysis, overall summative judgments of merit and worth, monitoring, or
knowledge-generation).
EVALUABLE
Definition: A high-quality principle must be evaluable. This means it is possible to
document and judge whether it is actually being followed, and document and judge
what results from following the principle. It is possible to determine whether
following the principle takes you where you want to go.
Meets this criterion by: Providing suggestions for how to evaluate the four
constituent principles.
Patton, M. Q. (2018). Principles-Focused Evaluation: The GUIDE. New York: Guilford Press.
