Module 13 – Monitoring – Protect Water

Monitoring and Tracking Progress Module.

Module 13 – Monitoring and Tracking Progress

“Setting a goal is not the main thing. It is deciding how you will go about achieving it and staying with that plan.”
– Tom Landry

Learning Expectations

Understand, apply five main stages of project monitoring and evaluation
Develop a proposed monitoring and evaluation plan
Define effective outcomes for progress
Assign monitoring roles and responsibilities
Determine useful examples of tracking documents (e.g., Watershed Report Cards)

Module Content

Learning Expectations and Learning Activities

Unit 1 – Why Monitor? Why Evaluate? Decoding the Underlying Messages

What’s it all about?

Unit 2 – The Theory and the Practice

The Fundamentals – Key Steps in a Monitoring and Evaluating System
Interview with a Staff Member

Unit 3 – Toward Recommendations for Monitoring and Evaluating the Implementation of the Source Protection Plans for the Maitland Valley and Ausable Bayfield Source Protection Areas

What are your main goals?

Who is responsible and who is involved?

What are the roles for implementation staff, Project Manager, General Manager, Source Protection Committee, Source Protection Authority, partners, stakeholders, general public?

Do we need an evaluation team?

Do we need outside expertise?

If so, at what stages?

Unit 4 – Wrapping it up

Field Learning:

There is one field learning task to undertake for next module. Develop a one-page ‘communications plan’ for source protection planning at this stage.

Answer these questions:

a) Who do you need to reach?

b) What should they know?

c) When and how do you communicate results?

Unit 1 – Why Monitor? Why Evaluate?

Activity 1 – Facilitated Dialogue – The Need for Monitoring

Source protection plans included polices that make use of implementation tools such as public education, incentives, municipal land use planning and bylaws, risk management plans and, in some cases, prohibition.

Source protection plans also included requirements for monitoring local progress on source protection.

Effective watershed management includes:
Scientific research
Source water protection and drinking water source protection planning
Source water protection and drinking water source protection implementation
Monitoring
Evaluation of success

Question and Answer

Answer the following True or False.

Monitoring and evaluating a plan or project’s progress:

Keeps projects on track, meeting objectives True or False
Ensures intended results are being achieved True or False
Identifies the need, if any, to change the project True or False
Involves identifying new concerns or issues and trends True or False

What do we mean by a ‘living plan’?

Define Project Monitoring and Evaluation:

What happens if effective project monitoring and evaluation does not take place? Discuss.

Are you aware of plans that were not monitored and evaluated? Discuss. Were they effective?

Activity 2 – Multimedia Presentation

Mini Quiz – What’s it all about?

What are the five recommended stages of project monitoring and evaluation:

What is an ‘output’ in a project?
What is an ‘outcome’ in a project?
What is the difference between an ‘output’ and an ‘outcome’?
Which is more important, the ‘output’ or ‘outcome’? Discuss.

Unit 2 – The Theory and the Practice

The Fundamentals – Key Steps in Project Monitoring and Evaluation

Activity 3 – Small Group Activity

Take a look at the above ‘planning cycle’ diagrams or other diagrams provided by your facilitator.

Which model do you like best for planning for the future?

What’s missing? As a small group develop your own diagram of a ‘planning cycle’ that should guide our local Source Protection Planning efforts.

Activity 4 – Small Group Activity

Look at a case study – real or hypothetical – provided to you by your facilitator of a past project.

Identify the project’s:

Outputs
Outcomes
Discuss if the outputs led to the desired outcomes:

Activity 5 – Small Group Activity

Look at a case study – real or hypothetical – provided to you by your facilitator of a past project.

Identify the project’s:

Expected Outputs
Expected Outcomes
Discuss if the outputs are likely to lead to the outcomes:

Activity 6 – Group Activity

Interview with a Staff Member

Instruction: Interview a General Manager, Project Manager or employee of the Ontario Ministry of the Environment, Conservation and Parks (MECP); the Ontario Ministry of Natural Resources and Forestry (MNRF); the Ontario Ministry of Agriculture, Food and Rural Affairs (OMAFRA); your local drinking water source protection planning project or a conservation authority. This may help you to identify monitoring and evaluation
successes and obstacles.

Review a Logic Model.

Consider some of these questions, and then add your own:

The nature of your work involves on-going learning and continuous improvement in water resource management. However when managing or implementing a project, do you systematically monitor and evaluate progress?

What improvements would you like to see in the approach to project ‘quality control’?
What kind of projects will you be involved with in drinking water source protection planning?
Is there a formal monitoring and evaluation process in place?
Do you favour a formal reporting system? Or is a less structured approach sufficient?
Have you been trained in project management? If not do you feel you would benefit from that?
When doing a project are you given sufficient time for monitoring, evaluating and, if necessary, revising your project?
For whom would you prepare a project monitoring and evaluation report?
Who or what groups in the organization should be responsible for project monitoring and evaluation?
Why is project monitoring and evaluation important?
Rate your organization’s commitment to and understanding of project monitoring and evaluation (considering all levels from Board, Committee to GM to Project Manager to implementation staff):
Completely adequate
Adequate, but in need of reinforcement in training and resources
Inadequate.

Activity 7 – Facilitated Group Dialogue

Discuss your committee or working group plan progress to date. What are your successes? What needs more work?
What are your main goals?

Unit 3 – Toward Recommendations for Monitoring and Evaluating the Implementation of the Source Protection Plans for the Ausable Bayfield Maitland Valley Source

Protection Region.

Activity 8 – Personal Reflection

How would you describe the local approach to Project Monitoring and Evaluation?

Activity 9

Consult the latest Watershed Report Cards or other reporting documents for Ausable Bayfield and Maitland Valley watersheds.

What are the advantages of these documents for tracking progress?
What are the disadvantages?

Activity 10 – Facilitated Dialogue – What are your main goals?

Say and Switch: Do you agree?

Instruction: Form two groups. Group A agrees with the goals of the plan for plan and project tracking. Group B disagrees. Select a debating partner from the opposite group and for five minutes defend your viewpoints. Then switch: i.e., Group A is asked to disagree and Group B is asked to agree.

A consensus will be sought as to the stated goals:

Are they satisfactory?

If not how should they be amended?

Activity 11 – Sub-Group Activity

Who is responsible and who is involved?

Based on input from your facilitator, and your own input, what are the roles for:

Implementation staff?
Project Manager or Program Supervisor?
General Manager?
Source Protection Committee?
Partners?
You?
General public?

Activity 12

Instruction: Groups are to list roles for each possible participant in Monitoring and Evaluation. Possible roles include:

Prepares plan for project Monitoring and Evaluation
Collects data and prepares commentary during project implementation
Writes output reports
Prepares outcome assessment reports
Prepares project evaluations at project completion
Receives, reviews, approves output reports
Is informed on output reports
Receives, reviews, approves outcome reports
Is informed on outcome reports
Receives, reviews, approves final project evaluation reports
Is informed on final project evaluation reports
Report to group as a whole.

Activity 13 – Facilitated Dialogue

Fundamental Questions in Deciding how to Monitor and Evaluate

Do we need a formal system?
Do we need training or more resources?

Activity 14 – Sub-Group Activity

More Fundamental Questions in Monitoring and Evaluation

Activity

Discuss the following quotations:

a) “No sir, our marsh will not be developed. You don’t kick mother nature in the shins.” – [Connecticut farmer John Whitman Davis, speaking to John G. Mitchell for National Geographic, ‘Our Disappearing Wetlands,’ October 1992.

b) “Act small and we’ll get bigger and better. Act big and we’ll get smaller and decline.”
– Airline executive.

c) In 1990, more than 20 toilets in a U.S. County Courthouse (in the State of Washington) erupted upon being flushed. A plumber had mistakenly switched an air compressor and a water line.
– Uncle John’s Third Bathroom Reader, page 67

Activity 15 – Small Group Activities

Instruction: Watch and listen to the film clip, “A Conversation with Joan Klaassen of Environment and Climate Change Canada,” DVD produced by the Ausable Bayfield Maitland Valley Drinking Water Source Protection Project. Consider Klaassen’s comments about why monitoring and evaluating progress is important.
Does she provide any examples of how we can proceed?

Questions for Consideration

What lesson do you take from Joan Klaassen’s comments regarding the importance of monitoring issues and tracking progress in protecting source water?
Other than being inspirational, what do the words of the old farmer, John Whitman Davis and his decision to retain his wetland, say to you about the importance of monitoring issues and tracking progress?

“Act small and we’ll get bigger and better. Act big and we’ll get smaller and decline.” Heb Kellehar was talking about the airlines business when he made this comment but he was also implying that his advice had general application. What’s the lesson for monitoring and evaluating progress in our Drinking Water Source Protection project?
What do exploding toilets have to do with monitoring?

Unit 4 – Wrapping it up

Activity 16 – Facilitated Dialogue

Input to the Source Protection Committee (SPC)

How can we confirm, enlarge upon, or amend our input to the Drinking Water Source Protection Committee?

Activity 17 – Small Group Activity – Summary

What input have you provided to the source protection committee to date?
What additional input do you have at this time?
You have three minutes to make a statement to the source protection committee? What will you say? (You can choose to write down your response or articulate it orally).

Do you require more information at this time? What information?
What should we stop?
What should we start?
What should we continue?

SECTION TWO

Priming the Pump

Notes, Definitions, Fact sheets

A plan has to be a ‘living’ plan if it is to be effective. We could come up with excellent ideas based on our best information but if all we do is publish a
report and put it on the shelf then all our hard work will evaporate. We need to implement a plan, certainly, but we also need to create mechanisms to ensure it is effective, that our foundational assumptions were correct, and that any changes that are required are made as the plan evolves.

Our planning process needs:
The appropriate data
A science-based plan
Monitoring of the plan
Evaluation of the plan
Adjustments to the plan

Defining Our Terms

The literature uses project monitoring, tracking, review, evaluation, assessment, and revision in different and sometimes conflicting ways.
For our purposes, we use the terms Monitoring and Evaluating together to include:
Monitoring the progress of the work plan (of the Source Protection Plans and their projects,
Describing the outputs (what is done)
Evaluating the outcomes (did we get the expected short-term results?)
Testing and evaluating impacts (determining long-term results)
Revising and updating the plan and its projects where indicated.

Occasionally, for convenience, we will use the term “tracking progress” to refer to various monitoring and evaluating activities.
Various sources say that effective project monitoring and evaluation requires:
Good advanced planning,
Measurable objectives and
A workplace culture that values learning and continuous improvement.

Most organizations strive to “measure up” but often tasks such as project monitoring and evaluation are eased to the back burner when other pressures heat up.
Why monitor and evaluate progress?

A number of experts argue that developing the plans is only half the battle.
Delivering the plan well is the other half. Plans can no longer be static
blueprints for action. The world is constantly changing and we need:
a) Reassurance we are doing the right thing, and;
b) An “early warning system” when we are not.

There are of course more specific answers to the question: Why monitor and evaluate progress? Some answers will seem obvious. Others may surprise. Taken together the answers seem to confirm the experts’ conclusion that tracking a project’s progress may be as important as the project itself.

Here are just some of the reasons for good project tracking. These might be considered to be good business decisions.

Ensuring Conformity with the Plans. Or, does tracking evidence suggest a plan shift is called for?

Maintaining accountability. It is important to demonstrate good value for the ratepayer and sometimes for funders.

Consult the Logic Model.

Verifying results. It is useful to know we are achieving our goals in order to learn what works, to build on success and to replicate successful activities
wherever else needed.
Identifying improvement. It is helpful to know what project activities need revision.

Identifying failure. This permits us to discontinue an unsuccessful project and to reassign resources to proven projects.

You may not have considered the following other benefits to project monitoring and evaluating:

Linking the Project (or plans) to day to day operations. An organization needs to maintain a strong sense of purpose.
Identifying new information, problems and/or trends. Recall that the data and research are incomplete. The plans are dynamic. While there are resources dedicated to research under the Plan, a tracking system can also help.

Finding new opportunities and new collaborators. A good tracking system fosters a learning workplace culture, creating the potential for these side benefits.
Comparative project review. A project may be successful but a good tracking system will reveal the comparative cost-benefits of projects having similar goals.

Maintaining Inertia and Building Momentum. When success is demonstrated, Committee, senior staff, implementation staff, benefiting parties and the general public may be motivated to continue to advance the cause.

What are the main challenges to effective monitoring and evaluation?

Experts in project management recognize that the allocation of resources to monitoring and evaluation is frequently inadequate. Beyond that, these common shortcomings are noted:
Getting commitment (and sometimes connected to that, affording sufficient time to project implementation staff to give to tracking progress).
Inadequate advance planning. For example failing to establish clear targets,
indicators or measurable at the project’s start.
Failure to obtain feedback from partners or stakeholders.
Failure to relay reports back to partners or stakeholders.

We want to satisfy the vision and goals of the source protection plans. But in ensuring that happens, what exactly are we to monitor and evaluate? Distilled down to basics, we are monitoring and evaluating the various plan implementation projects and initiatives. We are making sure we are getting the desired results from each plan project. And where we are not, we are making adjustments to ‘get back on track.’ The progress reports on the various plan projects will be compiled and reviewed in an evaluation of the progress of the plans themselves.
Is there a good system that might be used in project monitoring and evaluation?

A progress tracking system should be tailored to the particular project.
Nevertheless, there are common threads among systems and there are some good examples ‘out there’. A good Ontario example is the system advocated by the Province of Ontario in Strategic Planning for Economic Development, Rural Economic Development Data and Intelligence (REDDI).

Go to Track Progress.

Below, we have summarized the Ontario example (and amended it for our needs) to help the Working Group to grasp theory and to assess the approach being taken to project monitoring and evaluation by the source protection committee.

Five stages are recommended:

Describing the project
Tracking Output
Assessing Outcomes
Communicating Results
Preparing an Evaluation Report

1) Describing the Project

‘Describing the Project’ s is a key stage. One has to know what activities to perform and how much to budget. But more important is knowing what success should look like in the short and long run. In brief, the project description includes:

Setting goals. (A visionary statement, main challenges)

Setting Objectives. (Specific, trackable statements, timelines)

Detailing activities. (Specific priorities and tasks, with budget)

Expected Outputs. (Short-term results, the deliverables or concrete products, expressed in quantitative or qualitative terms). Outputs are linked to activities.

Expected Outcomes (changes, progress or improvements that result from the project). Outcomes are linked back to objectives.

Depending on the scale and nature of the project, expected impacts might also be described. What is the expected end result?

2) Tracking Output

‘Tracking Output’ is usually prepared by “implementation staff” for Project Managers. These are brief reports using forms or journals as to:

What was done

Immediate results.

The focus is on reporting only that which is needed to make project decisions.
Tracking reports are needed to understand both the effectiveness of activities and level of results.

The other three stages:
3) Assessing Outcomes;
4) Communicating Results and
5) Preparing an Evaluation Report will be addressed further in your upcoming module.

What are the Challenges in Monitoring and Evaluation?
Getting the commitment to do it;
Establishing base lines at the beginning of the project;
Identifying realistic quantitative and qualitative indicators;
Finding the time to do it and sticking to it;
Getting feedback from your stakeholders;
Reporting back to your stakeholders.

Communicating Results

A brief communications plan may be useful:

Whom do you need to reach?
What should they know?
When and how do you communicate results?

SECTION THREE

Handouts, Additional Readings, Field Learning Assignments, Findings:

Field Learning:

Develop a one-page ‘communications plan’ for Source Protection Planning at this stage. Answer these questions:

a) Who do you need to reach?
b) What should they know?
c) When and how do you communicate results?

Reading, Listening and Viewing Resources:

Evaluating [Health Canada, Community Action Resources for Inuit, Métis and First Nations publication, 1998]

REDDI Website, Province of Ontario Rural Economic Development Data and Intelligence [2007]

Strategic Planning [Fact sheet by Peter Fleming, Ontario Ministry of Agriculture, Food and Rural Affairs (OMAFRA), published in 1989, reprinted in 1991, reviewed in 1997].

Information here is provisional, subject to change, and posted for local information and education purposes. For current information visit Ontario.ca and sourcewaterinfo.on.ca. We would like to acknowledge the support of the Government of Ontario. Such support does not indicate endorsement of the contents of this material.