Quick guide: Monitoring and evaluation of ICT in education initiatives
Quick guide: Monitoring and evaluation of ICT in education initiatives
As infoDev's Knowledge Maps: ICT in Education notes, "It is generally believed that ICTs can empower teachers and learners, promote change and foster the development of '21st century skills', but data to support these beliefs are still limited". Indeed, "data on the nature and extent of these issues remain limited in most places because of the lack of monitoring and evaluation tools and methodologies dealing with the use of ICTs in schools and their impact on teaching and learning".
With this in mind, infoDev has gathered together some useful resources related to the monitoring and evaluation of ICT in education initiatives to assist those interested in the topic, but not sure where to begin.
- infoDev publications and activities
- UNESCO publications and activities
- Some notable studies related to the impact of ICT use in education
- Some additional monitoring and evaluation resources
- Favorite recommended resources related to ICTs & Education
1. infoDev publications and activities
- Monitoring and Evaluation of ICT in Education Projects - A Handbook for Developing Countries
- Monitoring and Evaluation of ICT4E: An Introduction
- A Review of Monitoring and Evaluation Impact Evidence
- Core Indicators for Monitoring and Evaluation Studies
- Developing a Monitoring and Evaluation Plan
- Capacity Building and Management
- Pro-Equity Approaches to Monitoring and Evaluation: Gender, Marginalized Groups and Special Needs Populations
- Dos and Don'ts in Monitoring and Evaluation
- Framework For The Assessment OF ICT Pilot Projects: Beyond Monitoring and Evaluation To Applied Research
- Knowledge Maps: ICT in Education
(quick snapshots of what the research literature tells us about a number of key areas)
- Using Technology to Train Teachers
- infoDev ICT/Education M&E-related activities
2. UNESCO publications and activities
Publications from UNESCO - Bangkok
- Developing and Using Indicators of ICT Use in Education
- Consultative Workshop on Performance Indicators for ICT in Education
UNESCO Bangkok ICT in Education Indicators projects
- Indicators for Assessing ICT Impact in Education
- Examples of ICT indicators for education
- Manual for Pilot Testing the Use of Indicators to Assess Impact of ICT Use in Education
- Performance Indicators on ICT for Education Matrix
from UNESCO Institute of Statistics (UIS)
- ICTs and Education Indicators: Suggested core indicators based on meta-analysis of selected International School Surveys
Note that this is linked to activities of the Partnership on Measuring ICT for Development
3. Some notable studies related to the impact of ICT use in education
Please note that this list is provided as a general reference for informational purposes; no endorsement of any particular M&E methodology or findings is implied (unless so noted).
- Becta ICT in schools research (links to many impact studies in the UK)
- Are Students Ready for a Technology-Rich World?: What PISA Studies Tell Us
- Assessing the Impact of Technology in Teaching and Learning: A Sourcebook for Evaluators
- Changing the Conversation about Teaching, Learning, & Technology: A Report on 10 Years of ACOT Research (Apple Computer report)
- Comparative International Research on Best Practice and Innovation in Learning
- Computers and Student Learning: Bivariate and Multivariate Evidence on the Availability and Use of Computers at Home and at School (aka 'Fuchs-Woessmann')
- Computers Make Kids Smarter — Right? (1998 article in Technos Quarterly by Kirkpatrick & Cuban)
- DEEP IMPACT: an investigation of the use of information and communication technologies for teacher education in the global south (from DFID)
- Effectiveness of Reading and Mathematics Software Products:
Findings from the First Student Cohort (2007 report to the U.S. Congress)
- Findings from the Teaching, Learning, and Computing Survey
- Freedom to Learn (Michigan, USA)
- Global Networked Readiness for Education. Preliminary Findings from a Pilot Project to Evaluate the Impact of Computers and the Internet on Learning in Eleven Developing Countries (Harvard, World Bank Institute)
- The ICT Impact Report: A Review of Studies of ICT Impact on Schools in Europe (European Schoolnet report)
- The Learning Return on our Educational Technology Investment - A Review of Findings from Research
- Maine Learning Technology Initiative: Research and Evaluation
- Programa Enciclomedia: Libro Blanco (the official 'white book' on Mexico's Enciclomedia initiative; in Spanish).
- Remedying Education: Evidence from Two Randomized Experiments in India (MIT Poverty Action Lab)
- A Review of the Literature on Computer-Assisted Learning, particularly Integrated Learning Systems, and Outcomes with Respect to Literacy and Numeracy (from New Zealand)
- Second Information Technology in Education Study: Module 2
- Technology, Innovation, and Educational Change: A Global Perspective
- Technology in Schools: What the Research Says (Cisco report)
- Texas Technology Immersion Project (eTxTIP)
- West Virginia Story: Achievement Gains from a Statewide Comprehensive Instructional Technology Program 1999
- World Links for Development: Accomplishments and Challenges. Monitoring and Evaluation Report
4. Some additional Monitoring and Evaluation resources
(Excerpted from the infoDev publication, Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries)
Monitoring and Evaluation of ICT for Education Impact: A Review
- School Factors Related to Quality and Equity (OECD/PISA)
A report on the effects of policies and the structure of education systems on educational outcomes, based on analyses of PISA 2000, the results of a multi-year, international study. The frameworks and assessment instruments from PISA 2000 were adopted by OECD member countries in December 1999, (see Annexes A & B).
- Monitoring and Evaluation: Some Tools, Methods, and Approaches (World Bank)
An overview of a sample of M&E tools, methods, and approaches, including purpose and use; advantages and disadvantages; costs, skills, and time required; and key references.
Core Indicators for Monitoring and Evaluation Studies for ICT in Education
- Young Children’s Computer Inventory Summary (Texas Center for Education Technology)
A 52-item Likert instrument for measuring 1st through 3rd grade children's attitudes on seven major subscales. Also contains links to the survey instrument and scoring.
- Tips for Preparing a Performance Evaluation (USAID) (also here)
This document outlines USAID’s framework for performance monitoring plans (PMP) used to plan and manage the collection of performance data (and occasionally includes plans for data analysis, reporting, and use). It describes the following components as essential to PMPs: a detailed definition of each performance indicator; the source, method, frequency, and schedule of data collection; how the performance data will be analyzed; and how data will be reported, reviewed, and used to inform decisions.
- Development Research Impact: REACH (International Development Research Centre), Paper by C. Sander (1998)
This report outlines issues in accountability and development research impact assessment; introduces “reach” as impact of development research; illustrates reach assessment with findings from impact studies; and concludes with suggestions for impact assessment as learning accountability and reach as a concept to facilitate assessing and designing for research impact.
Developing a Monitoring and Evaluation Plan for ICT in Education
- ICT Survey Questionnaire Summary (UNESCO-Bangkok)
This instrument was developed with the understanding that many countries are at different stages of ICT development and, hence, indicators may vary. Questionnaires are meant to serve as a basis, and evaluators may tailor them to their specific contexts. The overall framework includes four questionnaires to be completed: ministry of education, school heads/principals, teachers & teaching staff, and students.
- Open Source Monitoring & Evaluation Tool (International Institute for Communication and Development, IICD)
IICD provides advice and support to local organizations in developing countries to benefit from the potential of ICTs. This tool supports the collection of data and the analyses of results and includes surveys for project users, training & seminar participants, project team members & managers, information network members, and the global teenager project.
- Evaluation Planning in Program Initiatives (International Development Research Centre)
A series of guidelines to help IDRC managers, staff, and partners improve the quality and consistency of evaluations in IDRC and to enhance evaluation capacity, including guidelines for: searching for previous IDRC evaluation reports, program initiative evaluation plan tables, formatting evaluation reports, writing terms of references, identifying intended uses and users of evaluations, selecting and managing an evaluation consultant or team, and preparing program objectives.
- Resources for Technology Planning (Texas Center for Education Technology)
This is a listing of internet-based resources for technology planning, including tools, publications, templates, surveys, and checklists.
- Online Assessment Tools (enGauge) (North Central Regional Educational Laboratory)
This contains online assessments for use by districts or schools to conduct online assessments of system-wide educational technology effectivness. It includes sample surveys and profiles for educators, district & building administrators, district & building technology coordinators, board members, community members, students, and parents.
Capacity Building and Management in ICT for Education
- Educator Development for ICT Framework, Assessment and the Evaluation of the Impact of ICT (SchoolNet South Africa)
An educator development program that focuses on integrated, formative assessment of practice and competencies on four levels: ICT skills, integration, growth in the educator as a professional, and the whole school. Some indicators of competencies include: levels of computer efficiency; practical, foundational, and reflexive competencies; and self-assessment.
- Three-Step Technology Evaluation, Are Your Schools Getting the Most of What Technology Has to Offer? (Sun Associates)
With the goal of examining technology’s impact on student achievement district-wide, this evaluation focuses on three steps: setting goals, collecting and analyzing data, and recommendations and reporting, with several sub-steps.
- Computer Background Survey, Global Networked Readiness for Education (Harvard University/World Bank)
This Survey Toolkit is designed to collect experiential data about how computers and the Internet are being used around the world and in developing countries. The four surveys comprising this toolkit seek to address data deficits at both the school and policy levels, and give useful and actionable information about the integration of ICT in education. The four surveys are: teacher, student, head of school, and computer background.
Pro-Equity Approaches to Monitoring and Evaluation: Gender, Marginalized Groups and Special Needs Populations
- Advice on Special Education Needs (SEN) and Inclusion (National Grid for Learning)
A web site offering research, products, legislation and guidance materials, case studies, “ask the experts,” online communities, and recommended web sites for special educational needs and inclusion.
- Networking Support Program (Association for Progressive Communications)
A web site promoting gender equity in the design, development, implementation, access to and use of ICTs and in the policy decisions and frameworks that regulate them. It offers information on activities/conferences, policy, evaluation, training, and resources.
A web site focusing on ICTs’ contribution to the economic, political, and social empowerment of women and the promotion of gender equality, offering information on activities/conferences, policy, evaluation, training, clearing houses, and resources. The range of topics and accompanying resources include: economic empowerment, education, health, violence against women, women in armed conflict, cultural diversity and language, communication rights, universal access, strategic use and FOSS, and governance.
A reminder: Links change! Given the constantly changing nature of the Internet, we suggest that you copy the document or web site title (and author or organization name, as appropriate) of a resource below into your favorite search engine if a link on this page is not working.