Evaluating Educational Technology
Technology in Higher Education |
Government Initiatives Focused on Evaluation and Educational Technology
- The Office of Postsecondary Education sponsors several initiatives to promote the use of technology in higher education:
The Fund for the Improvement of Postsecondary Education
, stimulates innovation in education by distributing funds for curricular and pedagogical reform, dissemination of innovations, access to education, international education, assessment, and so forth. Although FISPE is not explicitly focused on digital resources, technology-based projects do fit its overall aims.
The Learning Anytime Anywhere Partnerships (LAAP)
provides funds for "asynchronous, innovative, scalable, and nationally significant distance education projects."
National Postsecondary Education Cooperative
aims "to identify and communicate on-going and emerging issues germane to postsecondary education, and to promote the quality, comparability and utility of postsecondary data and information that support policy development, implementation, and evaluation." It promotes the effective collection of data related to higher education, undertaking initiatives such as "Data Ramifications of Technology for Current Surveys" and "Student Outcomes."
Education Outreach and Training Project for the Advanced Computational Infrastructure
EOT-PACI strives to increase access to emerging technologies and the learning opportunities they provide. It is a joint effort of the National Computational Science Alliance and the National Partnership for Advanced Computational Infrastructure (NPACI). Although evaluation is not an explicit goal, EOT-PACI sponsors projects that document how students, researchers, and instructors interact with technology, such as the NISE College Level One Institute on Learning Technology, IATH's Humanities Research: Collaborative Tools in Networked Environments
project, and Michigan's Adapting Supercomputing Systems and Social Scientists. Rice's Richard Tapia is one of the co-chairs and directs the Center for Excellence and Equity in Education (CEEE). Program evaluation is built into the operations of EOT-PACI.
The Education and Human Resources Directorate of the National Science Foundation sponsors programs to improve math, science, health, and technology education. Its division of undergraduate education supports curriculum innovation, digital library projects, and advanced technological education, while its division of graduate education sponsors integrative graduate education and research and other programs.
Division of Research, Evaluation and Communication (REC)
REC, part of the Education and Human Resources Directorate, "contributes to the broad field of educational research and improvement by funding projects through grants, contracts, and cooperative agreements. It also provides conceptual and technical assistance to various EHR programs and principal investigators, through project and program evaluation, dissemination and implementation of knowledge and effective practices, and the utilization of technology in education." Current initiatives include programs to fund core research on learning education, the development of young faculty, and large-scale implementations of experimental approaches to education. To get a sense of emerging trends in educational technology (particularly in K-12 education), browse the database of past REC grants.
Institute for Technology Assessment and Support
aims to assess the impact of technology across higher education. More specifically, it proposes to:
- Provide Colleges and Universities with Tools, Processes and Instruments
- Perform Data Analysis and Reports
- Offer Assessment and Support Consultation
- Report on National Data Repository
- Enable Benchmarking Against Other Institutions
The emphasis on assessment seems to have been prompted by several factors: Seton Hall has undertaken a major mobile computing initiative, distributing IBM Thinkpads to all students and encouraging the integration of technology into the curriculum. It is required to perform regular institutional evaluations to fulfill requirements mandated by the state of New Jersey.
Most of the Center's efforts seem focused on Seton Hall itself, which won the 1999 EDUCAUSE Award for Excellence in Campus Networking .
EvNet: Network for the Evaluation of Training and Technology
This Canadian research consortium evaluates instructional technologies in worksites, schools, colleges, and
universities. To promote the effective use of technology in learning, EvNet recommends best practices, provides training, produces courseware and training modules, and builds education networks. Over 60 institutions participate in EvNet, including universities, K-12 schools, corporations, government, community colleges, and foundations. EvNet has defined four research themes: "Evaluating Administrative Practices," "Evaluating Design Roles," "Evaluating Delivery," and "Evaluating Collaboration." EvNet's web site also serves as a portal for information about assessment as well as for assessment tools. For further information, see the overview of EvNet at http://socserv2.mcmaster.ca/srnet/toc.htm.
The Flashlight Program at TLT (Teaching, Learning, and Technology Group)
The Flashlight Program has emerged as one of the leading centers for the assessment of technology in higher education. Flashlight provides assessment toolkits, stages training programs, offers consulting, organizes conferences, and provides access to free information. The TLT group, which sponsors Flashlight, is itself part of the American Association for Higher Education.
P-SITES: Planning for International Research on Exemplary Sites of Technology-Enhanced Instruction
is carrying out an international study of effective uses of technology in teaching. The study is a joint effort between the University of Minnesota and SRI. Its website features position papers on technology in education and a description of its research design.
University of Wisconsin,
LEAD Center: Learning through Evaluation and Assessment
"The LEAD Center provides support for funded research projects requiring student assessment activities to evaluate education reform efforts at both the undergraduate and graduate levels. The Center helps to enact the Chancellor's UW-Madison 'Vision for the Future' by creating horizontal links for faculty, departments and programs involved in educational reform projects, and thus helping the campus to reconceptualize itself as a learning community. The LEAD Center is the first of its kind in the nation, and is 'faculty-driven' and 'student-focused' in its assessment approach." Major research areas include: Educational Technology Projects, Curricular and Pedagogical Improvement Projects, Access and Inclusion Projects, K-12 Outreach Projects, and Program Evaluation. Research is performed by PhDs in anthropology, history, psychology, statistics, and education, and both qualitative and quantitative methods are used. LEAD designs and conducts evaluation studies, provides consulting and training services, and authors reports summarizing its findings.
For an example of an evaluation report prepared by LEAD, see Integrating High Performance Computing into the Undergraduate Curriculum: How PACI and the Education Center on Computational Science & Engineering Can Succeed
The Regional Educational Laboratories
research education reform and disseminate information about best practices in education. Funded by the U. S. Department of Education, Office of Educational Research and Improvement (OER), the REL is comprised of ten regional labs, each with a different specialty. The two most relevant to the assessment of educational technology are:
is regarded as one of the leading centers on K-12 evaluation. Its research on evaluation covers a wide range of subjects, from collaboration to technology.
North Central Regional Educational Laboratory "NCREL's 'specialty lab' work focuses on technology, specifically the examination of critical policy issues pertaining to the integration and application of technologies for teaching and learning, and the dissemination of information relevant to those policy issues."
Western Michigan University,
The Evaluation Center
"The EVALUATION CENTER'S mission is to provide national and international leadership for advancing the theory and practice of program, personnel, and student/constituent evaluation, as applied primarily to education and human services. The Center's principal activities are research, development, dissemination, service, instruction, and leadership." Most of the Center's projects involve assessing K-12 charter schools and community organizations, but it was recently awarded an NSF grant "to assess the impact and effectiveness of the National Science Foundation's (NSF) Advanced Technological Education (ATE) program."
Whereas educational research groups focus on whether students learn through technology, human-computer interaction (HCI) specialists conduct usability studies to measure the performance of both humans and machines (more specifically, humans with machines). The HCI special interest group of the ACM maintains a list of HCI laboratories around the world, as well as listings of journals, conferences, consultants, and so forth. Among the most significant HCI groups are:
Human-Computer Interaction Institute (HCII) is one of the leading HCI labs in the world.
Among the Institute's many projects are two devoted specifically to evaluation:
Understanding When, Where, Why and How Usability Assessment Methods Work and Evaluation Workstation,
which will capture data about HCI through "timestamped keystroke, mouse-click and event capture; digital audio for verbal protocols; digital video for video protocols; and eye tracking."
The Georgia Tech Research Institute's Intelligent Machines Group provides several
including the Evaluation Matrix, the Anecdotal Record Form, the Expert Review Checklist, the Focus Group Protocol, the Formative Review Log, the Implementation Log, and the User Interface Rating Form.
Significant Studies of Technology in Higher Education
Cornell, NOMAD Project
"This project is evaluating student collaboration and team-based development in a nomadic computing environment. Students in selected course are being issued with laptop computers with wireless modems. The research plan is both to evaluate the direct impact of these new technologies, and to create a very rapid feedback cycle for incorporating changes and improvements into the courses."
Wake Forest, Assessment of the Impact of Ubiquitous Computing
Describes the outcome of the ubiquitous computing initiative at Wake Forest, which provided laptops to every student and increased training and support available to faculty, students, and staff. Several assessment tools were used, including surveys of students and faculty and statistical studies of faculty, students, and facilities. Preliminary results indicate that the initiative improved academic achievement and attitudes toward computing.
Research on Evaluation
Ericae.net, one of the Department of Education's
information clearinghouses, compiles a bibliographic database about assessment in education, publishes an online journal, and offers full-text resources about evaluation.
AAHE Assessment Conference
To get a sense of the different ways that educators are discussing evaluation, it's worth browsing the abstracts of papers given at the 2000 American Association for Higher Education conference.
The Secretary's Conference on Educational Technology: Evaluating the Effectiveness of Technology
A conference hosted by the Department of Education on evaluating educational technology. It takes a K-12 focus, but makes available research relevant to other types of students as well. See in particular Critical Issues in Evaluating the Effectiveness of Technology, a summary of the conference by Mary McNabb, Mark Hawkes, and Üllik Rouk.
The National Postsecondary Education Cooperative (NPEC),
Technology and its Ramifications for Data Systems
This report collects papers "concerning the impacts of technology on data definitions and analytical conventions in the following areas: (1) new institutional and programmatic configurations, (2) understanding new faculty roles and work patterns, (3) measuring and analyzing student participation patterns, (4) assessing student progress and learning gains, and (5) analyzing revenue and expenditure flows." A brief summary of the report in available in Cause/Effect (22.2, 1999).
User-Friendly Handbook for Project Evaluation
"This Handbook was developed to provide Principal
Investigators and Project Evaluators working with the
National Science Foundation's Directorate for Education
and Human Resource Development (EHR) with a
basic understanding of selected approaches to evaluation.
It is aimed at people who need to learn more
about both what evaluation can do and how to do an
evaluation, rather than those who already have a solid
base of experience in the field."
Technology and Higher Education Statistics, Surveys, and Reports
A brief listing of statistics about the use of technology in higher education, last updated in 1999. Includes information from the U.S. government and from educational technology organizations such as Educause.
The Campus Computing Project
studies information technology in higher education, analyzing "qualitative and quantitative data to help inform faculty, campus administrators, and others interested in the use of information technology in American colleges and universities."
The Clearinghouse for Higher Education Assessment Instruments
provides surveys and other tools for conducting systematic evaluations of colleges and universities (where technology is only a component).
Alstrum, Vicki et al.
Evaluation: Turning Technology from Toy to Tool. Integrating Technology into Computer Science Education. ACM, 1996.
This report introduces some key terms and concepts for assessment in education.
Bruce, Bertram. Challenges for the Evaluation of New Information and Communication Technologies,
Journal of Adolescent & Adult Literacy, March 1999.
Hartson, H. Rex, José C. Castillo, John Kelso, and Wayne C. Neale, Remote Evaluation: The Network as an Extension of the Usability Laboratory. CHI 96 - Electronic Proceedings, 1996.
Lopata, Cynthia L. and Charles R. McClure, Assessing the Academic Networked Environment: Strategies and Options
Although this manual may be somewhat out of date, it offers useful strategies for evaluating academic computing and makes available measurement tools for assessment.
McClure is a leading expert on the management and evaluation of information services.
Research & Assessment
Collects research and interviews with experts on the effectiveness of using technology in teaching.
For a fine overview of funding sources available for education technology, see
UC Berkeley's RECOGNIZING AND FUNDING YOUR PROJECT THROUGH AWARDS AND GRANTS. Scroll down to "Extramural Grants for Higher Education."
The Consortium for the Advancement of Private Higher Education (CAPHE)
promotes reform and innovation at private colleges. "Founded by funders to strengthen the contributions to society of private colleges, CAPHE designs and administers competitive grant competitions; offers technical assistance to funders; and disseminates ideas resulting from its programs." In 1993, CAPHE released a report entitled Information Technologies in Independent, Liberal Arts Colleges.
The Mellon Foundation's Initiative on Cost Effective Uses of Technology in Teaching
sponsors projects that research whether universities can use technology to increase efficiency and lower costs. Led by Gilbert Whitaker, Dean of the Jones School at Rice, the initiative seeks to answer six questions:
- Can human resources be deployed more effectively by using technology to serve more students?--a question of great relevance to situations in which there is a need to expand higher education.
Can more material in basic courses be covered so that the number of courses required to reach advanced material is reduced?--enabling faculty to teach courses which are more exciting to them and to the students.
Can several campuses share scarce specialized talents in some subject matters? --reducing the need for adjunct faculty to replace faculty on leave.
Can the number of graduate assistants necessary to teach basic materials be reduced through mediated uses of technology?
Can some faculty time be redeployed by making better use of the time faculty spend in the classroom? What activities (for example, language drills and math exercises) are better done outside than inside the classroom?
On a more general level, the Mellon Foundation also offers grant support to private universities, emphasizing humanities scholarship.
The Kellogg Foundation
offers grants for higher education, including "educating a changing student body," "linking intellectual resources and community needs," "intellectual foundations of adult continuing education," and "comprehensive models." Programs funded under these initiatives include a million dollar grant to Portland State University to "Identify and support models of institutional transformation in select colleges and universities," a $1.7 million grant to the American Council on Education to "Strengthen higher education programs through a process of institutional change," and a $150,000 grant to Syracuse University to "Establish a National Academy for Academic Leadership." The Kellogg Foundation has also designated Information Systems/Technology as a cross-cutting theme for its funding efforts.
Home URL: <
Copyright © 2000 by CODE.
Last updated December 14, 2000 by Lisa
Spiro for CODE (Committee on the Digital Environment at Rice