|
Priming the Pumps: Developing and Assessing Research-Like Experiences in Courses
Janice E. Thornton, Department of Biology and Neuroscience
Judith Beinstein-Miller, Department of Psychology
Tysza Gandha, Department of Psychology
Patricia deWinstanley, Department of Psychology
Oberlin College
Oberlin, OH 44074
In 1998, Oberlin College was honored with an Award for the Integration of Research and Education
(AIRE) from the National Science Foundation. Oberlin College chose to use part of the monetary portion of this award to ‘prime the pumps’ of curricular innovation and assessment to further enhance the integration of research and education into the curriculum. The award provided Oberlin College with an exciting opportunity to extend curricular initiatives that integrate research-like activities into courses across the college and at all levels of the curriculum, and to promote campus-wide discussions on how to improve curriculum development. In parallel, we were able to develop and institutionalize methods to assess such curricular innovations, and disseminate data, models, and assessment information to the broader academic community, in part through workshops held at Oberlin.
Developing Research-Like Experiences in Courses: Curriculum Development Projects
Oberlin has sponsored 28 new curriculum development projects that provide more opportunities for students to engage in research-like activities in courses. Awards have been given to 33 faculty members from 13 different departments. Some projects have revised part of an existing course, and others have involved the development of a completely new course. In all cases, the projects developed discovery-based, research-like activities in which students actively sample, analyze, and interpret data. Applications that proposed library research alone were not funded.
The exciting projects that have been developed have stretched to all levels of the curriculum in the sciences, social sciences and humanities, and fostered communication and collaboration across departments and disciplines. These courses affect all classes of students, from frosh to seniors. Changes have been made to courses for general audiences and to both introductory and advanced level courses for specific majors. Some projects have incorporated research-like activities into a large lecture course, others into a small lecture/discussion course and still others in laboratory courses. In many cases, these new curricular initiatives have engaged students in the study of environmental, economic or social problems in the community.
Although disparate, these curriculum development projects all incorporate research-like experiences. Students in a non-majors physics course on solar energy have begun to use infrared and other equipment to do energy audits of homes in the community. Geology students are determining the water flow and water quality of surrounding streams. In a new course on Environmental Mathematics, students conduct modeling-based research to gain insight into real world problems and questions. In an Art History course, new laboratory exercises enable students to determine how, where and when certain objects were made and whether forgeries can be detected. In a new Psychology course for non-majors the students learn about experimental design and analysis and then run experiments of their own design. The research topics in these courses are derived from the course content rather than from faculty members' research programs.
 |
Figure
1: In the first year seminar course on the Neurobiology of the Mind,
students test the way the visual system works. The student with the
mirror is experiencing binocular rivalry; when he passes his hand in
front of the eye seeing blank wall, he will 'erase' the face of the
person sitting across him - the Cheshire Cat Illusion. |
The process we developed to award curriculum development grants was important to the success of this endeavor. All faculty members received an application for curriculum development proposals that encouraged curriculum development. To apply, faculty members submitted a title, a brief proposal that included articulation of the educational goals to be accomplished, and a budget. Importantly, the application announcement made clear that assessment was expected to be an integral part of the curriculum development activities and that awardees would also be expected to disseminate effective curricular innovations. Using AIRE funds, grants were awarded for four years (1999-2002). These grants provided 1-8 weeks of summer salary, some supply money if needed, and money for a semester-long student teaching assistant if needed. If the project required funds for supplies or a teaching assistant beyond the first year, the proposal had to address the source of future funds. As already mentioned, only those applications that proposed to develop active, discovery-based projects were awarded grants.
 |
Figure
2: Students in the Groundwater HydroGeology course are conducting a
pump test at water wells drilled on an environmental farm to determine
the permeability of an aquifer. |
Because faculty members from many disciplines are convinced that having students actively participate in research-like activities is a valuable pedagogical tool, they continue to maintain and develop such courses. Numerous faculty members who received no extra pay also attended the assessment workshop and revised their courses to include more research-like activities. However, even small infusions of money can facilitate the development of research-like activities and send the message that the institution values this kind of curricular innovation. Oberlin College continues to award a number of curriculum development fellowships each year that can be used for a variety of development efforts, including the development of curriculum that adds research-like activities to courses.
A Workshop to Teach Faculty How to Do Course Assessment
Although faculty members were informed that assessment should be an integral part of their
AIRE-sponsored curriculum development activities, we realized that most faculty members were not familiar with how to do assessment. We therefore developed an Assessment Techniques Workshop to help them learn how to assess the impact of the research-like activities they incorporated into their courses, or any other curricular innovation. In fact, effective curricular innovation requires that curriculum development and assessment be intertwined. Educational goals need to be clearly explicated and reliable assessment tools need to be developed to evaluate how well they have been achieved. Such assessment can also reveal avenues for future improvement.
Although diversity in faculty applicants and proposals is wonderful, it can create challenges for assessment. Proposals came from many faculty members, both within and between many different departments and the diversity of activities they proposed was tremendous. In addition, there was a wide range of motivation for assessment and knowledge of assessment principles and practice among these individuals. It was because of these challenges that we developed and conducted the assessment workshop.
The workshop was designed to bring faculty members to an understanding of the importance of assessment and of the basic principles of assessment. We also wanted to teach faculty members some basic assessment techniques and to help them develop assessment tools for their own curricular innovations. We realized that faculty members might resist the additional work created by assessment and might find assessment threatening when the success of their innovations was not certain. Consequently, an additional purpose of this workshop was to demonstrate the benefits of assessment work. As we indicated at the beginning of our workshop, we believed that relatively small effort could yield useful information about an innovation's effectiveness and ways to improve its impact. If we could get faculty members to appreciate and participate in assessment, we would gain information on the value of research-like experiences in courses, and we could firmly establish the assessment process so it persists at Oberlin College long after AIRE has run its course.
Organization of the Workshop: The Workshop was a short, one-day workshop help during summer, soon after classes ended. Faculty members whose proposals had been selected for funding were strongly encouraged to attend and were paid for their time. Other faculty from the College of Arts and Sciences were invited but not compensated. Faculty participants were told that this would be an active workshop during which they would develop their tools for assessment and were asked to bring in a list of educational goals for a course they wanted to assess.
The workshop began with a discussion of the purposes of assessment and of the participants' likely concerns about assessment. We said that the workshop was a relatively painless way to learn assessment principles and techniques. We emphasized that the first step in assessment is the clarification of educational goals and we described how these goals could be specified as measurable outcomes. We described how assessment tools could be used to measure these outcomes.
After discussing an example as a group, the participants engaged in a number of hands-on activities. First, they designed outcomes to be assessed for an imaginary course. Then, using their goals for one of their own courses, they designed outcomes and assessment tools for that course. Examples of these activities are outlined below.
Clarifying Educational Goals and Measurable Outcomes: Curricular innovation occurs because the course instructor either wants to introduce new educational goals or to develop new ways to help students achieve the existing goals. The first step of assessment involves the clarification of these educational goals. Once clarified, these goals must be further specified in ways that can be measured. For example, an educational goal might be to have students learn experimental design and analysis, and a measurable outcome might be that students can detect design flaws in an experiment.
We recognized that curricular innovations are often initiated with only general expectations of improvement, and that more specific outcomes are often not apparent until the end of the course. But we noted that such post facto, often unsystematic, evaluations preclude the use of procedures such as pre-tests and post-tests that can supply important information about the impact of the project. So we urged participants to clarify at the outset their reasons for the curriculum changes. Once they had recast their reasons as educational goals, finding measures of their achievement would not be difficult.
We emphasized that there are often multiple reasons for curricular innovation. It is seldom that we wish students to acquire only information. We also wish them to develop positive attitudes and intentions with regard to what we teach. For these reasons, we usually have multiple outcomes we want to evaluate, for which we need to design assessment tools.
We suggested that participants think in terms of four types of measurable outcomes - changes in attitudes, intentions, knowledge and skill. Attitudinal outcomes are so named because they involve students’ beliefs about course material and their feelings toward it. In general, we want our students to find the course material interesting, stimulating, and thought provoking so that they will feel positively toward it. The second type of outcome involves students’ intentions to use the material or seek out similar experiences in the future. Intentions could range widely, from application of ideas outside the classroom to the pursuit of additional courses, an academic major or even a career. One reason why student attitudes are important is because they can influence intentions.
The third type of outcome involves changes in knowledge. We want students to learn something they didn’t know before and to think differently about the material - less like they did at the beginning of the course and more like experts. Of all outcomes, this is the one most commonly evaluated through examinations and student research. However, for some outcomes, only pre- and post- tests can determine the impact of a particular course. The fourth type of outcome involves skill acquisition, including procedures for information collection, analysis and problem-solving, along with oral and written communication skills. As with knowledge outcomes, we grade student skills and therefore know, at least implicitly, what the desired outcomes are.
Therefore, the first step in developing assessment materials was to specify the educational goals and measurable outcomes that were relevant to the specific curriculum development project. Their specification would involve making implicit evaluation criteria explicit, so that these could be used to develop assessment tools.
Assessment Tools: We asked participants to think about two types of tools, those involving student self-reports and those involving instructor evaluations, and stressed the value of using both. While attitudes and intentions are subjective states that could be estimated by student self-reports of beliefs and feelings, attitudes could also be reflected in future course choices, which the instructor could investigate. Changes in knowledge and skill could be estimated by relatively objective tests, like midterm or final evaluations, or they could be estimated by student self-evaluation of learning in this course, relative to learning in other areas. When multiple measures yield similar answers, we have greater confidence that the educational goals were achieved.
We asked participants to remember four important points when constructing assessment tools. First we stressed the use of multiple items to measure each outcome. Rather than asking one question to measure an outcome, a handful of related questions could be asked. Answers to multiple items are more reliable than answers to a single item. Second, we stressed the use of multiple types of assessment tools to measure each outcome, because each supplies different estimates of change. Self-reports provide a subjective view and instructor evaluations a relatively objective one. Third, we encouraged the use of pre-test and post-test measures whenever possible, because this test pair can assess how much each student changed relative to his or her individual starting point. And finally, because changes in attitude, intention, knowledge, and skill could occur for a variety of reasons, we suggested the use of a relevant comparison group whenever possible. If there were two sections of a course and research projects were introduced in only the first, then the second could serve as a comparison group.
Hands-On Activities: Once we had covered the principles behind the development of curriculum goals, measurable outcomes, and assessment techniques, we discussed a detailed for a hypothetical curriculum development project (see Appendix A). We then split into small groups that each developed some measurable outcomes for a hypothetical course called General Introduction to Science. The course was described as an application of science to real world problems, such as conservation of environmental resources, disease prevention, and improvement in racial/ethnic relations. The five goals for the course were to learn applications of science to real world problems, to learn scientific procedures for addressing research questions, to learn computer-aided data analysis, to apply critical thinking in evaluating information, and to increase interest in applied science. Each group developed outcomes to be assessed and appropriate assessment methods for two of these goals. When they were finished, we reconvened as a large group to share and discuss the participant's measurement ideas. During lunch, one of the workshop presenters reported the assessment data from her course, as an example of the information that can be gained by using the principles discussed in the workshop.
Following lunch, groups of two, using ideas from the morning session, developed assessment tools for their own proposed curriculum development projects. Participants had brought a list of educational goals for a course they wished to modify/develop. Members of each group started by describing the course goals and explaining why those goals had been chosen. They went on to develop measurable outcomes and assessment tools for the stated goals. The large group reconvened one last time to address any unanswered questions. By the end of the workshop, participants had heard many examples of assessment efforts and had "enjoyed" two opportunities to practice developing assessment tools.
Impressions of the Workshop: The workshop appeared to be successful in a number of ways. Faculty members already familiar with assessment principles were encouraged to develop assessment tools and use them. These participants enjoyed talking with other faculty members about their goals and outcome measurements. Faculty members who were not knowledgeable about assessment learned the importance of making course goals explicit before planning an assessment strategy and became familiar with some basic tools to implement the strategy. These participants gained an appreciation of the value of assessment and appeared to be more willing to assess their own courses and to assist in our plans for a comprehensive assessment. Moreover, all the participants got to know each other better and learned more about the courses that their colleagues in other departments were teaching.
Fifteen faculty members attended the Assessment Workshop. Nine of them were faculty members who had been awarded AIRE curriculum development grants. The AIRE recipients who attended have all since participated in assessment of their curricular innovations.
Assessing Research-Like Experiences in Courses: Comprehensive Assessment Plans
The faculty workshop on assessment was only the first step in our assessment program. We have instituted a more comprehensive assessment plan to learn about the general impact of integrating research-like experiences in undergraduate courses. Because the AIRE has allowed us to introduce so many new research-like experiences into so many courses in such a relatively short amount of time, we have an unparalleled opportunity to assess the impact of these curricular innovations on the knowledge, skills, intentions, and attitudes of students. To achieve this comprehensive assessment, we have developed a set of instruments to use across the
AIRE-sponsored courses in the various departments. This overarching assessment meets three needs. First, we will be able to achieve a level of integration with our assessment that is impossible with the disparate approaches used by individual faculty. Second, we will be able to close any gaps in individual course assessments. This will allow us to draw general conclusions about the impact on students of integrating research-like experiences. Third, we will be able to ensure that we have developed sound, reliable instruments because we will have a database large enough to do psychometric analysis. Once we have developed reliable instruments, these instruments can be used by individual course instructors at Oberlin College and at other institutions as models for assessing courses with research-like activities.
Our comprehensive assessment includes, for those AIRE-sponsored courses initiated prior to our assessment efforts, a post-course survey of students about their research-like experiences within each course. For all other
AIRE-sponsored courses, the assessment includes a pre-course/post-course survey of changes in students' attitudes, knowledge, intentions, and skills, and a survey of the
AIRE-sponsored faculty regarding the design and the impact of their course innovations. We have also completed an experimental study on whether a lecture plus research-like experience versus a lecture-only experience differentially impact various aspects of learning such as knowledge level and retention.
Post-Course Surveys of Students: The post-course questionnaire consists of items that assess students’ attitudes, knowledge, skills and intentions (See Appendix B, items 2-27). In addition, students were asked to rank the contribution of several components of the course to their overall learning and enjoyment. Lastly, students were asked to rate the contribution of the research-like project to their learning experience. We designed the post-course questionnaire so that it could be used for any course - specific wording describing the actual research projects for a particular course is added to the introductory heading.
Pre-Course/Post-Course Surveys of Students: A major inadequacy of the post-course survey data is that it does not contain any information about the students’ knowledge and attitudes at the beginning of the course. The pre-course/post-course assessment method enables us to measure the change in students’ attitudes, knowledge, intentions, and skills over the semester. The pre-course/post-course questionnaire is a modification of the post-course questionnaire. For example, the question, “The research component of this course provided insight into issues in this area of study,” became, “I believe that research can provide insight into issues in this area.” Or, the question, “This research project taught me how to collect data systematically,” was changed to, “Rate your ability to collect data systematically.”
We now have pre-course/post-course surveys on 233 students in 11 classes distributed amongst 8 different academic departments. Based on current cumulative data from the pre-course and post-course surveys, the sections assessing students’ attitudes, knowledge, intentions and skills are reliable with alphas of 0.85 and 0.86 respectively. Factor analysis suggests that the items are measuring 5 distinct factors: 1) students’ value of research, 2) their research methods skills, 3) their data management and statistics skills, 4) their research plans and 5) interest in the discipline. Students in courses with integrated hand-on, research-like experiences generally report a significant positive growth in their value of research, their research methods skills, and their data management and statistics skills. However, they do not report a significant development in research plans or interest in the discipline. Based on their ratings of research components, students seem to learn the most from interpreting data, working closely with the instructor, data analysis and finding relevance in the research. They learn the least from library research, planning and doing presentations.
Even though the primary purpose of the comprehensive assessment is to get a general understanding of the impact of integrated research experiences, we have also begun to examine differences between categories of courses with different kinds of research involvement. For example, some courses conducted laboratory research while others engaged in field research. Some courses focused on the collection of data while others focused on data analysis. Instructors are being interviewed each semester about their course goals, activities, and desired research outcomes, so that we can determine the differential impact of different kinds of research activities.
Faculty Survey: A version of the faculty questionnaire is contained in Appendix C. The first part of the survey will allow us to better characterize the course goals, measurable outcomes, and assignments. In addition to characterizing the courses, we plan to use the information to modify the student questionnaires for each course. The second part of the faculty questionnaire included items that cover the instructors' assessment of their
AIRE-sponsored research projects. In this part we include knowledge, skills, intentions and attitude questions. Finally, we ask faculty to estimate the proportion of student learning from the research-like experiences relative to the other course activities. We are currently in the early stages on this part of the comprehensive assessment plan.
Experimental Studies on the Effects of Research-Like Experiences on Learning: Lastly, our assessment plan includes empirical studies of the amount of student learning in research plus lecture versus lecture-only conditions. We have completed a laboratory study designed to assess the impact of integrating research activities in lecture-based courses in psychology. The study participants were presented information in three formats. The first format was modeled after a traditional lecture; participants worked through a PowerPoint presentation of a lecture on a phenomenon, taking notes, as they would normally do for a lecture. For the next format, participants worked through the same lecture, and then participated as a subject in an experiment demonstrating the phenomenon. For the final format, participants also worked through the same lecture, then used a simple software package to design an experiment demonstrating the phenomenon and then participated in their experiment. At the end of the study, participants were given a quiz with knowledge-based questions very much like those that one might write for a course exam. We hypothesized that active participation in a research-like experience would enhance knowledge acquisition. We found that participants who designed an experiment fared significantly better in recall questions than the other participants, but there were non-significant differences in scores for recognition questions. We are replicating this study in physics. Concurrently, we are controlling for material-exposure time in a study that compares the differential impact on memory and understanding of the concepts of student participation in research experiences versus completion of additional reading.
Costs of Assessment: If faculty members undertake assessment of their own classes, assessment can be a relatively simple, inexpensive process. However, if a professional assessor is hired, assessment can be a much more elaborate, expensive process. Choosing between these will depend upon the goals of the assessment program. We decided that we did not need to hire a professional assessor from outside the college, but we did engage an outside consultant to get some advice early on in the process. Since 2001, our comprehensive assessment plan has been supervised by a faculty member and administered and analyzed by an assessment intern who is a recent graduate of the Psychology department. The faculty member is part of the Psychology Department. Most faculty members from Psychology have the training needed to carry out an assessment program, as do faculty members in education, and the staff of the Office of Institutional Research. In addition, upper-class psychology students have been hired to analyze the assessment data. Therefore, our costs for assessment have been minimal. When you plan to complete comprehensive assessment in a wide range of courses, you not only need to think carefully about how the evaluation will be done, but about evaluation personnel.
Final Reflections
Our comprehensive assessment is now well under way. We can already draw some conclusions about the general impact of integrated research-like activities on undergraduate education. Our pre/post surveys of students have indicated that the introduction of research-like activities in courses increases the value that students put on research, and also increases their perceived skills in research methods, data management, and statistics. Our experimental studies indicate that research-like activities increase students’ knowledge acquisition. In the future, because the
AIRE-sponsored courses contain such a wide range of research-like activities across many disciplines, we will try to clarify any connections between different kinds of research activities and specific student learning.
The Award for the Integration of Research and Education has given Oberlin College an unprecedented opportunity to “prime the pumps” of curricular innovation and assessment. We anticipate that this will benefit students and faculty for many years to come.
The Institution
Oberlin College is a select liberal arts college in northern Ohio, with approximately 2,800 students: 2300 in the College of Arts and Sciences and 500 in the Conservatory of Music. The college was founded in 1833 and has had a long-standing progressive tradition. For example, in 1835 Oberlin became the first college to admit black students and in 1837 was the first men's college to enroll women as degree candidates.
Oberlin College has a rich tradition of supporting the integration of research and education. As far back as the 1880s, professor Frank Jewett inspired and aided an undergraduate student Charles Martin Hall in research that led to the invention of the modern extraction method for aluminum. Since the 1970s, Oberlin College has placed a strong, systematic emphasis on the integral connection between research and teaching. According to the Faculty Guide, the ability to teach is the quality most prized in members of the faculty and scholarly and creative activities are essential to continued teaching excellence. In most natural and social science departments, this scholarly and creative activity takes the form of research.
One of the primary ways that research and education are integrated is through student-faculty collaborative research. In the process of mentoring student in research, faculty members maintain their scholarly excellence and also teach students about the research process. At Oberlin, students can do independent research with a faculty member as part of an honors project, for independent research credit, for summer pay, or for Winter Term credit during January when no formal classes are scheduled. Many students benefit from these opportunities. According to a 1998 survey of graduating senior science students, 76% reported having conducted science research at Oberlin. A systematic evaluation of Oberlin students involved in research projects from 1985-1989 indicated that collaboration with faculty on a research project had a profound and positive effect on students and numerous students called the experience the capstone of their undergraduate education. A 1998 survey of 1993-1997 psychology alums found that over 90% rated independent research as very good to excellent preparation for both graduate school and careers. No coursework consistently scored this high. Student-faculty collaborative research is well grounded at Oberlin and it is anticipated that it will continue to be.
Because student-faculty research is both time and resource intensive, it is difficult to offer student-faculty research opportunities to all students. However, a modified version of student-faculty research can be offered to the majority of students if active, discovery-based, research-like activities are incorporated into courses throughout the curriculum. These experiences are also important for scientific literacy. In the U.S. scientific community there has been a growing awareness of the importance of having both science majors and non-science majors experience doing scientific research to gain a personal understanding of the nature of science (e.g. Project Kaleidoscope, 1999). Also, doing research-like activities strengthens students' involvement in discovery processes and it is hoped that this will foster learning, positive attitudes toward research, and interest in course-related knowledge. The incorporation of research activities into lecture/discussion and laboratory courses is an important part of natural and social sciences education at Oberlin. For example, in 1996 80-100% of faculty members from each department of the sciences said that they had integrated inquiry-based exercises into at least some of their courses.
Acknowledgements
This work would not have been possible without the assistance of the National Science Foundation and its Award for the Integration of Research and Education.
References
Celebrating and anticipating a decade of reform. Project Kaleidoscope (PKAL) tenth anniversary publication, 1999.
Appendix A
Integration of Group Research Projects into the course Psychology of Close Relationships
Educational Goals
1. To enhance appreciation of psychological research as a source of knowledge about relationship processes and problems.
2. To increase intentions of doing further work involving social science research.
3. To impart elementary research skills.
4. To impart knowledge about close relationship processes.
Measurable Outcomes
Estimates of goal achievement will be based on the following course outcomes.
1. Student attitudes -- especially changes in attitude -- toward the role of research in understanding relationship processes.
2. Student intentions about taking future courses involving research or doing research in other ways.
3. Student research projects and their use of archival research, hypothesis construction, measurement, sampling, statistical analysis, and interpretation of data.
4. Student knowledge about close relationships, especially from the research project.
Assessment Tools
Student Attitudes (Measured before the course and at the end of the course).
Use the following scale to rate how much you agree/disagree with each of the following statements.
1 (Extremely disagree) 2 3
4 5 6 7 (Extremely agree)
1. Research about close relationships provides insight into reasons for interpersonal problems.
2. Research about close relationships can help people improve their relations with others.
3. Research about close relationships does not tell us anything more than common sense does.
4. It is impossible to make generalizations from research on close relationships because everyone is different.
5. Knowledge about close relationships is better acquired from experience than from research.
Student Intentions (Measured at the course's end).
Use the following scale to rate how much you agree/disagree with each of the following statements.
1 (
Extremely disagree) 2 3 4 5 6 7
( Extremely agree)
1. I am planning to take social or behavioral science courses in the future.
2. I am thinking about majoring in social or behavioral science.
3. I am planning to take at least one course in which I can learn more about research methods.
4. I will probably take an independent research course before graduating from Oberlin.
5. I will probably look for a research job or internship before graduating from Oberlin.
Students' Knowledge Acquisition (Measured at the course's end)
1. Please estimate the contribution that each of the following course components has made to your learning about close relationships.
___ % from lectures
___ % from reading assignments
___ % from my group research project
___ % from class discussions
___ % from student presentations
___ % from other (Please specify)
3. Please indicate how much you learned about your group research topic, relative to other topics that we covered in this course.
1 (Very much less) 2 3 4 5 6 7
(Very much more)
4. Please rate the importance of each of the following in what you learned from your group research project.
1 (Not at all important) 2 3 4 5 6 7
(Extremely important)
a. Doing the library research
b. Designing the study
c. Collecting the data
d. Analyzing/intepreting the data
e. Doing the presentation
f. Writing the report
i. Other (Please specify)
Instructor evaluation of knowledge and skill acquisition
1. Evaluation of the research project.
a. Was the research question/hypothesis developed from the research literature?
b. Were reliable and valid measures used?
c. Was an adequate sample selected?
d. Was data collection systematic?
e. Were appropriate statistics used?
f. Were results precisely and concisely described?
g. Were results made related to the research literature?
h. Were results qualified by limitations of the method?
2. Final exams: Knowledge of project-related questions versus non-project-related questions.
Appendix B
AIRE Assessment: Post-course Questionnaire
Oberlin College was recently honored with an award from the National Science Foundation for our past efforts to integrate research into our educational experience. You recently took a course that was funded, in part, by this award. As part of the award, we have agreed to do assessments of your experiences in this course. The
AIRE-sponsored course that you took was [Course XXX]. We are particularly interested in your reactions to the research
project[s] on [brief description of the AIRE-sponsored project or projects]. The following survey is part of our assessment efforts. Your time and effort in filling out this survey is greatly appreciated.
Name:________________________________________ Mailbox:__________
How much research experience have you had in and outside the classroom? (Count each course/experience separately.)
None 1 sem 2 sem 3 sem 4 sem or more
Doing the research project in [Course XXX] provided insight into relevant issues in the field represented by this course.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, I believe that doing research has practical value to the field.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, I believe that research is applicable to the area of study represented by this course.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, I believe that doing research can tell us more than common sense in this field.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, I believe that most research results can be generalized beyond the specific circumstances of a particular project.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, I believe that knowledge about this field is better acquired from doing research than from personal experience.
Not at all 1 2 3 4 5 Very much
I believe that doing research in this course contributed to my appreciation of research.
Not at all 1 2 3 4 5 Very much
I believe that doing research in this course contributed to my interest in research.
Not at all 1 2 3 4 5 Very much
As a result of my involvement in the research project in this course, I am comfortable with my research abilities.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, how much did you learn about the concepts taught in this course.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, rate your ability to describe the concepts in this course.
Not at all 1 2 3 4 5 Very much
As a result of my research experience in the course, I now feel like I learn more from courses that do not contain a research component.
Not at all 1 2 3 4 5 Very much
Please rate the contribution of the research experience in this course to your skills in each of the following areas.
Critical thinking skills.
No Contribution 1 2 3 4 5 Very Large Contribution
Developing research questions/hypothesis.
No Contribution 1 2 3 4 5 Very Large Contribution
Developing reliable and valid measures to address research questions/hypotheses.
No Contribution 1 2 3 4 5 Very Large Contribution
Selecting an adequate/appropriate sample for the purposes of doing research.
No Contribution 1 2 3 4 5 Very Large Contribution
Ability to collect data systematically.
No Contribution 1 2 3 4 5 Very Large Contribution
Applying appropriate statistics to analyze a set of data.
No Contribution 1 2 3 4 5 Very Large Contribution
Knowing how to describe results.
No Contribution 1 2 3 4 5 Very Large Contribution
Identifying and acknowledging the constraints of methods used in a research.
No Contribution 1 2 3 4 5 Very Large Contribution
In the future, I plan to take more courses of this type.
Definitely not 1 2 3 4 5 Definitely
In the future, I plan to take more courses within this department.
Definitely not 1 2 3 4 5 Definitely
I am now considering majoring in this field.
Definitely not 1 2 3 4 5 Definitely
Check here if you are already a major: _______
In the future, I am interested in taking at least one course in which I can learn more about research methods.
Definitely not 1 2 3 4 5 Definitely
I will probably do/already am doing some kind of independent research or research assistance.
Definitely not 1 2 3 4 5 Definitely
I will probably look/am already looking for a job or internship in which I can get experience doing research as an undergraduate.
Definitely not 1 2 3 4 5 Definitely
Please rank from 1 to 7, with 1 being the HIGHEST, the contribution of each following components with regard to how much you learned from each and how much you enjoyed each.
Rank for Overall Learning Rank for Enjoyment
Research projects _______ _______
Textbook and other readings _______ _______
Lectures _______ _______
Studying for exams _______ _______
Paper-writing _______ _______
Classroom discussions _______ _______
Discussion with other students _______ _______
Rate these components of doing a research project in terms of their contributions to your learning experience.
Working independently Unimportant 1 2 3 4 5 Extremely Important
Working in small groups Unimportant 1 2 3 4 5 Extremely Important
Working closely Unimportant 1 2 3 4 5 Extremely Important
with professor
Doing library research Unimportant 1 2 3 4 5 Extremely Important
Designing experiments Unimportant 1 2 3 4 5 Extremely Important
Collecting data Unimportant 1 2 3 4 5 Extremely Important
Analyzing data Unimportant 1 2 3 4 5 Extremely Important
Interpreting data Unimportant 1 2 3 4 5 Extremely Important
Writing a report Unimportant 1 2 3 4 5 Extremely Important
Planning and setting-up Unimportant 1 2 3 4 5 Extremely Important
presentations
Doing presentations Unimportant 1 2 3 4 5 Extremely Important
Applying knowledge Unimportant 1 2 3 4 5 Extremely Important
to a context relevant to oneself
Appendix C
AIRE Assessment: Faculty Questionnaire
Name:______________________________________________________
Is (are) the course(s) for majors or nonmajors?
What are the pre-requisites for the course(s)?___________________________________
What are the goals of your course(s)?
Which of the following research skills were you intending for students to learn?
Critical thinking
Developing research questions or hypotheses
Developing/using reliable and valid measures to test hypotheses
Selecting appropriate samples
Collect data systematically
Apply appropriate statistics
Describe results
Identify or acknowledge constraints of methodologies
Any others________________________________________
Which of the following components were present in your course?
Research projects (other than your AIRE-sponsored project)
Textbook or other readings
Lecture
Exams
Papers
Oral presentations given by students
Class discussions
Small group discussions
Review sessions given by you
Supplementary activities
Participation as subjects in research
Homework
Others___________________________________
Which of the following components were present in your AIRE sponsored class project?
Independent work
Small group work
Individual work with you
Library research
Designing studies
Collecting data
Analyzing data
Interpreting data
Writing report of data
Oral presentation of data
Applications of data to one's own life
Others_____________________________________________________
If you have already taught your course:
Are you planning to offer your course again with the AIRE sponsored research component present in the course?
Overall, how successful do you think the AIRE-sponsored research component was for reaching your course goals.
Very Successful Somewhat Successful Unsuccessful
What changes would you make to improve the success of the AIRE-sponsored research component?
Did the inclusion of the AIRE-sponsored research project in your course change your attitude about including research projects in other courses?
Did the inclusion of the AIRE-sponsored research project in your course teach you anything new about pedagogy?
NO
Yes if yes, What?
Did the inclusion of the AIRE-sponsored research project in you course require the development of any new teaching skills?
NO
Yes if yes, What?
What percentage of your students overall learning about research came from each of the following components of your course? There are four components:
Research projects
Textbook
Lecture
Discussion
Copyright
© 2003 Council on Undergraduate Research. All rights reserved.
|
|
Prev:
Next:
