 |
 |
 |
 |
 |
 |
 |
|
Project: Trailblazing for the World-Wide Web Student Researchers: Rachel Heck, Sarah Luebke, Weichao Ma, Hilary Mason Advisors: Samuel A. Rebelsky Institution: Grinnel College |
|
|
As is presumably the case in all CREW projects, there were both research goals and "personal" goals in the "Trail-blazers" research project. There was one primary research goal: to understand how people might understand, build, and use "trails", linked sequences of annotated pages, on the World-Wide Web. In support of that research goal, the students had a number of technical goals: to develop a file format for representing trails; to develop an interface or interfaces for presenting trails; to develop tools for building trails; and to do formative evaluation of the tools and interfaces.
On a more personal side, the students had goals of participating in a research project and learning not only research techniques, but also the joys and pitfalls of research.
Account of Process
Because the group had four student researchers, and the CREW grant only provided funding for three, an early order-of-business was to apply to the college for additional funds so that each student researcher could receive the recommended stipend of $1000. The faculty mentor was responsible for the proposal, which was funded through normal college channels. The faculty member was also responsible for communication with the CREW administration when their were delays in receiving funds.
With the funding arranged, our first real orders-of-business were to develop a plan and schedule and to determine roles within the research team. The team decided that weekly whole-group meetings (including the faculty sponsor) were appropriate, with additional smaller-group or "sponsorless" meetings when needed. Ms. Mason agreed to serve as director of the group, and Ms. Luebke volunteered to serve as "rappateur".
As is appropriate in all projects, we then reviewed the proposal that funded the project and began a literature search to identify related projects. Although the student researchers did read some of the papers referenced in the original proposal, they were initially not able to identify related papers. (It may also be that they wanted to move ahead in the project and I was perhaps too willing to let them do so.)
Our first goal was to identify the interface most appropriate for presenting trails. We began with discussions of potential interfaces for showing trails. These interfaces included a framed version, with "table-of-contents" on the left; a simple "next/previous" interface; a slightly richer "numbered pages" interface, which included a list of pages running across the top and bottom of each page; and a multiple-window interface, in which the table-of-contents was presented in a separate window. In preparation for discussions with the research subjects, the student researchers created a number of sample trails, and then manually developed each form of interface for each trail.
The student researchers then began some informal formative evaluation of the various interfaces using a variety of subjects culled from the college community. Rather than looking at particular numeric factors, such as which form of trail permitted readers to scan the material the fastest, the researchers decided to target more aesthetic qualities, asking subjects which they preferred, and determining which best indicate the "trail" quality to the clients. Before beginning these interviews (which are a form of research with human subjects), the researchers submitted a proposal to the Institutional Review Board which was approved. The faculty sponsor was responsible for much of this proposal, although the student researchers contributed to its development and provided the main questionnaire.
The evaluation revealed some surprising issues. For example, subjects who indicated that they didn't like frames in general nonetheless found the "framed with table-of-contents" interface appealing. In addition, many subjects preferred less information. In particular, few subjects wanted to see titles of all the pages in the trail. The research team developed a single interface, based on the information they had gathered from these subjects.
In the next phase, the students developed a format for "trail files" and then each student researcher chose a different development project to support the construction of trails. One student was responsible for turning "trail files" into a sequence of Web pages. A second was responsible for building a Web-based tool for constructing trail files. A third was responsible for building a tool that extracted links from an existing page and turned those links into a trail file. The fourth was responsible for support architecture, particularly support for downloading and manipulating files. The development and tuning of the tools took significantly longer than the students or the faculty sponsor expected.
During this development time, the student researchers also began to consider a number of other issues. In particular, they began to decide upon the next round of testing, which they decided would involve testing of the trail-construction tools. Towards the end of the development phase, they wrote a second proposal for the institutional review board. This time, the students were able to write the complete proposal, with little input from the faculty sponsor. Because they expected that the second study would involve more time and effort from the subjects, the researchers also began to look at mechanisms for compensating participants. They talked to a number of possible subjects, trying to identify appropriate compensation (enough to encourage subjects to participate; not too much, so that there were sufficient funds to support a number of subjects). They discovered that the compensation was more difficult than in many such experiments because of the odd funding of the CREW program. Typically, the institution would receive the funds and distribute them to the subjects when the researchers requested them to do so. Unfortunately, the CREW funds were handled from a separate institution, an arrangement not amenable to such small-scale funding. As a compromise, they settled on gift certificates from the college bookstore. This permitted mass purchase but individual distribution.
The tools neared completion at an awkward time; just before Spring break. It was clear that the college's long Spring break (two weeks) impeded progress. It was not possible to gather subjects before Spring break, and the fine-turning that was necessary right before Spring break took much longer than expected. In fact, assorted delays resulted in the interviews occurring too close to the end of the semester for all the recruited subjects to be available. Nonetheless, they gathered some interesting information, and the students expect to write a paper addressing their results in the near future.
The last half of the second semester also raised a number of related issues. The group gave an impromptu presentation to a visiting human factors scholar. The group identified some related research (Walden's Trails) and began to compare their own efforts and results to that related research. The group began to consider the form of the final report, and encountered the difficulties of developing a research report in the midst of other activities (e.g., classes, finals, work, the research itself). The group also began to develop a second CREW proposal, one intended to share their own good experiences by bringing less-experienced women into the research progress, and discovered the difficulty of coordinating grant writing and submissions within a group. The group also attempted to arrange funding from the Conference Experiences for Women program for travel to a conference on educational hypermedia, but were saddened to discover that that program no longer has funds. In the last week of the semester, the team also found an unexpected downside to their work: one of the trails that their subjects had constructed involved pages from another site, and the owners of the other site discovered the trail and objected to what the owners considered "thievery". Fortunately, a short explanation of the research project repaired the confusion.
Conclusions and Results
There are three main research outcomes from this project. First, we have evidence to show that both expert and novice Web users can understand the concept of trails, and can envision building and using trails. Second, we have some information on the interfaces that readers seem to prefer when using trails. Third, we have a number of tools that can be used for building and presenting trails on the World-Wide Web.
We do not consider our project complete. We clearly need more experience with "blazers" building new trails and readers following those trails. We also hope to do some comparative testing of the interfaces we and others have developed. We have applied to CREW for additional funding, but expect to continue the project in any case.
Lessons Learned
The students learned a number of important lessons about research and doing research. They worked on many of the technical skills necessary to be successful in research: writing grants, writing reports, coordinating subjects, meeting formal requirements (such as those of the Institutional Review Board), developing surveys, and otherwise building the infrastructure for research. Obviously, they have not yet mastered all of these skills, but they are on their way.
Many of the informal lessons will also stick with them. In particular, they learned that many things take longer than anyone expects, that it's difficult to coordinate many people and many parts of research, that it is important to work harder at identifying background materials, and that the rhythms of research do not always correspond conveniently to the rhythms of the academic year.
Other Outcomes
Happily, this was a good experience for all of the student researchers (and for the faculty mentor). All four indicate that they plan to go on to graduate school in computer science when they complete their undergraduate degrees. This summer, two of the students (Heck and Luebke) will be working on a related research project, one student (Ma) will be working on another research project at the college, and the forth student (Mason) will be participating in a research internship at AT&T labs.