To: Students in Psychology 384, Research Laboratory in Human Factors

From: Susan Brennan and Tony Weaver

Date: Thursday, April 2, 2009

Re: Final Project (Presentations due Tues. 5/5; writeups due before the final exam, Th May 14)

In class, we’ve critiqued bad designs. We’ve talked about good design methodology and evaluation techniques. Now it is time to put all these concepts into practice. You will work in small groups to design, build, and evaluate a spoken dialog system similar to the ones you reverse-engineered for Assignment #6.

In order to do this, you will be using various tools from BeVocal Café located on the web at: http://cafe.bevocal.com/, where you already have an account. You only need one account per group, so choose one group member’s account and provide the login information to all members of your group.

The next thing you need to do is to brainstorm together to choose a promising application domain for your spoken dialogue system. Your group should consider several different possibilities and take notes about your discussion. Once you have a domain in mind (or if you cannot agree on one!) check with one of us to make sure that what you have picked is feasible. Many of you complained that the dialogue systems that you reverse-engineered for Assignment #6 were poorly designed. Now’s your chance to design one properly!

Milestones: There are several milestones that your group will need to meet in order to stay on track. For each of the first 3 milestones, one of your members should turn in a short memo in behalf of the whole group.

1. Thursday, April. 16: First progress report due, end of class (describe your application domain).

2. Thursday, April. 23: Second progress report due, end of class. Say what you’ve done for your task analysis and who your users will be, describe how you approached your design (scenarios and flow chart), report progress so far on your BeVocal program, and preview how you plan to evaluate your system).

3. Tuesday, May 5 AND Thursday, May 7: Project presentations in class (presenters for the 5th will be randomly chosen if there are not enough volunteers)

4. Thursday, May 14: Last chance to turn in final report.

During each of your group meetings, have at least one person take notes. You will need to document the process of how you analyzed the task, created flow charts and scenarios, designed the system, evaluated it, and improved it. This information should be reported in one of the preliminary reports of how the project is going and then summarized for the final project.

Part 1: Analyzing the task. Review course material on task analysis (Assignment #8). Then analyze your task by observing or interviewing several potential users of the system to see how they accomplish that task now.

Part II: Design. Next, come up with some alternatives to support this human activity. Since this is a spoken dialogue system, your interface will use an agent metaphor. Discuss some different agent metaphors and decide what kind of agent your system will be (examples: reservations agent, advisor, coach, guide, salesperson, etc.). Sketch out some alternatives for screen designs and discuss the pros and cons. Decide on the scope of the task (what the agent can do for the user). Then draw some alternative flow charts to represent the dialogue you are designing (or key parts of it). Discuss the flow charts and reach a consensus on an initial design. Pay attention to all the details: what feedback do users get that things are on track, what happens when there’s a problem that needs to be repaired, what happens when they do something unexpected, etc. Review relevant handouts and your writeups for previous assignments.

Part III: Prototyping. Next you need to develop a prototype. To create the prototype, visit the BeVocal Café located on the web at: http://cafe.bevocal.com/. Have your group call up and try out all 3 of your members’ VoiceXML programs from Assignment #7. Then get to work creating your own dialogue design. Assign different members of your group to work on different parts of the dialogue or aspects of the prototype—e.g., messages (what the user will hear), grammar (what the user will say), designing different parts of your flow chart. Stay in close touch with each other—email your team whenever you get something to work on BeVocal and have them try it out and give you feedback. Keep Nielsen’s 10 heuristics in mind as you critique the system; take notes about any problems or weaknesses you discover. After you’ve all tried to break the system, discuss your experiences and come to a consensus about how to improve the system. Then make the necessary improvements.

Part IV: User study or heuristic evaluation. When your group is reasonably content with the design of your prototype, conduct either a user study OR a heuristic evaluation and invite others from the class (who are not in your group) to participate either as naive users OR as heuristic evaluators. For a user study, have one person in your own group act as observer and scribe. This person should present your group's prototype to the naive user (see Gomoll’s steps), who will then try it out and try to find problems. The observer/scribe should keep a record of the testing session and the data collected. Watch what the naive users do. Get them to think out loud. Note any errors or confusions. If you’ve given them a task to do, note how long it took, as well as any errors. At the end, collect additional data by getting them to critique your system (with an interview, questionnaire, etc.). If you do a heuristic evaluation, review the procedures for Assignment #2; ideally you should have at least 5 volunteers act as users or evaluators (see Nielsen’s articles and the lecture slides on this). Combine your evaluators’ findings into an array to identify the problems that need to be fixed (ask one of us for help if need it), or else summarize the findings from your users in a systematic way. While one (or two) of your group members is busy with the user study or heuristic evaluation during lab time, you can lend the rest of your group to other groups to help with their evaluations.

Part V: Iterate. After the observers/scribes collect reactions from the naive users, or after the evaluators finish their reports, meet with your group to discuss the information you got from the naive users. Summarize the findings so far and the changes you need to make to your design, and then make them. If there is time, test your improved prototype again. Make sure that you take notes about the details of your original and final designs, your evaluation sessions, and what needed to be changed along the way to your final design, as you will need to discuss this in your project report.

Part VI: Final presentations and project report. Your group will produce a single and complete project report that summarizes all the phases of your project. You will present this orally in class (using PowerPoint slides) on Tuesday, May 5th and Thursday, May 7th. Everyone should take responsibility for part of the report, and everyone needs to make part of the oral presentation. Document all 5 parts listed above, and make sure to include the following info, summarized from your notes during your meetings:

1. A description/analysis of the task and potential users. Include a brief summary of how you believe your application is likely to be an improvement over other ways of doing the same task.

2. A description of your design and details about how you came up with it. Include flow charts. Also include any necessary documentation required to for us to understand or use your dialogue application, as we will be dialing it up. Discuss any tradeoffs you needed to make during the design process.

3. A description of your evaluation procedures. Include: How did you evaluate the prototype? Did you learn anything surprising or useful? Did different users/evaluators yield similar or different results, and were these different from what you expected? What revisions did you have to make to your original design and prototype? Etc.

4. At the very end of your report, include a statement breaking down who did (or wrote) which part of your project, and have all the group members sign it. Your project cannot be graded with this info.

*************************************************************************************************************

NOTE: Working as a group is an essential component of this project. Although you will split up the responsibilities among individuals in your group, the product must represent an integrated effort by the group as a whole. The most successful groups work together to ensure the success of each of their individual members; the whole group will lose credit if anyone is left out. Don’t forget to submit the signed group statement at the end of your project writeup (see #4, above).

**************************************************************************************************************