September 2002
The player underwent several stages of evaluation. The initial stages were iterative and the resulting issue reports informed the ongoing development of the player tool. Testing with focus-groups enabled the investigators to identify further issues of access or player use that are specifically related to mode of access or disability. After further development, usability testing with specific user groups was carried out.
All of the usability test participants reported that they were daily computer users. Furthermore 12 of the 17 participants reported that they were somewhat (7) or very familiar (5) with their computer video player. Only five of the participants had taken a web-based course before. All of the participants were given three content questions (see p. 51) to answer that would force the participant to use different player features in order to find the answer. The results showed that all of the participants were able to try hyperlinks to additional material. However, linking for the blind participants was very difficult and brought forward some issues related to using the player with screen readers (e.g. inability to have links read out, difficulty accessing some windows). These were communicated to the development team. Most of the participants struggled with a formula content question although 15 of the participants did navigate correctly to the calculations content page that contained the link to the correct answer. Once again participants who were blind were not able to navigate to the answer. These findings show that participants (excluding those using screen readers) were able to successfully navigate through the course content and any difficulty finding the correct answer was more likely due to the question rather than user navigation abilities.
The content questions of the session helped us to understand how participants used the player features in their learning as well as to identify further uses for the video accommodations. For example, all of the participants used the captions to search for answers. A variety of methods were employed from stepping through the captions using the video controls, or by reading and/or searching the full text of the captions. Several participants who were not hard of hearing and therefore not reliant on captions reported that they liked having captions with the video. This finding demonstrates the widespread benefit of inclusive design where an accommodation such as captions for one user group has benefits for other groups as well. Transcriptions of the verbal protocol from each session were made and used to create issue reports for the development team. These reports include user likes, dislikes and difficulties with the player. Software is typically used in ways that is not foreseen by the developers and this protocol helps to inform the team of the rationale behind user decisions as well as expose difficulties experienced by the user (see pp. 52 & 53 for examples).
Authoring Tool Evaluation Evaluation Process
Our goal was to evaluate the usability of the authoring tool as described in section 2. Three stages of evaluation of the authoring tool were planned: heuristic evaluation, beta testing and usability testing. Unfortunately, time constraints restricted full usability testing of the authoring tool. The identified test group was educators who already utilize the internet for delivery of course material. A number of this group were unable to set aside the required time to review and try the authoring tool at the beginning of the school year. However, a heuristic evaluation and beta testing was carried out.
Methods Heuristic Evaluation
The evaluation of the authoring tool also followed the usability heuristics developed by Jakob Nielson (1994). These heuristics are outlined on page 42 of this report. Beta Testing Further evaluation of the authoring tool involved beta-testing of the player by usability experts. Researchers used and analysed the authoring tool in the role of educators or course developers. The analysis included usability heuristics, user expectations, and intuitiveness. From this data, an issue and bug report was created and given to the development team.
Results and Discussion
The evaluation of the authoring tool identified 12 issues (see p. 56). Once an issue was identified a suggestion for addressing the issue was made and reported to the development team. Although full summative user-testing could not be carried out within the project time-frame, initial evaluation shows that the authoring tool is not complicated to use and compilation of new and existing media tracks is intuitive. There were some issues around the use of technical language that confuse novice users. Changes to the current structure of the meta-data input were recommended. Meta-data is a particularly important component for the sharing of learning objects and requires precise language and categories. A simple but structured means to draw the relevant information from learning object creators that will then automatically populate the meta-data fields was recommended.
Impact on project
The results of all of the evaluations provided a significant contribution to the project. The formative evaluations for the player and the authoring tool provided valuable information during the development cycle when it was simpler and less expensive to make changes. Issues discovered during the evaluation process informed and helped developers to make design improvements and decisions that were best suited to users. It is often difficult for developers to predict user reaction to and usability of designs particularly when target user populations are so diverse. As a result of the evaluations, the player and authoring tools can be successfully used by people who are deaf, hard of hearing, vision impaired, learning disabled and others.
Impact on community
The evaluation carried out to explore alternative presentation and customized controls for American Sign Language translations of video material did not inform the design of the player or authoring tools but it did provide important information about customized controls and ASL preferences for video.
Shared with the partners in the Barrier-free project, The Canadian Hearing SocietyÕs mission is to provide services that enhance the independence of deaf, deafened and hard of hearing people and that encourage prevention of hearing loss. The implications of and results from the ASL portion of the Barrier-Free project, will greatly enhance the independence of the Deaf.
We were interested in finding ways that could better enhance the comprehension of online material for deaf individuals. Deaf individuals have a right to receive information in their native language Ð sign language. It is also their right to access video material that has the same richness and entertainment value as with an oral video. Many people assume that written material including closed captions is sufficient for the Deaf. In majority of cases this is not so and therefore we needed to explore options for access to video material.
The experimental results show that the Deaf are interested in new variations of video formats (rather than the conventional closed captioning). Having these formats available online will allow deaf individuals to have improved educational opportunities such as online distance courses. It will also allow content developers to produce effective ASL translations of their video material. It also demonstrated that customizable formatting for ASL display spaces is desirable and worthwhile including in player or presentation systems, although a complete cost-benefit analysis remains to be carried out to determine the most cost effective.
The Internet has become an essential communication tool for the Deaf. By developing and delivering web-based material in sign language the potential is immeasurable. Educational courses online could be translated into any sign language (over 100 around the world). Important government forms (often difficult to comprehend) could be explained in sign language with a glossary of difficult terms explained in sign language as well. Deaf individuals with lower levels of literacy often rely on interpreters, family, friends, volunteers and CHS counsellors to explain how to fill out forms such as the annual income tax forms. With explanations in sign language, these deaf individuals would rely less on others and increase their independence.
The impact on other communities of people with disabilities is also significant with the development and deployment of the player and authoring tools. Not only is the Internet important for people who are Deaf but also it is important to a wide range of communities who find access to traditional or conventional learning systems difficult or cumbersome. The player and authoring tool have been design and evaluated with various populations of users and found to be accessible and usable.
Allowing content designers and creators to easily produce accessible on-line learning material for many different types of learners increases the potential benefit to all users as evidenced by the use of the closed captions in our experiments. It also means that on-line content can meet legislative requirements and standards for accessibility.
