"For people without disabilities, technology
makes things convenient,
for people with disabilities, it makes things possible."
Evaluation Team Activities
- Formative Evaluation for Broadband Accessibility Study
- ASL Video Evaluation
- Summary of Interface Evaluation Activities
- Additional Evaluation Initiatives :
Literature Review and Expert Survey
Formative Evaluation for Broadband Accessibility Study
The purpose of the formative evaluation component is to obtain user input during the development process through a simple methodology with results that will inform design (i.e. problems, sources of error, barriers to access, learnability). The evaluation team is undertaking an analysis of the entire project and providing relevant feedback to the tools development team. Specific evaluation protocols for evaluating player, editor functionality and user interface are to be utilized. Activities are as follows:
- For usability testing video documentation is utilized. Short fifteen-minute clips of users performing representative tasks will be analyzed.
- Heuristic evaluation using Jakob Nielson’s methodology is based
on expert users evaluation of design based on prescribed usability heuristics,
with specific focus on disability issues for each heuristic.
(http://www.useit.com/papers/heuristic/heuristic_evaluation.html)
ASL Video Evaluation:
A portion of the evaluation for this project has looked at the addition of an American Sign Language (ASL) video translation of an existing educational video and video setting preferences of individuals who use ASL for communication. The objectives of this evaluation were:
- to determine what enhancements to ASL translation are effective to viewers
- to examine the effects of different ASL presentation formats on learning
- to inform design of courseware software
In order to accomplish these objectives, the team presented ASL-speaking participants with two different ASL versions of the same educational video, The Law, from the Room for Five series. One of the videos was a direct interpretation of the law video to ASL while the other video was an acted ASL interpretation of the video. Our goal was to see if learners preferred one format over the other and to see if either format had an effect on their learning.
Our participants were also provided opportunity to arrange the ASL and original video appearance through settings options. These options allowed the viewer to make one video window larger than the other video window, to set the position of the two videos relative to each other (i.e. top, bottom, side, near, far) and to choose to place borders around the video windows.
Participants provided information to the evaluation team through pre and post questionnaires, signed feedback and test scores. Additional sources of information to the team include video tapes of the sessions and key-stroke data that records use of the video controls by the participant.
With the help of experimenters and translators from the Canadian Hearing Society, we have been able to collect the relevant data and are in the process of analyzing the information that has been collected. Test scores show no trend towards better learning with either type of translation, however all of the data must be examined before the potential effect of each option is fully understood. At the completion of the analysis we expect to have a better idea of the kind of ASL adaptation that is most cost-effective and valuable to learners. We also expect that the design of the learner interface will reflect the needs and preferences of learners who are deaf particularly with regards to the degree of control the learner will have over video size and positioning.
Summary of Interface Evaluation Activities:
- Two different ASL versions of the Room for Five learning video were produced with CHS and Marble Media. The first version is acted ASL where the ASL equivalent of the video is provided by actors. The second version is an ASL interpretation of the video material.
- The team participated in the definition and specification of enhancements required for the original physics video material.
- The team participated in defining and specifying the user interface requirements for the editor and player.
- An evaluation template to evaluate the various ASL video production methods was designed and an ethics application was submitted.
- Ethics approval was given to the evaluation process.
- Training materials were created and two experimenters were trained to carry out the experiment: one experimenter in Sudbury and one in Toronto.
- Sixteen ASL subjects completed the study.
- Data analysis of the ASL component has commenced
- The evaluation team will continue to monitor the literature and incorporate new results into current work.
- Developed specific evaluation protocols for evaluating player, editor functionality and user interface.
- Provided player interface feedback to developers in preparation for planned usability evaluation
Additional Evaluation Initiatives:
Literature Review and Expert Survey
An expert survey on distance education practices, recommendations and concern for accessibility of content has been undertaken. This process includes inter-rater reliability of the survey data. A paper based on the findings is being prepared.
In addition, a literature review of distance education practices and accommodations for accessibility has been completed. The evaluation team are continuing to monitor the literature and incorporate new results into current work and recommendations.
This project is funded by CANARIE
Inc. - Learning Program
