Kanjingo is a post-editing application for iOS devices developed at the ADAPT Centre (formerly CNGL) at Dublin City University (DCU). The first stage of user testing was con-ducted in 2014 (reported in O'Brien et al., 2014), and improvements were made based on the initial feedback. This abstract describes further exploratory testing based on this new iteration of the Kanjingo application. The new tests were designed with several aims: (1) testing Kanjingo usability; (2) testing Kanjingo interface design; (3) testing Kanjingo for revision of post-edited texts; (4) testing Kanjingo for post-editing with voice input and (5) testing Kanjingo for post-editing using the phone’s keyboard.
The testing was carried out at three spaces in DCU in July 2016, using a methodology adopted from Krug (2009). The first space was a testing room, equipped with an iPhone with the Kanjingo app, a webcam fixed to a ‘sled’ to record the subjects’ hand move-ments and audio, phone screen mirroring software (Mirroring 360 ) and a computer with Camtasia screen recording software to record the webcam and the phone mirroring activi-ty.
Some days before the test, five subjects filled in an online pre-test questionnaire on their professional profile and their relationship with machine translation (MT) and post-editing. During testing, the subjects came to the testing room, signed consent forms, and were handed task scenarios. The testing facilitor welcomed each subject and showed them the basic features of Kanjingo. Each subject was given a couple of sentences to post-edit as a warm-up exercise before beginning the test tasks. Once the subject understood how the app worked and could use it on his/her own, the first task was presented (revision of post-edited sentences), then the second task (post-editing with voice input), and finally the third task (post-editing using keyboard). Subjects were asked to comment on their experiences during post-editing (using Think Aloud Protocol), and comments were recorded via the webcam. After the test, subjects answered some general questions on the app and its use.
The second space was the observation room. The test was observed here in real time by the Kanjingo architect and a group of selected observers with research interests in transla-tion technology. The observation room was equipped with a large screen which mirrored the computer screen in the testing room. Audio from the testing room was streamed using Skype. Observers were provided with snacks and asked to provide reflections based on usability issues and ideas for further developments. The third space was a debriefing meeting room where the testing team and some observers met to discuss their notes im-mediately after the test.
This paper presents the results of the various tests, issues identified, and ideas for im-provements obtained. For example, the use of Kanjingo for post-editing with voice input, one of the most innovative forms of interaction with MT in the test, worked much better than subjects expected, and this mode of input was preferred for translating from scratch when MT quality was very poor, whereas post-editing short words or phrases was found to be faster with the iPhone keyboard. In addition, we present some reflections on the strengths and weaknesses of the testing methods employed.