fp2010_acmProceedings

Gameplay Evaluation of the Trackball Controller

Daniel Natapov

I. Scott M acKenz ie

Department of Computer Science and Engineering

York University, Toronto, Canada

{dnatapov , mack}@cse.yorku.ca

ABSTRACT

We present a study of user performance in a First Person Shooter

game comparing a prototype trackball controller, a standard game

controller, and a keyboard and mouse. The prototype controller

replaces the right analog stick of a standard game controller (used

for pointing and camera control) with a trackball. To measure the

performance of the three input devices, participants played two

games. Penguin Hunt measured the number of target hits per

minute, which was 28.1 with the keyboard and mouse, 22.9 with

the trackball controller, and 21.7 with the standard controller.

Modern Warfare measured average completion times, which were

26.8 s with the keyboard and mouse, 31.8 s with the trackball

controller and 35.5 s with the standard controller. The trackball

controller represents a 5.5% target hits increase over the standard

controller in Penguin Hunt, and a 10.4% speed-up in trial

completion time in Modern Warfare.

Categories and Subject Descriptors

H.5.2 User Interfaces. Input devices and strategies.

General Terms

Design, Experimentation, Human Factors.

Keywords

Game controllers, trackball, joystick, analog stick, pointing

devices, performance evaluation, video games, game input

1 INTRODUCTION

Game playing on the PC offers a different experience than game

playing on a console. Not surprisingly, there is considerable

Internet debate on which is better. The differences are numerous,

ranging from insignificant to major. For example, the TV screen

used with consoles is generally larger than the PC’s smaller

screen. In terms of performance, a good PC has more processing

power and better graphics than a console. In terms of price,

consoles are generally less expensive than a PC. The list of

differences goes on, but an important difference, which is the

focus of this research, is the input method.

PC games are played using the keyboard and mouse, on a

supporting surface such as a desk. Console games are played using

handheld console controllers, either sitting or standing, and

without a supporting surface. Console controllers come in a

variety of types. The standard input device is a gamepad, such as

the Microsoft Xbox 360 controller, or the Sony PlayStation 3

controller. We refer to these as “standard controllers”, as they

have similar arrangements of buttons, triggers, and analog sticks.

Another method of input is motion sensing, such as the Nintendo

Wiimote or the upcoming Microsoft Project Natal and Sony

PlayStation Move. Other methods of input include specialty

controllers, such as guitars, mats, guns, etc.

This paper focuses on the difference between PC controls and

standard game controllers. We are not investigating specialized

controllers or motion sensing controllers, as they are task-specific

and often used for tasks of a different nature than standard PC

controls, making for an invalid comparison. On the other hand, PC

controls share a lot of similarities with standard controllers. Both

types are generally used for similar genres of games, with the

same games often playable both on consoles and the PC. Both

types of controllers are also similar in their use of buttons for

input. The main differences between the two are the analog thumb

joysticks used by gamepads, which are not present on PC controls.

Analog sticks became standard on game controllers when gaming

became three-dimensional. The first analog stick was introduced

on the Nintendo 64 controller in 1996. Analog sticks are useful for

3D games, since they allow for motion in more than eight

directions (as offered by the directional pad). A second analog

stick was added, since in 3D games camera position and

orientation is player controlled. The second analog stick was used

to pan the camera in a larger number of directions than possible

with buttons, and at varying speeds. At present, in most 3D games,

and First-Person Shooters (FPS) specifically, the left analog stick

is used for character movement, while the right analog stick is

used for camera control and pointing. On the PC, the

W, S, A, and D

keys are used for motion, while the mouse is used for camera

control and pointing.

The left analog stick performs well compared to the

W, S, A, and D

keys. It specifies a direction of motion, and allows for movement

in more directions and at varying speeds. The right analog stick,

on the other hand, performs poorly compared to the mouse for

pointing and camera control. The mouse specifies a position, while

the analog stick specifies a panning direction, with the former

being faster and more accurate. In terms of throughput, which is a

measure of pointing speed and accuracy [6], the mouse is vastly

superior to the analog stick – the throughput of the analog stick

was measured to be 61% lower [7]. The purpose of our research is

to bridge the gap between gamepads and PC controls in terms of

pointing.

Using the mouse is not feasible with gamepads, as a mouse is

operated on a supporting surface, while gamepads are handheld.

Instead, we created a prototype which replaces the right analog

stick of a gamepad with a trackball (Figure 1). Earlier, we

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided

that copies are

not made or distributed for profit or commercial advantage and that

copies bear this notice and the full citation on the first page. To copy

otherwise, or republish, to post on servers or to redistribute to lists, re-

quires prior specific permission and/or a fee.

FuturePlay @ Vancouver Digital Week 2010, May 6-

7, 2010,

Vancouver, Canada.

167

978-1-4503-0235-7/10/05…$10.00

evaluated the trackball controller in terms of pointing in a Fitts’

law experiment [8] and found that the trackball controller offers

approximately a 60% increase in throughput over the analog stick

for novice and advanced gamepad users. We also found that cursor

path was less direct with the trackball as compared to the analog

stick. The previous study involved a simple pointing task, and as a

consequence a number of questions remain unanswered. It is

uncertain whether the controller offers performance benefit in a

realistic task, as opposed to a pointing task (as pointing and

camera control are similar but not identical). It is also unclear

whether a less direct cursor path has a detrimental effect on

camera control.

Figure 1: Trackball controller

To address these questions, this study compares the trackball

controller to a standard controller and a keyboard and mouse, in a

realistic gameplay task. Participants used the three input methods

in two FPS games: Penguin Hunt [3] and Infinity Ward’s Call of

Duty 4: Modern Warfare [1]. Our hypothesis is that while the

keyboard and mouse are still the best method of input

(performance wise), the trackball controller will yield an

improvement over the standard controller. Our aim is for this new

prototype to bridge the gap between standard gamepads and PC

controls, offering an alternative that is better for camera control

and pointing, but is still usable in a handheld fashion.

1.1 Related Work

Klochek and MacKenzie presented five new performance metrics

to quantify differences between video game controllers [5]. The

experiment compared an Xbox gamepad and a standard PC mouse

in a target-tracking task. Performance was compared according to

five cursor path metrics. The significant finding was that the

mouse allowed participants more control over acceleration than

the gamepad, which in turn helped correct errors in position. The

trackball will also allow more control over acceleration than the

analog stick.

Isokoski and Martin report on a number of studies comparing

performance of input devices for FPS games. In one, they

compared an Xbox 360 controller (gamepad) to a mouse and

keyboard combination for use in FPS games [4]. Participants

played a simple FPS game called Penguin Hunt. The number of

targets hit in a trial was used as the measure of performance.

Results showed that the keyboard and mouse combination

performed better in terms of the number of target hits than the

Xbox 360 controller. In a similar study, they compared the

Xbox 360 controller to a keyboard and mouse, and to a keyboard

and mouse aided by eye tracking [2]. As before, performance was

measured in terms of target hits. The keyboard and mouse again

performed better than the Xbox 360 controller, which confirms

their previous result. The eye tracking assisted keyboard and

mouse performed on par with the Xbox 360 controller.

In the following section, we describe our methodology. The goal

was to compare the prototype trackball controller to the standard

controller and to the keyboard and mouse, in terms of performance

in FPS games.

2 METHOD

2.1 Participants

The problem with comparing our prototype to existing devices is

the confounding influence of participants’ prior experience. It is

reasonable that participants will perform better with the device

with which they are familiar. Unfortunately, as no participants

could be experienced with our prototype, it created the risk of a

biased comparison. As a consequence, we initially planned to only

use participants who had no experience with FPS gaming using

any input device in order to eliminate prior experience as a

confounding variable.

Unfortunately, such a participant pool turned out to be unusable.

We performed a pilot study with novice game players, who had no

experience with either input method, and found that the task of

playing even simple FPS games is too complex for novices to

obtain any clear results during a short study. Participants were

frustrated, confused, and often unable or unwilling to complete the

task. The resulting data had too much variance from participants

learning the task; the variance overshadowed any differences

resulting from the input methods used.

As a result, we used participants who were advanced FPS players,

despite this biasing the comparison against our prototype. As a

consequence, the differences presented here are understated. For

the final study, we recruited 18 participants, self assessed as

advanced FPS players (on either consoles or the PC). Participants’

adeptness at FPS gaming was also validated based on the task.

Some participants who performed below expectation, or were

outliers, were removed. In the end, we used the scores of 16 of the

18 participants (discussed below).

Prior to beginning, all participants completed a pre-experiment

demographic questionnaire. A question asked whether participants

had previously played Call of Duty 4: Modern Warfare. Although

all participants were advanced FPS players, they ranked their

platform experience on PCs and consoles, and with gaming in

general using the following categories:

 Beginner (use/play never or rarely)

 Intermediate (use/play occasionally)

 Advanced (use/play often)

Participants were recruited from the local campus and the local

community, and paid for their participation. We divided

participants into three groups, based on platform experience:

players advanced on consoles, players advanced on the PC, and

players advanced on both. Each group had 6 participants for a total

of 18 participants. The average age was 23.2 (SD = 4.13). On

average, participants took one hour to complete the study. Eleven

of 18 participants stated they previously played Modern Warfare.

168

2.2 Apparatus

2.2.1 Hardware

The experiment was conducted on a Sony VAIO laptop running

Windows 7. For the keyboard and mouse conditions, the laptop’s

keyboard was used with a USB mouse (Logitech Laser). A

Microsoft Xbox 360 wired controller was used for the standard

controller condition (connected through USB). For the trackball

controller we built a prototype using a Microsoft Xbox 360 wired

controller and a wired Logitech Trackman Wheel trackball. We

used the Xbox 360 controller as the housing for the prototype so

that the form factor for both is the same, except for the trackball

replacement.

The prototype controller replaced the right analog joystick with a

trackball. The optical camera and original housing of the trackball

were glued to the underside of the top half of the Xbox 360

controller. The trackball was placed in the housing, and a plastic

lid was attached so that the trackball was moveable but secure.

The rest of the trackball, excluding all unnecessary parts (such as

button and scroll wheel controls), were attached to the bottom of

the Xbox 360 controller, where the battery pack is on a wireless

controller.

The resulting prototype is seen in Figure 1. The prototype

connects to the PC with two USB cables: one for the trackball for

mouse motion, and the other for the game controller for all other

inputs.

2.2.2 Software

The first task involved shooting targets in a game called Penguin

Hunt. This game was previously used to measure the FPS

performance of input devices [2, 4]. We used the same software

for our analysis so that our results will be comparable to previous

work. The game world in Penguin Hunt consists of a square-

shaped randomly generated terrain, with different height levels,

and randomly placed trees. The game also contains 10 randomly

placed penguin targets, which move slowly in a random direction.

The goal is to shoot the targets. Trees block the movement of the

player and are also impenetrable to the weapon. When a target is

shot it disappears and another appears at a random location.

Ammunition is unlimited. A screen shot of the game is shown in

Figure 2.

Figure 2: Screenshot of Penguin Hunt

For the standard controller condition, the left analog stick was

used for movement, while the right analog stick was used for

turning/aiming. The right trigger was used for shooting. The

developers of the software reported that the angular rotation

velocity was 270 degrees per second, following the Xbox version

of Halo. Following the instructions of the Xbox controller

programming manual, the displacement-velocity transfer curve

was cubed [4]. For the trackball controller condition, the left

analog stick was used for movement, and the right trigger for

shooting, exactly like in the standard controller condition. The

trackball was used for turning and camera control. For the

keyboard and mouse condition, the

W, S, A, and D keys were used

for movement and the mouse was used for camera control. The

left mouse button was used for shooting. The mouse and trackball

had the same sensitivity, and pointing acceleration (gain) was

turned on for both.

Each game trial lasted 120 s. The top right corner of the game

screen showed the remaining time, and the number of targets hit

so far. Hits and misses were accompanied by sounds to provide

auditory feedback. The game logged the number of targets hit, the

distance between the player and the target when a target was hit,

and the number of misses.

The second task involved playing the PC version of Infinity

Ward’s Call of Duty 4: Modern Warfare, run at a resolution of

1366  768. This commercial game was chosen due to its training

level. Part of the game’s introduction involves the player

completing an army-like obstacle course called the cargo ship

mockup. The mockup involves navigating a “ship”, moving from

one station, or “position”, to the next and completing objectives in

each. Each position involves some offensive action, such as

shooting cardboard targets, or throwing flashbangs (blinding

grenades) into rooms. In total, there are 6 positions, 2 of which

require throwing a flashbang, and 10 cardboard targets spread out

across the positions. The entire event is timed, with the goal to

complete all objectives as quickly as possible.

Looking directly at a thrown flashbang blinds the player

(represented by the screen going white for a few seconds, and

sounding a ringing noise). This has a negative impact on

completion time. To avoid being blinded, the player either moves

behind a wall or pans the camera away from the blast area.

Overall, this level was a good comparative task as it is

representative of actions in FPS games, while being short and

identical between trials. Specifically, it involves navigation,

camera control, aiming and shooting, taking cover (to not be

blinded by flashbangs), sprinting, etc.

Completion of the obstacle course represents a trial, which

participants repeated multiple times. Before a trial, participants

restocked on flashbangs and ammo. The gun held 30 bullets,

which was enough to shoot all targets without reloading, but

reloading was still possible. At the end of a trial, the game

displayed the time to finish the course and an accuracy score from

0 to 3. The accuracy score is a “black box”, in that we do not

know how it is calculated. The accuracy score is subtracted from

the time to obtain the final time. In essence, the final time

represents both speed and accuracy. Having no access to the game

code, we manually recorded the time and accuracy of each trial.

Figure 3 shows a screenshot of the cargo ship mockup level.

We used the PC version of Modern Warfare, with the default

control configuration used for the keyboard and mouse condition.

Specifically, the

W, S, A, and D keys were used for movement,

while the mouse was used for camera control. The right mouse

button was used for looking down the sight of the gun (zooming

in) and the left mouse button was used for shooting. S

HIFT was

used for sprinting,

R for reloading, and ‘4’ for throwing

flashbangs.

169

Figure 3: Screenshot of Call of Duty 4: Modern Warfare

To use the standard and trackball controllers with the PC version

of Modern Warfare, we used an emulator called XPAdder [9],

which allows emulation of game controller input as mouse and

keyboard input. For the standard controller condition, we used the

default key configuration of the Xbox 360 version of the game.

Specifically, we mapped the left analog stick to movement (

W, S,

, and D) and the right analog stick to camera contro (mouse

motion), with the left trigger for aiming down the sights and the

right trigger for shooting. Pressing down the left analog stick was

used for sprinting, ‘

X’ was used for reloading, and the left

shoulder button was used for throwing flashbangs. For the

trackball controller condition, XPAdder was configured exactly as

it was for the Xbox 360 controller, except for aiming and turning,

for which the trackball was used instead.

XPAdder allows setting the analog stick sensitivity when

emulating a mouse. The emulated sensitivity needed to match the

sensitivity on the Xbox 360 version of the game. To that end, we

compared the turning velocity of an emulated controller on the PC,

to a controller on the Xbox 360 version of Modern Warfare. We

set the sensitivity so that turning 360 degrees by maximum analog

stick displacement takes as long with the emulated controller on

the PC as on the Xbox 360 game.

Despite our attempt to emulate the Xbox 360 controller on the PC,

the mapping was not exact. The Xbox 360 version of the game has

a number of features, or “handicaps”, which are not present on the

PC version of the game. Specifically, aiming down the sights, or

zooming in, on the Xbox 360 version of Modern Warfare slows

down the camera panning speed, but it does not do so on the PC

version. Additionally, pulling the left trigger to aim down the

sights “locks on” to the nearest target (positions the crosshead on

it) on the Xbox 360 version of the game, but not on the PC version.

That Infinity Ward includes targeting assistance mechanisms on

the console version of the game, but not on the PC version,

indicates that developers are aware of the shortcomings of

standard controllers in terms of accurate pointing.

2.3 Procedure

The purpose of the experiment was explained to participants. They

were then asked to give informed consent for their participation

and fill out a pre-experiment questionnaire. The first task was

playing Penguin Hunt. Before each control scheme was used,

participants were given a 120 s practice trial to familiarize

themselves with the controls. The results of the practice session

were discarded. Participants performed four recorded trials of

120 s each, with each controller.

After completing all trials on Penguin Hunt, with all controllers,

participants switched to Modern Warfare. Participants were given

a number of practice trials to familiarize themselves with the

game, the goals, and the obstacle course. These familiarization

trials were not recorded. Once participants were reasonably

familiar, recording started. The results of the first trial with each

controller were also not recorded, as they too were considered

practice trials. Participants performed 10 recorded Modern

Warfare trials with each controller. Each trial took approximately

30 s, which allowed us to have more trials than with Penguin

Hunt. After completing both tasks, participants were asked to fill

out a post-experiment questionnaire.

2.4 Design

The Penguin Hunt part of the experiment employed a 3  4  3

mixed design. There were three independent variables:

 Input Device (PC, standard, trackball)

 Trials (1 to 4)

 Experience Group (PC, console, both)

Input Device and Trials were within-subjects variables.

Experience was a between-subjects variable. Input Device was

counterbalanced according to a Latin square. Each trial lasted

120 s. In total, participants performed 4 (Trials) × 3 (Input Device)

× 3 (Experience Groups) × 6 (participants per group) = 216 trials.

The dependent variables were hits (count), misses (count), and

distance to target (pixels).

The Modern Warfare part of the experiment employed a 3  10 

3 mixed design. The independent variables were:

 Input Device (PC, standard, trackball)

 Trials (1 to 10)

 Experience Group (PC, console, both)

Input Device and Trials were within-subject variables. Experience

was a between-subjects variable. Input Device was

counterbalanced according to a Latin square. Each trial consisted

of a completed cargo ship mockup level run. In total, participants

performed 10 (Trials) × 3 (Input Devices) × 3 (Experience

Groups) × 6 (participants per group) = 540 trials. The dependent

variables were final time (seconds) and accuracy (score).

3 RESULTS AND DISCUSSION

3.1 Outliers

After completing the obstacle course in Modern Warfare, the

game provides a recommended playing difficulty, based on the

player’s final time. The recommended difficulties are as follows:

 Veteran (Expert): If the final time is under 20 s.

 Hardened (Advanced): If the time is under 26 s.

 Regular (Intermediate): If the time is under 40 s.

 Recruit (Novice): If the time is over 40 s.

We feel that this recommendation is accurate and we used it to

validate participants’ self-assessed proficiency. To be included in

the study, each participant had to reach the Hardened difficulty on

at least one trial, with any input device. In other words, at least

one trial had to be completed in less than 26 s.

All but one participant met this requirement, which indicated that

overall, participants’ self-assessment was accurate. The one

participant (from the advanced PC players group) who did not

meet the requirement, and was likely not an advanced FPS player,

was disqualified. We removed the participant’s scores from the

170

data in order to reduce variability. Additionally, a player from the

advanced console players group was removed due to being an

outlier. The participant had scores which were more than 3

standard deviations away from the mean. As a consequence, the

data in the results are based on 16 participants instead of 18: five

players advanced on the PC, five players advanced on consoles,

and six players advanced on both.

3.2 Penguin Hunt

3.2.1 Hits

We first present the results for the three experience groups

combined. Though a trial lasted 120 s, we present the results as the

average number of hits per minute. Other research has used

different trial lengths with Penguin Hunt, so reducing the score to

a ratio makes it more comparable. The results are shown in Figure

4. All figures show standard error bars.

Figure 4: Penguin Hunt hits per minute, across the four trials.

Overall, the number of hits per minute using the keyboard and

mouse was 28.1. This is 18.5% higher than the trackball

controller’s 22.9 hits per minute, and 22.8% higher than the

standard controller’s 21.7 hits per minute. Comparing just the

console controllers, the trackball controller represents a 5.5%

improvement over the standard controller. The difference between

the input devices was statistically significant overall

2,30

= 37.63, p < .0001), but a post hoc Tukey-Kramer test

revealed the difference to only be significant between PC controls

and standard controller, and between PC controls and trackball

controller. The difference between the standard controller and the

trackball controller was not significant. The order effect was also

not significant (F

2,30

= 0.33, ns), indicating that counterbalancing

was effective. It is not surprising that the keyboard and mouse

outperformed both console controllers. It was the expected result

since, in terms of throughput, the mouse (4.5 bps) is known to be a

faster and more accurate pointing device than the trackball

(3.0 bps) and the analog stick (2.0 bps).

Based on the results, participants not advanced with PC controls

still performed well with the keyboard and mouse, but participants

not familiar with the standard controller performed poorly using it.

This may be due to participants’ familiarity with the keyboard and

mouse for non-game tasks. Advanced PC players had an average

of 26.9 hits per minute with PC controls. Advanced console

players also performed well with PC controls, and had an average

of 27.6 hits per minute. This indicates that it is easy to learn to

play with the keyboard and mouse.

On the other hand, while advanced console players performed well

with the standard controller, and had an average of 26.0 hits per

minute, advanced PC players struggled with the standard

controller, and had a lower average of 18.2 hits per minute. This

indicates that it is more difficult to use the standard controller

without practice. The Experience  Input Device interaction effect

was significant (F

2,30

= 11.83, p < .0001), with the Tukey-Kramer

test revealing that there was a difference between console players

and PC players using the standard controller, but no difference

between the two when using PC controls.

The gap between the trackball and standard controllers is not as

large as was hoped. This stands to reason, as two thirds of our

participant pool were experienced with the standard controller, but

none were experienced with the trackball controller. Despite this,

the trackball controller had a higher average of hits per minute

than the standard controller.

Based on the figures in previous research using Penguin Hunt, the

average number of hits per minute using the Xbox 360 controller

was between 13 and 18 [2, 4], lower than our average of 21.7. On

the other hand, using PC controls, participants in previous research

averaged between 23 and 28 hits per minute [2, 4], which is

comparable to the 28.1 hits per minute observed with our

participants. The difference in scores with the Xbox 360 controller

may be due to prior participant experience. Two thirds of our

participants were advanced players using standard controllers. The

experience level of participants in previous research is not

specified.

To better illustrate how each group of participants performed with

the various controllers, we provide separate analyses based on

experience group. The results for the advanced PC players are

seen in Figure 5 (left). The average number of hits per minute was

highest with PC controls, at 26.9. This is not surprising, since

participants in this group were only experienced with keyboard

Figure 5: Penguin Hunt hits per minute per group: advanced PC players (left), advanced players on both systems (middle), and

advanced console players (right).

171

and mouse gaming. The trackball controller had an average of

21.3 hits per minute, which was higher than the standard

controller’s average of 18.2. The trackball controller represents a

17.0% improvement over the standard controller for participants in

this group. The difference based on Input Device was statistically

significant, (F

2,8

= 54.16, p < .0001), and the Tukey-Kramer test

revealed that it was significant between all pairwise comparisons.

This result is important, as it indicates that participants who are

unfamiliar with either standard console controllers or the trackball

controller, perform significantly better with the trackball

controller. The value of 18.2 hits per minute obtained with the

standard controller is comparable to the range of results seen in

previous research [2, 4], which indicates that their participants

were not advanced players with standard controllers.

The results for participants who were advanced with both consoles

and the PC are shown in Figure 5 (middle). PC controls once

again perform best, with an average of 29.5 hits per minute.

Participants again performed better using the trackball controller

than the standard controller, with 23.4 hits per minute and 21.1

hits per minute respectively. The difference is statistically

significant (F

2,10

= 34.83, p < .0001), but the Tukey-Kramer test

revealed no significance between the trackball controller and the

standard controller. Though significance was not obtained

between the two controllers, on average, participants performed

better using the trackball controller, despite being advanced

players with the standard controller.

Finally, we analyzed the results for participants who were

advanced console players. These are shown in Figure 5 (right).

Participants in this group did not perform best with the standard

controller, which is surprising considering their experience.

Instead, the highest number of hits per minute was obtained with

PC controls: 27.6. Average hits per minute were 26.0 with the

standard controller and 24.0 with the trackball controller. The

difference was statistically significant (F

2,8

= 12.33, p < .005), but

a Tukey-Kramer test revealed significance only between the PC

controls and the trackball controller. Despite participants in this

group being only advanced with standard controllers, they did not

perform better with it. This alone indicates deficiencies with the

standard controller. It is likely that with equal experience the

trackball controller, as well as PC controls, will perform better

than the standard controller.

3.2.2 Misses

In addition to the number of hits, we counted the number of times

participants missed with each device. On average, participants

missed 6.6 times per minute using the keyboard and mouse, 4.1

times per minute using the trackball controller, and 6.2 times per

minute using the standard controller. The difference was not

statistically significant (F

2,30

= 0.96, ns).

3.2.3 Distance to Target

For each target hit, we logged the distance in pixels between the

target and player. We did not expect to obtain interesting results,

due to participants’ varying preferences and play styles, but a

relationship did surface. The results for distance to target per trial

are shown in Figure 6. The average distance to target was 130.9 px

using PC controls, 114.6 px using the trackball controller and

101.4 px using the standard controller. The difference was

statistically significant, (F

2,30

= 11.43, p < .0005), but the Tukey-

Kramer test revealed significance only between PC controls and

the standard controller.

Figure 6: Penguin Hunt average distance to target, per trial.

Targets appear smaller when they are shot from a distance. This

result signifies that participants felt most comfortable shooting at

smaller, farther away targets when using the mouse, and least

comfortable when using the standard controller. This in itself is a

statement about the different levels of precision and accuracy of

the three devices.

3.2.4 Summary

We have shown that advanced PC players, as well as advanced PC

and console players, perform better in Penguin Hunt with the

trackball controller than with the standard controller. We expect

that with practice the benefit of the trackball controller will grow,

and further increase the performance gap, even for advanced

console players.

3.3 Call of Duty 4: Modern Warfare

3.3.1 Final Time

We present the results for the final time in Modern Warfare next.

As noted, final time is the time for a participant to run the cargo

ship mockup level, minus the accuracy bonus. Faster (lower) times

are better. The average times per input device, across the 10 trials,

are shown in Figure 7.

Figure 7: Modern Warfare final time per input device.

On average, participants were fastest using PC controls, with an

average final time of 26.8 s. As hypothesized, the average time

using the trackball controller was faster than using the standard

controller. The average time using the keyboard and mouse is

18.6% faster than the trackball controller’s average time of 31.8 s,

and 32.5% faster than the standard controller’s time of 35.5 s.

Comparing just the console controllers, the trackball controller

represents a 10.4% speed-up in trial completion time over the

172

standard controller. The difference was statistically significant

2,30

= 14.50, p < .0001), and the Tukey-Kramer test revealed

significance between all pairwise comparisons. The effect of

group was not significant (F

2,30

= 1.64, p > .05), indicating that

counterbalancing was effective.

This is an important result. Despite none of the participants having

experience with the trackball controller, participants performed

better using it than the standard controller. There is also an

improvement across the trials for all devices, as participants

become familiar with the task. Indeed, the difference based on trial

is significant (F

9,135

= 7.41, p < .0001), with the earlier trials (1

and 2) being slower than the later trials (5 - 10). Despite obtaining

significance in the overall results, we analyzed the results

individually by experience.

Participants in the advanced PC players group, and unfamiliar

with either gamepad, performed better with the trackball than the

standard controller by a significant margin. The results are shown

in Figure 8 (left). The average time with the keyboard and mouse

was fast: 26.7 s. The average time with the trackball controller was

33.9 s, almost 8 s faster than the standard controller’s average time

of 41.7 s. The trackball controller represents an 18.7% speed-up

over the standard controller for this group. The difference was

statistically significant (F

2,8

= 3.61, p < .005), and the Tukey-

Kramer test revealed that the significance was between the PC

controls and the standard controller, and between the trackall

controller and standard controller. The difference between the

trackball controller and PC controls was not significant. This is

similar to the results obtained with Penguin Hunt; participants

who are unfamiliar with either console controller perform better

with the trackball controller.

The results for participants who are advanced players on both

consoles and the PC are shown in Figure 8 (middle). The same

pattern appears: the keyboard and mouse is fastest, and the

standard controller is slowest; but the gap between the conditions

is narrower than for the PC players’ group. The average time was

25.1 s with the keyboard and mouse, 30.8 s with the trackball

controller, and 33.4 s with the standard controller. The difference

was statistically significant (F

2,10

= 11.16, p < .005), but it is only

significant between the PC controls and the two console

controllers, and not significant between the trackball and the

standard controller. Despite this, as with Penguin Hunt, the

standard controller, on average, still performed worse than the

trackball controller, despite participants’ prior experience. We

expect that given equal experience, the trackball controller will

perform better by a larger margin.

The results for participants who were advanced console players

are shown in Figure 8 (right), and it is clear that no well defined

relationship exists. The average final time was 28.7 s for the

keyboard and mouse, 29.6 s for the trackball controller, and 30.3 s

for the standard controller. Not surprisingly, the difference is not

statistically significant, (F

2,8

= 0.52, ns). But the fact that

participants, who are advanced at using the standard controller and

nothing else do not perform any worse with other controllers, is

telling.

3.3.2 Accuracy

We recorded the accuracy bonus awarded per trial. Though we do

not know how the accuracy bonus is calculated (Infinity Ward has

not disclosed this information), we know that it is based on the

amount of misses, and that the minimum bonus is 0 and the

maximum bonus is 3. The results for the average accuracy bonus

are shown in Figure 9. As with other dependant variables, the best

performance and highest accuracy bonus was achieved using the

keyboard and mouse, and the worst using the standard controller.

The average accuracy bonus was 2.6 using the keyboard and

mouse, 1.7 using the trackball controller, and 1.1 using the

standard controller. The difference was statistically significant

2,30

= 41.51, p < .0001), with the Tukey-Kramer test showing

significance between all pairwise comparisons.

Figure 9: Modern Warfare, accuracy bonus per trial.

3.3.3 Summary

These results mirror the results from Penguin Hunt, with the

keyboard and mouse offering the best performance, the standard

controller offering the worst performance, and the trackball

controller offering midway performance. As before, the most

visible gap between the two console controllers is for participants

who are unfamiliar with both, while there is no difference for

Figure 8: Modern Warfare final time per group: advanced PC players (left), advanced players on both systems (middle), and advanced

console players (right).

173

participants familiar with one but not the other. When all groups

are combined, on average, participants were faster by 3.6 s with

the trackball controller than the standard controller. This may not

seem like a big improvement; however, it is important to

remember that participants were largely experienced with one

controller but not with the other. It is also worth noting that the

Modern Warfare task was complicated, with a lot of randomness

and variability, including luck, participant strategy, mistakes,

reload times, flashbang blindness, sprinting or walking, aiming

down the sights or firing from the hip, etc. The fact that despite all

these variables, and by changing only the method of aiming with

the controller, participants were able to shave 3.6 s off their time

(a 10.4% speed-up), while using an unfamiliar controller indicates

a considerable improvement indeed!

3.4 Qualitative Results

At the end of the experiment participants completed a

questionnaire. They were asked to rank the input devices in terms

of playing preference for each game. For Penguin Hunt, 15 out of

16 participants (93.8%) rated the trackball controller ahead of the

standard controller in terms of preference – an overwhelming

majority. For Modern Warfare, 11 out of 16 participants (68.8%)

rated the trackball controller ahead of the standard controller. It

appears that the increase in performance offered by the trackball

controller over the standard controller is also reflected in

participants’ preference of each.

4 CONCLUSION

This paper presents the gameplay evaluation of a prototype

trackball controller. The prototype replaces the right analog stick

of a standard game controller with a trackball. We compared the

trackball controller’s in-game performance to that of the keyboard

and mouse, and the standard controller in two game tasks. In

Penguin Hunt participants were asked to shoot as many targets as

possible with each controller, in a 120-second trial. In Modern

Warfare participants were asked to complete an obstacle course as

quickly as possible with each controller.

Results confirmed our hypothesis. In both tasks, PC controls

offered the best performance, while the standard controller offered

the worst performance. The trackball controller provided a

performance compromise, bridging the gap between the two.

Considering just the standard controller and the trackball

controller reveals that the trackball controller offered better

performance in both games. The gap in performance was widest

for participants with no experience with either controller, and

narrowest for participants experienced with the standard controller

but not the trackball controller. This is not surprising, and we

expect that with practice, users in all experience categories will

perform better with the trackball controller than the standard

controller. Overall, the results conclusively show that our trackball

controller offers a significant performance improvement over the

current standard of game controllers. Though it does not offer the

accuracy and precision of a keyboard and mouse, it is fully usable

in a handheld fashion without a surface to operate on (as with a

mouse).

Additionally, though our experiment only involved First Person

Shooters, we are confident that the results are applicable to any

genre of games involving camera control or pointing, including

Real Time Strategy (RTS) games, and Role Playing Games

(RPGs). Since gaming is competitive and goal-oriented, we

believe that the observed improvement in performance is broadly

applicable to game playing.

5 ACKNOWLEDGMENTS

We thank Poika Isokoski and Benoît Martin for permission to use

the Penguin Hunt software. We also thank Andreea Izdraila for

her editorial help. This research is sponsored by the Natural

Sciences and Engineering Research Council (NSERC) of Canada.

6 REFERENCES

[1] Infinity-Ward, Call of Duty 4: Modern Warfare: Activision.

[2] Isokoski, P. and Martin, B., Eye tracker input in first person

shooter games, Proceedings of the 2nd conference on

Communication by Gaze Interaction. Turin, 2006, 76-79.

[3] Isokoski, P. and Martin, B., Penguin Hunt, 2006.

[4] Isokoski, P. and Martin, B., Performance of input devices in

FPS target acquisition, Proceedings of the International

Conference on Advances in Computer Entertainment

Technology. New York: ACM, 2007, 240-241.

[5] Klochek, C. and MacKenzie, I. S., Performance measures of

game controllers in a three-dimensional environment,

Proceedings of Graphics Interface 2006. Toronto: Canadian

Information Processing Society 2006, 73-79.

[6] MacKenzie, I. S., Fitts' law as a research and design tool in

human-computer interaction, Human Computer Interaction 7,

1992, 91-139.

[7] Natapov, D., Castellucci, S. J., and MacKenzie, I. S., ISO

9241-9 evaluation of video game controllers, Proceedings of

Graphics Interface 2009. Toronto: Canadian Information

Processing Society, 2009, 223-230.

[8] Natapov, D. and MacKenzie, I. S., The Trackball Controller:

Improving the analog stick, Proceedings of FuturePlay 2010,

Accepted for publication.

[9] XPAdder, http://xpadder.com. Accessed September 2009.

174