Overview
For my Advanced Usability Testing course, I had collaborated with three other students to conduct a usability study on the performance of the WMATA Fare Vending machine. Our aim was to examine this critical touch point in the user journey and suggest improvements to enhance user experience and in turn work towards promoting the adoption of a sustainable transport system. In order to suggest improvements, we wanted to gather user opinions on various factors ranging from mental model of the system during particular tasks to the overall experience. Our experiment consisted of task evaluation at the College Park metro station. After the task evaluation, users were asked to complete a post-study survey. We also observed users using the vending machines to understand micro interactions in a natural context.
Methodology
Our usability study comprised of (a) contextual on-ground observation (b) task evaluation and post-study survey. We conducted the study at the college park metro station, after seeking the permission of the station manager. We utilized the SmarTrip vending machine and the fare vending machine for the purpose of our test.
We had a total of 7 participants for the task based evaluation who were representative of most user groups, novice users, such as visitors and regular users, such as commuters who use the metro system very often. The test duration was around half an hour for each user, and the test session included user briefing, task completion, and post-test interview. Additionally, we observed 7 users using the vending machines to understand micro interactions in a natural context. Furthermore, we gathered field notes to assist data analysis.
To conduct the usability study, we took on the role of facilitators and observers during the user sessions. While completing the tasks, users were asked to apply the think-aloud method only if they were comfortable. We provided participants with this choice to ensure they are in their most comfortable state and prevent any bias from creeping in. After the task evaluation, users were asked to complete a post-study survey.
To capture nuanced information during the experiment, our team deployed video recording and captured photographs of the users using the system after receiving consent from participants. Five participants were part of the initial phase of our experiment. 3 out the 7 participants were new users of the WMATA vending machine. The other four were experienced users.  Participants were given reimbursements to the amount they spent during the task completion.

After completing the consent form we had our participants complete the 9 different scenarios. During the tasks we took notes, captured photographs and recorded the sessions which we later used for our analysis.

Our team came up with a list of recommendations for improving the overall experience with using both the Metro fare and SmarTrip machines.
Provide Additional Language Support
One of our recommendations for the SmarTrip machine is to give the user the option of selecting a language before completing any other task. Once the user selects his or her language, then the screen should only show instructions in that language.
Provide Flexible Payment Options
The SmarTrip machine currently appears to provide users with the option of purchasing a SmarTrip using either a credit card or cash. However, the machine only allows a user to pay for the SmarTrip with cash. If this is the case, then it should only display that the user must use cash to buy the SmarTrip. Otherwise, the machine needs to be fixed so that it is able to recognize when someone inserts a credit card into the slot of the machine.
Enhance Descriptive Labeling on the Interface
The affordances on the machine are not supported by the proper external signifiers or labels. For example, the machine doesn’t have any label that says a user should tap their card on the machine in order to see their available balance. Likewise, some users expected to receive coins from the receipt hole and vice versa. The proper use of labels will help eliminate any confusion.
Categorize the Instructions on top of the Kiosk
The fare machine should only show the user what they need to complete a task. Instead of showing all the instructions at the same time. For example, instead of showing the cost of both times (peak and off-peak) for a trip, our team suggests using LEDs lights or some indicator that would highlight how much a particular trip would cost based on the current time.
Provide Assistive Features To Reduce Task Completion Time
Time should be displayed somewhere on the interface so that users know what time is it as they are adding money to their cards.
One of the limitations of our study was that we only tested users at one Metro stop location. Many of the metro stops are unique in the usability of their machines and the average number of users that use the machines. For example, the SmarTrip machine at Union Station works with both cash and credit which contrasts with the results we found at the College Park Metro. Another limitation was that we conducted our study with few users and we had each user complete different tasks. 
Although we’ve obtained rich data from our observations, we limited ourselves by not having more users complete all of the tasks. With more users completing every task, we may have had the even better insight of the usability of the Metro Fare system. Metro users are normally on the go so it was challenging to get everyday users to do the “think aloud method” for example, while hastily adding money to his or her card. Instead, we had to have participants volunteer for our study and complete the tasks. While this turned out to be successful, it was also not natural. The tasks were taken out of context and this could have in fact skewed some of our data.
Furthermore, our usability test would have been robust had we received permission to analyze data collected via the security camera at the station, as it would have provided richer details. In addition, our usability test could have been improved by incorporating accessibility principles.
Conclusion
In our attempt to evaluate the WMATA vending machines, we used observation and task-based evaluation to collect qualitative data about the experience of using the vending machines. Our analysis of the responses from the user, observations and the media collected indicate that the fare vending machine is usable for an intermediate user with minor design flaws that delay the completion of a task. On the other hand, we realized the SmarTrip card vending machine was more difficult to use especially for new users. We have provided recommendations to improve the usability of both the fare vending machine and SmarTrip vending machines. Considering that the WMATA is discarding the use of paper fare cards, it is important for the SmarTrip vending machine to provide a good user experience. We believe that extensive usability testing is needed in the phase where old machines will be replaced with more usable machines.
Back to Top