Remote but real life

This article was first published in The Ergonomist

The need to develop new, and in many cases, life changing therapies has not stopped, and neither has the progress of developing medical devices and drug delivery combination products. Fundamental to making safe and effective medical devices and combination products is HF testing of the user interface. Should human factors testing have to stop, it is possible that many new medical devices and combination products would be delayed from reaching those in need.

Lockdowns and social distancing could not be allowed to halt HF testing, and through the internet, and video conferencing and collaboration platforms, HF engineers have found new ways to continue testing. Although these platforms have been used previously in limited ways, in essence we all had to start from scratch, and through trial and error learn new ways of operating.

Over the past year, having faced the new challenges and complexities of remote testing, we have overcome difficulties and gained some unexpected benefits. Here are some of the things we have discovered:

A rare opportunity to watch a device being used in real homes

A traditional usability study, conducted at a viewing venue, usually simulates just one home setting. Thanks to remote interviews, we have seen a spectrum of home settings such as living rooms, kitchens, bedrooms, and dining rooms. This has proven to be a rich source of insights, as we were able to observe participants use a device in a variety of situations, such as sitting on sofas; at cluttered tables or desks; or perched on the edge of a bed. Real-life distractions such as children interrupting, barking dogs, loud noises from outside and delivery drivers impatiently ringing the doorbell, only enhance the realities of actual home use.

This has enabled us to gather data from across a broader range of home settings, helping to evaluate and understand how a device’s user interface performs in a multitude of home use situations.

Enjoying the comforts of their own home

Participants will still experience some nerves and anxieties normally seen at a traditional usability study, but the comfort of a familiar environment and the removal of unknowns such as finding the venue, dealing with traffic problems or the vagaries of public transport, all help reduce the participants’ stress levels, helping them to  relax quickly and settle into the interview. Some participants felt so at ease when attending an interview, they turned up in pyjamas! A certain unintended indulgence many of us may have experienced while working from home in lockdown.

Whilst many participants may enjoy the benefits of being at home, they still have to perform in front of a camera, which can cause a small number of participants to remain anxious and hold a defensive frame of mind. For an experienced moderator, putting such participants at ease is a normal part of the introductory phase of the interview. However, now all the participant can see of the moderator is a face on their screen. No longer being in the same physical space, non-verbal communication such as eye contact, open body language and the impact of a smile is diminished, so more time and effort, on occasion, may be required to put the participant at ease.

Much greater flexibility to see the participants

At a viewing venue, the interviews have a hard stop. A parking ticket might be running out, there is a job to get back to, or a train to catch. This all puts pressure on the moderator to gather the data within an allocated time. Without such concerns, we have found there is often more flexibility with participants’ time. Combined with the additional contingency time built into the scheduling of the interview, we have found timing compromises that sometimes occur can be avoided. Occasionally, for example a participant can be particularly slow with tasks; in such cases we have often found, rather than forcing the pace of the interview to finish on time, the participant is able and willing to continue beyond their scheduled finish.

Rescheduling is also far simpler. When a viewing venue is involved, there tends to be just one opportunity to interview the participant. However, when done virtually, we have found participants to be very accommodating when it comes to rescheduling. Participants who would typically be lost if they cancelled or didn’t show up, are now just rescheduled into a vacant time slot. This flexibility has proven valuable as we have experienced a wide range of reasons for a participant to be unavailable at their allotted time, from power cuts to tropical storms. We even had a Californian participant explain they had to leave quickly due to an encroaching forest fire!

The logistics required to run a study have increased

Whilst remote studies can save time for participants, they do create extra logistical demands, from additional project management to preparing study materials. Ensuring each participant’s study package contains all the test items demands double and triple checking. If anything has been forgotten, it may result in the interview needing to be rescheduled. There are also no guarantees that supplies will be delivered on time so advance shipping is a necessity. This has taken away the option for last minute study changes in the days preceding the interviews. However, should supplies get damaged in transit, the increased flexibility of rescheduling participants can avoid any dropouts.

Technology – a blessing and a curse

The internet has made remote interviewing possible, and over the last year we have used a range of viewing platforms such as UserZoom, Miro, GoToMeeting and Microsoft Teams. Whilst some of these platforms are better than others for certain types of research, there are still some technological challenges that can interfere with the interview process such as Internet connections, technical IT difficulties and camera angles.

Internet connections have been taxing. Even though we would contact participants ahead of their session to conduct ‘tech checks’ to test their computer set up, some participants would still experience a poor internet connection during their interview. Poor connections have hampered the moderator and observers’ ability to see and hear what the participant was doing. To compensate for this, the moderator would have to resort to longer, sometimes more difficult, conversations with the participant to establish what the participant was doing.

Although we have seen participants experience various IT problems, these have tended to be matters which the participant could be talked through to fix the problem. However, IT problems at the start of the session could be quite time consuming and occasionally made a participant feel stressed at the beginning of the interview.

Regardless of the camera used, (laptops’ built-in camera; entry level webcams, or high-quality video equipment) they were all limited by the field of view they captured. Consequently, despite spending time with the participant to carefully set the camera so that a wide enough angle was created to fully view the participant, and sufficient detail was also captured, some participants would still manage to move out of shot.

Remote testing is challenging, but it’s here to stay

With the moderator not being in close proximity to the participant, remote testing has certain inherent limitations. Consequently, when the COVID-19 pandemic has passed, traditional face-to-face usability tests will return, particularly in the case of summative evaluations. However, having conducted remote testing for a year, and gone through a very steep learning curve, we have found that although it is not necessarily a cheaper or quicker option than face-to-face usability studies, it has a number of distinct advantages. Consequently, remote testing will continue after COVID-19 has passed.