This semester, I’m working with a team of Usability Assessment students and a team of engineering students on an IV Drip Monitor. Throughout the semester, I will be performing usability testing on this device. Before I can get into testing participants, I need to prepare a heuristic evaluation and cognitive walk through. This will give me a better idea about the product as a whole.
The following was co-written with my teammates (Julia Hossu and Hao Yun).
Heuristic evaluations and cognitive walkthroughs are widely accepted “discount” methods for usability assessment of products. Usually conducted before user testing, these methods help evaluators inspect a product according to a set of usability heuristics, set usage error categories, define tasks and understand deficiencies. We performed a heuristic evaluation and a cognitive walkthrough for our target product, Maji the IV Drip Monitor, as part of our usability assessment in preparation for the user testing of this product.
Maji, Swahili for “water”, was designed by a group of Rice University engineers to regulate the volume of fluids during intravenous drip therapy. It is a low-cost mechanical device designed to be used in low-resource settings to prevent over-hydration during the treatment of babies and children. Within the context of administering IV therapy, the administrator would have to hang an IV bag onto the Maji system then set the amount of fluid they wanted to dispense. Any other IV bag specific setup, such as flow rate adjustments, would then occur. When the specified amount of fluid is dispensed from the bag, Maji clamps the tubing together and stops further flow.
A heuristic evaluation is a method where the evaluators assess a product to determine its compliance to a set of known usability principles (heuristics). We, the three evaluators, first evaluated the product independently, then worked cooperatively to combine our findings. We used the ten usability principles proposed by Molich and Nielson (1991).
This principle was originally designed to assess a software interface. For our non-interactive mechanical device, we modified this principle into “simple and natural protocols”. We specified seven steps in the process of using Maji:
The order of the operating steps is critical but is intuitively very difficult to remember. For example, users may set the equilibrium before hanging the IV bag because they may want to have the measures set up and ready for use before bringing in the medications.
The paper volume tape does not have units marked (supposedly mL), and this may cause mistakes if the user is more familiar with other units (e.g. “oz”). For the instructions, all the steps are in graphic form. The only words on the instructions are “Step 1” through “Step 4”. It lacks important information that cannot easily be explained without words, e.g. it is difficult to grasp the idea of equilibrium from the graphical instructions, or the fact that the counterweight needs to be at the furthest right point that causes equilibrium.
Since Maji has only one function, it does not require users to remember the specifics of device usage per situation. However, the user does have to remember the somewhat complicated processes of setting Maji up. Plus, if the steps are done out of orders then the system won’t work appropriately. As discussed before, the steps are not intuitively natural to follow, so an inexperienced user may have to refer to the instructions frequently to use Maji. There was no mark or word on the device to minimize user’s memory load by reminding them of the steps.
The equilibration process is very inconsistent. The counterweight can be placed in multiple positions that make the lever arm appear horizontal. There was about a 150 mL range (three notches for the counterweight) where the system would stay horizontal in equilibrium position. Since no instructions were given to inform the user of the fact that the counterweight has to be put in the rightmost position that brings the lever arm to a horizontal state (this usage information was mentioned in a manuscript which introduces the device), a user could easily miss true equilibrium. Additionally, it is sometimes difficult to determine a “horizontal position” just by eyeballing it. The user needed to shift the lever arm up and down constantly to see if it would ‘hang’ horizontally. Plus they needed to have a feel for what ‘hanging’ or balancing with a sense of stability, was. When testing the system ourselves, the equilibrium step seemed to account for the most variance in dispensing a certain amount of liquid. An unpublished paper argued that the accuracy of the system is 12mL to 25 mL, but the evaluator was an experienced user of Maji, and should be excellent in determining the horizontal status of the lever arm. We can anticipate that ordinary users may encounter much larger variances in the actual dispensed volume.
The inconsistency with equilibration discussed above has to do with the fact that there is no clear feedback in this process. The only feedback of any form is the “visual horizontal status” of the lever arm. But, as was pointed out above, this status is very difficult to measure. There is good feedback when the target amount has been dispensed. The user can see when the system has run its course because it pinches the tubing and there is an audible click, then the flow of the fluid stops. However, during dispensing, users cannot get any feedback in terms of how much fluid has already been dispensed or how much more fluid will be dispensed and how much more time before the system is supposed to shut off.
There is no marked exit on Maji, although we found that users can exit the dispensing process by pressing down on the counterweight to stop the flow.
We found no shortcut in using Maji, because all operating steps need to be completed in order for the system to work properly. In fact, we believe the instructions should incorporate warnings against any temptation to use a shortcut in order to prevent errors in use.
The IV Drip Monitor is a low-cost mechanical device with no interactive graphical user interface. Therefore, we understand that it can barely give any precise error messages. The only error indication of any kind is that when the user sets the system over equilibrium (the counterweight is too far right), he cannot continue to set the knobs and load the spring in Step 5. This device is exactly in the same state as when it stops dispensing. There is no error indication whatsoever when the system is under equilibrium (the counterweight is too far left). The user can still set the tubing but the device will over dispense.
There are two pages of instructions. The first page has a photo of the system next to its graphical representation, its name and some key benefits of the Maji system (automatic, durable, accurate and using no electricity). The second page has pictures to illustrate the steps of setting up the system. These were the instructions used in Malawi for usability testing and apparently need to be updated because the device design has been further developed since that time. The instructions are missing the step where you set the red volume indicator and the graphics are missing the volume tape entirely. As mentioned before, it is difficult to grasp the idea of equilibrium from the graphical instructions. Finally, the layout of the steps in the documentation is in a zig, zag so if one wasn’t paying attention to the numbers, they could easily get confused.
Maji is a one-trick pony in that it can really only be used for small variations of one task: dispense X amount of fluid from the amount in the IV bag. Therefore, for our cognitive walkthrough, we decided to test two different situations. The first task was to dispense a small amount of of fluids, 50 mL, from a 600 mL bag. The second task was to dispense a larger amount of fluid, 400 mL, from a 600 mL bag. The reasoning in choosing these tasks was that in our heuristic evaluation, we found it very difficult to set the system at equilibrium if the target volume was very small. So we wanted to at least partially focus on setting the device to dispense small amount of fluids. We would have liked to do additional tasks with different initial volumes, but we only had one IV bag containing 600 mL. We also would have liked to have a task that dispensed in succession ( for example dispense 200 mL from a 1000 mL bag then dispense another 200 mL from the same bag which would then contain 800 mL), because it mimicked the real scenario of use. However, given the time constraint and the lack of additional IV bags, we were not able to wait through the whole dispense time for such a task.
First, we hung the IV bag on the hook attached to the Maji system. This had good feedback because the bag did not fall down and the system looked as it did in the instruction diagrams.
Next, we put the system in equilibrium by moving the counterweight to the rightmost notch that allowed the lever arm to remain horizontal. We found multiple ways to accomplish this task. One could start the counterweight at the furthest left point and keep moving it to the right until the bar fell to equilibrium. Alternately, one could start the counterweight at the furthest point to the right and keep moving the counterweight to the left until, when raised to a horizontal position, the bar hung horizontally. Finally, one could keep resetting the bar to a perceived equilibrium and see whether it hung horizontally in the equilibrium position. There was a limited amount of feedback presented in two ways: if the counterweight was set under equilibrium (too far to the left), the bar would remain tilted upwards; if set above equilibrium (too far to the right), the bar would drop below the horizontal. This form of feedback alone was not sufficient to ensure consistent equilibration. We had to shift the lever arm up and down to guage if it was truly hanging in equilibrium.
Once equilibrium was found, we went on to move the red volume indicator to match the position of the counterweight. The volume indicator moved as expected and we could visually see the red notch lined up with the counterweight. Then we moved the counterweight to the 50 mL notch on the lever arm. Once again, we could visually see the counterweight matched up with the 50 mL mark on the volume indicator.
Next, we moved the black knobs towards each other until they hooked together. There was good feedback because the knobs stayed together. Next, we put the IV tubing in the plastic tubing holders with clips and made sure it was taut and unoccluded. While testing the device, our team had an extremely difficult time knowing whether the tubes were secured properly or not. In fact, we unknowingly made it impossible to crimp the tubes by not inserting them fully. Also, the clips moved very easily making it necessary to manually re-adjust them back into the holders. A newer model of Maji seems to have different types of clips and tubing holders but we are unsure if these fixed the issue. Another problem came when we tried to place the tubing in the holders given our small target value to dispense, 50 mL. The way that Maji works is that the mechanical stop motion takes place when the specified weight is removed from the IV bag. As we placed the tubes in the plastic tubing holders, the holding clips would disengage and attempt to crimp the tubing because we were relieving the weight of the bag of the bag by a large enough amount. Therefore, it was difficult for one person to set up the device successfully to dispense an amount as small as 50 mL without help from another person the lever arm in place while the tubing was inserted into the holding clips.
Once the tubing is secured, IV therapy can begin. All that remains is IV bag specific tasks such as setting the flow rate and releasing the wheel to get fluids flowing. When the set amount has been dispensed, the mechanical stop will engage, the lever arm goes down and the knobs release causing the tubing to be pinched shut by the holding clips. One critical problem that we found was that the tubing was not always successfully shut after the lever arm went down. As mentioned above, the holding clips in the tubing holders seem to move easily and although the tube appeared to be crimped flow did not cease because the bend was not tight enough. There is no feedback from the device warning the user that it was not stopped entirely.
The process for this task is the same as for the previous task, but had 400 mL as the target volume. We found the same set of problems as in the other task, except that the placing of IV tubing in the holders was much easier in this task and gave our team fewer problems.
In unusual cases, the fluid flow could be stopped if, for example, the patient accidentally pulled the tubing and disturbed the equilibrium. There would be no warning message or any indication from the device about this early termination. The device cannot reset itself automatically either. The graphics in the instructions need to be revisited for the reasons mentioned above. Also, the image with the tubing kinked is incorrect because the clips are at a different angle in the picture than they are on the device.
Although we believe that trained medical workers should be able to successfully set up Maji very quickly and have it properly work in most of the cases, our assessment of the usability of Maji using heuristic assessment and cognitive walkthrough have found various problems with it. These usability problems would make new users very confused at the device, because the steps in setting up are complicated and the sequence is not very intuitive. The instructions are far from adequate, because they lack critical information, and the presentation of the graphs in the instructions is also hard to follow. Furthermore, even experienced users would have difficulty in performing certain tasks using Maji, e.g. dispensing small target volumes. Also, Maji is prone to usage mistakes. The clips in the tubing holders, for example, are not well-designed, and can easily go off the place so the device will not stop properly. Finally, once a mistake happens, there is often no clear feedback given, so the user may not easily notice that the device has stopped before the target volume is dispensed, or that the device continues to dispense fluids because the holders have not shut the tubing entirely.