Background

Point-of-care ultrasound (POCUS) is a powerful tool that has been shown to reduce procedural complications [1,2,3,4], improve bedside diagnostic accuracy [5], reduce diagnostic testing [6], and improve patient satisfaction [7, 8]. Despite the benefits, lack of access to an ultrasound machine has been a top barrier to POCUS use reported by multiple specialties [9,10,11,12,13,14]. Historically, the cost and size of cart-based ultrasound machines has limited their use in POCUS imaging. Since the 2010s, a surge of pocket-sized handheld ultrasound devices (“handhelds”) has dramatically improved clinicians’ access to portable ultrasound technology, especially in resource-limited settings [15, 16]. For the first time, handhelds have allowed clinicians to buy a personal ultrasound device for training and clinical use [17].

Although handhelds often have lower image quality, several studies comparing handhelds and cart-based ultrasound machines have demonstrated similar accuracy for common procedures and diagnoses, and any discrepant findings were not clinically significant [18,19,20,21,22,23,24,25,26,27,28,29,30,31]. However, few studies have compared different brands of handhelds in a head-to-head comparison [20, 32, 33]. One study compared 3 handhelds for gynecological ultrasound exams in a resource-limited setting [20], and another study evaluated 5 handhelds for ophthalmologic and facial aesthetics [32]. Based on our literature review, our group conducted the only head-to-head comparison of 4 handhelds for common general medical applications in December of 2021. Since then, major hardware and software updates have occurred to nearly all handhelds, and new handheld devices have become commercially available.

The objective of this study was to compare the performance of 6 common handheld ultrasound devices that are readily available in the United States to guide purchasing decisions. A multidisciplinary group of physician POCUS experts compared performance of handhelds to acquire 3 specific views (right upper quadrant, cardiac apical 4-chamber, and superficial neck and lung views) and rated the image quality. Afterward, experts rated the overall ease of use, image quality, and satisfaction of each device, and ranked the devices against each other. Additionally, we sought to identify the most important characteristics of handhelds per POCUS experts to guide selection of a device for use in clinical practice.

Methods

Subjects and setting

We conducted a cross-sectional study during a 2-day POCUS continuing medical education course in January of 2024. Thirty-five POCUS experts specializing in adult hospital medicine, critical care medicine, pulmonary medicine, emergency medicine, and nephrology acquired 3 standard POCUS views (right upper quadrant, apical 4-chamber, and superficial neck and lung views) using 6 commercially available handheld ultrasound devices on the same set of adult standardized patients with a body mass index (BMI) < 24. POCUS experts scanned the same patient with all devices for each of the 3 standard POCUS views. The University of Texas Health San Antonio Institutional Review Board reviewed and deemed this study to be non-regulated human research (STUDY00000326).

Protocol

Six handheld ultrasound devices with both low- and high-frequency transducer capabilities were compared (Table 2): Butterfly iQ +™ (Butterfly Network, Inc.) all-in-one probe (referred to as “Butterfly iQ +™”) connected by a Lightning® cable to an Apple iPad® (iPad Pro® 11-inch, iPad Air® 11-inch); Clarius™ (Clarius Mobile Health) phased-array (PA HD3), linear (L15 HD3), and Convex (C3 HD3) probes (referred to as “Clarius™”) connected wirelessly to an Apple iPad® (iPad Pro® 11-inch); Kosmos™ (EchoNous, Inc.) linear (Lexsa) and phased-array (Torso-one) probes (referred to as “Kosmos™”) connected by a USB-C cable to an Apple iPad® (iPad Pro® 13-inch); TE Air™ (Mindray) phased-array probe (referred to as “Mindray”) connected wirelessly to an Apple iPad® (iPad Pro® 11-inch) and an Apple iPhone® (iPhone 11 Pro®); Lumify™ (Philips Healthcare) probe (referred to as “Lumify™”) connected by a USB-C cable to a Samsung Galaxy S9 11-inch tablet™, and Vscan Air™ (GE Healthcare) SL (sector-phased array + linear) and CL (curved + linear) probes (referred to as “Vscan Air™”) connected wirelessly to a Samsung Galaxy A9 + 11-inch tablet™. Eight companies were requested to provide loaned handheld equipment only for this comparative study, but 3 companies (Exo, Vave Health, and Butterfly Network, Inc.) declined to provide equipment. Three Butterfly iQ +™ devices were provided by POCUS experts participating in this study; however, a sufficient number of handheld devices from Exo and Vave were not available for inclusion in the study.

Nine standardized patients were assigned to one of three POCUS views: (1) Focused Assessment with Sonography in Trauma (FAST) right upper quadrant (RUQ) view (diaphragm, liver, hepatorenal recess, and right kidney), (2) apical 4-chamber and 5-chamber views of the heart, (3) superficial view of the right neck (thyroid, internal jugular vein, and common carotid artery) and lung along the anterior chest wall (ribs, pleural line with lung sliding). Standardized patients were pre-scanned by 2 POCUS experts with a cart-based machine (Sonosite PX™ Fujifilm-Sonosite) and selected if high-quality images of one of the 3 views could be easily obtained based on their expertise.

Using the 6 handheld devices, all 35 POCUS experts independently acquired the same views on the same standardized patients. For the RUQ view, experts were instructed to use the curvilinear transducer, except for Mindray and Kosmos™ which only had phased-array transducers and Butterfly™ which had an all-in-one transducer. All RUQ views were acquired with an abdominal preset and focused on the liver, kidney, diaphragm, aorta, and spine. Color flow Doppler was applied over the vessels in the renal pelvis. For the apical 4-chamber view, experts were instructed to use the phased-array transducer with a cardiac preset to acquire views of the mitral valve, aortic valve, and right and left atria and ventricles. Experts were instructed to focus on the resolution of the endocardial lining and cardiac motion. Color flow Doppler was then applied over the mitral valve and left ventricular outflow tract. For the transverse view of the neck and superficial view of the lung, experts were instructed to use the high-frequency linear transducer with a venous or vascular preset to acquire transverse views of the internal jugular vein, common carotid artery, and thyroid gland, and color flow Doppler was applied over the common carotid artery and internal jugular vein. Next, a lung preset was used to acquire longitudinal views of the lung on the anterior chest wall to visualize lung sliding. All handhelds, except Mindray, had a high-frequency linear transducer or lung preset.

Data collection

This study was conducted in two phases (Fig. 1). First, experts rated the image quality of the 6 handheld devices for each of the 3 views as 0 (“poor”), 1 (“interpretable”), 2 (“good”), or 3 (“excellent”). Specific anchors were provided to rate the image quality for 5 characteristics of each view on the data collection forms (Additional files 1–3). An overall ranking of each device from 1 (“best”) to 6 (“worst”) was performed for each view. Second, data were collected on the overall ease of use, image quality, and satisfaction of each device (“overall survey”) (Additional File 4). For ease of use, experts rated the physical characteristics, software navigation, maneuverability of the probe/tablet for imaging, and overall satisfaction. For image quality, experts rated the detail resolution, contrast resolution, penetration, clutter, and overall satisfaction. The overall ranking assessed satisfaction and recommendation for purchase. Ratings were made using standardized statements on a Likert scale of 1 (”strongly disagree” or “very dissatisfied”) to 5 (“strongly agree” or “very satisfied”). Qualitative feedback was collected in each category using free text. Experts completed all data collection forms immediately and the overall survey no later than 72 h after scanning each standardized patient. Data were captured electronically using REDCap™ (Vanderbilt University, Nashville, TN, USA).

Fig. 1
figure 1

Study Flow Diagram. POCUS, point-of-care ultrasound. A4C apical 4-chamber, TV tricuspid valve, LVOT left ventricular outflow tract, MV mitral valve

Full size image

Data analysis

Descriptive statistics about the experts were reported as frequencies with percentages, without any statistical analysis. Ratings of ease of use and image quality were compared using the Kruskal–Wallis rank sum test, with the Dwass-Steel-Critchlow-Fligner post hoc method to control the familywise-error rate. Rank analysis was performed via Friedman’s test, followed by a post hoc Sign test for paired data, using the Holm’s step-down procedure to control the familywise-error rate. For image quality ratings of the 3 specific views, scores were calculated by finding the mean score of each characteristic across raters and then adding the 5 means within a view, while the comparison of devices was done using a non-linear mixed model to predict the rating scores, with the device and view characteristic as fixed factors and rater as a random factor.

Potential bias due to prior experience with a handheld was assessed by having experts rate their past experience with a device as none (1), some (2), or proficient (3). Spearman correlation coefficients were calculated to evaluate the correlation between experts’ prior experience with using each handheld device and ratings for ease-of-use, image quality, and overall satisfaction, with a modified independent sample t-test used to test for statistical significance. A p-value < 0.05 denoted statistical significance. All analyses were performed with SAS software version 9.4.

Free text responses were analyzed using a qualitative deductive and inductive coding process based on a framework method approach. Advantages and disadvantages of all 6 handhelds were coded and tabulated. Two investigators independently applied the coding framework to the free text responses, resolved coding differences through discussion, and assigned a final code based on that discussion.

Results

POCUS experts

Thirty-five POCUS experts specializing in internal medicine/hospital medicine, critical care, emergency medicine, and nephrology that care for adult patients participated in this study. Most experts (80%) had either completed a POCUS training certificate through a national specialty society, achieved certification through the National Board of Echocardiography, or completed a dedicated POCUS fellowship, and 75% had > 5 years of experience using POCUS to guide patient care (Table 1). Right upper quadrant, apical 4-chamber, and superficial neck and lung views were acquired and rated by each of the POCUS experts using 6 different handhelds on the same adult standardized patients.

Table 1 Characteristics of the point-of-care ultrasound experts
Full size table

Handheld characteristics

Characteristics of the 6 handhelds compared in this study are shown in Table 2. All handhelds had M-mode and color flow Doppler imaging modes, but only Kosmos™ had continuous-wave Doppler. All handhelds, except Mindray, were compatible with both iOS and Android tablets. Clarius™, Mindray, and Vscan Air™ were wireless. Butterfly iQ + ™ and Vscan Air™ were multifunctional transducers allowing acquisition of cardiac, abdominal, and superficial images from the same transducer, while Mindray allowed acquisition of both cardiac and abdominal images.

Table 2 Characteristics of Handheld Ultrasound Devices
Full size table

Specific views

Abdominal right upper quadrant view

The specific characteristics evaluated in the RUQ view were the difference in echogenicity of the renal cortex and liver, clarity of blood vessels in the liver parenchyma, distinction of the medullary pyramids in the renal cortex, far-field resolution, and color flow Doppler of vessels in the renal pelvis. For the abdominal RUQ view, the top 3 highest-rated handhelds were Vscan Air™, Lumify™, and Mindray (Fig. 2) which was consistent with the overall ranking for the RUQ view (Additional File 5: Table S1).

Fig. 2
figure 2

A) Abdominal Right Upper Quadrant View ratings of image quality by handheld (5 domains displayed were rated on a scale from 0 to 3); B) Abdominal Right Upper Quadrant View acquired from the same standardized patient showing kidney, liver, and diaphragm from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™

Full size image

Cardiac apical 4-chamber view

The specific characteristics evaluated in the apical 4-chamber view were endocardial definition, clarity of valve leaflets, clarity of the lateral tricuspid valve annulus, far-field resolution, and color flow Doppler of the left ventricular outflow tract and mitral valve. For the apical 4-chamber view, the top 3 highest-rated handhelds were Mindray, Vscan Air™, and Lumify™ which was consistent with the overall ranking (Fig. 3 and Additional File 5: Table S2). Compared to the RUQ view and superficial neck and lung views, the total rating scores for the apical 4-chamber view were lower with all handhelds. Parasternal long-axis views were not rated in this study, but sample images acquired from a standardized patient post-study are provided for the benefit of readers (Fig. 3C).

Fig. 3
figure 3figure 3

A) Cardiac Apical 4-chamber View ratings of image quality by handheld (5 domains displayed were rated on a scale from 0 to 3); B) Cardiac Apical 4-chamber View acquired from the same standardized patient in mid-diastole with the mitral and tricuspid valves open from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™; C Cardiac Parasternal Long-axis View acquired from the same standardized patient in early systole with the mitral valve closed and aortic valve open from 6 handheld devices: A Butterfly iQ + ™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™

Full size image

Superficial neck and lung views

The specific characteristics evaluated in the superficial neck and lung views were clarity of the carotid artery/internal jugular vein, color flow Doppler of carotid artery/internal jugular vein, difference in echogenicity of thyroid, contrast of chest wall vs. pleural line, and clarity of lung sliding. For the superficial views, the top 3 highest-rated handhelds were Lumify™, Vscan Air™, and Clarius™ which was consistent with the overall ranking (Fig. 4); however, the difference in image quality between the Vscan Air™ and Lumify™ was not statistically significant (Additional File 5: Table S3). Notably, the Mindray handheld lacked a linear probe and was excluded from the comparison of superficial views.

Fig. 4
figure 4figure 4

A) Superficial Neck and Lung Sliding View ratings of image quality by handheld (5 domains displayed were rated on a scale from 0 to 3); B) Superficial Neck Views acquired from the same standardized patient displaying the thyroid, common carotid artery, and internal jugular vein from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™; C) Superficial Lung Views acquired from the same standardized patient showing the pleural line from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™. Mindray lacks a linear probe and was excluded from the comparison of superficial views of the neck and lungs. The Mindray images in sections B (panel E) and C (panel E) are only displayed for demonstration purposes

Full size image

Overall survey

After rating the specific views, all 35 POCUS experts completed an overall survey on ease of use, image quality, and satisfaction of each device. Specific characteristics and ratings for ease of use and image quality are shown in Table 3. Vscan Air™ and Mindray were rated the highest on physical probe characteristics and maneuverability, while Vscan Air™ and Butterfly iQ +™ were rated highest for ease of use of their software. For overall satisfaction with ease of use, Vscan Air™ was rated highest followed by Lumify™ and Mindray.

Table 3 Overall ease of use & image quality ratings of handheld ultrasound devices per experts (n = 35)
Full size table

For image quality, there were fewer statistically significant differences compared to ease of use. Vscan Air™ was rated highest in all categories (detail resolution, contrast resolution, penetration, clutter). For overall satisfaction with image quality, Vscan Air™, Lumify™, and Mindray were rated highest, and the differences were not statistically significant. A comparison of mean ratings for ease of use vs. image quality is illustrated in Fig. 5.

Fig. 5
figure 5

Mean Ratings of Handhelds by Ease of Use and Image Quality

Full size image

The final survey asked experts about their overall satisfaction with each handheld (Fig. 6). Vscan Air™, Lumify™, and Mindray received the highest number of “satisfied” responses and ranked highest in order from 1 (“best”) to 6 (“worst”). When experts were asked which handheld they would purchase today as their personal device to carry in their coat pocket, a majority selected the Vscan Air™ (66%).

Fig. 6
figure 6figure 6figure 6

A) Overall Satisfaction with each Handheld Device; B) Overall Comparison Rankings of Handhelds by POCUS Experts; C) Purchasing Decision of Handheld to Carry in Pocket by POCUS Experts

Full size image

The 6 most important characteristics of handheld devices per experts were image quality, ease of use, portability, probe size, battery life, and availability of different probes. The least important characteristic was inclusion of artificial intelligence (AI) technology (Table 4).

Table 4 Importance of Characteristics of Handhelds per POCUS Experts
Full size table

The qualitative data based on free-text comments from POCUS experts revealed a few important themes (Table 5). First, image quality is the most critical characteristic of handhelds because poor-quality images preclude making any clinical decisions. Thus, if an image of adequate quality to make a clinical decision cannot be obtained, it is not worth having the handheld. Second, after an adequate image quality can be acquired, it is desirable to have a small, multifunction (2- or 3-in-1), wireless probe. However, wireless probes that have connectivity issues, such as difficult, slow, or unreliable pairing with a tablet, are less desirable than wired probes. Finally, all 6 handhelds had notable advantages and disadvantages, and no single handheld was perceived as having all desired qualities or features.

Table 5 Advantages and Disadvantages of Handhelds per Comments of POCUS Experts (n = 35)
Full size table

Bias evaluation

Potential bias due to prior experience with each handheld was assessed. Mindray and Clarius™ had a mean experience score < 1.1, indicating near total lack of experience with these devices. Kosmos™ and Vscan Air™ had mean experience scores of 1.5 and 1.6, respectively, indicating about half of experts had some experience. Lumify™ and Butterfly iQ + ™ had average experience scores of 2.1 and 2.4, respectively, indicating most users had some experience and several were proficient in their use.

No statistically significant association between experts’ experience levels and their ratings for image quality were seen (Additional File 6: Table S4). For ease of use, Vscan Air™ and Lumify™ had a small positive association with experience (correlation coefficient = 0.33, p = 0.05 for Vscan Air™ and correlation coefficient = 0.53, p = 0.001 for Lumify™), Thus, experts with more experience with Vscan Air™ and Lumify™ tended to rate them as being easier to use.

For overall satisfaction, there was no association with experience for five of the handhelds, but for Lumify™ there was a small positive association identified (correlation coefficient = 0.56, p = 0.001), indicating that experts with more experience tended to report more overall satisfaction with it. However, it is noteworthy that Butterfly iQ + ™ had the highest number of experts proficient in its use, yet it scored low in overall satisfaction. On the contrary, Mindray had virtually no experts with experience using it, yet it scored nearly equivalently as Lumify™ in overall satisfaction.

Discussion

We compared the performance of 6 common handheld ultrasound devices for image quality, ease of use, and overall satisfaction. For image quality, the highest-rated handheld for the RUQ view was Vscan Air™, for the cardiac apical 4-chamber view was Mindray, and for superficial views of the neck and lung was Lumify™. The overall satisfaction with image quality was highest with Vscan Air™, Lumify™, and Mindray. The Vscan Air™ was rated highest for overall ease of use and was the most preferred handheld for purchase by POCUS experts. The most desirable characteristics of handhelds were image quality, ease of use, portability, probe size, battery life, and availability of different probe types.

Several studies have compared handhelds to cart-based ultrasound machines and have demonstrated similar accuracy for common diagnoses and procedures [18,19,20,21,22,23,24,25,26,27,28,29,30,31]. However, few studies have directly compared different brands of handhelds. [20, 32, 33] A study in 2020 compared 3 handhelds (GE Vscan™, Sonosite Iviz™, Philips Lumify™) for gynecologic measurements and common pathologies in patients and concluded that Lumify™ was the best handheld overall in this resource-limited setting [20]. Another study in 2024 evaluated 5 handhelds (Butterfly IQ +™, Clarius™ L15 and L20 probes, Lumify™, and Vscan Air™) with 3 ophthalmologists acquiring views of facial arteries, ocular/periocular structures, and areas for filler injections and concluded the Clarius™ L20 had the highest image quality for superficial facial structures [32]. Based on our review of the literature of handhelds, our group conducted the largest (n = 24) head-to-head comparison of handhelds for common general medical applications in December of 2021. [33] Building on our past work, the current study compared image quality based on specific characteristics of 3 common views, included new handhelds, and incorporated important hardware and software updates of existing handhelds. Further, by having a large number of POCUS experts (n = 35) conduct the handheld comparison on the same standardized patients, we were able to minimize potential patient, device, and operator variables that could confound results. Also, experts acquired and evaluated image quality in real-time as they would in clinical practice. Both high- and low-frequency transducers were used to assess abdominal, cardiac, and superficial views that are broadly relevant to clinical practice in multiple specialties.

Comparing data from our 2021 and present study revealed important similarities and differences. Most important, the distribution of data points in the graph comparing mean ease of use vs. image quality of handhelds has narrowed, signifying differences between handhelds appear to have become more subtle (Figs. 5 and 7). We anticipate the differences in image quality and ease of use between handhelds will continue to narrow and subsequently, other important characteristics, like battery life, probe ergonomics, and availability of different imaging modes, will differentiate the ratings of handhelds. Additionally, from 2021 to present, the Vscan Air™ surpassed Lumify™ with respect to overall satisfaction, and the Vscan Air™ continued to be the preferred handheld that experts would purchase “today as a personal device to carry in my coat pocket.”

Fig. 7
figure 7

Comparison of Handheld Devices from December 2021 and January 2024. Mean ratings of ease of use and image quality are shown for Butterfly iQ +™, Kosmos™, Lumify™, and Vscan Air™ from 2021 and 2024. Mindray and Clarius™ were not included in the 2021 comparison study

Full size image

Experts’ ratings of the most and least important characteristics of handhelds did not change significantly from 2021 to the current study. Among the 20 characteristics of handhelds, the 3 most important characteristics were image quality, ease of use, and portability, and the 5 least important characteristics only changed slightly in rank order. Although the most and least important characteristics of handhelds did not change significantly, no single handheld was perceived to have all desired characteristics, and all handhelds had important advantages and disadvantages (Table 5). For instance, wireless connectivity appeared to be preferred and advantageous, but when pairing between a handheld and tablet was slow or unreliable, wireless connectivity became a disadvantage. Furthermore, new AI functions have been added to most handhelds in recent years; however, experts rated AI technology as one of the least important characteristics of handhelds. Beyond handhelds, the current role of AI in medicine is unclear, and how clinicians will use AI in POCUS is yet to be determined. Perhaps AI will help facilitate self-directed POCUS training or allow less skilled clinicians to acquire and interpret POCUS images more accurately. For example, several handhelds and cart-based machines now perform automated cardiac calculations, and it is plausible that trainees or nurses could acquire cardiac measurements daily, similar to recording vital signs and other clinical parameters. Finally, adding new features to handhelds demands a critical balance of probe characteristics. If adding new features changes the probe size, weight, or costs substantially, the new feature may not be attractive to clinicians.

We acknowledge our study has limitations. First, we used standardized patients with a BMI < 24 and easily acquired views to minimize patient variables as confounders in the assessment of ease of use and image quality, but performance of these handhelds on patients with pathologic findings and higher BMIs may differ. For instance, we were unable to compare lung ultrasound performance using a low-frequency transducer to assess common lung pathologies, such as pneumonia and pulmonary edema. Second, bias from prior experience with some of the handhelds may have been a component in experts’ overall evaluation, but we did not identify a statistically significant correlation between experts’ prior experience and overall ratings of image quality of the devices. Bias from prior experience may have been a factor in the expert’s overall evaluation for ease of use of the devices, as the ratings for ease of use for Vscan Air™ and Lumify™ had a small positive association with experience. However, it is worth noting that Butterfly iQ + ™ had the highest number of experts proficient in its use, yet it scored low in overall satisfaction, and on the contrary, Mindray had virtually no experienced users, yet it scored nearly equivalently in satisfaction as Lumify. Third, the ultrasound manufacturers supplying handhelds for this study were requested to provide a tablet that best demonstrated their handheld device’s capabilities. However, handhelds were paired with tablets that varied in brand, operating system (iOS vs. Android), size, and resolution, and the selection of tablets may have affected experts’ ratings of image quality. Fourth, handheld purchasing decisions are complex, and this study focused on the 2 most important characteristics, image quality and ease of use. However, several device characteristics were rated as important, and though many of these characteristics appeared in our qualitative data, they were not addressed directly in our study, such as tablet connectivity and battery life from prolonged use. Notably, institutional approval of handhelds and integration with the local image archiving software or picture archival and communication system (PACS) may be the deciding factor for purchase of handhelds, regardless of the clinicians’ preferences.

Conclusion

In our comparison of 6 handheld ultrasound devices, the overall satisfaction with image quality was rated highest with Vscan Air™, Lumify™, and Mindray. Specifically, image quality was rated highest with Vscan Air™ for the RUQ view, Mindray for the cardiac apical 4-chamber view, and Lumify™ for superficial views of the neck and lung. No single handheld ultrasound device was perceived to be superior in image quality for all 3 views. Vscan Air™ was rated highest for overall ease of use and was the most preferred handheld for purchase by POCUS experts. The most desirable characteristics of handhelds were image quality, ease of use, portability, probe size, battery life, and availability of different probe types. As differences in image quality and ease of use become less significant between handhelds, secondary characteristics, including portability, probe ergonomics, battery life, imaging modes, and costs, will become the distinguishing features of handhelds.