The evidence behind CommCare

A study by Mathematica Policy Research provides statistically significant evidence of behavior change in clients whose FLWs use CommCare [Borkum, 2015]. The setting was Saharsa district in Bihar, a district with persistently low health outcomes in one of India’s poorest and most populous states. The study surveyed 1,500 women across 70 sub-centers, half randomized to intervention and half to control.

All 70 sub-centers already received an extensive health-system-strengthening package. The only difference in the intervention arm: FLWs used a CommCare application with a scheduler supported by MOTECH (a separate digital health platform).

The CommCare intervention produced statistically significant improvements across antenatal care, delivery and newborn care, child nutrition, and reproductive health; child immunizations were the only domain without a significant effect. All four significant impacts were positive. These results are especially notable given the strong non-ICT health system strengthening already in place in the control arm, showing CommCare's value on top of extensive community health support.

Most notably, women whose FLWs used CommCare were 73% more likely to attend at least three antenatal care visits. Interestingly, improvements in client health outcomes were not always matched by increases in client knowledge of those behaviors, suggesting CommCare also reduced non-knowledge barriers such as cultural norms.

The report also suggests several areas for improvement in implementing CommCare. There were a few indicators where behaviors decreased, though these were not statistically significant results. On a programmatic level, the study reported challenges with data connectivity and broken mobile phones, which sometimes resulted in poor coordination between FLWs and supervisors. FLWs also experienced an increased workload due to the continuation of paper documentation required by the government.

Despite these logistical challenges and the lack of improvements in supervision, equipping FLWs with CommCare created a significant impact on client health outcomes and behaviors.

The results from an RCT in rural Tanzania report that mothers tracked by FLWs using CommCare had increased facility-based delivery rates, especially among first-time mothers with low antenatal care uptake [Hackett, 2018]. As with the study in Bihar, both the control and intervention groups received health system strengthening services; in this case, those services were administered by World Vision. The FLWs in the intervention group were also equipped with CommCare applications (implemented and supported by D-tree International) to assist with data collection, education, danger sign identification, and referrals.

The study included 32 villages that were cluster-randomized to control or intervention. The study surveyed a total of 572 women and found that “The smartphone intervention was associated with significantly higher facility delivery: 74% of mothers in intervention areas delivered at or in transit to a health facility, versus 63% in control areas. The odds of facility delivery among women counseled by smartphone-assisted health workers were double the odds among women living in control villages (OR, 1.96; CI, 1.21±3.19; adjusted analyses).” First-time mothers with low antenatal care uptake saw the biggest gains: their facility-based delivery rates were 32% higher than the control group.

In India, Catholic Relief Services (CRS) deployed the ReMiND (‘Reducing Maternal and Newborn Deaths’) project developed on the CommCare platform. The intervention required development and implementation of a mobile application used as a job aid by ASHAs for registering pregnant women and for providing real-time guidance through key counseling points, decision support, timely alerts and referral algorithms for various maternal and child health issues.

A study of the effectiveness of ReMiND found a statistically significant increase in coverage of iron and folic acid supplementation (12.58%), self-reporting of complications during pregnancy (13.11%) and after delivery (19.6%). The coverage of three or more antenatal care visits increased significantly in the intervention area (10.3%) [Prinja, 2017].

A study on TulaSalud’s implementation of CommCare in Guatemala provides evidence of reduced maternal mortality rates (MMR) and infant mortality rates (IMR), as compared to the control areas and the provincial average [Martinez-Fernandez, 2015].

The ICT intervention took place in Alta Verapaz, a predominantly rural region of northern Guatemala with high maternal and infant mortality rates. 125 FLWs in Alta Verapaz were equipped with mobile phones and the Kawok system developed by TulaSalud, which was built on CommCare, to assist with making client consultations, collecting epidemiological data, receiving continuous training, and performing community health promotion and prevention activities. After five years (2008 to 2012) of this intervention that included CommCare, an observational study was conducted to compare the MMR and IMR between the districts with FLWs using CommCare, and those without it. Both the intervention and control areas had similar hospital access, racial/ethnic makeup, age, education, etc.

The study found that in intervention areas, MMR decreased by 18% from 309 to 254 maternal deaths per 100,000 live births (p<0.05), and IMR decreased by 48% from 25 to 13 infant deaths per 1,000 live births (p=0.054). Figure 3 describes the IMR rates over time in the areas with and without the ICT intervention, as well as the entire region between 2008 and 2012.

Several other studies also showed client health-seeking behavior change in the areas of antenatal care, institutional delivery, and postnatal care. While the above studies are each more rigorous than the ones described in this section, these additional studies suggest that the findings are generalizable to other contexts.

Several other studies reported high institutional delivery rates among clients of FLWs using CommCare [Amoah, 2016] [Battle, 2015] [Hoy, 2015] [World Vision, 2013]. For example, D-tree International implemented a project using CommCare in Zanzibar to increase the institutional delivery rates, especially in cases of complicated pregnancies [Hoy, 2015]. Traditional birth attendants (TBAs) were equipped with CommCare to identify danger signs, refer clients, record family members’ permission to transport the women to a health facility in case of emergencies, and facilitate transportation payment to local vehicle owners. The intervention reported a 71% institutional delivery rate, compared to the regional average of 32%.

These studies demonstrate that the effects of CommCare hold over a wide range of geographies and maternal and child health programs. The results show that women with previously lower antenatal care uptake [Hackett, 2018] and lower castes [Borkum, 2015] experience a higher positive impact in CommCare interventions. Taken together, there is a strong pool of evidence indicating that pregnant women who are tracked by FLWs using CommCare have improved health outcomes and behaviors in the areas of antenatal care, institutional deliveries, and child health.

Several studies have shown that CommCare improves FLWs’ personal credibility and the credibility of the health messages they deliver [Bhavsar, 2014] [Medhi, 2012] [Schwartz, 2013] [Braun, 2016] [Gopalakrishnan, 2020]. These findings have emerged from qualitative interviews with FLWs, who have reported that CommCare enhances their credibility in their communities, and that clients and clients’ families perceive recorded messages as more trustworthy. CommCare is widely viewed as an independent, objective source of information, which greatly benefits FLWs’ ability to deliver sensitive messages. FLWs and clients in Tanzania reported that CommCare is a highly acceptable counseling tool, particularly for its improved sense of privacy and trust with clients [Braun, 2016].

A study of 50 FLWs in India found that home visits with pregnant women were more inclusive and interactive when CommCare was used. The client’s husband and mother-in-law were 60% and 110% more likely to participate during the visit, respectively, and the client was 33% more likely to ask questions when CommCare was used by the FLW [Mohamed, 2014]. The Mathematica study in Bihar also found that FLWs who use CommCare were 37% more likely to report a high level of confidence in their skill and ability to do their job, and 20% more likely to run regular FLW meetings by themselves [Borkum, 2015]. In a qualitative study examining both FLWs’ and beneficiaries’ perceptions of a CommCare intervention in Bihar and Madhya Pradesh, FLWs reported being better able to engage and build trust with influential household members beyond their direct clients, such as mothers-in-law, which in turn increased direct clients’ ability to interact with FLWs and receive necessary care. FLWs also reported that features of this intervention intrigued other community members and encouraged them to gather around FLWs to receive information, thus allowing FLWs to engage with more clients [Gopalakrishnan, 2020].

FLWs who use CommCare for maternal and child health interventions have been shown to be more knowledgeable about the health topics and services they provide. A study on CommCare in India showed improvements in FLW knowledge of at least three to five pregnancy danger signs by 22% [Kumar, 2012]. In Nigeria, FLWs combating the Ebola Virus Disease improved their knowledge of the disease, with statistically significant improvements (p<.05) on questions about human transmission of the virus, common symptoms, and whether Ebola fever is preventable. The study also noted reinforcement against risky behaviors such as contact with Ebola patients, eating bush meat, and risky burial practices [Otu, 2016].

FLWs whose tasks are tracked through CommCare have been found to perform more efficiently and consistently. A study in South Africa observed an increase in adverse event form submission from 5% to 27% when switching from paper forms to CommCare [Chaiyachati, 2013]. Two studies in India found that CommCare improves FLW performance during home visits, particularly in their frequency and timeliness [Borkum, 2015]. In particular, CRS found that, after introducing CommCare, the percentage of women ever visited by a FLW increased by 15%, the number of FLW visits per pregnant woman nearly doubled, and the percentage of women receiving counseling from their FLW increased by 28% [Murless, 2015]. In Bihar, India, Mathematica surveyed clients about FLW home visit consistency and found that FLWs using CommCare were more likely to conduct visits at critical times throughout pregnancy and early childhood than in the control group (Figure 4) [Borkum, 2015].

Home visit consistency improved at every recorded stage of maternal and newborn care among FLWs who used CommCare in the Mathematica study in Bihar. In a study conducted in Indonesia, the ability to track FLW tasks through CommCare was cited as a mechanism for motivation and engagement [Walton, 2020]. These findings reinforce the notion that CommCare’s capacity to track when and how FLWs perform tasks in the field increases performance and accountability.

Two RCTs have been conducted to measure the added impact of supplemental features to the CommCare application on FLW performance. The first study uses SMS reminders while the second uses web- and voice-based performance feedback to evaluate the added impact of the CommCare additions on FLW performance.

SMS feedback. An RCT in Tanzania found that SMS feedback generated from data collected by CommCare increased FLW visit frequency. The approach hinges on the fact that FLWs’ visits are reported in near real-time to CommCare’s web portal, where visit data is monitored by FLW supervisors. The study found that SMS reminders that were escalated to a supervisor in the case of a missed visit improved FLW visit timeliness by 86%, compared to CommCare-using FLWs who did not receive SMS reminders [DeRenzi, 2012].

Self-Tracking Tool. An RCT in India was conducted to measure the impact of phone-based motivational messages on FLW performance [DeRenzi, 2016]. The intervention was randomized into two groups, one who received generic advice and encouragement messages irrespective of their performance, and the intervention group who was given a self-tracking tool allowing FLWs to monitor their own performance through visual graphs and audio messages. The study found that the intervention group made 24% more visits than the control group within the 12-month intervention period. The study also found a correlation between FLW performance and usage of the self-tracking tool. Within the intervention group, most FLWs used both web- and voice-based feedback channels, highlighting the demand for diverse feedback mechanisms in FLW programs.

CommCare has proven effective in improving antenatal care and delivery. In Nigeria, Pathfinder International found that CommCare increased the antenatal care visit quality score from 13.3 at baseline to 17.2 (out of 25) at end-line in antenatal care clinics. The study found that CommCare improved the quality of health counseling during ANC, and the frequency at which healthcare providers performed more technical aspects of care during antenatal care visits, particularly the increased provision of HIV tests from 67.5% to 82.2% [McNabb, 2015]. In India, a mobile partograph (mLabour) built on the CommCare platform was created to overcome barriers to partograph use in under-resourced health systems. mLabour provides labor unit staff with decision support, automatic graphing to replace the paper partograph, and reminders prompting clinicians to conduct patient exams [Khalid, 2015]. A preliminary study found that data collected through mLabour was more complete than the corresponding paper charts, and users reported that the application reduced patient neglect [Schweers, 2015].

CommCare has also been used to improve adherence to protocols by FLWs in low-resource settings. An RCT in South Africa found that FLWs using CommCare for cardiovascular disease screenings had no errors in calculating risk scores, compared to 3.8% error when using the paper tool [Surka, 2014]. Clinicians using a CommCare precursor for Integrated Management of Childhood Illness (IMCI) classification completed 20% more of the required steps on average [DeRenzi, 2008]. They also were able to more accurately classify diseases (90.9% versus 82.7%) that were more prevalent across clinics [Mitchell, 2013].

More recently, Terre des Hommes and partners implemented the Integrated eDiagnosis Approach (IeDA) interventions at large scale in Burkina Faso, which includes non-technology components and technology components, including using CommCare to guide health providers through the IMCI protocols. An independent, stepped-wedge cluster randomized study was conducted to assess whether the IeDA increased adherence to the IMCI guidelines during under-five child consultations [Sarrassat, 2019]. The intervention districts showed a much higher adherence rate, including that the FLWs in the intervention completed 79% of their tasks compared to 54% in the control areas, from a baseline of 48% (p=.0002).

FLWs were compared to nurses who classified the children separately. The FLWs in the intervention completed 79% of their tasks compared to 73% in the control areas, from a baseline of 75% (p=.04), showing a smaller but still important improvement in classification accuracy than the improvement in following protocols.

Additionally, significant improvements were found while classifying and prescribing malnutrition and dysentery. Though the sample size was small, the data also showed consistency around improvement in danger sign identification, correct referrals/hospitalisations and management of severe malaria or severe febrile illness.

In Wajir, Kenya, an intervention assessing the effect of a CommCare app on integrated management of acute malnutrition (IMAM) service delivery among 40 health facilities found a 25% reduction in reporting errors, as well as an improvement in adherence to treatment protocols. Among a control group using paper-based systems, 28% of weight-for-height Z-scores (WHZ) were miscalculated, and 17% of children were misdiagnosed, leading to their wrongful admission to the IMAM treatment program. Use of the CommCare app, which automatically calculated WHZ scores, allowed for 99.3% accuracy [Keane, 2018].

CommCare has also been used to improve the accuracy of disease screenings for HIV in South Africa [Mitchell, 2012], Acute Malnutrition in India [Chanani, 2015], nutrition-related stunting and anemia in children under the age of 2 in Indonesia [Htet, 2019] and Rheumatic Heart Disease (RHD) in Zambia [van Dam, 2015]. Two studies have found that CommCare improves medicine dosing [Segal, 2015] [Palazuelos, 2013]. In Guatemala, CommCare was used to improve efficiency in calculating prescription dosages by 20%, and decreased consultation time [Segal, 2015]. In Mexico and Guatemala, CommCare use resulted in a higher medicine dosing accuracy compared to paper-based tools [Palazuelos, 2013].

Several studies discuss the impact of CommCare on data collection and transmission in projects across water, sanitation, and hygiene, malaria, and HIV/AIDS programs. A water, sanitation, and hygiene study across Vietnam, Cambodia, and Mozambique found that CommCare improved the efficiency of water data quality transmission from water supply structures to upper administrative levels [Ball, 2013]. Research on the use of algorithms to evaluate data submitted by FLWs found that CommCare is able to identify false data with 80% sensitivity and 90% specificity, validating CommCare as a monitoring and evaluation (M&E) and data collection tool for frontline programs [Birnbaum, 2012]. Two studies discuss the impact of CommCare’s improved data collection on infectious disease control efforts. In Zimbabwe, improved transmission of malaria test data through CommCare resulted in faster and more accurate diagnoses [Dell, 2014]. A study by MEASURE Evaluation in Mozambique found CommCare to be a more efficient, effective, and cost-effective tool for monitoring HIV/AIDS patient adherence to treatment programs and appointments than paper-based systems [Nascimento, 2014]. An RCT in South Africa found that FLWs using CommCare for cardiovascular disease screenings took 75% less time to be trained than FLWs using paper-based screening tools, and 41% less time to diagnose patients for cardiovascular disease than those using paper tools [Surka, 2014].

As a result of improved data collection and accessibility, several studies have found that CommCare can impact supervision and communication within frontline programs. In one project in India, improved data completeness via CommCare resulted in a reduction in the average data transmission time from FLWs to supervisors from 48 days to 8 hours [Medhi, 2012]. The Mathematica study in Bihar found that FLWs who used CommCare were 31% more likely to communicate about coordinating home visits in their catchment area than FLWs who did not use CommCare (p=.018). It also highlighted challenges with supervision, such as low understanding and usage of a specialized CommCare application for supervisors, resulting in no substantial increase to FLW supervision (although none of the supervision indicators were statistically significant) [Borkum, 2015]. Both studies in India saw improved communication with the use of CommCare by FLWs, although only minor improvements in direct supervision were observed.

The study examining the use of CommCare on IMAM service delivery in Wajir, Kenya saw a large decrease in the amount of time for data to become available in the national Health Management Information System (HMIS) when using CommCare (1.3 days on average) compared to paper-based systems (approximately 40 days). Use of paper-based systems necessitated physically sending reports to offices for personnel to then enter into the HMIS, which sometimes led to data loss throughout the process. Use of CommCare eliminated this need for physical data transfer [Keane, 2018].

USAID commissioned an independent cost-effectiveness analysis for the ReMiND project in Uttar Pradesh (described above). The purpose of the study was to estimate the incremental cost per disability adjusted life year (DALY) averted and the cost of each death averted as a result of ReMiND intervention, as compared to routine maternal and child health programs without ReMiND [Prinja, 2018]. The researchers built an Excel model to estimate change in DALYs and cost from both a health system and societal perspective, assuming ReMiND over a 10 year period if ReMiND were scaled to all of Uttar Pradesh. Probabilistic sensitivity analysis was undertaken to account for parameter uncertainties. The analysis found that over a 10-year time period, implementing the ReMiND intervention in Uttar Pradesh averted 312 maternal and 149,468 neonatal deaths. This implies that the ReMiND program led to a reduction of 0.2% maternal and 5.3% neonatal deaths. From a health system perspective, ReMiND incurs an incremental cost of USD $205 per DALY averted and USD $5,865 per death averted. The cost-effectiveness of the program is higher than that of vaccines for cholera, typhoid, and haemophilus influenza type ‘b’ at current market rates in India.

A study by USAID’s Health Finance and Governance Project evaluated the cost-effectiveness of a CommCare-based Integrated Management of Childhood Illness (IMCI) application implemented by D-tree International [Kukla, 2015]. The study evaluated the ratio of the additional cost per FLW to the improved effectiveness per FLW of a pilot project involving 50 FLWs. The study concluded that “compared with the existing paper-based system, the mobile tool costs an additional $10.43 per annum for an HSA to improve his/her diagnostic and treatment accuracy by 1 percent.” Given the low marginal cost of increasing the number of FLWs using the tool, and assumed consistency in relative effect per FLW, the authors also estimated a cost-effectiveness of $1.07 per 5,000 FLWs, concluding that a much higher return on investment is achieved when taking mobile tools to scale.

Another study evaluated the costs of collecting and processing 24-hour dietary recall data in both Vietnam and Burkina Faso, comparing a CommCare-based dietary assessment platform (INDDEX24) to traditional pen-and-paper interviews (PAPI). Compared to PAPI, the cost of using INDDEX24 was lower in Vietnam ($820/respondent vs. $755/respondent, respectively) and slightly lower in Burkina Faso ($541/respondent vs. $538/respondent, respectively). In both countries, the initial costs associated with INDDEX24 (preparing surveys, purchasing devices) were offset by the higher costs associated with PAPI (data entry, cleaning, and processing) [Adams, 2021].

Independent evaluations have also put numbers on the savings that come from running a program on CommCare rather than on paper. Two separate studies found that launching CommCare would save the state of Colorado $15 million over 3 years [Colorado, 2021], and between $830,000 and $1.7 million a year for a government program in Burkina Faso [London School of Hygiene, 2018]. Separately, an analysis by USAID’s Development Innovation Ventures team found that one CommCare-supported attendance-monitoring program returned more than $24 in social value for every dollar invested [USAID DIV, 2021].

There are various studies that describe challenges with implementing CommCare. Several studies cited technical issues, such as broken phones that remained unfixed, lack of convenient power source to charge devices, and inconsistent connectivity which limited data sharing and synchronization among FLW teams [Chaiyachati, 2013] [Borkum, 2015] [Style, 2017] [Adler, 2020] [Gopalakrishnan, 2020] [Nigussie, 2021].

Other studies also identified programmatic issues that limited CommCare’s impact. One challenge with large-scale digital health implementations is that adopting organizations may have limited IT capacity to adequately maintain technology [DeRenzi, 2012] [Chaiyachati, 2013]. Low adoption rates among users or a decrease in usage over time, often referred to as the ‘novelty effect’ of new technology, has also been observed in CommCare projects [Borkum, 2015] [Chaiyachati, 2013] [DeRenzi, 2012] [DeRenzi, 2016] [Segal, 2015]. Chaiyachati attributed low adoption rates to a disconnect between the application design and the actual value the application provided to FLWs in their everyday work. FLWs reported that they did not find the primary functionality of the application useful, although using the mobile phones for SMS and phone calls did improve communication between FLWs and their clients and coordination among FLWs [Chaiyachati, 2013].

It is also worth noting that what may be perceived as beneficial to FLWs through the use of digital health tools may not always be perceived the same way by or be most impactful for the client if careful considerations are not made. For example, the study examining FLW and client perspectives on a digital health intervention in Bihar and Madhya Pradesh, India found that although the majority of interviewed FLWs and clients saw the tool as a facilitator to engagement and communication, some clients felt that the use of certain features intended to improve healthcare delivery, such as video, ultimately limited interpersonal communication and led to rushed interactions, which may discourage clients from being comfortable enough to share health history or ask questions. Additionally, there may be socio-cultural factors that could impact service delivery when implementing digital health interventions (such as customs related to leaving the home, or the practice of women returning to their natal homes during pregnancy and the postpartum period); these may be seen as barriers that can limit uptake or FLW engagement with intended clients if not considered ahead of time [Gopalakrishnan, 2020].

The Mathematica study also reported that CommCare’s supervisory functionality had several challenges (though it has since been improved), and that CARE had to provide extensive training sessions (16 3-hour sessions over the course of 8 weeks) to achieve an adequate level of impact [Borkum, 2015]. Other programmatic issues highlighted in CommCare projects include delayed top-up payments to FLWs [Borkum, 2015], limited stock of medical supplies (in this case, Tetanus Toxoid injections) [McNabb, 2015], concerns about long-term sustainability due to the cost of devices or desired features (i.e. centralized SMS delivery) [Adler, 2020], and the risk of inaccuracy in self-reported data [DeRenzi, 2012].

Additionally, a study of mWellcare did not find positive results. The mWellcare system used CommCare as a clinical decision support system and an e-health record storage system to drive integrated management of five chronic conditions. The trial found no statistical difference in this setting vs. primary care setting, which had only enhanced usual care for patients with diabetes mellitus and hypertension [Prabhakaran, 2019]. While no improvement was found across secondary outcomes of fasting blood glucose, depression score, total cholesterol, predicted 10-year risk of cardiovascular disease, or tobacco and alcohol use; however, the mWellcare arm had higher self-reported adherence to medications. The null result highlights the potential value of using nonphysician providers and improving access to needed medications which can be used while drafting public health interventions by policy makers.