Electronic copy available at: http://ssrn.com/abstract=2198749

COMMENTARY

january 5, 2013 vol xlviIi no 1 EPW Economic & Political Weekly24

Sukumar Vellakkal (sukumar.vellakkal@phfi .org) and Shah Ebrahim (Shah.Ebrahim@lshtm.ac.uk) are with the South Asia Network for Chronic Diseases, Public Health Foundation of India, New Delhi.

Publicly-Financed Health Insurance Schemes Concerns about Impact Assessment

Sukumar Vellakkal, Shah Ebrahim

This article stresses that any impact assessment of health insurance schemes is sensitive to the methodology as well as the data used for analysis. It is based on two recent studies evaluating the impact of publicly-fi nanced health insurance schemes on benefi ciaries.

In the past few years, governments have launched many publicly-fi nanced health insurance schemes targeting

people in the informal sector in several low and middle income countries (LMICs), including India. This has, in turn, led researchers to evaluate these schemes in terms of their impact on utilisation, out-of-pocket spending, and health outcomes. Two relevant and noteworthy Indian studies – “Why Publicly-fi nanced Health Insurance Schemes Are Ineffective in Providing Financial Risk Protection” by Selvaraj and Karan (2012), and “State Health Insurance and Out-of-pocket Health Expenditures in Andhra Pradesh, India” by V Y Fan, A Karan and A Mahal (2012) – are important attempts to con-tribute evidence on the impact of health insurance schemes, especially at a time when governments are infusing massive amounts of public money into these schemes but with limited rigorous assess-ment of their impact.

Recently, we concluded a systematic re-view on the impact of publicly-fi nanced health insurance schemes for the infor-mal sector in LMICs in which we exam-ined 34 relevant studies from different countries (Acharya et al 2012). In general, we found no clear evidence of protection from fi nancial risk, healthcare utilisation, or health outcomes for the insured popula-tion. In this context, we would like to com-ment on a few key issues of impact assess-ment of health insurance with special ref-erence to these two new studies on India’s public health insurance schemes and the subsequent commentary by T R Dilip (2012a, b) on the methodological issues.

Contrasting ResultsSelvaraj and Karan found that publicly-fi nanced health insurance schemes (including the Rashtriya Swasthya Bima Yojana (RSBY), Aarogyasri, and the public

health insurance schemes in Tamil Nadu and Karnataka) have increased out-of-pocket spending on healthcare. On the other hand, Fan et al found that in the fi rst phase of Andhra Pradesh’s Aarog-yasri scheme, out-of-pocket inpatient ex-penditure, and to a lesser extent outpa-tient expenditure, was signifi cantly re-duced. In our systematic review, we also noted contrasting fi ndings. For example, while examining evidence from Viet-nam, Axelson et al (2009) and Wagstaff (2010) found reduction in out-of-pocket spending for the insured, Wagstaff (2007) showed no overall impact on out-of-pocket spending on healthcare. Simi-larly, in China, Lei and Lin (2009) and Wagstaff et al (2009) found no evidence of lower levels of out-of-pocket spending on healthcare for the insured.

Selvaraj and Karan conclude that since publicly-fi nanced health insurance schemes are ineffective in providing fi nancial protection to the benefi ciaries, alterna-tive fi nancial mechanisms need to be ex-plored. However, before making any policy decision to continue or discontin-ue the schemes, it is worth examining the potential reasons for the low level of welfare impact on the benefi ciaries. We noted in our systematic review that sev-eral studies give various reasons that can undermine the welfare impact of health insurance in LMICs, including the pitfalls in design and implementation of the schemes, lack of scientifi c criteria for selecting private healthcare providers and fi xing the package rates/prices for various healthcare services under the schemes (provider payment mechanism), and the absence of proper public awareness of the schemes. As India’s healthcare system, like several other LMICs, is relatively un-regulated, less organised and pluralistic in healthcare delivery, these underlying factors may be relevant to the Indian scenario and require further in-depth investigation for informed policy decisions.

Methodological ConcernsIn a response to Selvaraj and Karan’s study, Dilip (2012a) questions their methodology of difference-in-differences (DID) and argues that the fi ndings are not robust. We also noted in our review that the results are quite sensitive to

Electronic copy available at: http://ssrn.com/abstract=2198749

COMMENTARY

Economic & Political Weekly EPW january 5, 2013 vol xlviIi no 1 25

various comparison methods adopted by studies. (For example, Wagstaff (2010) outlined different results by using sin-gle, double and triple DID methods with statistical matching). Since the basic principle of the methodology of impact evaluation is counterfactual analysis (what would have happened to the ben-efi ciaries in the absence of the interven-tion (control group)), impact is estimated by comparing counterfactual outcomes to those observed under the intervention (treatment group.) Therefore, the strength of any study on impact evaluation depends on how reliably the chosen control group mirrors the counterfactual situation.

There are several quantitative methods available for impact evaluation conducted either in experimental or non-experi-mental settings. However, no method is considered as the gold standard. Since a simple comparison of outcomes occur-ring pre- and post-insurance periods may be affected by extraneous factors occur-ring during the time period studied, ran-domised control trials (RCTs) conducted in experimental settings, with a base line survey conducted in parallel to launch-ing the programme and then repeated later at the time of measuring the impact, is often considered the preferred method. (King et al (2009) used RCTs for evaluat-ing Mexico’s health insurance programme in collaboration with the government.) RCTs are clearly hard to mount and require considerable collaboration with policy-makers and insurance agencies.

We noted in our review that in the absence of RCTs, quasi-experimental study designs are widely used. These include: (1) Regression discontinuity design, or RDD (comparing the healthcare-related outcome of those who are eligible for insurance at the margin to those who are just above the eligibility); (2) statistical matching methods, which include exact matching, propensity score matching, and coarsened matching, for generating a sub-sample of the control group that match with the treatment group (case) based on observable characteristics; (3) DID by comparing the healthcare outcome between the treatment and control groups of the post-insurance period to the pre-insurance period; and (4) instru-mental variable methods that require the

selection of suitable instruments. Based on the availability of suitable data sets, resear chers sometimes combine the DID methods with stat istical matching ones to generate treat ment and control group in both periods.

Selvaraj and Karan have used the DID method, comparing the health expendi-ture of 2004-05 (pre-insurance period) and 2009-10 (post-insurance period). Fan et al (2012) also used the DID method, but used two baseline periods: 1999-2000 and 2004-05 (double DID rather than a single DID). In our opinion, Selvaraj and Karan could have provided a more robust analysis had they considered 1999-2000 and 2004-05 as baselines for measuring the trends in health expenditure. It is rather more likely that further applica-tion of a statistical matching method and combining it with the DID with more baseline period data (as used by Wagstaff et al 2009 and Wagstaff 2010) may have yielded more robust results. However, the main issue here is that the available survey data, i e, the consumer expendi-ture survey (CES) of the National Sample Survey Offi ce (NSSO) does not provide information on whether the household is covered by health insurance. In this regard, the classifi cation of the sample as treatment and non-treatment group in both periods by distinguishing the districts to with and without enrolment in health insurance by Selvaraj and Karan is a good attempt to defi ne groups with the potential to benefi t from insurance from those who could not.

But to what extent can the interven-tion districts and control districts serve as the treatment and control group in each period? This question is related to the health insurance enrolment rate also, an issue raised by Dilip (2012b). Selvaraj and Karan have studied not only RSBY but also three state-level schemes. How-ever, RSBY is a national level scheme with enrolment in several states and has relatively more coverage than state government schemes. A recent study (Dror and Vellakkal 2012) found that RSBY covered only 10% of India’s population (as on 31 March 2011), which is only 28% of the below the poverty line (BPL) households (as per the Tendulkar Com-mittee estimate). Furthermore, this study

found that although RSBY operates in 24 of the 30 states/territories, it covers all the districts in only 10 states, and with focused coverage in few states, e g, over 55% of the total in three large states (Bihar, Uttar Pradesh and West Bengal). Moreover, not all those districts with RSBY intervention have signifi cantly cov-ered a majority of their BPL target bene-fi ciaries. This may lead to several biases with the impact evaluation, viz, (1) the interstate variation of health expendi-ture in general would affect the compar-ison of out-of-pocket spending between the intervention districts and control districts because the intervention dis-tricts will be over-weighted by a few states, and (2) the treatment group in the intervention district also includes peo-ple without health insurance (diluting any possible effect of insurance) and thus challenges the validity of the com-parison between treatment and control group at the district level.

Another issue raised by Dilip (2012b) is that Selvaraj and Karan’s study com-pares the out-of-pocket spending on healthcare of 1.35% of households in the 2009-10 population to the remaining 98.65%. In our opinion, this is not true; in fact, they compared the healthcare expenditure between the treatment and control group. More importantly, the analysis was made by classifying the population into income quintiles. As the main target benefi ciaries of the schemes are BPL households, the policy implica-tions of the results from the analysis relevant to the two bottom income quin-tiles are worth considering. Dilip also raises the related issue that some struc-tural and policy changes that happened between both time periods due to a plethora of changes in the health system might lead to changes in health expend-iture. However, Selvaraj and Karan’s study uses DID, and one of the reasons for using the DID method instead of a

available at

NEBS208, West High Court Road

Dharampeth, Nagpur 440 010, Maharashtra

COMMENTARY

january 5, 2013 vol xlviIi no 1 EPW Economic & Political Weekly26

simple pre- and post-comparative analysis is to control for these extraneous factors.

Data IssuesAs we know, RSBY and several state-level schemes have been implemented through multiple insurance companies, and the Aarogyasri scheme in Andhra Pradesh was also run by an insurance company during the study reference period of Fan et al. As none of the publicly available data-bases have the characteristics to facilitate sound impact assessment, the authors of both studies made good attempts to use the CES round undertaken by the NSSO. The general understanding about the CES round is that the information collected refl ects the expenditure incurred by the respondents and, therefore, these are essentially out-of-pocket spending.

The CES round collected information on health expenditure under two main categories: institutional (inpatient) and non-institutional (outpatient), with vari-ous subcategories. However, the health expenditure was recorded without any distinction in out-of-pocket spending by households, and the reimbursement from health insurance companies (or from the governments) to the household directly or through the hospitals. The utility of the CES round data for impact assess-ment would have been more if such a distinction of the health expenditure is made at least for the data of the post-insurance period. However, if it is as-sumed that the health insurance schemes are fully cashless, as both studies did, such a distinction is not necessary because the health expenditure data reported in the NSSO are essentially out-of-pocket spending alone as cashless reimburse-ments from the health insurance schemes do not refl ect in the CES round.

However, there is the possibility that the CES round of NSSO data 2009-10 did not comprise out-of-pocket spending alone but combined both out-of-pocket spend-ing and payments received by households from the schemes (insurance companies) directly or through the hospitals. The supporting documents distributed along with the CES round data lead us to believe that the health expenditure reported in the survey is not just out-of-pocket spending alone but a combined total of

both out-of-pocket spending and the contribution received from insurance companies.

Of the eight documents in the folder titled “Instructions to Field Staff” with the CES round of NSSO data 2009-10, the document named “ins66chap3.doc” (or the document titled “ins.66_1.0.pdf” in the NSSO website (NSSO 2010)), page number C-2, states:

The following are part of Consumer Expend-iture and should not be missed... Payments for medical care reimbursed or directly paid by insurance company.

More elaborately, in the same docu-ment (page number C-33, paragraph number 3.9.14):

...On the other hand, when an insurance company makes a payment to the sample household (or directly to a hospital under the “cashless” system) in settlement of a claim made by the household for medical reimburse-ment, the amount is to be shown as medical expenditure of the household against items 410 to 414. In other words, the value of medi-cal goods and services on which expenditure is incurred will be recorded in Block 9 or Block 10, EITHER if incurred by the house-hold itself, whether or not reimbursed by employer or insurance company, OR if paid by the employer or by the insurance com-pany directly to the hospital.

(In the instructions, “items 410 to 414” refer to the institutional (in-patient) health expenditure.)

The above statements clearly demon-strate that although the reported health expenditure in the NSSO survey was not out-of-pocket spending alone but a mix of this and the contribution from insur-ance schemes and other sources, the re-searchers have treated it as out-of-pocket spending only. Therefore, the CES round of NSSO data 2009-10 is invalid for mak-ing any impact assessment of health insurance schemes.

Furthermore, in similar context, as several studies have used the previous rounds of the CES round data (when the health insurance schemes were not in place) to measure the fi nancial burden of households due to health expenditure because the reported health expendi-ture was just out-of-pocket spending on healthcare at that time, there can be a possibility of doing similar analysis by misreading the CES round of NSSO data 2009-10 as it is out-of-pocket spending

on healthcare, so we invite the attention of other researchers also to consider the above issues before making any analysis with this data set.

Concluding RemarksIn recent years, several publicly-fi nanced health insurance schemes have been launched in the country and attempts are ongoing to assess their impact. Our main concern is that the impact assess-ment of health insurance schemes is sen-sitive to the methodology as well as the data used for analysis. Although the two studies have taken a practical approach, for planning purposes and to enhance more impact value of interventions, it is better to consider the limitations of the analysis and the generalisability of the fi ndings prudently.

Furthermore, the absence of a dedicated data set is one of the main constraints for robust impact assessment. As huge funds have been invested in several health insurance schemes, it is impera-tive that the data sets on the schemes should be designed with inputs from the research community and made available to them by the concerned authorities for an independent and unbiased assess-ment of the schemes. Further, we also appeal to the NSSO to collect more infor-mation relevant to the health insurance schemes and other similar social security schemes in the forthcoming surveys so that rigorous evaluation of these schemes can be made to inform policies.

References

Acharya, A, S Vellakkal, F Taylor, E Masset, A Satija, M Burke and S Ebrahim (2012): Impact of National Health Insurance for the Poor and the Informal Sector in Low and Middle-income Countries: A Systematic Review (London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London).

Axelson, H, S Bales, P Minh, B Ekman and U Gerdtham (2009): “Health Financing for the Poor Produces Promising Short-term Effects on Utilisation and Out-of-pocket Expenditure: Evidence from Vietnam”, International Journal for Equity in Health, 8(1): p 20.

Dilip, T R (2012a): “On Publicly-Financed Health Insur-ance Schemes: Is the Analysis Premature?”, Econo-mic & Political Weekly, 5 May, XLVII(18): 79-80.

– (2012b): “Why Use Consumer Expenditure Sur-veys for Analysis of the RSBY?”, Economic & Political Weekly, 1 September, Vol XLVII, No 35.

Dror, D and S Vellakkal (2012): “Is RSBY India’s Platform to Implementing Universal Hospital Insurance?”, The Indian Journal of Medical Research, 135(1): 56.

COMMENTARY

Economic & Political Weekly EPW january 5, 2013 vol xlviIi no 1 27

Fan, V Y, A Karan and A Mahal (2012): “State Health Insurance and Out-of-pocket Health Expenditures in Andhra Pradesh, India”, Inter-national Journal of Health Care Finance and Economics, pp 1-27.

King G, E Gakidou, K Imai, T Lakin K Moore, C Nall, N Ravishankar, M Vargas, M M Téllez-Rojo, J Hernández Ávila, M Hernández Ávila and H Hernández Llamas (2009): “Public Policy for the Poor? A Randomised Assessment of the Mexican Universal Health Insurance Pro-gramme”, Lancet, 373(9673): pp 1447-54.

Lei, X and W Lin (2009): The New Cooperative

Medical Scheme in Rural China: Does More Coverage Mean More Service And Better Health?”, Health Economics, 18 Suppl 2: pp S25-S46.

NSSO (2010): “Instructions to Field Staff, Consumer Expenditure Survey Round 66”, National Sample Survey Offi ce the Ministry of Statistics and Programme Implementation, http://mospi.nic.in/Mospi_New/upload/nsso/ins.66_1.0.pdf, accessed on 31 August 2012.

Selvaraj, S and A K Karan (2012): “Why Publicly-fi nanced Health Insurance Schemes Are Inef-fective in Providing Financial Risk Protection”, Economic & Political Weekly, Vol XLVII (11): 61-68.

Wagstaff, A (2007): Health Insurance for the Poor: Initial Impacts of Vietnam’s Health CareFund for the Poor”, The World Bank, Policy Research Working Paper Series: 4134.

– (2010): “Estimating Health Insurance Impacts under Unobserved Heterogeneity: The Case of Vietnam’s Health Care Fund for the Poor”, Health Economics, 19(2): pp 189-208.

Wagstaff A, M Lindelow, G Jun, X Ling and Q Juncheng (2009): “Extending Health Insurance to the Rural Population: An Impact Evaluation of China’s New Cooperative Medical Scheme”, Journal of Health Economics, 28(1): pp 1-19.