Economic evaluation of hospital management information systems in Tamil Nadu , India

Introduction: Huge investments have been made in implementing computerized Hospital Management Information Systems (HMIS) across the world, including developing countries. Economic evaluation studies have shown that HMIS implementations are benefi cial. While measuring effectiveness, studies have compared the quality of data generated in the HMIS with paper-based systems and have shown that HMIS is more advantageous. However, not all HMIS implementations are successful. Many successful implementations are the result of a continuous process of re-engineering, with inputs obtained from the end-users at every stage of the implementation process. The aims of this study were to compare cost-effectiveness, end-user satisfaction and perspectives between HMIS and traditional paper-based system (TPBS). Materials and Methods: This cross-sectional study evaluates the HMIS in government hospitals in Tamil Nadu using a mixed method approach. The study sample included 24 hospitals (13 HMIS and 11 TPBS), and hospital staff in four professional categories (doctors, staff nurses, pharmacists and lab technicians). Results: Cost-effectiveness analysis showed that HMIS implementation has high cost and high effectiveness with an ICER value of 3301.33. The Likert scale used to measure end-user satisfaction levels found that end-users were highly satisfi ed with HMIS as compared to TPBS. Conclusions: The study demonstrated the high cost and high effectiveness of HMIS implementation. The ‘accuracy of data’, ‘content of the system’ and the ‘timely availability’ characteristics of HMIS were highly appreciated by the users, whereas in TPBS, the users appreciated the ‘user friendly’ nature of this system.


INTRODUCTION
In the recent years, there has been growing global emphasis on the need for Hospital Management Information Systems (HMIS).[3][4][5] These wide-ranging benefi ts include decreasing medical documentation errors, preventing adverse drug reactions, decreasing expenditures on manual and paper costs, preventing billing-related errors and ensuring proper follow-up care for patients seeking long-term treatment.There has been strong lobbying for investing in HMIS in health sector.For example, HIMSS recommendation to Obama administration and the 11 th Congress in US has recommended that a minimum investment of USD 25 billion be made in health information technology (IT) to help non-governmental hospitals and physician practices to adopt electronic medical records. [6]Developing countries have also started implementing HMIS in their healthcare setup.The 12 th draft 5-year plan in India, mentions the need to strengthen HMIS across the country [7] and a potential investment is expected in health IT in public healthcare sector in the next fi ve years.In south India, Tamil Nadu through Tamil Nadu Health Systems Project (TNHSP) has already started implementing a state-wide HMIS, which was started as a pilot in fi ve hospitals in 2009 and was expanded to cover 222 hospitals in 2011, and has had an investment of more than INR 100 crores.Since such a huge investment was made and TNHSP plans to further upscale and implement HMIS in tertiary-care government hospitals, there is a critical need to perform an economic evaluation at this juncture.This HMIS project was aimed at improving oversight and management of the While the initial cost for implementing HMIS is very high, the benefi ts are expected to accrue and a break-even is expected to be achieved in 3 to 13 years. [4]Since health sector resources are fi nite and scarce, a careful analysis is needed and strong evidence in terms of resource allocation should be generated.Economic evaluation in terms of cost-effectiveness analysis, will be useful in comparing two alternatives: Measuring the cost and effect and interpreting them in the form of an Incremental Cost Effectiveness Ratio (ICER).
Cost-benefi t studies try to measure outcomes only in monetary terms and this has limitations in approach, methodology and interpretation.Hence, a cost-effective analysis has been attempted in this study in order to address some of the limitations of doing a cost-benefi t exercise.In addition to generate comprehensive information useful for decision makers, evaluations of HMIS need to be multi-dimensional, covering many aspects beyond technical functionality and economic evaluation.Many studies have shown that not all HMIS implementations are successful and a critical failure was the HMIS implementation in Limpopo district of South Africa. [8]The lesson learnt was that success signifi cantly depends on the end-users who are expected to use the HMIS and have a signifi cant infl uence on the outcomes of implementation.Published literature shows that end-user satisfaction level is a proxy indicator of success of an Information System. [9]11] However, very few studies emphasize the importance of qualitative research methodology for evaluating an HMIS implementation.Merely implementing an HMIS will not automatically increase organizational effi ciency.Strategic, tactical and operational actions should be taken including management involvement, integration in healthcare workfl ow, establishing compatibility between software and hardware and most importantly, user involvement, education and training. [12]Hence, this study combined quantitative and qualitative approaches to do an evaluation of the HMIS implementation, costeffectiveness, user satisfaction and the perspectives of the major stakeholders i.e. the end-users.
Tamil Nadu has implemented a state-wide HMIS for all the secondary-care hospitals in the state.HMIS in Tamil Nadu was implemented in a phased manner in 2009 followed by a major expansion and by April 2012; only two districts were still using the traditional paper-based system (TPBS).However, no scientifi c evaluations have been conducted on this implementation project till date.Since two districts were still using the TPBS in their secondary care hospitals, it was possible to conduct this comparative study.This comparative study evaluates the cost-effectiveness of HMIS and also serves as a means to provide vital feedback to the implementing authority for current and future implementations.

Objectives
1. To compare the cost-effectiveness of HMIS with the TPBS in terms of data reliability.2. To compare the satisfaction levels of end-users where HMIS has been implemented with end-users using TPBS.
3. To compare the perspectives of end-users where HMIS has been implemented with end-users using TPBS.

MATERIALS AND METHODS
This is a cross-sectional study where quantitative research methodology was used for studying the fi rst two objectives, while qualitative research methodology was used for analyzing the last objective.The study was conducted in the state of Tamil Nadu during 2012.Data was collected in April, May, September and October 2012 from four districts.
The study sample for the fi rst objective was individual hospitals and for the second and third objective, it was the hospital staff in four professional categories namely doctors, staff nurses, pharmacists and lab technicians.24 hospitals (13 HMIS and 11 TPBS) were studied in total from four districts for the fi rst objective, 216 respondents were evaluated for the second objective and 202 respondents were interviewed for the third objective.
[15][16] The effectiveness of the systems measured in this study is based on the characteristics of data reliability, user satisfaction levels and the time saved for predetermined comparable activities in both systems.The cost calculation included capital costs like server, hardware, software and recurrent costs including the electricity, salary of staff, stationary, etc.For the calculation of ICER, the effectiveness measure of data reliability alone was taken into account.
For the purpose of this study, the following assumptions are made: Data reliability is directly proportional to the effectiveness of the system; user satisfaction is directly proportional to user participation levels and effectiveness of the system is directly proportional to the amount of time saved for carrying out a given task.Also, positive perspectives of the end-users would add to the user participation and overall effectiveness of the system, whereas a negative perspective from end-users would adversely affect effectiveness.However, both types of perspectives would provide a feedback mechanism for the system.The variables studied were data reliability, enduser satisfaction, time taken for comparable activities and user perspectives in both the systems.For calculating cost-effectiveness, 'data reliability' was measured as the common effect in both the systems.The cost-effectiveness study was done with a provider perspective.The cost for operating HMIS and TPBS in the study hospitals were obtained from secondary data from the departments.The costs were calculated in Indian rupees (INR).
For quantifying data reliability, records were checked in four outpatient departments: Outpatient (OP) registration, OP doctor's consultation, pharmacy and clinical laboratory in both the hospital groups.Data reliability is the sum of data completeness and data accuracy.Data reliability = 50 % data completeness + 50 % data accuracy.
For each complete record, a score of '1' was assigned and for each incomplete record, a score of '0' was assigned.Data completeness in a given record was matched against the mandates prescribed by the government.Similarly, a score of '1' was assigned for an accurate record and a score of '0' was assigned for an inaccurate record.Data accuracy was evaluated by verifying the information from various sources.The department-wise data completeness and data accuracy fi gures were calculated for each hospital.Similarly, department-wise data calculations were undertaken for all the hospitals included in this study.The department-wise data for all hospitals was added and the average department-wise data completeness and accuracy fi gures were determined.Since data was collected from four departments, 25 % weightage was given to each department.The average data reliability scores namely E hmis and E tpbs were calculated for hospitals in HMIS and TPBS groups, respectively.

Time measurements
All observed time durations were measured in seconds.For each department in each hospital, the average time spent for three activities (OPD registration, OPD consultation and pharmacy drug dispensation) were calculated.Then the average times spent for each activity in all the hospitals were added and a total average obtained.

Measuring cost
The ingredients approach to costing for both the systems was used.
The cost of any input to a production process is the product of the quantity used and the value (or price) of each unit. [17]In HMIS group of hospitals (13 hospitals), the cost for desktops, dot matrix printers, UPS and electrical cabling, local area network (LAN), TNSWAN connectivity, furniture, server, HMIS application development and training were considered as capital costs.All costs were proportioned over the lifetime of the respective hardware or till such time the warranty or AMC is covered.
For calculating recurrent cost, the following items were considered: Cost of electricity for the items directly used under HMIS, cost of broadband connectivity, cost for hosting the server at data center, cost toward salary of the district IT technical coordinator and the cost for printer cartridges.All the costs were proportioned for a single day.
Total Cost of operating HMIS, in a single day for 13 hospitals The average cost for operating HMIS in a single day for one hospital C HMIS = C hmis /13.
For the TPBS group of hospitals (11 hospitals), the cost for the registers, paper OP records, slips/tokens that were used in the 4 outpatient departments were taken into consideration and per day costs were calculated.The average cost for operating TPBS in each hospital was then calculated.
Average cost in TPBS per hospital per day C TPBS = C tpbs /11 Since the effectiveness data was calculated per day, the cost of implementing HMIS and TPBS was also calculated per day.
The ICER was calculated using the formula: This ICER score would be the incremental cost that is paid by THNSP for each hospital per day, for every additional unit of effectiveness gained.
Although there are several ways to measure end-user satisfaction, a scale developed by Doll and Torkzadeh [10] having a seven-point Likert scale [9] was used to quantify the end-users satisfaction for 10 different attributes [10] of the information systems in both the systems.216 end users (125 HMIS and 91 TPBS) were evaluated.
The seven-point scale ranged from 'extremely satisfied' to 'extremely dissatisfi ed'.The 10 different attributes for which enduser satisfaction was evaluated were precision, content, suffi cient information, just exact information, accuracy of data, output presentation format, clarity, user friendly, timely availability and upto-datedness.The end-user satisfaction levels were evaluated across four professional groups at each hospital namely doctors, staff nurses, pharmacists, and lab technicians.In order to quantify the user satisfaction level, the seven points on the scale were assigned numerical values starting with 7 for extremely satisfi ed, 6 for quite satisfi ed, 5 for slight satisfi ed, 4 for neither, 3 for slight dissatisfi ed, 2 for quite dissatisfi ed and 1 for extremely dissatisfi ed.This data was analysed using SPSS version 20.
In order to study the end-users perspectives on the advantages and disadvantages of their respective systems, the effect of the respective system on their individual performance and productivity, their view about computerization in general and with respect to the government hospitals, interviews were conducted using a semi structured interview schedule.All those who participated and responded to the end-user satisfaction questionnaire were invited to participate in the interview.This data analysis was done using ATLAS Ti software version 5.0.

RESULTS AND DISCUSSION
The day-wise cost for operating TPBS per hospital (C tpbs ) was INR 78.97 [Table 1] and for the computerized HMIS (C HMIS ) was 2511.80 [Table 2].The effectiveness scores E hmis and E tpbs were 0.98365 and 0.246725 respectively.Substituting these fi gures in the ICER formula, the ICER was calculated to be 3301.33,implying that the government has to spend INR 3301.33 for operating HMIS in a single day for each hospital for every additional unit of data reliability gained in the four outpatient departments.On plotting the ICER as shown in Figure 1 to pay by the implementing agency, in this case the government has to be taken into consideration.
By performing sensitivity analysis on parameters like changes in unit cost of electricity, consumption of electricity, consumption of stationery, cost of stationery, the change in ICER was minimal ranging from -1.78 % to +3.75 %, which indicated that the ICER calculated was robust over the changes in the cost or quantity of components that were varied.
Table 3 shows the time taken to perform four activities in both the systems The average time taken for registration of new patients in HMIS was 0.86 seconds lower for each patient compared to TPBS.Assuming there were 1000 new patients at OP per day, this would translate into 14.33 minutes of time saved on that day.Also, the average time to register each old patient in TPBS was 13.66 seconds, whereas this was completely avoided in HMIS.Assuming even 100 old patients attend OPD per day, this would translate to 22.76 minutes saved in the HMIS.At the same time with 14.33 minutes and 22.76 minutes of time saved for registering 1000 new patients and 100 old patients in HMIS, the average reliability score in HMIS (0.98365) is almost four times more compared to TPBS (0.246725).The doctors in HMIS spent 1.25 seconds more for each patient.Assuming each doctor consulted 200 patients per day; each doctor will spend an additional time of 4.16 minutes when compared to TPBS.Even though the doctor spends additional time, the data is stored permanently and can be reliably retrieved in the future, ensuring proper follow-up treatment of the patients.The pharmacists under HMIS spent an average of 5.18 seconds more per patient when compared with TPBS.Translating these numbers for 1000 patients per day, the pharmacist in HMIS has to spend an additional 86.33 minutes for the same patient load compared to TPBS, but the advantage for the pharmacists is that they can eliminate the need to count and account for the manual drug tokens after OP hours.
The mean score for the user satisfaction levels was 5.492 for the HMIS group with standard deviation of 0.992 and for the TPBS group, the mean was 4.731 with standard deviation of 1.39.The difference in mean was statistically signifi cant (P < 0.0001).Reliability testing on the seven-point Likert scale showed highly reliable with Cronbach's alpha score of 0.923.
There were highly signifi cant interactions between type of system (HMIS or TPBS) and professional categories (doctor, nurse, pharmacist, lab technician) independently on the user satisfaction, but they also have a highly significant interaction with user satisfaction when combined together.
For assessing the user's perspectives regarding HMIS and TPBS, a qualitative assessment was done.A total of 202 users participated in this assessment-114 from HMIS group and 88 from TPBS group.The perspectives of the users were arranged into broader themes and linked to advantages or disadvantages in the respective groups.

TPBS group
The end-users said that the familiarity with this system and the ability to make corrections (rewrite/overwrite) as well as review anytime and anywhere were the major advantages of TPBS.The disadvantages of TPBS included loss or damage to records while handling and storage, undue delays in accessing information, excessive writing work and the tendency to postpone work, which hampers maintaining up-to-date records.

HMIS group
The end-users perceived the ease in account maintenance and system transparency, improved data quality, improved access to information, better clinical follow-up, stress-free work nature, building individual responsibility, one-time registration for the patients and the improved prestige of the hospitals as the advantages of HMIS.Infrastructural issues regarding unstable connectivity, intermittent slowing of server speed during peak OP hours, strain on the doctor-patient relationship during down times of connectivity or server issues; insuffi cient training, computer illiteracy among hospital workers and the absence of an offi cial guideline or departmental code of conduct for working in HMIS were perceived as the disadvantages of the HMIS.
Most of the cost-effectiveness studies undertaken on HMIS have calculated the effectiveness based on reduction in medication errors, reduction in adverse drug reactions, improved follow-up of patients with system-generated appointment reminders, etc.There were no cost effectiveness studies in HMIS which had technically evaluated reduction in documentation errors and subsequent improvement in data reliability in outpatient settings, as effectiveness indicators in India.This study has limitations, as the effectiveness has been measured only in terms of data reliability and time saved and also for calculation of ICER only data reliability has been taken into account.
If a multitude of benefi cial factors are studied then probably the incremental cost shall comedown.The cost of implementing HMIS and the cost involved in operating it are high in this study.Based on these fi ndings one cannot contradict the widely prevalent notion that HMIS is cost effective.However, in the context of TNHSP, HMIS implementation has a higher effectiveness than the TPBS but with a higher cost.

CONCLUSIONS
The study demonstrated the high cost and high effectiveness of HMIS implementation.The ICER showed that the government has spent INR 3301.33 per day, for every additional unit of data reliability in the four outpatient departments for each hospital, which was robust to changes in cost/quantity components of some items.The study identifi ed higher user satisfaction levels in HMIS compared with TPBS, indicating that the effectiveness of HMIS is superior to TPBS.The 'accuracy of data', 'content of the system' and the 'timely availability' characteristics of HMIS were highly appreciated by the users, whereas in TPBS, the users appreciated the 'user friendly' nature of this system.
Considering that HMIS would eventually replace TPBS and since high investments have gone into this implementation, the implementing agency should adequately train the end-users.The government can initiate a common high quality training program on computer hardware, connectivity troubleshooting and the HMIS application for new recruits in the healthcare system.The implementing authority should establish a 'departmental code of conduct' for working in HMIS, for all the professional categories.The standard practices in operating HMIS should be made available to the end-user community.The implementing authority should build measures to ensure patient compliance and responsibility in retaining their respective registration details, which will ensure proper followup and also avoid duplication of records.Future research aimed at examining cost-effectiveness of HMIS, by calculating alternate measures of effectiveness would also prove useful.Policy makers and program mangers need to consider the other benefi ts, which may accrue from HMIS before taking a decision.

Study limitations
The study considered only data reliability as a measure of effectiveness for calculating cost-effectiveness.Introduction of HIMS has many more advantages than what was analyzed in this study e.g.quality of care, management and monitoring.Taking data accuracy, data completeness and time taken may not capture effectiveness in  all domains.Hence, the interpretation of ICER from this study has to be done with caution.
healthcare International Journal of Medicine and Public Health | Jul-Sep 2014 | Vol 4 | Issue 3 Equal weightage was given to both completeness and accuracy.Thus, for this study International Journal of Medicine and Public Health | Jul-Sep 2014 | Vol 4 | Issue 3 , the intervention HMIS lies in the north-east quadrant, thereby indicating that HMIS implementation is a high-cost, high-effectiveness intervention and the willingness International Journal of Medicine and Public Health | Jul-Sep 2014 | Vol 4 | Issue 3

Table 1 : Costing for the TPBS group
This centralized procurement and supply was made to all hospitals in the year 2007-08 and the unit cost was also derived from this supply.After this supply, there were no further centralized procurement and supply to the government hospitals, Source-Purchase order copy from Tamil Nadu Medical Services Corporation *

Table 3 : Comparison of average time taken to perform certain activities between the two groups Activity Average time in HMIS (seconds) Average time in TPBS (seconds)
International Journal of Medicine and Public Health | Jul-Sep 2014 | Vol 4 | Issue 3