Introduction

The Notifiable Diseases Information System (SINAN) was created in the 1990s¹ and has been evaluated according to attributes of quality, coverage, completeness of form filling and reliability, based on secondary data on specific diseases.^2,3 Disease control institutions recommend that surveillance systems evaluations be focused on specific diseases,⁴ in order to ensure the efficient and effective monitoring of important Public Health problems.^5,6

When evaluations have specific dimensions their usefulness is restricted,⁷ with gaps remaining in knowledge about the complete process of information production. There is a scarcity of evaluation research addressing the totality of health surveillance information systems, such as studies that have identified weaknesses in organizational settings and in the data collection, processing, transmission and dissemination stages,^8,9 so as to contribute to the development of strategies favouring better information coverage, regularity and quality.^9,10

Accurate and timely information is needed in order for epidemiological surveillance to be effective.^1,11 The globalized world, characterized by individual mobility and the constant flow of groups of people between countries and regions, requires structured services capable of providing quick responses to Public Health emergencies and monitoring national and international agreements and commitments.^11,12 The shortage of SINAN evaluation studies hinders identification of flaws in information generation, causing repercussions for the decision-making process.

The diversity of methodological approaches, in turn, enables more in-depth studies as to the operational adequacy of systems, ranging from data collection to information being publicized, whereby studies that encompass the entire information production process are useful.⁹ The objective of this study was to evaluate SINAN implantation in the state of Pernambuco in 2014.

Methods

This is an evaluation study using implantation analysis, with the purpose of examining the influence of variation in the degree of implantation of an intervention on the effects observed.¹³ The single-case study strategy was used,¹⁴ focused on the state of Pernambuco and its central and regional health levels.

The SINAN system is in operation in the twelve political/administrative regions of Pernambuco, corresponding to its regional health divisions which cover the state’s 185 municipalities. The purpose of the system is to collect, process, transmit and disseminate epidemiological data which are generated by health professionals as part of the service routine. Cases of compulsory notifiable diseases are registered on investigation forms which are sent to the municipal epidemiological surveillance service which is responsible for inputting them on SINAN, taking control measures and closing investigations based on case evolution.¹

Our evaluation was carried out in four stages:

Stage 1 - Preparation of the logic model for SINAN

In order to detail the intervention we evaluated, we designed the logic model for SINAN at state level (Figure 1) based on the trinity of structure-process-result¹⁵ within the five technical components of an information system: management; notification and investigation; monitoring; data processing; information analysis and publicizing. The analysis was built based on the following normative documents: Normative Instruction SVS/MS No. 2/2005; manuals (SINAN Net, norms and routines 2007; SINAN Online Operation Manual, SINAN Reports); and ministerial ordinances (SVS/MS No. 201/2010, issued by the MoH Health Surveillance Secretariat; and GM/MS No. 1.271/2014, issued by the Health Minister’s Office).

a) CND: compulsory notifiable disease.

Figure 1 - Logic Model for state-level Notifiable Diseases Information System (SINAN), Pernambuco, 2014 

Stage 2 - Building of the indicator and judgement matrix

The indicator matrix and judgement criteria were prepared based on the state-level SINAN logic model. All the indicators were submitted to system technical staff and managers in order to validate the construct and the criteria. An adapted version of the nominal group technique was used in two meetings with the participants, to whom these documents had been sent previously (Figure 2). When selecting the indicators we considered content validity, relevance, availability, ease of obtaining, calculation simplicity and timeliness, dividing the scores pre-established for the components between them. When defining the judgement criteria, we used the parameters set out in the legal instruments and/or scientific studies and if there were no parameters we created them in accordance with the service routine.

a) CIEVS: Health Surveillance Strategic Information Centre.

b) CND: compulsory notifiable disease.

c) ICND: immediate compulsory notifiable disease.

d) EW: epidemiological week.

Note: Scores are not attributed to results, nor are they included in the degree of implantation.

Figure 2 - Indicator and judgement matrix by Notifiable Diseases Information System (SINAN) compnents, Pernambuco, 2014 

Figure 2 shows the matrix of the 65 indicators by component, with 17 in the ‘structure’ dimension, 28 in the ‘process’ dimension with the judgement criteria and 20 in the ‘result’ dimension, which was not considered in the definition of the degree of implantation. Based on the indicators selected, we created the data collection instrument according to the SINAN system components.

The primary data were collected in November and December 2014, by means of interviews in which a questionnaire with 13 open questions was administered with all 13 epidemiological surveillance managers and 13 technicians responsible for SINAN at the central level and at the 12 regional health divisions; during this period we also undertook non-participant direct observation of structural and procedural aspects. The secondary data were taken from the normative documents and also retrieved from the SINAN database for the reference year (2014), in March 2015, in order to analyze the results indicators.

Stage 3 - Classification of degree of implantation

In order to define the degree of implantation, we used structure and process indicators according to SINAN’s five technical components. The participants of the nominal group technique gave a score to each of the components according to its relevance in making the state-level SINAN operational: management (25 points); notification and investigation (10 points); monitoring (20 points); data processing (20 points); and information analysis and publicizing (25 points). Degree of implantation was calculated taking the sum of the scores obtained in relation to the maximum scores foreseen for dimension, component, regional health divisions and central health level. Degree of implantation at the regional level was obtained by the arithmetic mean of the scores found for each of the 12 regions, while degree of implantation for the state as a whole was obtained by the arithmetic mean of the regional and central level scores. Degree of implantation was classified as follows: implemented, when percentages varying between 80.0 and 100.0% were achieved; partially implemented, 60.0%-79.9%; incipient, 40.0-59.9%; and not implemented, below 40.0%. This classification was defined by the authors based on a prior evaluation study of an information system.⁹

Stage 4 - Analysis of the results and analysis of the influence of degree of implantation on the effects observed

For analysis of the results (effects), we considered the indicators contained in the SINAN indicators and judgement matrix (Figure 2). When classifying the degree of completeness of the tuberculosis and leptospirosis notification forms we used the model proposed by Malhão et al.¹⁶ The analysis of degree of implantation done in stage 3 was compared with the result indicators, using a deductive process based on the SINAN logic model to identify elements bringing influence to bear on result achievement.

In order to increase the robustness of the system implantation analysis, we triangulated information regarding the structure and process dimensions contained in the interviews with the non-participant direct observation at the 12 regional health divisions and at the central health level, as delineated in the single-case study of SINAN/Pernambuco. Triangulation was used as a strategy capable of adding rigour, amplitude and depth to the investigation.¹⁷

The study project was approved on November 21^st 2014 by the Prof. Fernando Figueira Institute of Integral Medicine Human Research Ethics Committee - Opinion No. 4488/14; Certificate of Submission for Ethical Consideration (CAAE) No. 488214.400005201. All the participants agreed to take part in the study and signed the Free and Informed Consent form.

Results

SINAN was found to be partially implemented in Pernambuco (69.2%), at the central level (77.2%) and the regional level (61.2%). The system was incipient in seven of the twelve regional health divisions. Implantation of the structure dimension was better than that of the process dimension at the central and regional level, with the exception of the notification and investigation component (Table 1).

With regard to the state as a whole, the notification and investigation (90.0%) and data processing (84.1%) components were implemented; however, management (56.8%) and monitoring (53.4%) were incipient, with 40.0-58.0% variation between the health regions for the former and 26.3-70.0% for the latter. The monitoring component was not implemented in six regions; information analysis and publicizing was partially implemented at state level (61.6%), varying from 40.0-84.0% between the regions (Table 1).

Table 2 shows the result indicators according to the SINAN components for the central and regional health levels. The management component indicator ‘number of SINAN plannings undertaken annually’ was inexistent for the state as a whole; and the notification and investigation component indicator ‘Regional consistency of notification volume’ was 28.7% implemented on both levels - regional and central -, varying from 24.1-36.9%. With regard to the monitoring component, the ‘timely closure of compulsory notifiable disease investigations’ (CND) indicator on the state level was 78.1%; while for the data processing component the ‘municipalities with ≥80% regular batch sending’ indicator showed considerable differences between the health regions, ranging from 0% to 100.0%. In relation to the information analysis and publicizing component, ‘completeness of tuberculosis notification forms’ reached a rate of 10.1 for the state as a whole, varying between 9.1 (good) and 12.4 (excellent) between the health regions, while ‘completeness of the leptospirosis notification forms’, reached a rate of 8.8 for the state as a whole, varying between 5.2 (poor) and 13.1 (excellent) between the health regions.

In Table 3, coherence was found between degree of implantation and results indicators. In the case of the notification and investigation component, which was found to be implemented (90.0%), the result for one of the three indicators, ‘regional consistency of notification volume’, was lower than expected (28.7%). With regard to the nine indicators of the analysis and publicizing component, which was identified as being partially implemented (61.6%), only one indicator, ‘proportion of epidemiological bulletins and profiles prepared and publicized’, had no score.

Discussion

SINAN in Pernambuco was found to be partially implemented, with variations between health regions generally coherent with the low effects achieved. The system does not fully meet its objectives owing to organizational shortcomings, despite being in operation for more than 20 years¹⁸ and despite being an important information production tool.^1,19

The routine activities undertaken by central level technicians to improve indicators of data completeness, duplication and inconsistency may conceal negative results in the health regions. In turn, the SINAN logic model and indicator and judgement matrix, which were validated by the state-level team without being appraised by specialists at other hierarchical levels, may have weaknesses in the study judgement criteria and may be subject to changes, given that the nominal group expresses the opinion of the participant without the inconvenience of interactions with multiple stakeholders. Moreover, the validity of the logic model is related to the quality of theoretical articulation and to the complexity of the interdependence between components. For this reason, the logic model can be applied to other places, with some adaptations; notwithstanding, the results cannot be extrapolated.^14,17

Management and monitoring were found to be the critical components of the state SINAN, this being reflected in the inexistence of planning and insufficiency of timely notification, case closure and data input. However, health information systems should provide reliable data identifying relevant events and enabling rapid responses to Public Health problems. In view of the flow of people between countries with a multiplicity of diseases, accurate information reinforces the importance of these systems for epidemiological surveillance.^18-20

In our case study, the SINAN management component in Pernambuco as a whole and in all its health regions was found to be incipient, with negative repercussions on system coordination. This may possibly have been observed because, although in Brazil information systems are coordinated at the three management levels (national, state and municipal), there is no full joint accountability between the different levels of the Brazilian federation.¹ This has repercussions on the quality of data produced, especially at local level. In Pernambuco, the evaluation of SINAN structure was better than the evaluation of SINAN process, despite the need for good quality internet access. These structural aspects of some of the health regions studied are similar to the finding of a study conducted in the state of São Paulo on the Mortality Information System (SIM), whereby there was greater availability of computers and staff and better access to fast internet in municipalities with larger populations.²¹

Well-structured health information systems are essential for accompanying the progress of national and local strategic indicators;²² including those agreed to in the Sustainable Development Goals (SDG),²³ in particular SDG 3 in relation to diverse communicable diseases, including neglected ones. On the international level, the lessons learned from the results of the Millennium Development Goals relating to mortality and morbidity show the need for developed information systems with compete and accurate information. The health challenges of the forthcoming decade cannot be addressed without effective information system management.^20,22-24

It was expected, given the length of time SINAN has been in existence, that the work process would be adequate, but this was not what our study found. Inconsistencies in the attributions of each management level⁹ can be related to the historical centralizing conception of systems the function of which, at the local level, has been restricted to data collection.²⁵ State management takes on the responsibilities of municipalities when they do not undertake them. The same occurs with the health regions,⁹ which perhaps explains incoherencies revealed in our study such as the incipient degree of implantation in the majority of these regions, which diverges from the good quality of the indicators of effects relating to the information analysis and publicizing component.

Shortcomings in completeness owing to poor filling in of forms by health service workers were not minimized by information being recovered at the state level. If monitoring and recovery of incomplete or inconsistent variables were a routine hospital or municipal epidemiological surveillance activity, this would enable greater knowledge of disease magnitude and profile, thus favouring planning and execution of strategic actions.^11,26

Despite monitoring being a relevant attribute of the state and health region levels, the effect indicators relating to timeliness were lower than expected. This fact arose from irregular periodicity and reflected on the promptness of the surveillance system and on the taking of measures for the prevention and control of health events.^5,26-28 These monitoring indicator results may indicate low acceptance of the surveillance system by the health workers involved. Usually, acceptance is greater when the usefulness of the information produced is recognized. To achieve this, health workers who produce the information need system managers to provide awareness-raising and training.^19,29

The degree of data processing implantation was coherent with the result indicators, probably because this is an essential routine for the system and which has been improved in the face of criticism as to coherence between variables, correction of mistakes and enhanced quality of information.^1,27-30 Notwithstanding, the state-level SINAN system should not be restricted only to these activities.

SINAN in Pernambuco was found to be partially implemented. This had repercussions on the results achieved and presented in our study. The management and monitoring components were the main obstacles to full system implantation, especially the aspects relating to work process, whilst its strengths related to notification and investigation and data processing. In order for SINAN to fully meet its objectives, services need to be reorganized and there needs to be greater mobilization of resources and greater investment in the qualification of surveillance and disease information systems involving epidemiological rationality, associated with new evaluation research to deepen knowledge regarding the weaknesses of health information systems.