Introduction

In Brazil, the first known case of COVID-19, a disease caused by the SARS-CoV-2 virus, was recorded on February 26^th 2020, and the first COVID-19 death was announced on March 17^th.¹ As at September 1^stthe pandemic had caused 122,596 deaths based only on those reported to the to the Brazilian Ministry of Health.¹ On August 28^th 2020, Brazil was the world’s second leading country in terms of COVID-19 deaths and cases.²

The University of Washington Institute for Health Metrics and Evaluation, based in the United States, began publishing COVID-19 projections on March 26^th 2020, initially focusing on that country. In May the Institute included Brazil in its projections, the estimates of which consist of daily and cumulative deaths attributed to COVID-19, number of infections and number of tests performed, hospital capacity and hospital resource needs, along with estimates of facemask use and social mobility for the forthcoming months.³

Continuous generation of data and projections on the course of the pandemic, using different control measure scenarios, can aid the formulation of policies intended to contain the pandemic. Notwithstanding, projections have intrinsic errors and their accuracy needs to be known.

Considering their potential utility for the country, the objective of this study was to describe the Institute for Health Metrics and Evaluation (IHME) projections for COVID-19 in Brazil and its states, present their accuracy and discuss their implications.

Methods

This was a descriptive study of the series of IHME projections for COVID-19 in Brazil, from May to August 2020, and the subsequent notification of deaths.

Since May 12^th, IHME has been releasing iterations of its projections for the COVID-19 pandemic (https://covid19.healthdata.org/brazil) in Brazil and in the Brazilian states.⁴Figure 1 illustrates daily deaths in Brazil according to the August 28^th projection. The models adopted were based on population size estimates retrieved from WorldPop 2020.⁷ The daily and cumulative number of deaths were retrieved from the Brazilian Ministry of Health website (https://covid.saude.gov.br/).¹ The number of cases and number tests performed were retrieved from federal and state government websites, expressed as the average of the last three days in order to minimize fluctuations arising from data update delays at weekends and on public holidays.⁵ Hospital capacity data were retrieved from websites of the Brazilian government, the Organization for Economic Cooperation and Development (OECD) and the World Health Organization (WHO), as well as from published studies.⁴

a) The solid line is the course of the pandemic as at August 28^th 2020. The two almost overlapping upper dotted lines to the right of the August 28^th reference axis, represent the base projection (the lower of the two lines) and the alternative projection in the context of continuing mandate relaxation, regardless of the course of the pandemic. The dotted line further down shows the projection considering adoption of universal facemask use.

Figure 1 – Projection of the daily number of COVID-19 deaths in Brazil, made by the Institute for Health Metrics and Evaluation (IHME) and provided on August 28th 2020, considering three scenariosa 

The Global Burden of Disease (GBD) Brazil network, comprised of Brazilian researchers affiliated with IHME, along with IHME’s central team, studied the social distancing measures decreed by the state governments, including decrees and ordinances published weekly on state government websites.⁵ The social distancing measures were classified using an adapted version of the New Zealand 4-Level Alert System.⁸ They included education institutions and non-essential business ordered to close, people ordered to stay at home, and strict travel restrictions.⁸ More recently they have included partial restriction orders: restrictions on any level of crowding and on any type of business closure. At the time of this report´s conclusion, only measures that apply to the entire population of each state were being used in the estimates.

Mobility measurements based on anonymous cellphone data were obtained from Facebook and Google for all states, as well as from Apple for some states.⁶ Facemask use was self-reported, based on periodical surveys of people’s adherence to this personal protective equipment (PPE) when going outside of their homes, as part of the Facebook Global Symptom Survey.⁵

The model used by the IHME in August 2020 is a hybrid model. It combines statistical modeling of case curves and death curves with modeling of disease transmission based on estimates of fractions of the population – at each location, ‘susceptible to’ or ‘exposed to’, ‘infected by’ or ‘recovered from’ SARS-CoV-2 (the SEIR model).⁵

Initially, death trends were modeled to estimate the number of deaths. Taking this number, the number of cases at projection start date was then estimated. Cumulative death data was smoothed using an algorithm based on splines with randomly distributed nodes.⁵ The effect of factors that influence the SARS-CoV-2 transmission rate was estimated, based on United States data, by testing for associations between these factors (covariables) and the course of the pandemic. The fixed covariables considered in the model were (i) population density, (ii) tobacco smoking prevalence, (iii) environmental pollution and (iv) altitude; while the covariables subject to variation considered in the model were, (i) social mobility, (ii) facemask use, (iii) number of COVID-19 tests performed and (iv) seasonality. The ratio between weekly pneumonia mortality and average annual pneumonia mortality for each location was assessed by IHME as being the best way of estimating the effect of seasonality on transmission.⁴ These covariables were included in the SEIR model, thus enabling assessment of variations in the projections of future transmission according to possible changes – e.g. increased facemask use.⁵

In order to minimize the impact of inconsistent reports on the number of deaths, the forecasts were based on the mean of several projection iterations and were published with uncertainty intervals.⁴

Need for hospital resources was estimated for resources available only for hospitalized COVID-19 cases.⁴

The first two projections, released on May 12^th and 25^th, were based on the IHME May 4^th model and included eight and nineteen Brazilian states, respectively. The third projection, dated June 5^th, was based on the May 29^th model and included all 26 states and the Federal District.³

The June 24^th model added two alternative scenarios to the base projection:³ the first scenario assumes that decrees (mandates), once removed, will not be reimplemented; the second scenario adds universal facemask use in public places to the base projection, with an increase to 95% use within 7 days.³ The assumptions included in the assessment of the base projection rates were (i) facemask use as at the projection date, (ii) flexibilization of social distancing mandates, based on recent trends, and (iii) reimplementation of restrictive decrees if daily mortality rates reached 8 deaths per 1 million inhabitants.

Based on publicly available data, the evolution of the IHME projections for COVID-19 deaths was described per day and cumulatively, for Brazil as a whole and for four states.

The projection data were retrieved from the IHME website (http://www.healthdata.org/covid/data-downloads) on August 28^th 2020; while the data on observed deaths were retrieved from the Brazilian Ministry of Health internet portal (https://covid.saude.gov.br/), also on August 28^th 2020.

Data on four states – one state for each of four of Brazil’s geographic regions: North, Northeast, Southeast and South – were assessed right from the first forecast. The four states were chosen because at the time of the first projections they had the highest estimated number of deaths among the other states in their respective regions.

The projected death estimates were compared, using graphs, with the number of subsequent deaths reported by the Brazilian Ministry of Health. This was done using projections 1, 3, 5, 7, 9 and 11 from the 11 projections published as of August 28^th 2020. The error rates were also calculated for the 1^st nine projections, comparing the projected number of cumulative deaths – at two, four and six weeks after the projections were released – with the cumulative death count according to the Brazilian Ministry of Health in the same period. In order to assess the overall accuracy of the IHME projections, the Mean Absolute Percentage Error (MAPE) was calculated for these projections.

The Metrics package of the R 4.0.2. software was used to perform the analyses.

Results

As of August 28^th 2020, the IHME had published 11 projections for Brazil and its states. The May 12^th projections estimated that a total of 88,305 deaths would have occurred in eight states by August 4^th; while the May 25^th projections estimated a total of 125.833 deaths in 19 states. The June 5^th projection estimated a total of 165,960 deaths for all the Brazilian states by August 4^th. The June 24^th projection was extended until October 1^st and estimated a total of 166,362 deaths. The July projections were extended until November 1^st and the August projections were extended until December 1^st. The cumulative number of deaths per state and for Brazil as a whole, according to the IHME projections, are shown in Supplementary Material 1.

The August 28^th base projection for the epidemic curve estimates a total of 182,809 deaths [uncertainty interval (UI): 165,415 – 202,948] by December 1^st. When it is expressed in deaths/day (Figure 1), the forecast is that after a long peak the epidemic curve will fall, with the fall accelerating from the second fortnight of September. The dotted lines to the right of the August 28^th reference axis are the projections: the slightly lower line of the two upper and almost overlapping dotted lines corresponds to the base projection, reaching approximately 350 deaths/day by December 1^st. The dotted line further down the graph, however, shows that universal use of facemasks could lead to a much lower number of deaths: 160 deaths/day. The base projection, projection, though not shown in Figure 1, also estimates that between the end of August and December 1^st universal facemask use could avoid a total of 17,351 deaths; and even with universal facemask use, on December 1^st there would still be >25,000 standard hospital beds and >5,000 ICU beds occupied by individuals with COVID-19 complications in Brazil as a whole.

Figure 2 shows the evolution of the IHME mortality projections for Brazil as a whole, by date, along with the official Ministry of Health death figures represented by the thicker black line. The upper panel shows daily deaths; while the lower panel shows cumulative deaths.

Figure 2 – Projections made by the Institute for Health Metrics and Evaluation (IHME) and observed data provided by the Ministry of Health (MoH) for daily number of deaths (upper panel) and cumulative number of deaths (lower panel) due to COVID-19, Brazil, March – December, 2020 

Figures 3 and 4 and Supplementary Materials 2 and 3 show the evolution of the projections and observed deaths in four states: Amazonas, São Paulo, Paraná and Pernambuco. The thicker black lines represent observed deaths. In each figure the upper panels indicate the daily values; while the lower panels indicate the cumulative values. The results show a variable pattern in correct projections for the course of the epidemic between the states. Generally, the projections are better and more accurate in the short and medium term, in comparison to the long term. The August 28^th projection shows a gradual decline for all four states, although less so in Pernambuco.

Figure 3 – Projections made by the Institute for Health Metrics and Evaluation (IHME) and observed data provided by the Ministry of Health for daily number of deaths (upper panel) and cumulative number of deaths (lower panel) due to COVID-19, state of Amazonas, March – December 2020 

Figure 4 – Projections made by the Institute for Health Metrics and Evaluation (IHME) and observed data provided by the Ministry of Health for daily number of deaths (upper panel) and cumulative number of deaths (lower panel) due to COVID-19, state of São Paulo, March – December 2020 

Figure 5 shows errors for the IHME estimates of deaths at two, four and six weeks after the projections were published. At two and four weeks, the errors estimated for Brazil as a whole varied between 11% below and 52% above observed death rates; at six weeks, the error variation raged between 6% below and 48% above. The projections for the four states assessed were less reliable. The mean absolute percentage error for the cumulative number of deaths at two, four and six weeks was, respectively, 13%, 18% and 22% for Brazil as a whole, and greater for each of the four states, varying between 17% and 36% at two weeks, between 20% and 44% at four weeks, and between 22% and 81% at six weeks (Figure 5).

Note: The darker color indicates a higher error value.

Figure 5 – Percentage projection error, comparing cumulative COVID-19 deaths as estimated by the Institute for Health Metrics and Evaluation (IHME) at two, four and six weeks after the projection date, with the cumulative number of deaths reported by the Ministry of Health for the same period, and mean error (mean absolute percentage error: MAPE) between all projections, Brazil and the states of Amazonas, Paraná, Pernambuco and São Paulo, May – August 2020 

Discussion

Among the projections of the evolution of the pandemic focusing on Brazil, the IHME projections stand out due to their sophistication and level of detail. They provide estimates for all the Brazilian states and are frequently updated. Mathematical prediction models do not get future reality 100% right, especially with regard to new and complex etiological issues, such as COVID-19. The IHME projections contain errors which frequently are not small.

The errors, which are greater in the projections for the states, appear to stem from the difficulty in characterizing the variable and volatile contribution of society’s response to the epidemic curve. In relation to the state projections, this response is only captured at the state level by means of surveys of social mobility and facemask use, as well as by state government decrees on distancing. The approach taken by the IHME requires a decree to be in force throughout an entire state in order for it to be considered implemented, whereas many state decrees allowed differentiated implementation between municipalities or regions.⁹ In these cases, the modeling considered that the decree had not been implemented, which may explain the overall trend of hyper estimated deaths in the projections. Moreover, adherence to the decrees varied between places and times, and municipal decrees were not taken into consideration. Another difficulty that may have contributed to projection errors was heterogeneity between states in underreporting cases and deaths attributed to COVID-19.¹¹ An additional source of error could lie in the difficulty in adequately taking into consideration the impact of social determinants – extensive and varied inequality present in Brazilian society and the real impossibility for many Brazilians to isolate themselves from the virus in view of their living and working conditions.¹² Furthermore, despite the frequency of their publication (fortnightly), there is also the difficulty of the projections capturing the huge volatility present in the state epidemic curves.

The variable accuracy of the IHME forecasts, inherent to projections based on mathematical models, illustrates the importance of analyzing them together with other projections and/or regional/local data. The volatility of the epidemic curves requires estimates from other sources and regional/local data to also be taken into consideration for planning COVID-19 prevention, control and care actions.

Other sources of projections, possibly useful to health service managers, include certain national and international institutions. The websites of the Federal University of Minas Gerais (UFMG) (https://dest-ufmg.shinyapps.io/app_COVID19//) and the Federal University of Rio Grande do Sul (UFRGS) (https://covid19.ufrgs.dev/tools/predictions) provide short- and long-term forecasts. The University of São Paulo (USP) website offers a SEIR model for forecasting the continuity of the pandemic in Brazil and in the Brazilian states (https://ciis.fmrp.usp.br/covid19/).¹⁴ The projections made by Imperial College London (ICL) (https://covidsim.org/v2.20200806/?place=Brazil) and by Youyang Gu (https://covid19-projections.com/brazil), the latter using a machine learning tool, as well as those made by the Los Alamos National Laboratory (https://covid-19.bsvgateway.org/) and by Delphi (https://www.covidanalytics.io/projections) are recognized internationally.

In terms of accuracy, available data suggest that errors in relation to COVID-19 death estimates for Brazil made by international groups are similar, except those made by ICL which present a much greater overestimation.³ Compared to the cumulative mortality predictive validity of the other forecasts, the IHME projection had the least errors for the Latin American and Caribbean region.³

The utility of the IHME projections is facilitated by the availability on the IHME website of easy to view graphs of the states and by the models being frequently updated, incorporating novelties regarding exposure rate and spread of the disease. An example of this is the rapid incorporation of seasonality estimates into the projections which enables, for instance, forecasts (generally correct) up to August, in view of increased spread of the disease given, among other factors, to the increase in indoor social interaction during the winter in the South Brazilian states.

The estimates are, however, not up to date with regard to availability of hospital resources, as they do not take into account the increase in hospital beds and equipment, such as temporary field hospitals, installed in response to the pandemic.

Of additional importance is the fact that the COVID-19 pandemic projections in Brazil and worldwide are generally underestimated, given that diagnostic tests are far from universal, especially among deaths not occurring in hospital.¹¹ In this aspect, Brazil stands out due to the low coverage of tests performed. Underreporting of deaths can be estimated by comparing the number of deaths recorded in civil registries during the pandemic with the number of expected deaths, i.e. the average number of deaths expected for the same period based on the five preceding years. Indeed, this comparison shows that deaths attributed to COVID-19 as at June 20^th 2020 accounted for just two thirds of the 74,172 excess estimated deaths at that date.¹⁹ As the models aim to predict the official death count, deaths not registered officially generate underestimates and thus limit the accuracy of the models.

Moreover, the rates of confirmed cases in the official statistics are underestimated, not just by the limitation of the number of tests performed, but also because many asymptomatic or milder cases tend not to seek medical care. A nationally representative seroprevalence study suggests that for each officially reported case, there are five undetected cases.²⁰ The IHME estimates of prevalent cases, however, are less affected by this problem, since they are calculated indirectly based on case fatality ratios.

Some limitations of this article are worthy of note. Firstly, as mentioned above, problems with official estimates of cases and deaths limit the ability to compare the IHME forecasts with the reality observed. Secondly, a simple accuracy measure was opted for, instead of other more sophisticated measures, with the aim of presenting results that are easy to visualize and interpret. Finally, it is not possible to compare the accuracy of projections made by Brazilian institutions with the IHME projections, since the Brazilian institutions do not provide their previous estimates. These are errors and limitations that should not overshadow the utility of the IHME projections, whose models, when taken together with other information, offer valuable data for guiding public policies in the short and medium term.

Of particular interest for health authorities are the estimates of hospital beds needed, based on the two alternative mortality scenarios produced: (i) continuing mandate relaxation; (ii) rapid increase in facemask use.

A contribution to be highlighted at this pandemic time relates to the appreciable gain that could be achieved by greater facemask use. Based on emerging evidence of the importance of facemask use,²² the World Health Organization has recommended their use. Use of cloth facemasks, which has been frequent in Asian countries where containment of the pandemic has met with greater success, may have contributed to avoiding transmission, especially when it takes place via asymptomatic cases.²⁴ The IHME projections show that facemask use by 95% of the population could avoid approximately 17,000 COVID-19 deaths in Brazil up to December 1^st. Use of masks in Brazil and Latin America is in general greater than that found in the United States and several European countries;³ even so, the projections show that even greater use of masks would produce more favorable results for Brazilian society.

A second contribution of the IHME projections, in line with those of other sources of analysis and research, is the forecast of the pandemic being slow to end and probably extending into 2021. Although less reliable, when taken together, long-term estimates made by several institutions suggest that the pandemic will continue to impact Brazil for at least a further six months. The IHME estimate projected at the end of August, of around 350 deaths/day as at December 1^st, shows an even longer path to be trailed, with renewed impact and consequent challenges for Public Health. The Youyang Gu projections support this prediction.³ The UFMG model released on August 30^th forecasts that the epidemic will extend into 2021, with approximately 50 deaths/day at the beginning of next year.¹⁵ Imperial College London projects a significant second wave at the same time next year in Brazil and in other countries.²⁶

New positive factors, such as efficacious treatment and mass vaccine availability, as well as rapid antigen tests earlier than expected, or negative factors, such as virus mutations and introduction of new and more infectious or virulent strains, are equally important issues to be considered. Unless there are unexpected positive changes, the outlook provided by current projections is one of intense work for several months before the pandemic in Brazil can be overcome.

Finally, International Monetary Fund (IMF) estimates show even higher falling gross domestic product (GDP) rates in countries more affected by the pandemic, including a 9.1% fall in the Brazilian GDP, this being a compelling illustration of the fallacious dichotomy spread by sectors of government and public opinion between protecting health and protecting the economy. The projections made by the IHME and other sources indicate that there is no short-term solution to the pandemic, suggesting that harm to the economy will be worse if control of the pandemic is delayed.²⁷ Comparison of Brazil’s economic perspectives with those of other countries, especially China and South Korea, which have been more capable of controlling the virus, demonstrates the success of actions which although very economically harmful in the short term, have been essential for the health of the economy in the long term. The more public officials and service managers and society understand the need to invest for limited periods in Public Health measures to contain the virus, the more both the nation’s health and its economy will be benefitted.

We conclude that the IHME projections for COVID-19 in Brazil demonstrate figures which, although not perfect, are close to reality. The bigger differences found in the medium and long term indicate that in order to make use of projections, one has to be aware of their imperfections, and always emphasize the importance and the need for the model to be constantly updated. Notwithstanding the limitations found, the projections available on the Institute for Health Metrics and Evaluation (IHME) website can be incorporated into the evidence base for decision making to address the pandemic.