Document of

The World Bank

Report No: ICR00003114

IMPLEMENTATION COMPLETION AND RESULTS REPORT

ON A

LOAN

IN THE AMOUNT OF US$300 MILLION

TO THE

STATE OF RIO DE JANEIRO, BRAZIL

WITH THE GUARANTEE OF THE FEDERATIVE REPUBLIC OF BRAZIL

FOR A

FISCAL EFFICIENCY FOR QUALITY OF PUBLIC SERVICE DELIVERY

DEVELOPMENT POLICY LOAN

July 17, 2014

Governance Global Practice

Brazil Country Management Unit

Latin America and Caribbean Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

CURRENCY EQUIVALENTS

(Exchange Rate Effective July 17, 2014)

Currency Unit = Brazil Real

R$ 1.00 = US$ 0.45

US$ 1.00 = R$ 2.21

FISCAL YEAR

January 1 – December 31

ABBREVIATION AND ACRONYMS

AGE Internal Audit Department Auditoria Geral do Estado

CPS Country Partnership Strategy Estratégia de Parceria com o País

DPL Development Policy Loan Empréstimo para Políticas de

Desenvolvimento

FY Fiscal year Ano Fiscal

GDP Gross Domestic Product Produto Interno Bruto

GIDE Integrated School Management Gestão Integrada da Escola

GORJ Government of Rio de Janeiro Governo do Estado do Rio de Janeiro

IBRD International Bank for Reconstruction

and Development

Banco Internacional para Reconstrução e

Desenvolvimento

ICMS Tax on Goods and Services Imposto sobre Circulação de Mercadorias e

Prestação de Serviços

IDEB Index of Development of Basic

Education

Índice de Desenvolvimento da Educação

Básica

LOA Annual Budget Law Lei de Orçamento Anual

LRF Fiscal Responsibility Law Lei de Responsabilidade Fiscal

PAC Growth Acceleration Program Programa de Aceleração do Crescimento

PAHI Program to Support Hospitals Programa de Apoio aos Hospitais do

Interior

PAF Program of Fiscal Adjustment Programa de Ajuste Fiscal

PFM Public Financial Management Gestão de Finanças Públicas

PPA Multi-year Plan Plano Plurianual

PPP Public-Private Partnerships Parcerias Público-Privadas

PSF Program for Primary Health Care Programa Saúde Família

REGIN Integrated Recording System for taxes Sistema de Registro Integrado

SAERJ Education evaluation system of the

State of Rio de Janeiro

Sistema de Avaliação da Educação no

Estado do Rio de Janeiro

SEEDUC State Secretariat of Education Secretaria de Estado de Educação

SEFAZ State Secretariat of Finance Secretaria de Estado da Fazenda

iii

SEPLAG State Secretariat of Planning Secretaria de Estado de Planejamento

SES State Secretariat of Health Secretaria de Estado de Saúde

SESDEC State Secretariat of Health and Civil

Defense

Secretaria de Estado de Saúde e Defesa

Civil

SIL Specific Investment Loan Empréstimo para Investimento Específico

STN National Treasury Secretariat Secretaria do Tesouro Nacional

SUS Public Health System Sistema Único de Saúde

SWAP Sector Wide Approach Abordagem Setorial Ampla

TAL Technical Assistance Loan Empréstimo para Assistência Técnica

UPA 24 hour emergency care unit Unidade de Pronto Atendimento

VAT Value Added Tax Imposto sobre o Valor Adicionado

WB World Bank Banco Mundial

Vice President: Jorge Familiar

Country Director: Deborah L. Wetzel

Senior Practice Director: Mario Marcel Cullel

Practice Manager: Arturo Herrera Gutiérrez

Team Leader: Roland Clarke

ICR Team Leader: Lorena Viñuela

ICR Lead Author Eduardo Somensatto

iv

CONTENTS

Data Sheet ........................................................................................................................................

1. Program Context, Development Objectives and Design ....................................................... 1

2. Key Factors Affecting Implementation and Outcomes ......................................................... 7

3. Assessment of Outcomes......................................................................................................... 10

4. Assessment of Risk to Development Outcomes .................................................................... 16

5. Assessment of Bank and Borrower Performance ................................................................ 17

6. Lessons Learned ...................................................................................................................... 19

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ................... 21

Annexes ........................................................................................................................................ 22

Data Sheet

A. Basic Information

Country: Brazil Program Name:

RIO STATE

DEVELOPMENT

POLICY LOAN III

Program ID: P126465 L/C/TF Number(s): IBRD-81910

ICR Date: 07/10/2014 ICR Type: Core ICR

Lending Instrument: DPL Borrower: THE STATE OF RIO

DE JANEIRO

Original Total

Commitment: US$US$ 300.00M Disbursed Amount: US$US$ 300.00M

Revised Amount: US$US$ 300.00M

Implementing Agencies:

Secretaria de Fazenda

Cofinanciers and Other External Partners:

B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 07/28/2011 Effectiveness: 11/30/2012 11/16/2012

Appraisal: 02/09/2012 Restructuring(s):

Approval: 08/30/2012 Mid-term Review:

Closing: 01/31/2014 01/31/2014

C. Ratings Summary

C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Low or Negligible

Bank Performance: Moderately Satisfactory

Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)

Bank Ratings Borrower Ratings

Quality at Entry: Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Moderately Satisfactory Implementing

Agency/Agencies: Moderately Satisfactory

Overall Bank

Performance: Moderately Satisfactory

Overall Borrower

Performance: Moderately Satisfactory

i

C.3 Quality at Entry and Implementation Performance Indicators

Implementation

Performance Indicators

QAG Assessments

(if any) Rating:

Potential Problem

Program at any time

(Yes/No):

No Quality at Entry

(QEA): None

Problem Program at any

time (Yes/No): No

Quality of

Supervision (QSA): None

DO rating before

Closing/Inactive status: Satisfactory

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

General education sector 20 20

Health 20 20

Sub-national government administration 60 60

Theme Code (as % of total Bank financing)

Education for all 20 20

Health system performance 20 20

Managing for development results 10 10

Public expenditure, financial management and

procurement 30 30

Tax policy and administration 20 20

E. Bank Staff

Positions At ICR At Approval

Vice President: Jorge Familiar Hasan A. Tuluy

Country Director: Deborah L. Wetzel Deborah L. Wetzel

Sector/Practice Manager: Arturo Herrera Gutierrez Arturo Herrera Gutierrez

Program Team Leader: Roland N. Clarke Roland N. Clarke

ICR Team Leader: Lorena Viñuela

ICR Primary Author: Eduardo Somensatto

ii

F. Results Framework Analysis

Program Development Objectives (from Project Appraisal Document)

The objective of the proposed operation is to assist the Government of Rio de Janeiro in

strengthening its tax administration, improving the efficiency of public financial management,

increasing the quality of public service provision in education and health and ensuring that

policies adopted are both consistent with priorities of the State Government and with resources

likely to be available in the medium term.

Revised Program Development Objectives (if any, as approved by original approving

authority) Not applicable.

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval documents)

Formally Revised

Target Values

Actual Value

Achieved at

Completion or Target

Years

Indicator 1 : ICMS tax annual revenue

Value

(quantitative or

Qualitative)

24.8 billion 37.7 billion 28.67 billion 30.75 billion

Date achieved 12/31/2011 12/31/2013 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved.

The baseline and target value included in the approved Program Document were erroneous.

The original baseline for 2011 was R$32.6 billion, but official data showed that it was R$24.8

billion. The target was 15.6% increase and the result 23.99%.

Indicator 2 : ITD tax annual revenue

Value

(quantitative or

Qualitative)

418 million 470 million 644.6 million

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved.

The target was 12.4% increase and the result 54%. This is not a normalized indicator.

Indicator 3 : Establishment of Unified General Taxpayer ledger, with access of individual taxpayers to their

unified accounts

Value

(quantitative or

Qualitative)

No Unified General

Taxpayer Ledger

Unified General

Taxpayer ledger, with

access of individual

taxpayers to their

unified accounts

Unified General

Taxpayer ledger has

been created, but it is

still in experimental

phase for internal

purposes. GoRJ

Authorities estimate

that individual

taxpayers will have

access to their unified

accounts in 2015.

iii

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was partially achieved.

The program required advancing through a series of sequential steps, before the ledger could

be fully operational. Those steps took longer and proved more difficult to effect than initially

anticipated.

Indicator 4 : Number of indicators adopted as part of the progressive implementation of performance based

management in tax administration

Value

(quantitative or

Qualitative)

0 indicators adopted 20 indicators adopted 20 indicators adopted

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved. The target number of indicators adopted was 10 by 2012 and 20 by

2013. Both goals were met.

Indicator 5 :

Number of public investment projects estimated to cost over R$50,000,000 subjected to the

new technical screening process and with consolidated information showing annual

commitments published

Value

(quantitative or

Qualitative)

0 prior investment

projects

shown in the annual

commitments of

investments

5 prior investment

projects

shown in the annual

commitments of

investments

0 prior investment

projects

shown in the annual

commitments of

investments

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target not achieved.

Three investment proposals were identified for screening. However, these were subjected to

an ex-post evaluation rather than ex-ante process and the costing information was not included

in the budget.

Indicator 6 : Number of major policy programs with detailed costing and key results defined and published

Value

(quantitative or

Qualitative)

0 major policy

programs with costing

and results published

4 major policy

programs with costing

and results published

3 major policy

programs with costing

(ex-post) and results

prepared but not

published

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was partially achieved.

With the analysis, SEFAZ was able to link the costs of the programs to the number of

beneficiaries, and begin to identify those factors, such as eligibility criteria, that could drive

future expenditures.

Indicator 7 : Implementation of an action plan to improve internal control system is developed on the basis

of the annual report from State Audit on the performance of internal control systems

Value

(quantitative or

Qualitative)

No plan or regular

reporting.

Action plan to improve

internal controls

developed on the basis

of the annual report of

the State Audit agency

to the State

Government on the

performance of

internal controls

Action plan to improve

internal controls

published and annual

report on human

resource reforms and

performance of

internal control

systems corresponding

to 2013 published

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved.

Additional reforms recommended by the 2013 report are under implementation. Indicators to

measure and monitor the process have been developed and used since 2012.

iv

Indicator 8 : Number of regional directors selected using new merit-based process

Value

(quantitative or

Qualitative)

0 28 35

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved.

18 regional supervising directors and 17 regional pedagogical directors represent a 100% of

the regional positions.

Indicator 9 : Number of school directors selected based on merit

Value

(quantitative or

Qualitative)

0 82 164

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved.

Additionally, 184 assistant directors were selected based on merit.

Indicator 10 : Annual bonuses paid to school personnel for achieving 80% of improvement targets.

Value

(quantitative or

Qualitative)

No bonuses paid Bonuses for 2012 paid

by June 2013

Bonuses for 2012 paid

by June 2013

Date achieved 12/31/2011 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Target was fully achieved.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval documents)

Formally Revised

Target Values

Actual Value

Achieved at

Completion or Target

Years

Indicator 1 : Annual school targets communicated to schools (before March)

Value

(quantitative or

Qualitative)

No annual

improvement targets

communicated

Annual school targets

communicated to

schools (before March)

2013 and 2014 annual

school targets

communicated to

schools (before March

of the previous year)

Date achieved 12/31/2011 12/31/2013 06/30/2013

Comments

(incl. %

achievement)

Target was fully achieved.

G. Ratings of Program Performance in ISRs

No. Date ISR DO IP Actual

v

Archived Disbursements

(US$ millions)

1 11/14/2012 Satisfactory Satisfactory 0.00

2 03/20/2013 Satisfactory Satisfactory 300.00

3 02/08/2014 Satisfactory Satisfactory 300.00

H. Restructuring (if any)

Not Applicable.

1

1. Program Context, Development Objectives and Design

1.1 Context at Appraisal

1. The Development Policy Loan (DPL) formed part of a continuous engagement of the

World Bank with the State of Rio de Janeiro. The operation was the third in a quasi-

programmatic series of DPLs that started in 2009 and supported important public sector and

service delivery reforms in the State. This engagement continues to date with follow-up policy

based operations and targeted technical assistance.

2. The initial discussions of a possible third DPL took place in the early stages of the

second term of Governor Sergio Cabral’s administration. The Governor had been re-elected at

the end of 2010, and had begun his second term with the intention of accelerating the public

sector reforms started during his first term (2007-2010). The new set of reforms was motivated

by the realization of the need to show greater results, to effect more fundamental changes, and to

consolidate earlier policy changes.

3. During the administration’s first term, and in the midst of the financial crisis, the

Government of Rio de Janeiro (GORJ) embarked on a set of much needed reforms aimed at

regaining control of the State’s fiscal position, improving public sector management, and

increasing the efficiency in the provision of public services. The administration had inherited a

difficult fiscal situation and a deficient administrative capacity in many areas. Prior to 2007, the

annual budget process was essentially inoperative, with funds allocated in an ad-hoc basis, the

revenue base had eroded, and planning was almost non-existent.

4. The operation was designed to assist the GORJ accelerate and deepen its reform

process. The loan built on the prior two DPLs, approved in 2010 and 2011, that had supported

key measures in fiscal management, competitiveness, service delivery and urban infrastructure.

These areas had been identified as priorities in the Government’s Multi-year Plan 2012-2015.

5. The continued engagement by the Bank helped to build a solid long term relationship

with the GORJ, which proved crucial in advancing the reform process in Rio. The Bank

extended significant knowledge support during the preparation of the loan and worked closely

with the administration in identifying measures that would consolidate previous reform efforts.

6. The Bank’s close engagement in the definition of the program was the result of

accumulated experience in the State, such as in the health, education and financial management,

and also drew on experience built up elsewhere. The Bank’s involvement with the GORJ was

also strengthened by the close working relationship it had developed with the Municipality of

Rio de Janeiro and its work with other states in Brazil.

7. The Bank had accumulated considerable experience with public sector reforms at the

sub-national level in Brazil. It has worked with (and continues to work with) many states to help

improve their fiscal management capacity and, more importantly, to pursue results-based public

sector management reforms. This helped to build the Bank’s reputation as a trusted and

knowledgeable partner in the area of public sector reforms. As a result, the authorities requested

the operation and the continued participation of the Bank in the reform process of the State.

8. The time to prepare the operation was very compressed and a rapid preparation was

possible because of the rich on-going dialogue between the Bank and the GORJ. The program

encompassed several important areas of the public administration and required considerable

2

work effort by a large team from both the Bank and the State Government. Even though

discussions on some topics had taken place over an extended period of time, the definition of the

details of the program occurred over a short period of time, from late 2011 and early 2012. To

ensure the strength of the program, it was necessary to reach quick resolutions and agreements

by the involved parties. This included also establishing estimates of expected results two years

hence, which are now the basis for the evaluation of the program in this ICR.

1.2 Original Project Development Objectives and Key Indicators

9. The main objective of the operation was to support the State Government to strengthen

its tax administration, improve the efficiency of public financial management, and increase the

quality of public service provision in education and health. The overarching goal was to promote

broader changes in the administrative structure of the State Government, to improve its

institutional capacity, and to consolidate and secure the advances made since 2007.

10. The key outcome indicators and expected results under each policy area are summarized

below.

11. In the area of Tax Administration: The objectives were to strengthen and increase the

efficiency of tax administration, to raise additional resources, and maintain a solid fiscal position.

The prior actions included legal and administrative measures designed to strengthen the basis for

the introduction of results based management, for establishing standards to improve tax

collection, and to strengthen the tax payer registration system. The key outcome indicators and

expected results were: a) an increase in the revenue of ICMS and ITD taxes by 15.6 and 12.4

percent by 2013; b) the establishment of the general tax payer ledger, allowing individual

taxpayers to access their unified accounts; and c) the gradual introduction of performance

indicators for tax administration, from 0 to 10 indicators by the end 2012 to 20 by the end of

2013.

12. In the area of Public Financial Management: The objectives were to build the

institutional capacity and introduce the systems and procedures to improve the management and

controls of public expenditures, particularly in large investments and programs. The prior

actions included the adoption of a methodology and procedures for evaluating investment

projects, the introduction of pilots to test methodologies for costing social and economic

development programs, and the restructuring of the internal audit institutional arrangements. The

key outcome indicators and expected results were: a) at least five investment projects (with costs

over R$ 50 million) to be subjected to a screening process for approval, according to the adopted

techniques; b) publishing the annual commitments for five large selected investment projects; c)

publishing the cost and defining key results for at least four major policy programs; d) improving

the internal control systems used by AGE in accordance with the 2013 Action Plan developed on

the basis of its 2012 Annual Report.

13. In the areas of Public Education and Health Services: The objectives were to: first, in

education, to contribute to raise the State of Rio to the top five performers in the national

development index of basic education by 2013; and second, in health, to improve the

management and efficiency of service delivery. The prior actions entailed formally adopting the

merit based systems for selection of personnel in the education sector, and introducing the school

level targets and bonus program. In the health sector, the measures included adopting the legal

and administrative framework for contracting social (non-public) organizations to deliver

services, and regulations governing the incentives, transfers and allocation of services by local

3

hospitals. The key outcome indicators and expected results in the education and health included:

i) the merit based selection of all 28 regional directors and 164 school directors by 2013; ii) the

application of school level targets for both 2013 and 2014 (established and communicated in the

previous year); iii) the payments of performance bonuses in 2013 based on 2012 targets; iv)

improve the participation of non-profit providers through the signing of contracts with social

organizations (OS) to manage 15 health units and strategic services (e.g. lab services, imagery

services) by 2012 and 22 by 2013; iv) raising the number of hospitals under the performance-

based transfer PAHI1 program with ombudsman services from 44 to 63 by the end of the period;

v) increasing the number of hospitals with infection evaluation committees from 46 to 66 by the

end of the period; and vi) having 24 percent of the complex treatments in hospitals (compared to

current 17 percent) be for patients from municipalities other than the location of the hospital.

1.3 Revised Program Development Objective and Key Indicators, and Reasons

14. Not applicable.

1.4 Original Policy Areas Supported by the Program (as approved)

15. They policy areas covered by the operation were well aligned with the priorities and

objectives outlined in the Government’s Multi-year Plan (for the period 2012-2015). The main

areas of the plan were economic management, infrastructure, social policies, human capital, and

quality of life. The primary goal was to accelerate and deepen the reforms began in the first term

and effect more fundamental changes.

16. The present operation was also consistent with and closely linked to the objectives of

the 2012-2015 Bank’s Country Partnership Strategy (CPS) for Brazil, which emphasized sound

macroeconomic management, fiscal consolidation, increasing the efficient public sector

management, and improving quality of education and health expenditures for low income

households, especially at the sub-national level, as key pillars for achieving sustainable economic

development and inclusive economic growth in Brazil.

17. The selection of the components emerged from the Bank’s engagement with the State of

Rio, and the joint work that had been done in the earlier phases of the reform program. While

not a formal programmatic series, the sequence of loans that supported different phases of the

reform process and the nature of the continuous policy dialogue essentially makes the operations

a quasi-programmatic series (the main operations are summarized in Figure 1 and more details

are available in Annex 6). The sequencing of the reform process and the developing priorities of

the Government provided a good rationale for the content of the third DPL. These were the

context in each policy area of the DPL.

18. The first State DPL focused on fiscal management and service delivery in the health and

education sector. It established the basis for the current operation by supporting earlier measures

to consolidate the fiscal position, improve the efficiency of tax administration, strengthen budget

procedures and financial management, and raise the quality and efficiency of the education

system, along with increasing access to health services. The third DPL built and deepened the

work started under the first DPL by addressing the same broad areas of fiscal reform, health, and

education. The operation was also accompanied by important technical assistance in the areas of

public financial management, tax administration, education, and health, and by knowledge

services, in both the health and fiscal areas. Technical assistance has focused on improving

information systems for service delivery, strengthening the evidence base of policy, and

4

expanding performance-based management. The third DPL helped the Government launch an

important new phase of reforms and initiate new ones in public investment management.

Figure 1: Rio de Janeiro Quasi-Programmatic Series

Note: Implementation Completion/Support Reports’ ratings: HS = Highly Satisfactory, S = Satisfactory, MS = Moderately

Satisfactory, MU = Moderately Unsatisfactory

19. Policy Area 1 – Strengthening Tax Administration. The process of improving tax

administration was already underway when the operation was designed and negotiated.

Previously, the Secretariat of Finance had professionalized the Revenue Secretariat and adopted

electronic invoices. The GORJ had seen an important increase in tax collection since 2006,

primarily due to economic growth and the performance of key sectors in the economy, but also

to modernization of tax administration by the Authorities. The actions included in the operation

to support the on-going reform process were: (i) the introduction of performance management in

the tax administration unit of the Secretariat of Finance; (ii) the integration of taxpayer

information; (iii) the introduction of digital certificates for taxpayers; (iv) the development of the

first stages of the taxpayer ledger; and (v) the consolidation of the collection system.

20. Policy Area 2 – Improving Public Financial Management. Improved budgetary

management and administrative reforms were central to the overall program, and had been a

main component of DPL I. Since 2007, the GORJ devoted considerable attention to enhance

fiscal control and make budget execution more orderly. They put in place financial management

systems that regularized payments and eliminated arrears. They also strengthened the

institutional capacity of SEFAZ, through new staffing, policies and procedures. However, there

was still much to be done to improve public investment management, to better understand the

factors driving the costs of different programs, and to modernize internal controls; areas in which

the Government had not until that date engaged with the Bank.

5

21. In the area of Public Investment, the institutional capacity of the SEPLAG had

declined over time. There were major deficiencies in the preparation and implementation of the

investment program. Even though State’s budget planning has been guided by the Multi-Annual

Plan, in practice, investment decisions were fragmented among different agencies, driven by

political and sectorial considerations and budgetary allocations done in an ad hoc manner. There

were few formal mechanisms for filtering and prioritizing investment proposals, and hardly any

feasibility studies done. While sector review committees were responsible for reviewing

proposed investments, in many cases they lacked the basic planning information on costs to

make an informed recommendation. They also did not have information on the operation and

maintenance costs of new investments, which in some instances were not part of budgetary

planning exercises. These deficiencies made imperative the need to improve the GORJ’s

investment planning capacity, which became a crucial part of the reform supported by the Bank.

22. As a result, the Authorities and the Bank agreed on including key measures under the

DPL in the investment area. These included the adoption of formal procedures for evaluation,

selection and approval of public investment projects that had a total estimated cost of over R$50

million. In addition, the State was to carry out detailed analysis of all ongoing major investment

projects to document costs and future operating and maintenance costs. The process began with

five investment projects during 2012 that served as pilots.1 This was to be followed up with ex-

ante analysis for other projects above the R$50 million threshold. It was recognized that it was

going to be a major undertaking, since there were 26 investment projects over R$50 million in

the budget for 2012, accounting for 68 percent of the R$6.8 billion State investment budget.

23. Costing of Programs. In recent years the GoRJ had improved service delivery by

expanding or introducing new programs, in transport, housing, health, and income subsidies.

These programs included basic cost estimates when first introduced, but these estimates did not

fully capture the future fiscal implications of the programs. To improve the planning, the

authorities piloted a program to improve their costing methodology. This was done prior to the

preparation of the loan, and was applied to four small programs. As part of the DPL, the costing

methodology was expanded and was to be applied to four large programs. The objective of the

new approach was to derive methodologically consistent data for future cash outflows.

24. The development of costing methodologies for selected programs was initially produced

by two working groups and served as basis for the follow up work. These methodologies

included standard cost parameters, basic scenario analysis and expected costs on an annual basis

for at least the next three years. The methodology was to be applied to four major programs: the

Bilhete Único, Renda Melhor, Distribuição de Remédios, and Aluguel Social, during 2012. This

exercise was to provide the GORJ with a baseline for evaluating the cost-effectiveness of

expanding services and for increasing the predictability of budget and fiscal sustainability.

25. Auditing and Internal Controls. Finally, the authorities initiated the modernization of

the internal control and audit system. Prior to the preparation of the DPL, the Auditoria Geral do

Estado (AGE), the agency responsible for internal control in the State, focused mainly on ex-post

reviews of accounting and procurement documents for actions already taken by a myriad of

1 Including: Arco Metropolitano (R$230 million); Ampliação da Rede de Ensino (R$222 million); Ampliação da Via

Light (R$161 million); Conservação de Rodovias (R$133 million); Projeto Iguaçu (R$89 million); Projeto

Saneamento (R$80 million) projects.

6

entities and satellite State organizations. Essentially all of AGE’s work was compliance based

and had limited capacity to carry out risk-based, performance or value for money auditing. As

part of the preparation of the DPL, the Bank team helped the agency identify the process

required to modernize and shift its focus away from compliance to risk based performance audit.

26. Prior to the approval of the DPL, the Government issued a new decree governing the

role of AGE and adopted associated initiatives, allowing the agency to rebuild its workforce and

to undertake a more effective work program. The decree created a new management and career

structure for the accounting and audit personnel of the State Government. It also established a

process for AGE to develop annual audit plans and report annually on its accomplishments. This

sub-component effectively helped change the focus and modus operandi of AGE.

27. Policy Area 3 – Quality and Efficiency of Public Education and Health Services. The

final main component of the operation was to support important changes in the provision of

public services in education and health for lower income groups. The actions represented an

important change in the models for the provision of quality public services.

28. Human Capital (Education). Improving education was a key element in the GORJ’s

agenda to promote social inclusion, labor productivity and growth. The Government wanted to

tackle poor educational outcomes and improve student learning through accountability reforms.

Rio had set an ambitious target of becoming one of Brazil’s five highest-performing states on the

Federal Government’s IDEB education quality index by 2013 (in contrast to 26th position or

second to last in 2009). To achieve this, the State Secretariat of Education had made some

progress prior to the loan preparation by building a solid administrative technical team and

developing a comprehensive reform agenda. This work was supported by the Bank through a

Technical Assistance Loan and a close working relationship among the teams. The reforms were

in line with global best practices and research on how to improve school systems. They started

with curriculum reform and changes in the training for teachers, and eventually included wider

management and governance reforms.

29. The reforms had two important and overarching goals. The first was to raise the quality

and accountability of school managers. In January 2011, the State adopted legislation that

established new processes for the appointment of school district managers (regional directors)

and school directors. For the first time, these appointments were to be made through a

competitive selection process, and not through political appointments. The same legislation also

introduced the first-ever process of annual performance evaluation for regional and school

directors. These politically sensitive measures were expected to increase the transparency of the

selection processes and to upgrade the technical quality of the state’s top education managers.

30. The second goal was to strengthen the incentives for schools to focus on results, with

the aim of improving the State’s low test scores and graduation rates. The Authorities introduced

in January 2011 annual school-level improvement targets and bonus pay linked to these results.

The State also adopted an annual learning assessment test, which was based upon the national

test, to monitor learning outcomes more effectively. The personnel in the schools that achieved at

least 90 percent of their targets received a salary bonus (of up to 25 percent of their annual

salary). The targets for each year are based on a linear projection of the improvement required at

the local level in order to raise the state’s overall national ranking.

31. Health. The primary focus of health policy in the State was to improve the

management in the sector and increase its efficiency, while expanding the coverage and quality

7

of services. To achieve these goals and to promote better governance, the GORJ introduced new

contract models based on payments for performance. The Government also sought to improve

the coordination between the primary health services and hospitals in small municipalities, by

setting-up better referral and counter-referral processes and by increasing the capacity of regional

hospitals to respond to more complex health care needs. In addition, the Government expanded

access to urgent and emergency care by extending the 24-hour emergency care unit (Unidade de

Pronto Atendimento–UPA) model beyond the Rio de Janeiro Metropolitan Region.

32. The Government adopted foundational measures in the early phases of the reform

program. It created the legal framework for implementing new models of contracting for health

services from non-profit providers. In addition, it provided support and financial incentives for

hospitals in small municipalities in the interior to improve primary health care and the quality of

hospital care (PAHI1). It introduced incentives for regional hospitals to respond to medium and

high complexity care needs, including for patients from outside the municipality where the

hospital is located (PAHI2)2. Finally, it created the State Health Foundations to introduce

flexibility and strengthen performance orientation and accountability in the system.

33. The phase of the program supported by current operation entailed the launching of

many complementary initiatives, particularly those of subcontracting social organizations

emergency local units and some public hospitals. This new management model was to provide

the state with increased flexibility to expand the supply of health care services and was expected

to stimulate innovation and improved efficiency in service delivery. In addition the DPL

supported improvement of the system of incentives to municipal governments and hospitals

under the Program to Support Hospitals in the Interior (PAHI1 and PAHI2). Specifically, new

regulations established indicators, administrative systems, and other operational features of the

program, with the aim to strengthen primary care and hospital services in small municipalities

(less than 115,000 population), and enhance the quality, efficiency and inter-municipal

coordination of medium- and high-complexity services provided by regional hospitals.

2. Key Factors Affecting Implementation and Outcomes

2.1 Program Performance

34. This was a single-tranche operation which disbursed on effectiveness. All policy actions

required by the Loan Agreement and described in section 1.2 were met by GoRJ prior to

negotiations. The loan was approved by the Board on August 30, 2012 and signed on October

30 of the same year. Effectiveness was declared on November 16, 2012 and the project closed

according to its original schedule in January 31, 2014.

2.2 Major Factors Affecting Implementation

35. The implementation experience of this program was consistent with many similar

operations and reinforced lessons already learned. As with any set of comprehensive reforms,

implementation always poses a considerable challenge. This is more pronounced when the

program entails building institutional capacity in challenging environments. The implementation

2 PAHI1 (Programa de Apoio aos Hospitais do Interior 1); PAHI2 (Programa de Apoio aos Hospitais do Interior

2) is dedicated to support improvements on management and infrastructure for regional state hospitals or those

hospitals with medium complexity covering more than one municipality according to the health regionalization plan

of the Rio de Janeiro State.

8

issues under the DPL III can be grouped into four different categories: (i) those of political

leadership, commitment, and personnel capacity; (ii) those of learning by doing; design updates,

and adjustment; (iii) those of unexpected developments, technical setbacks, and unrealistic

timetables; and (iv) those of technical assistance, close supervision and follow up.

36. The success of the program in different areas was clearly associated with the

administrative capacity of individual agencies, the political leadership within those agencies, and

the degree of commitment and focus by the implementing agencies. The greatest

accomplishments occurred in those areas where the responsible authorities were confident on the

nature of the reforms and the goals to be attained, and were able to garner the full commitment of

the involved personnel to achieve the established results, and had adequate administrative

capacity. In all successful instances, the reforms were also central to the goals of the agencies.

This was most evident in the case of SEFAZ, where there was a strong commitment to

strengthen fiscal controls and improve tax administration, consistent with the design of the loan.

The same can be said about the reforms in education, where strong leadership provided guidance

to implement reforms that faced considerable political resistance. It took persistence in this case

to begin to ease opposition to the reforms.

37. The most difficult area was the public investment program, where the institutional and

political economy constraints were quite significant. Both the Bank and the authorities

recognized this was a critical component, given the deficiencies in investment planning and cost

control. The Bank team noted the importance of introducing changes in the manner investments

were considered, selected, and funded over time. This was reflected in the proposal that five new

large investment proposals had to be reviewed under a new set of evaluative and screening rules.

The authorities agreed, but had reservations on the ability of the Government to carry out the

proposed reforms. They entailed a significant change in the operations of the investment agency

and the collaboration of powerful executing line agencies, as well as confronting strong political

interests and institutional constraints in a short period of time. Still, the authorities agreed to

developing a more coherent process, in part as a managerial strategy to promote changes in the

Secretariat of Planning and the line agencies, while recognizing that those were going to be

difficult to achieve.

38. The actions in the investment and costing areas proved to be even more challenging

than expected. SEPLAG did not focus on implementing the new screening process, as day to

day responsibilities and budget execution trumped planning. The Secretariat has a mandate to

carry out investment decisions of the Government, and to ensure that proper financing be

available to the chosen projects. The GORJ did not take additional measures to ensure that the

Secretariat’s capacity was strengthened to properly evaluate proposed investments, which

emerged from powerful executing agencies. These agencies had greater capacity to promote and

obtain support for their investments. Furthermore, often the investments came with associated

financing that were outside the standard budget process. All of these factors contributed to

SEPLAG’s inability to achieve the expected results. In this component, clearly there was a need

to focus on strengthening the institutional capacity of the agency, before establishing goals that

required considerable cooperation and coordination with many other agencies in Government.

39. Another important implementation issue was the adjustments made through a process of

learning by doing. In several areas of the program, there were adjustments made after the initial

measures were carried out. Many of these adjustments improved the program, especially in the

tax administration and health areas. Following the adoption of performance indicators in the tax

9

area, staff was able to identify a number of measurements, practices, and procedures that needed

to be further reviewed. This led to an analysis of how they could be adjusted to improve tax

administration. The same was in the case of health, where the experience with the UPAs’

contracts led to a constant updating and adjustments of their content, and an application of a

similar approach to local hospital services, which had not been contemplated initially. Another

illustration of the learning by doing process was the expansion of the criteria and indicators used

to determine the incentives provided to hospitals. Initially, two basic indicators were included in

the operation (the introduction of ombudsman services and infection evaluation committees).

Soon after the loan was approved, the responsible staff for implementation recognized those

indicators were not sufficient to bring about the expected changes in hospital management, and

introduced other indicators to give financial recognition to hospitals for their efforts.

40. All areas of the program confronted some unexpected events and a few areas some

technical setbacks. While not significant to derail any major part of the program, they introduced

some delays in implementation. This was present mostly in the areas that required gathering

information from different agencies, such as in the costing of expenditure programs and the

investment planning area, and areas that required introducing technically complex processes,

such as in the development of a taxpayer single account (or general ledger). The latter has been

difficult to construct and is not likely to reach its final stage until a year after the original

expected date. Just after the closing of the loan, the technical specifications of the proposed

ledger would be tested and validated. The final stage of making the information available to the

taxpayers still has to go through several steps, including a review of security considerations. The

anticipated results in this area did not foresee many of these implementation issues.

41. Finally, the implementation experience clearly shows that a close engagement with the

authorities, either through technical assistance or close supervision, significantly increased the

chances of achieving the expected results. This is shown by the fact that in those areas where the

Bank had an ongoing technical assistance program closely aligned with the DPL, results

exceeded expectations. Nowhere is this more evident than in the education sector, where the

technical support predated the preparation of the DPL and continued during its implementation.

The same is the case, albeit to a lesser degree, with the engagement in the health sector. Whereas

in those areas where the Bank followed or assisted the Government less closely with the

implementation of the program, the results were mostly not satisfactory, especially in the area of

public investment management. Bank staff lacked resources (budget and time) to support closely

the actions by the Government and had to rely on the ability and willingness of the respective

agencies to reach the expected results. The same Bank staff were also tasked with the preparation

of new operations (in Rio and elsewhere), thus diverting some of their focus from

implementation support. A closer engagement with the implementing agencies could have

prompted earlier corrective actions and ensure better performance.

42. Still, the assessment of the program’s implementation experience is positive, since it

hinges on measuring the degree of accomplishment against the expected outcomes. Most goals

were achieved, and the results were essentially according to plans. The preponderance of the

positive results reaching or surpassing the original goals substantiates the generally positive

evaluation of the program.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

43. The monitoring and evaluation of the program was based on the expected results for

each component. Those results were fairly detailed and had specific dates attached to them to

10

follow the achievements under the program. There was, however, no formal M&E arrangement

for the whole operation. While the authorities formed an ad-hoc committee within SEFAZ to

monitor the actions agreed under the loan, the group worked mostly to satisfy the needs of the

periodic supervision missions of the Bank. There were dedicated groups that closely monitored

the advances of the reforms in the Education and health sectors, but those were essentially set up

for implementation purposes. In the education sector, there are ongoing impact evaluations

supported by the Technical Assistance that are expected to provide additional information on the

results from implementing a new management model and results-based pay in secondary

education. A closer scrutiny and monitoring of the actions of those responsible for carrying out

the measures agreed under the program could have helped the program performance.

2.4 Expected Next Phase/Follow-up Operation

44. The Bank is providing continued support for the implementation of reforms in the major

areas of the DPLIII. This is occurring through a follow up operation, DPL IV (Enhancing Public

Management for Service Delivery) and two Technical Assistance Loans. The new DPL operation

is assisting the GORJ to enhance public management for the delivery of key public services for

vulnerable populations by instituting new policies and regulations to improve: the medium term

planning and monitoring of public expenditures (Policy Area 1); the accessibility, quality and

affordability of urban mobility services for the poor (Policy Area 2); and the availability of

targeted social services aimed at reducing domestic and gender based violence (Policy Area 3).

45. The Bank also has two Technical Assistance Loans in place—Public Sector

Modernization and Strengthening Public Management and Integrated Territorial Development.

They support the GORJ’s efforts with performance-based management, and the introduction of

information technologies aimed at improving service delivery in key areas, including secondary

education and hospital care. The loans are also supporting the modernization of the tax

administration and public finance management.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation

46. The objectives of the loan were and continue to be highly relevant for improving the

public administration in the State of Rio. The objectives were consistent with the Government’s

reform program, but also with the identified needs of the State in important areas of the public

sector. The relevance is enhanced by the continuity of the program. Consistency in pursuing

different phases of the reform over an extended period adds to the significance of the program.

Furthermore, many of the objectives entailed fundamental changes in the operation of the

sectors, in the way that resources are allocated, incentives are structured, performance is

evaluated, and services are rewarded. One of the most important aspects in terms of both

objectives and design was the expansion of the use of indicators and modern management

methods.

47. The design of the program was also very relevant for achieving the expected results,

focusing mostly on the policies and measures that were critical to the success of the

Government’s reform program. It ensured continuity and furthered the transformation of key

areas of public administration. Even in the investment area, where there was recognition that the

program was quite ambitious, there was a need to promote reforms, given the critical role of

better investment planning for the budget process.

11

48. Implementation did not match the ambition of the design, but can still be considered

good, particularly when evaluated from the perspective of the achievements under the overall

program. There were considerable efforts to focus actions on the most important components,

and the result was, on balance, very positive. Fundamental reforms were undertaken in the

education area, substantial improvements were made in tax collection, there were some gains in

the efficacy of tax administration, and important performance based management tools in both

the health and education sector were adopted. The major shortcoming remains in the area of

public financial management, particularly as it pertains to investment decisions.

3.2 Achievement of Project Development Objectives

49. Achievement of the PDO as a whole is rated moderately satisfactory, as most of the

results indicators were achieved or exceeded their targets. This is a weighted rating, which takes

into consideration the importance of the achievements in key service areas, such as Education, as

compared to the disappointing progress in some parts of the public financial management area. A

very significant part of the reform program was successfully implemented. The operation

identified 15 specific results and outcome indicators to be achieved by the end of 2013. Ten of

those were fully met; three were mostly achieved; and two failed to satisfy basic expectations.

The achievements were most significant in the education and health sectors, as well as in tax

administration. Results failed to meet initial expectations primarily in the public investment

program.

50. The positive evaluation of the program is also based on the consistency and persistence

of the reforms over an extended period of time. In this context, the DPL should be seen as part of

a sequence of operations supporting earlier phases of the reform program. The current operation

continued these achievements by supporting deeper, and in some cases more significant, reforms.

Policy Area 1 – Efficiency of Tax Administration.

Rating: Satisfactory

51. The overall rating for this policy area is based on the fulfillment of the original

objective, which was to strengthen and increase the efficiency of the tax administration, to raise

additional resources, and maintain a solid fiscal position. The State improved on all fronts. The

rating is reinforced by the fact that in two of the sub-components (revenue generation and the

introduction of performance indicators) the achievements exceeded the expectations established

at the time of the loan preparation. In the third sub-component (the establishment of a general tax

payer ledger), delays revealed the difficulties of introducing such an extensive administrative

reform, when both political and technical factors can alter the best plans.

52. Tax Revenues. Performance in this sub-component was highly satisfactory. The final

revenue figures for both ICMS and ITD exceeded the 2013 targets by 8.4 and 41.8 percent points

respectively. The initial targets, following a correction of the baseline, were for increases of 15.6

percent for ICMS and 12.4 percent for ITD. The good performance was due to both internal

actions taken to improve tax collections and external developments, including the rise in food

prices and higher vehicle sales. These developments offset the lower than expected revenues in

other areas such as the electricity sector, where the authorities expanded exemptions. It is

difficult at this stage to determine the extent to which the increase in revenues was due to the tax

collection measures or simply better economic conditions. The actions aimed at improving the

efficacy, effectiveness and efficiency of tax collection are in their early stages. It will take time

for them to be fully operational and to show solid results. The initial targets for increasing ICMS

12

and ITD considered some gains in the efficacy of tax administration, but those cannot be

substantiated with the current information. A proper analysis of the impact of efficiency

improvements in tax collection would require knowing the initial assumptions and calculating

the effects of unexpected economic developments, data that is not available for the preparation of

the report.

53. General Tax Payer Ledger. The result of this prior action was expected to be the

establishment of a unified general taxpayer ledger by the end of 2013 that would be available to

individual taxpayers online, as the final result of a broader process of improving and upgrading

the tax collection systems, called Gestão do Crédito Tributário (GCT). Its achievement can be

considered moderately satisfactory. The program required advancing through a series of

sequential steps before the ledger could be fully operational, which took longer and proved more

difficult to effect than initially anticipated. The platform and information consolidation has been

accomplished, and the authorities are now entering the stage of testing and certifying the

technical adequacy of the new system. It is not yet certain when the ledger will become fully

operational. The tentative date is the beginning of 2015. The ledger will first be used internally to

analyze the tax situation of individual business entities. The slippage in the calendar is also

explained by unanticipated events, such as changes in tax laws that demanded urgent changes in

many systems and caused the pace of GCT project to be significantly slowed.

54. Tax Administration Performance Indicators. The sequenced introduction of

performance indicators was carried out as envisaged. The final batch of indicators was done in

March of 2014. The last phase occurred a bit later than expected, but the delay, due mostly to

personnel changes, did not affect the overall thrust of the program. Hence this sub-component is

considered satisfactory. Overall, SEFAZ’s experience with the indicators has been positive.

Their introduction has led to a better understanding of the deficiencies in the processes and also

to an adjustment in their management. The simple act of collecting information prompted the

authorities to investigate the factors behind certain outcomes and to adopt measures to improve

the results. However, most of the indicators are geared towards looking at the efficacy of the tax

administration. They are internally oriented and tend to focus on administrative enforcement;

few measure efficiency and effectiveness, particularly in terms of the quality of client services.

The indicators were constructed depending on the availability of data and administrative

demands, and are now being organized according to the logic of administrative procedures. The

calculation and the disclosure of the indicators remain restricted to a small group within SEFAZ.

There seems to be room to improve in both of these areas, by automating the calculations, and

making them more broadly available to all tax administration employees. In addition, there are

benefits to creating formal arrangements such as regular high level meetings to discuss the

indicator outcomes, and defining appropriate actions.

Policy Area 2 - Improving the Efficiency of Public Financial Management

Rating: Moderately Unsatisfactory

55. The evaluation of results under this policy area is heavily influenced by the failure to

achieve the original goals in one important area, not necessarily the progress that has been made

in each sub-component. As a whole, there have been improvements in the public financial

management of the state, in part due to actions contemplated under the loan. But those actions

yielded results well below expectations, and failed to reach a key goal of the reform program.

13

The shortcomings were most pronounced in the public investment area and to a minor extent in

the costing of Government programs.

56. The failure to properly establish an investment review, screening and selection process

implies a major gap in the Government’s ability to improve its fiscal management. The measures

in this area were critical for the broader objective of the operation, which was to solidify fiscal

control. The overall achievements in this policy area are in the right direction, but they fall short

of expectations and much remains to be done in this area.

57. Public Investment Management. The implementation of this sub-component was

under the purview of the State Secretariat of Planning (SEPLAG), which has considerable

responsibilities for administering the information of a large and diverse public investment

portfolio. The reforms supported by the loan anticipated a significant change in the manner in

which proposed investments, particularly large ones, were considered by the agency. The

program not only attempted to change the procedures for reviewing, evaluating and selecting

investments, but also introduced a pilot program, where 5 new investment proposals would be

assessed under the new procedures. However, for the 2013 budget only 3 new investment

proposals were identified and subjected to the proposed process. The evaluation was done only

ex-post and was not included in the ordinary budget process, thereby defeating the purpose. The

objective became a mere bureaucratic obligation. The proposed methodology was not

incorporated into the 2014 budget cycle. SEFAZ introduced a parallel process to calculate

possible future commitments emerging from maintenance and operational costs. The duplication

of efforts shows the importance of improving the process and garnering greater support from

SEPLAG and improving cross-agency cooperation. The achievements under this sub-component

are considered unsatisfactory.

58. Costing of Programs. Most of the targets under this sub-component were achieved,

but overall the effort fell short of the broader goal of analyzing the cost-effectiveness of the

programs. The efforts to estimate the costs of four large public programs helped the authorities to

gain a better understanding of the features of those programs. With the analysis, SEFAZ was

able to link the costs of the programs to the number of beneficiaries, and begin to identify the

factors, such as eligibility criteria, that could drive future expenditures. This provided important

information and was a first step in developing the required budget tools for improved

management of the programs.

59. Despite the achievements, however, the initiative is not yet an effective tool of budget

analysis. The costing only covered actual expenditures and number of beneficiaries. There are no

well-developed projections of future costs. There are also no means to undertake a cost-benefit

analysis, given the lack of information on benefits provided. Officials recognized many of these

shortcomings, and are hoping that the technical assistance provided by the Bank will spur

advancements in the application of the methodology. Finally, there is the important issue of

transparency, which was to be addressed by the publication of the estimates and analysis. The

costing reports were not made available beyond the working group, due to delays in obtaining

and refining the information needed to produce the report. The authorities, however, reaffirmed

their commitment to make the report public after it is finalized. As a whole, therefore, the sub-

component can be classified as being moderately satisfactory.

60. Internal Audit Agency and Controls. This sub-component proceeded according to

plan, and its achievements were satisfactory. The measures taken included: developing an annual

14

plan of action and reporting on their accomplishments; issuing key audit regulations; training

additional staff; creating an internal career plan; and hiring a new cadre of younger professional

auditors. In addition, staff has been decentralized to the executing agencies. New procedures

have been established to create oversight on the basis of results and ex-ante procedures, moving

away from ex-post reviews. These are the first steps in a long and slow process of incorporating

modern audit and accounting systems in the State Government.

Policy Area 3 - Quality and Efficiency of Public Education and Health Services

Rating: Satisfactory

61. Education Sector. The reforms undertaken in this area were quite significant, and all of

the expected results have been fully met. Hence, the achievements under this sub-component can

be considered highly satisfactory. The core elements of the program were designed to change the

incentive structure for educators and school administrators, with the purpose of improving

student learning performance and reducing repetition and drop-out rates. Administratively, the

Education Secretariat was able to appoint all new regional directors under a competitive system,

along with 15 percent of all school directors in the first three years. The Secretariat also

streamlined the regional structures, consolidating 30 regional supervisor positions into 14 new

districts, reducing the number of regional directors by more than half. To signal a stronger focus

on school quality and student learning results, the Secretariat created a new position of regional

pedagogical supervisor. On incentives, all schools received individual performance targets. More

importantly, the staff in about 30 percent of the schools received bonuses for meeting their

targets. The bonuses were substantial, amounting to approximately 25 percent of the annual

salaries. The number of schools achieving 80 percent of their annual targets, and therefore

receiving bonuses, increased by 23 percent in comparison with the first year.

62. These reforms were politically sensitive and required considerable determination to

carry out. The authorities had to overcome opposition from many groups, particularly those who

benefited from the previous arrangements, and had to be convincing to overcome doubts about

targets and bonuses based on standardized tests. Given this environment, the achievements under

the program are even more noteworthy. While it is early to assess the overall improvement in the

quality of education in the State, very preliminary indications show an impressive rise in the

State’s ranking for secondary education quality, going from 26th in 2009 to 15th in the latest

national IDEB scale; one of the largest improvements of any state. Results have consistently

improved in the annual Index of Educational Development of the State of Rio de Janeiro as well.

Crucially, there are indications that governance changes are being accepted, and a new culture

focused on results has taken hold among school leaders and administrators. Among other

innovations, the Secretary of Education introduced meetings of the entire state leadership team

(approximately 600 state and district level administrators) every six months to review progress

against district goals, to highlight progress and disseminate lessons from the principals of schools

that have made exceptional gains, and to develop strategies for intensive support to schools with

low performance. While the teachers’ strike for a 19-percent wage increase in late 2013 are

expected to have a negative effect on the State’s results in the 2013 IDEB (to be released in

August 2014), they are unlikely to offset the substantial improvements registered since 2010.

63. Health Sector Reforms. The achievements in this area were also satisfactory.

Essentially all but one of the expected results was achieved. The primary focus of health policy

in the State was to improve the management in the sector and increase its efficiency, including

15

expanding coverage and improving services. In the area of contracting social organizations (OS)

to manage emergency services, the Government exceeded the targets, and now has 30 UPAs

under contract, as compared to the 22 originally expected by this time. The process of

contracting service providers has dramatically changed the modus-operandi of the Secretariat of

Health, which is now much more focused on contract management than oversight of direct

provision of services. In fact, management contracts have now been extended to some local

hospitals, which were not planned initially. Contracts still need to be improved, in areas such as

reimbursement for performance targets instead of the cost for providing services. But, still

significant progress has been made. The goals on referrals and coordination of treatment were

also achieved. More than 25 percent of complex treatments handled by regional hospitals were

for patients outside the immediate jurisdiction.

64. Finally, the Government has successfully handled the PAHI program, which provides

financial support to hospitals that are managing patient care according to certain targets and

indicators. Initially, the DPL operation focused on two indicators to gauge progress; the number

of hospitals that adopted an ombudsman program to review patient treatment, and the number of

hospitals with an infection evaluation committee. The number of ombudsman has been met, but

the created committees were found not to be very effective in achieving their goals. Because they

did not operate as expected, the Government expanded the indicators for this purpose, to include

hygiene standards and hospital waste disposal, among others. The broader approach served to

accomplish the goals initially established for the committees and a better indicator to track

results than the original one. While there are preliminary indications of improvement in hospital

disease management, those are not readily and consistently available for outside evaluation. A

new monitoring system is under implementation as part of the first Technical Assistance Loan to

address this gap.

3.4 Justification of Overall Outcome Rating

Rating: Moderately Satisfactory

65. The justification for the overall rating is based on the magnitude of the program and the

results achieved. As noted in section 2.1, the operation supported reforms in important areas of

the public sector in the State of Rio de Janeiro. Two thirds of the target results were fully met.

While the results were not uniform, the main achievements, particularly in the education sector

and to some extent in health and tax administration, were significant and likely to be sustained.

These results offset any shortcomings in other areas, such as public investment.

66. The operation was well designed to help launch a new phase of reforms that included

some noteworthy changes in the nature of key public sector programs. While the overall

approach was to strengthen results-based management, the more immediate achievement was to

help personnel to focus on targets, to review processes and procedures, and to improve service

delivery. More importantly, the program helped set up new incentive structures that over time

could help change the culture within certain sectors. Many of the initiatives helped to build the

institutional structure of the State Government and are likely to continue in the future.

67. The positive evaluation of the entire program is also based on the consistency and

persistency of the reforms over an extended period of time. As noted, the operation was part of a

series of loans that ensure the continuity of the reform process in Rio. Even in those areas where

the outcomes were not as good as expected, such as with the public investment program, the

continuous involvement and inclusion of the issues on the debate with the authorities raised their

16

profile and led to some improvements. Better management of the public investment remains a

critical issue for the State Government and still need to be addressed.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Poverty Impacts, Gender Aspects, and Social Development

68. The evaluation of the main beneficiaries of the reforms supported by the DPL was quite

extensive and thorough. It showed that those in the lower quintiles of the income distribution and

the most vulnerable groups in society would be the main beneficiaries from the reforms and the

improvement in the service delivery attained in education and health. It is not possible to validate

these assertions ex-post, given the early stage of the reform process. The information for a

deeper analysis is not available at this time, although an impact evaluation is underway in the

education sector. But, it is safe to assert that the reforms should have a positive social impact,

given the characteristics of the primary users of the public services being reformed.

(b) Institutional Change/Strengthening

69. Unquestionably, the operation has contributed to build the institutional capacity of the

GORJ. In some cases, such as in the education sector, this has been done with the close

assistance of the Bank. In other cases, this has taken place through a process of learning by

doing, as the executing agencies carry out the reforms aimed at achieving the targets established

under the program. Furthermore some of the changes, such as in the adoption of the contracting

for services in the health sector, led to a change in the focus of the agency away from direct

provision of services toward oversight and enforcement of contracts. In the case of tax

administration, the institutional strengthening (which was initiated in earlier operations) is taking

place in part through the adoption of performance indicators. These have highlighted a number

of issues and provided new insights about operational efficiency. With the new information, staff

are now reviewing processes and procedures in order to improve institutional efficiency. Finally,

the most important change is taking place through the incentive structures that are being

introduced in the education and health sectors. They are leading to a new culture of focusing on

targets, and being remunerated accordingly. Still, one of the most evident lessons from the

operation is that creating institutional capacity is a slow methodical process that requires

considerable persistence and close support from principals, which was not always present.

4. Assessment of Risk to Development Outcomes

Rating: Moderate

70. The second term of the current administration is coming to an end in 2014 and there are

some reversal risks. This is particularly the case in the area of tax and costing of programs where

the policy changes were instituted through directives and have not been fully consolidated.

Continued strong leadership at the executive level and focus on seeing through the reforms

taking place would be critical to preserve the gains made.

71. Conditions are more positive, however, in the education and health sectors where the

changes are taking root and their benefits are becoming more apparent. This is particularly the

case for those involved in the service provision, who are now benefitting from additional

financial incentives. The reforms have built constituencies that have an interest in the

continuation of the programs. The cultural change taking place is an element that should reduce

the possibilities of reversal. A similar development is taking place in the Secretariat of Health,

where the new focus on oversight of contracts is seen by staff as a much better use of their

17

talents. The same can be said for the introduction of new systems, processes and procedures in

tax administration and internal audit. The staff responsible for the introduction of these new

tools is already developing follow up improvements. All these factors contribute to making more

difficult to reverse the policies and return to the previous management practices.

5. Assessment of Bank and Borrower Performance

Rating: Moderately Satisfactory

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry

Rating: Satisfactory

72. As noted in Section 2.1, the Bank worked closely with the GORJ during the preparation

of the loan, providing important advisory support in the design of the program. The Bank also

worked to ensure timely financing. The time-frame for the preparation of the loan was driven in

great part by the financing plan of the Government.

73. The preparation of the loan and the design of the program benefitted enormously from

the experience of both the Authorities and Bank staff. Both sides understood the challenges of

the program and generally agreed on the follow up measures needed to accelerate the process.

There were solid diagnostics of the existing conditions and very good analysis of required

policies. The Bank had a group of dedicated professionals for each area, and also brought top

experts to discuss the experiences of other states and countries. The advice was followed in some

areas more than others. For example, the modernization of the audit agency continues to date

according to initial recommendations. In the health component, an important recommendation

was to develop an information system with cost standards for service providers (a dashboard

with scorecards). This has yet to be done, in part because of magnitude of the task and the

technical challenges. But it reveals the breadth of the advice provided during preparation. The

experience with the DPL underscores the fact that programs are more likely to succeed when

there are high level professional teams involved in the design stage.

74. There was also sufficient preparation time to develop a sound program. Initial

discussions on the nature of a loan began in late 2010 and pre-appraisal missions formally began

in the later part of 2011. These involved both sides and resulted in detailed sector background

information and indicator monitoring data. In particular, the World Bank technical staff helped

assess the proposed policy actions against international best practices. They also carried out a

thorough review of the fiscal situation of the State and a beneficiary analysis.

75. The evaluation of the Bank’s performance during preparation is partially affected by the

adoption of ambitious targets for some parts of the program. The Bank team promoted strong

measures and bold goals, in part to propel the executing agencies to move expeditiously and also

to strengthen the quality of the program. In most instances, such as in education, this approach

was successful, whereas in others, such as in the goals for the investment reforms, it failed. The

final decision to have ambitious targets fell to the central authorities, who understood the

challenges that the line agencies would confront. It was based on a calculus that setting

ambitious goals would drive actions. It turned out that without the proper political support,

leadership, and technical assistance the goals were simply not achievable. Bank staff recognized

that agencies needed more time and assistance to proceed with the basic elements of the reforms,

but was not able to provide it uniformly because of staff and budget restrictions.

18

76. Nevertheless, the Bank team responded to the client’s needs, with timely financing and

advice; it was an adequate and timely response to the challenges faced by the GORJ. The actions

ensured continuity and the adoption of measures that will have lasting impact, particularly on the

provision of and management of public services.

(b) Quality of Supervision

Rating: Moderately Satisfactory

77. While Bank support and involvement during the preparation was highly significant,

more resources and staff time would have improved outcomes. The loan established a period of

18 months to meet the targets. This provided a short window for the Government to demonstrate

results. During that time there were (two) full formal supervision missions, which were mainly

geared at obtaining information on the advances being made. Only in a few areas did the Bank

accompany closely the progress of the implementation of the program and assisted the

authorities on a continuous basis. In many other areas, such as with tax administration and public

investment, the Bank did not have a close presence. The missions were able to ascertain the

progress made, but did not have the resources (time and budget) to engage closely with the

authorities and help develop remedial measures or assist with the details of implementation.

More could have been achieved with closer supervision.

78. The rating here is not necessarily based on the effort by Bank staff or for that matter the

quality of the dialogue with the authorities. Instead, it is a reflection of the fact that supervision

of a single tranche loan disbursed on the basis of prior actions does not carry the same priority to

Management as other loans. The nature of the supervision per se is different, simply to monitor

results. But, the needs under the loan were also to help build the institutional capacity, which

required a much stronger focus on the processes and procedures, not just the results.

(c) Justification of Rating for Overall Bank Performance

Rating: Moderately Satisfactory

79. Bank guidelines suggest that a moderately satisfactory in one of the dimensions of Bank

performance, should automatically translate also into a moderately satisfactory rating for the

overall Bank performance. This presupposes that each dimension, quality at entry and

supervision, should carry similar weight. While the guidelines are followed here, it should be

noted that in this operation, as in many other single tranche loans with short periods of execution,

supervision is not a major element of the Bank’s involvement. Most of the work by the Bank in

this operation took place as part of an on-going dialogue with the GORJ as part of other loans,

and not so much as part of the supervision of this particular operation. Hence, when considering

the total effort and involvement by the Bank, perhaps greater weight should be given to

preparation and design.

80. The Bank involvement and support to the State under this operation was very sound. It

assisted the State to deepen a reform process that improved many aspects of the public

administration in the GORJ. The Bank has maintained a close policy dialogue with the State, and

has developed follow up operations that will ensure continuity in the process. The Bank is

committed with technical assistance to help build the institutional capacity of many agencies.

The only aspect that detracts, from an otherwise, very good performance is the constraint faced

by the public sector and fiscal teams to be even more closely engaged during the implementation.

19

5.2 Borrower Performance

(a) Government Performance

Rating: Moderately Satisfactory

81. The government’s performance is evaluated on the basis of the strength of the program,

the breadth of the results achieved, and the role of different agencies involved in the process.

The reforms entailed significant efforts to improve the managerial capacity, to build better

information systems, to confront political opposition, to change the modus operandi of certain

sectors, and to build a better institutional structure. Nevertheless, the Government’s performance

was uneven. It varied by agency and different areas of intervention, hence the moderate rating.

Still, on the whole there was an important advancement that reflects on the commitment by the

Authorities to the overall objectives of the operation.

(b) Implementing Agency or Agencies Performance

Rating: Moderately Satisfactory

82. While there were several agencies responsible for individual components of the

program, formally SEFAZ was the coordinating agency. SEFAZ demonstrated very strong

leadership and cooperation throughout preparation and implementation of the program. It

administered the improvements in tax administration. It also followed the actions of the other

agencies and spurred their greater involvement in the program. AGE, the audit agency,

systematically pursued its modernization plan and begun to change its oversight approach. Many

of the other agencies performed well, broadly in accordance with expectations, except the

Secretariat of Planning, as explained in more detail in section 3.2. The Secretariat of Education

exceeded expectations by assiduously executing a far reaching program that had detractors and

required leadership. The Secretariat of Health embraced the program under its purview and

dedicated considerable efforts to performance based management, leading to the UPA and PAHI

programs meeting their targeted results. SEPLAG had challenges carrying out the expected

reforms of the public investment program. The agency has remained more focused on its

considerable day-to-day responsibilities of administering the investment program, and did not

devote the time to develop the information and procedures expected under the reform program.

(c) Justification of Rating for Overall Borrower Performance

Rating: Moderately Satisfactory

83. The overall performance is deemed as moderately satisfactory as a result of the

shortcomings in the implementation of the program. Despite the moderate rating, the

achievements by the Borrower should be recognized, given the scope of the reforms pursued and

the number of targets met. The operation signified an important effort to accelerate the reforms

started earlier, and demonstrated a commitment to show much better results, particularly in

service provision. While the execution was somewhat uneven, the aggregate effort was

significant and merits recognition.

6. Lessons Learned

84. There are a number of lessons under this operation that reinforce those learned in

similar programs. The implementation experience with the DPL highlights the importance of the

quality of personnel capacity, of learning by doing, of design updates, dealing with technical

20

setbacks, and the need for technical assistance and close supervision. Here, only three key

lessons are noted.

85. The first is the importance of considering the dynamics of the political economy of

reforms in establishing performance targets, even when dealing with individual executing

agencies. This was most evident in the establishment of ambitious targets for parts of the

program, such as in the investment area. The Bank felt it was important for the planning

secretariat to show improved results in the identification, selection and tracking of investment

projects. The lead agency negotiating the loan, SEFAZ, also had a particular interest in agreeing

with those targets. This approach was seen as means to induce the line agency to make greater

efforts in handling the evaluation of public investments. All parties recognized, however, that

without greater political leadership and without greater commitment by all involved agencies,

those goals had little chance of being achieved. The decision to proceed, despite considerable

evidence of the political roadblocks, proved to be over ambitious. The goals were not fulfilled

and little progress has been made to date. In retrospect, it might have been better to help build the

institutional capacity of the agency through other means.

86. Conversely, in those areas where there was clear leadership and commitment to the

reforms, the program achieved much greater success. The greatest accomplishments occurred in

those areas where the responsible authorities were able to garner the full commitment of the

involved personnel to achieve the established results. In all successful instances, the reforms

were also central to the goals of the agencies.

87. The second main lesson concerns building institutional capacity. Successful public

sector reforms are often accompanied by measures to create stronger institutions. This includes

both better organizational structures and incentives frameworks. The DPL was a good example

of the importance of institution building. Those areas that saw sustained improvements, such as

in education, introduced new incentive systems and more importantly new methods for

improving service provision. The same occurred, but to a lesser extent in the health sector. The

shift to a system of contracts for urgent care services transformed not only the service provision

but also the Secretariat of Health, which is now much more focused on supervising the contracts

and following up on the standards of hospital services.

88. Another example along the same lines is the modernization process that is just starting

in AGE. There is a new vision on the role of the agency and the manner in which it can perform

its tasks, away from ex-post compliance reviews to much more significant ex-ante risk based

standard setting tasks. In general, the areas of success under the DPL program where those that

saw improvements in their institutional capacity. Improvements occurred through the methodical

introduction of new incentives and slow changes in behavior that helped to create a new culture

in the institutions and the acceptance of modernization processes. As in most cases,

implementation could have been better if there were greater focus and resources to assist in

institutional strengthening, including through supervision. Achievements were more limited in

those areas where institutional deficiencies were not addressed. The experience shows that

greater focus and resources to assist in institutional strengthening remains a key to development,

particularly as it pertains to public sector reforms.

89. Finally, one last important lesson is the value of consistency and persistency in the

reform process. The DPL was the result of a long engagement with the State of Rio, as well as

the city of Rio. The engagement occurred through the preparation and supervision of several

21

operations that focused generally in the same areas. This helped to build expertise and

knowledge through a process of learning by doing, which proved to be critical. This knowledge

not only applied to the design of the programs and the adjustment to measures over time, but

equally as important to overcome the resistance to reforms. It was through a slow and methodical

process of trial and error along with showing and explaining that changes are eventually being

accepted. The focus on results-based management also helped to make the merits of the reforms

more transparent. The continuous engagement by the Bank and the persistency of the Authorities

to pursue key reforms contributed significantly to build a solid basis for future, and to make the

program a relative success.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners

(a) Borrower/Implementing Agencies

90. No issues were raised. The GoRJ formally agreed with the ratings and provided

editorial comments that have been incorporated in the final version of the report.

(b) Cofinanciers N/A

(c) Other Partners and Stakeholders NA

22

Annexes

Annex 1. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team Members

Names Title Unit Responsibility/

Specialty

Lending

Barbara Bruns Lead Education Economist LCSHE Education

Andre C. Medici Sr Economist (Health) LCSHH Health

Alberto Coelho Gomes Costa Senior Social Development Specialist LCSSO Social

Development

Pablo Fajnzylber Sector Manager, PREM AFTP2 Economic Policy

Joseph Kizito Mubiru Lead Financial Management

Specialist

LCSFM Financial

Management

Yaye Seynabou Sakho Adviser MDI Economic Policy

Mariana Margarita Montiel Senior Counsel LEGLE Legal

Patricia Miranda Senior Counsel LEGOP Legal

Magnus Lindelow Sector Leader LCSHD Human

Development

Roland N. Clarke Lead Economist and Sector Leader,

PREM

LCSPR Public Sector

Governance

Maria Florencia Liporaci Senior Program Assistant MNSSD Program Assistant

Fanny Weiner Public Sector Mgmt. Spec. LCSPS Public Sector

Governance

Edith Kikoni Economist LCSPE Economic Policy

Flavia Nahmias da Silva

Gomes

Program Assistant ECRBX Program Assistant

Claudia Rocio Manrique LCSPS Program Assistant

Cindy Audiguier LCSPE Economic Policy

Rafael Chelles Barroso Economist LCSPE Economic Policy

Ezau Pontes Senior Health Specialist LCSHH Health

Rogerio Bianchi Santarrosa LCSPP Poverty

Fabio Sola Bittar Research Analyst LCC5C Economic Policy

Leandro Oliveira Costa Economist LCSHE Education

Ana Mie Horigoshi Reis Junior Professional Associate LCSPS Public Sector

Governance

Supervision

Barbara Bruns Lead Education Economist LCSHE Education

Andre C. Medici Sr Economist (Health) LCSHH Health

Mariana Margarita Montiel Senior Counsel LEGLE Legal

Patricia Miranda Senior Counsel LEGOP Legal

Magnus Lindelow Sector Leader LCSHD Human

Development

Roland N. Clarke Lead Economist and Sector Leader, LCSPR Public Sector

23

PREM Governance

Maria Florencia Liporaci Senior Program Assistant MNSSD Program Assistant

Fanny Weiner Public Sector Mgmt. Spec. LCSPS Public Sector

Governance

Edith Kikoni Economist LCSPE Economic Policy

Flavia Nahmias da Silva

Gomes

Program Assistant ECRBX Program Assistant

Claudia Rocio Manrique Program Assistant LCSPS Program Assistant

Cindy Audiguier Consultant LCSPE Economic Policy

Laura De Castro Zoratto Economist LCSPS Public Sector

Governance

Rafael Chelles Barroso Economist LCSPE Economic Policy

Ezau Pontes Senior Health Specialist LCSHH E T Consultant

Ana Mie Horigoshi Reis Junior Professional Associate LCSPS Public Sector

Governance

Angela Nieves Marques Porto E T Temporary LCSPS E T Temporary

Eduardo Somensatto Consultant LCSPS Economic Policy

Lorena Viñuela Public Sector Specialist LCSPS Public Sector

Governance

(b) Staff Time and Cost

Stage

Staff Time and Cost (Bank Budget Only)

No. of staff weeks US$ Thousands (including

travel and consultant costs)

Lending

Total: 79.09 286,513.80

Supervision/ICR

Total: 19.56 59,625.87

24

Annex 2. Beneficiary Survey Results

Not applicable.

25

Annex 3. Stakeholder Workshop Report and Results

Not applicable.

26

Annex 4. Summary of Borrower’s ICR and/or Comments on Draft ICR

The GoRJ formally agreed with the ratings and provided editorial comments that have been

incorporated in the final version of the report.

27

Annex 5. Comments of Cofinanciers and Other Partners/Stakeholders

Not applicable.

28

Annex 6: Recent Operations in Rio de Janeiro

While not a formal programmatic series, the sequence of loans that supported different

phases of the reform process and the nature of the continuous policy dialogue essentially makes

the operations a quasi-programmatic series. Details are available in Table A.1.

The first State DPL (Fiscal Sustainability, Human Development, and Competitiveness DPL–

P117244) was approved in February 2010. The first DPL was done partially in response to the

crisis when the Government was concerned that the fall in revenues could destabilize the overall

reform agenda by jeopardizing financing for the health and education programs. It focused on

the initial GORJ efforts to improve fiscal management, competitiveness, and social service

delivery. The first DPL established the basis for the current operation by supporting earlier

measures to consolidate the fiscal position, improve the efficiency of tax administration,

strengthen budget procedures and financial management, and raise the quality and efficiency of

the education system, along with increasing access to health services.

The Second State DPL (Rio de Janeiro Metropolitan Urban and Housing Development Policy

Loan – P122391), approved in March 2011, was in part a response to the natural disasters from

heavy rains, which highlighted the issue of urban housing. The operation was designed to

support the GORJ to strengthen its policies for planning and managing territorial growth in the

Rio de Janeiro Metropolitan Region, and promoting the provision of affordable housing.

The third DPL built and deepened the work started under the first DPL by addressing the same

broad areas of fiscal reform, health, and education. The operation was also accompanied by

important technical assistance, for example in the education area, and by knowledge services, in

both the health and fiscal areas. This DPL helped the Government launch an important new

phase of reforms.

The Bank’s involvement with the GORJ was also strengthened by the close working relationship

it had developed with the Municipality of Rio de Janeiro and its work with other states in Brazil.

The Bank had a similar operation with the Municipality of Rio that focused on many of the areas

that formed part of the State’s DPLIII. The experience of working at the municipal level

revealed the importance of coordinating the reforms at different levels of government and

contributed to identifying the measures that needed to be taken by the State to improve the joint

provision of public services, such as in the health and education sectors.

The Bank is providing continued support for the implementation of reforms in the major areas of

the DPLIII. This is occurring through a follow up operation, DPL IV (Enhancing Public

Management for Service Delivery) and two Technical Assistance Loans. The new DPL operation

is assisting the GORJ to enhance public management for the delivery of key public services for

vulnerable populations by instituting new policies and regulations to improve: the medium term

planning and monitoring of public expenditures (Policy Area 1); the accessibility, quality and

affordability of urban mobility services for the poor (Policy Area 2); and the availability of

targeted social services aimed at reducing domestic and gender based violence (Policy Area 3).

The Bank also has two Technical Assistance Loans in place (Public Sector Modernization and

Strengthening Public Management and Integrated Territorial Development) to support the GORJ

to support performance-based management and introduce information technologies to drive

service delivery improvements in key public services, including secondary education and

hospital care. As well, the loans are supporting modernization of the tax administration and

public finance management.

29

Table A.1: Rio de Janeiro Operations, 2009-2013

2009 2010 2011 2012 2013 2014

CPS 2012-

2015 Priority

Areas

DPL 1

Fiscal Sustainability,

Human Development

and Competitiveness

(US$ 485 million)

TAL (P106768)

Public Sector

Modernization

(US$ 18.6 million)

DPL 2

Metropolitan Urban and

Housing

(US$ 485 million)

DPL 3 (P106465)

Fiscal Efficiency for

Quality of Public

Service Delivery

(US$ 300 million)

DPL 4 (P147695)

Enhancing Public

Management For

Service Delivery

(US$ 500 million)

TAL II (P126735)

Strengthening Public

Management and

Integrated Territorial

Development

(US$ 48 million)

Fiscal

Consolidation

and Public

Financial

Management

Capitalization of

Social Security

System for State

public servants

Reorganization of

budget processes

Adoption of Public

Sector Performance

Indicators

Real Estate

Property

Management

System

Social Security

Management

System

Piloting of a

methodology for

costing public

policy programs

Restructuring of the

internal audit

agency.

Treasury’s

operational risk

policy

Adoption of budget

preparation

practices to

improve its

accountability and

fiscal discipline

Adoption of

financial and debt

management tools.

Tax

Administration

Strengthening of

the Revenue

Secretariat.

Adoption of

electronic invoices.

Strengthening of

synchronized

taxpayer

registration system.

Adoption of

standards to

improve collection

of State taxes.

Implementation of a

results-based

management

system for tax

administration.

Strengthening core

taxation functions

30

Public

Investment

Adoption of a

methodology and

procedures for

evaluating,

selecting and

approving

proposals for large

public investment

projects.

Private Sector Streamlining of

business

registration process

Education Implementation of

learning program to

reduce of age-grade

distortion in basic

education.

Implementation of

new school

management

information

program

Building of

SEEDUC Technical

Capacity for

Studies and

Evidence-based

Policy.

School-level

Performance

Targets for State

Schools.

Adoption of a

merit-based

selection process

for regional and

school directors.

Establishment of

annual school-level

targets for

improving student

learning and

graduation rates and

a bonus pay system

for schools.

Developing In-

Service Teacher

Training Network.

Mapping of

demographic and

infrastructure

needs.

Health Adoption of

national standard

practices in 24-hour

operating Urgent

and Emergency

Care Units (UPAs).

Establishment of a

system of

performance-based

transfers from the

State to

Information System

for Health Supply

Chain Management

Evaluation of

Financial Incentive

Programs for

Municipalities

Framework to

assess and contract

social organizations

to manage public

hospitals and health

units.

Evaluation

mechanisms to

transfer bonuses

and monetary

incentives (i) to

Cost-Accounting

Methodology for

UPAs and

Hospitals.

31

municipalities municipalities and

municipal hospitals

and (ii) to regional

State hospitals.

Housing and

Land

Management

Enhancement of

low-income

housing policy and

financing

mechanisms.

Strengthening land

management and

titling.

Enhancement of the

governance of the

State Housing Fund

(FEHIS).

Strengthening of

ITERJ’ land tenure

regularization

capacity

Restructuring of the

Secretariat of Social

Assistance and

Human Rights.

Improving living

conditions in the

most vulnerable

social settings

Urban

Management

Strengthening

metropolitan

management

through improved

integration and

coordination in

urban development

and transport.

Establishment of a

Directive

Committee for

Metropolitan

Strategy and an

Strengthening

metropolitan

management

through integration

and coordination in

urban development,

housing, transport,

environment and

disaster risk

management

32

Integrated Program.

Targeted and

integrated social

development

programs to the

urban poor.

Establishment of

the UPP Social

Program, on a pilot

basis, in the State

slums.

Transport Creation of the

Single Ticket

(Bilhete Unico) for

public

transportation in the

RJMR.

Adopted of the

Urban Transport

Master Plan

(PDTU) and a

stakeholder

mechanism to

accompany

implementation.

Implementation of

a policy to

physically integrate

municipal and

State- managed

mass transit

systems.

Adoption of a new

policy to promote

non-motorized

transport.

Adoption of a

policy for

competitively

awarding licenses

for bus routes,

instituting a results-

based management

system for

33

concessions, and

increase

transparency on bus

service

performance.

DRM Radar system. Establishment of a

State disaster risk

policy.

DRM.

Natural

Resource

Management

Strengthening of

the Institute of

Environment to

capacity to protect

environmental

assets.

Improvement of

effective watershed

management and

decentralization.

PDO Rating Satisfactory Satisfactory Moderately Satisfactory Moderately Satisfactory Satisfactory (ISR) Not yet initiated

34

Annex 7. List of Interviewees

The following individuals were interviewed during the preparation of the report.

Paulo Tafner Deputy Secretary for Finance, Secretariat of

Finance, GORJ

João Marcos Meneses Simas de Souza Finance Analyst, Secretariat of Finance, GORJ

Luisa Mazer Finance Analyst, Secretariat of Finance, GORJ

Eugenio Manoel Chief Auditor, Secretariat of Finance, GORJ

Robson Ramos Audit Manager, Secretariat of Finance, GORJ

José Correa Superintendent of Finance Information Records,

Secretariat of Finance, GORJ

Sergio Abramovich IT Director, Secretariat of Finance, GORJ

Julio Montovani Finance Planning Director, Secretariat of

Planning, GORJ

Eduardo Brandão Finance Analyst, Secretariat of Finance, GORJ

Pablo Villarim Superintendent of Fundraising, Subsecretary of

Finance, GORJ

Fátima Leite Planning Manager, Secretariat of Planning,

GORJ

Claudia Bezerra Health Director, Secretariat of Health, GORJ

Deborah Linhares Assessor Subsecretariat of Finance, Secretariat

of Finance

Roland Clarke Task Team Leader, World Bank

Rafael Chelles Barroso Economist, World Bank

Barbara Bruns Lead Education Economist, World Bank

35

Annex 8: Map State of Rio de Janeiro