Impact assessment of public policies: guidelines for the evaluation of the PEIEX program

The objective of this study is to propose general guidelines for evaluating the impact of the Export Qualification Program (PEIEX), which considers the delivery of a theory of change and guidelines for evaluating the impact of this program, using the propensity score-matching and Difference in Differences methods to infer the eventual effect. To obtain information about the program, a survey was carried out including publications with PEIEX results, as well as on the website of the agency that created it, the Brazilian Agency for the Promotion of Exports and Investments (ApexBrasil). It is concluded that the guidelines delivered for the impact assessment of the PEIEX may contribute to studies aimed at empirically inferring about the program and thus contributing to its growth and improvement. In addition, it is suggested the application of interviews with those responsible for PEIEX to obtain more detailed information about it, in order to expand the scope of the proposed theory of change.

Keywords: Impact assessment; Public policy; PEIEX; Apex Brasil

Through the formulation and application of public policies, governments seek to boost the economic and social development of countries [1-4]. One of the areas in which these public policies are formulated and applied is entrepreneurship [5-10].

This study focuses on public policies aimed at fostering International Entrepreneurship (IE). In Brazil, one of these initiatives is the Export Qualification Program (PEIEX in the acronym in Portuguese and hereafter) of the Brazilian Trade and Investment Promotion Agency (Apexbrasil). Among the expected benefits of applying policies that encourage and support IE are the increase in exports, the development of an entrepreneurial culture and the increase in job and income generation [11]. In addition, it is expected that this promotion will help companies to be competitive in foreign markets.

Considering the international competitiveness of Brazilian companies, according to the National Confederation of Industry (CNI in the acronym in Portuguese and hereafter), Brazil ranked second last position among 18 countries analyzed in a study carried out between 2019 and 2020. Among the Latin American countries analyzed in that study, Brazil (17) was far from the best placed in the ranking, Chile (8) and only surpasses the last placed, Argentina (18). Regarding Brazil’s share in the production of the world manufacturing industry, a drop from 1.31% in 2020 to 1.28% in 2021 was recorded (CNI, 2022).

With regards to exports from manufacturing industry specifically, the CNI reports that Brazil’s global share fell to 0.84% in 2019, to 0.77% in 2020 and recorded an improvement in 2021, reaching 0.81%. Also, according to the CNI report (2022), the figures confirm the negative and backward performance that the country’s manufacturing has been showing since 2012, still failing to reach pre-covid-19 pandemic levels. Sebrae (2018) reports that about 8 out of 10 Brazilian companies that exported in 2015 did not do so again in the following years, and among the most cited reasons for this is the lack of international demand, and along with that, the fact that 83% of the companies did not receive any type of institutional support.

In this sense and with the aim of encouraging and supporting exports by Brazilian companies, policies/initiatives such as PEIEX are formulated and applied. Regarding public policy as a research area [12] suggests that research should support its formulation. For [13], public policy is an area that lacks more studies, whereas for [14] it is a field of knowledge that simultaneously pursues two things. First, to put the government into action and/or review that action, and second, when necessary, to propose changes to those actions. In addition, [14] states that these actions are expected to produce results and changes in the real world. As a field, for [15], public policy is the analysis of the State in action.

Gertler et al. [16] argue that there is a crucial point in public policy, although not frequently addressed, namely: knowing whether the changes intended by a public policy are actually achieved. The authors state that, in general, program managers and those responsible for formulating public policies focus on controlling and measuring the immediate inputs and outputs of a program, instead of evaluating whether these programs achieve the intended objectives of improving the well-being.

In what concerns PEIEX, a search carried out in February 2023 in the Google Scholar database showed the scarcity of scientific production on the application and results of this program and as reported in the literature about this program [17-21]. It is not possible to identify studies that make statistical inferences of the impact of this program adding to qualitative analyses.

In addition, no information was found that could show in detail the methodology applied by the program. Despite being mentioned by the aforementioned authors, the methodology was not described in detail, except for mentioning the steps, the directions given to program partners and the areas of organizations in which the method is applied. It is understood that this lack, in part, can be caused by the lack or difficulty in obtaining information from the program. From the above, there is a crucial gap to be filled related to the impact assessment of public policies and programs in general and PEIEX in particular.

Thus, the objective of this study is to propose general guidelines for assessing the impact of PEIEX using a Theory of Change (TC), as well as guidelines for one of the possible approaches for quantitatively assessing the impact of this program. To this end, a survey was carried out of publications with PEIEX results, as well as on the website of the program and the agency that created it, Apexbrasil. The present paper presents a review of public policy and its evaluation, followed by information about PEIEX and the guidelines for evaluating its impact. Lastly, the final considerations are presented.

Mintzberg and Jorgensen [22] understand that public policies are for the public sector what strategy is for the private sector, arguing that in both sectors, policies and strategies should be taken as a guide or set of guidelines to influence decisions and acts in the future. In relation to public policies aimed at entrepreneurship in general, they are relatively new, even though their connection to economic growth dates back to the era of mercantilism [8].

In Brazil, public policies aimed at smaller companies that need the most support, date from the 1960s, passing through a timid period until the 1990s, when a change was observed in terms of taxation for Small and Medium Enterprises (SMEs) and a greater focus on technological development in relation to tariff protectionism, demonstrating a very low number of these companies with access to aid programs [11].

An important milestone in the history of public policies to support Brazilian companies is the birth of the Brazilian Support Service for Micro and Small Companies (Sebrae in the acronym in Portuguese and hereafter). [23] reports that Sebrae was founded in 1972 as a private entity of public interest, representing the main support body for Brazilian micro and SMEs. When it comes to the formulation of public policies for entrepreneurship, Acs and Szerb (2008) warn of some challenges such as fiscal, educational, science and technology, as well as regulations linked to entrepreneurship. For [7], public policies to encourage entrepreneurship are not always effective or focus on a clear understanding of the costs and benefits of entrepreneurship.

At the end of the 1990s, changes were made in support policies for SMEs in Brazil, directing support to groups of companies called productive arrangements, which despite representing progress, still denotes a long way to go and adjustments to be made [11]. In this context, there are questions about the application and evaluation of the impact of public policies.

White and Raitzer [24] mention 2 purposes for their evaluation, namely: the first is accountability, which refers to ensuring that the actions taken actually lead to development results. The second is learning, which aims to provide evidence for selecting and designing interventions that are likely to be effective in fostering outcomes of interest.

White and Raitzer [24] understand that impact evaluation can answer the following questions: what difference a policy or program can make with its application and which program designs are most effective for one or more specific quantifiable results. In addition, impact assessment offers understanding about how these results differ among different populations, as well as about what factors condition these results.

Regarding techniques for impact assessment, [16] mention the following ones without claiming to be exhaustive: random selection methods, instrumental variables, discontinuous regression, Difference in Differences (DD) and matching. According to the aforementioned authors, all these approaches share the common goal of creating valid comparison groups so that the true impacts of a program can be estimated.

Another point to be highlighted in relation to the formulation and implementation of an intervention is what [16] suggest before implementing an initiative, that is, that there should be a TC that details how the intended results should be achieved and that serves for the intervention evaluation stage. The TC is a description of how an intervention is designed to generate the desired results. In addition, it describes the causal logic of how and why a given program should achieve the intended results [24].

Efforts have been made in terms of formulating policies and actions that favor different areas at the national level, including IE. In what concerns this study, it will focus on Apex Brasil’s PEIEX program, presented in what follows.

According to information available at the program website, PEIEX is the export qualification program offered by ApexBrasil to aid companies in the export process in a planned and safe way [21]. According to PEIEX, it is implemented in all regions of the country, through partnerships between Apexbrasil and teaching institutions (Universities, Technological Parks or Research Support Foundations) or Industry Federations, called program executing entities which are responsible for applying the PEIEX methodology to qualify companies. Service to companies is provided by professionals specialized in foreign trade (technicians) who guide businessmen in the most appropriate ways for the foreign market [21].

As explained on the website “PEIEX’s assistance is provided by the Operational Nucleus team based on visits to companies: the 1st visit aims to measure the export potential; the 2nd visit is when the PEIEX technician carries out a business diagnosis to identify the company’s level of readiness for export. Based on the information gathered, the PEIEX Nucleus team prepares a work plan with the implementation of necessary requirements for export” [21].

Regarding the PEIEX methodology, the program’s assistance in the aforementioned steps seeks to help companies to export and be more competitive through improvements in the areas of Strategic Management, Human Capital, Finance and Costs, Sales and Marketing, Foreign Trade, and Product and Manufacturing [17-19].

Regarding the PEIEX results, in the study by [18], the authors report that of the 191 demands suggested to entrepreneurs by the PEIEX Operational Nucleus, 77 were prioritized (40%), 42 partially prioritized (22%) and 72 not prioritized (38%). In addition, these authors mention that the PEIEX technician (responsible for service) did not have to prove the implementation of the program’s suggestions. In fact, [19] conclude that the commitment of the entrepreneur assisted by the program is essential. Also, the efforts that are needed (from the executing entities) are important so that the activities directed by the technicians are fulfilled.

Dornelas and Carneiro [17] highlight the support that PEIEX provides to participating companies but make a caveat related to the preparation of those who are responsible for serving the companies participating in the program. The aforementioned authors state that these professionals should have more specialized knowledge than most in relation to export processes. Thus, the contribution of initiatives and programs such as PEIEX is considered necessary and relevant, but so is its evaluation. In what follows, Table 1 shows the states, cities and executing entities where the program is applied.

Table 1: PEIEX in Brazil – 2022.

First, it is necessary to define what impact evaluation is. For [16], it is a particular type of evaluation that seeks to answer the specific question of cause and effect about the impact (or causal effect) of a program on an outcome of interest. The authors add that this basic question incorporates an important causal dimension, in which the focus is only on impact, that is, the changes directly attributable to a program. For [24], evaluating impact is quantifying the causal effect of an intervention for a specific population.

It is worth highlighting an important point in an impact evaluation, namely: the preparation, in collaboration with stakeholders, of a TC (ideally before) for the intervention of a given program [25]. The TC is a statement of how inputs, such as funding, people and legal changes, lead to outcomes and impacts [24]. In addition, it identifies the steps in the causal chain of the intervention and the underlying assumptions that need to be fulfilled for the TC to operate as planned [26].

A TC helps to identify variables where data should be collected, as well as deviations from the initial plan that could lead to different results [24]. In this sense, it is opportune to clarify that the TC is not a complex theory, for example, from the social sciences or psychology, but rather an objective and pragmatic framework that describes how an intervention affects a change [26].

In the case of PEIEX and considering the information available about the program, it was not possible to confirm the existence of a TC for this initiative. This may be an indication that the program may be part of the large percentage of managers and programs that do not assess the impact of their programs [16]. Thus, based on the information provided by PEIEX, as well as on studies that analyzed the program [17-20] and taking into account discussions in this regard [26, 24, 16], a TC proposal is elaborated for PEIEX.

A TC should include some basic elements such as a causal chain, a specification of external influences and conditions, and the main assumptions. A basic results chain maps the following elements: inputs, the resources available to the project, including staff and budget; activities, the actions taken or work done to convert inputs into outputs; products, the tangible goods and services that project activities produce, being directly under the control of the agency responsible for implementation; the results that are likely to be obtained after the beneficiary population uses the project’s products. They are generally achieved in the short to medium term and are typically not directly under the control of the implementing agency; the final results achieved indicate whether the project objectives were met. They can normally be influenced by multiple factors and are achieved after a longer period [24].

According to information in the program, PEIEX offers a service (which lasts approximately 38 hours) through technicians who, together with the company, will build an export plan based on the product/service of the enterprise. This service takes place in stages developed in a company profile, a management diagnosis, a work plan, monitoring the implementation of the work plan, the evaluation and conclusion of the works. All these services do not generate costs for the company.

Other relevant information about PEIEX is related to the main objectives of the program, “to increase the competitiveness of companies; disseminate the export culture; expand access to products and support services available from government institutions and the private sector; contribute to raising employment and income levels; promote capacity building for innovation; expand the number of exporting companies [19, 21].

In addition, as a requirement to be part of the initiative “Companies of any size can participate in the program (except Individual Micro-entrepreneur -IME), as long as they have exportable products or services with added value, capacity to serve the foreign market and who are committed to meeting the stages of the project”. It is worth mentioning what could be an adjustment of the program in terms of requirements, and according to information from APEX, in the year 2022, in its news section, PEIEX would be serving companies cataloged as Individual Microentrepreneur (IME) [21]. This information is important because a program’s eligibility criteria and their clarity for impact evaluation purposes is directly related to the method to be used for an impact evaluation.

Considering the aforementioned PEIEX methodology [17-21], as well as the information obtained on the program’s website, a preliminary analytical scheme of TC was elaborated (Figure 1).

Note: due to the limited information provided by PEIEX, more detailed information is required to feed the TM, which can be obtained from the program’s stakeholders. *Establish indicators to measure suitability as appropriate.

When dealing with impact evaluation, it is necessary to keep in mind some fundamental concepts. According to [16], impact evaluations determine the extent to which a program and only that program causes a change in an outcome variable. These authors state that in impact evaluations, two concepts must be understood (also addressed later), causal inference - in which it is possible to infer the cause of a change the result of a specific program and not to other factors - and the problem of counterfactual - in which there is a situation where there are no problems to know the result for those who participate in a program (i.e. company or person), but rather to establish the result for that same participant in the absence of the program.

About the impact evaluation methods mentioned by [16], that is, random selection methods, instrumental variables, discontinuous regression, DD and matching, the first 3 produce estimates of the counterfactual when there are clear and explicit rules for allocation of program beneficiaries, which is not the case from PEIEX. Regarding the matching and DD methods, these offer additional tools that can be applied when the beneficiary selection rules are less clear or in case none of the other three methods mentioned above are plausible to apply (random selection methods, variables instrumental, discontinuous regression).

One difficulty that may arise in applying the matching method is the need for large databases and the fact that it can only be applied considering observable characteristics in the sample, which makes defining a comparison group more difficult. In the case of this work, focused on PEIEX, with the available information it is estimated that the application of the DD method is the most advisable, which is why the guidelines for impact assessment will be directed in this direction.

PEIEX is the Export Qualification Program [21]. According to information from the III national meeting of this program in 2022, by the end of that year, PEIEX should have reached the figure of 2,200 companies served. In addition, it is reported that the program had 300 technicians responsible for serving companies, of which 70% were micro and small and 30% medium and large companies that were starting their export processes [21]. During its 18 years of existence, 25,000 enterprises that have never exported or that had little export culture were assisted by PEIEX [21].

Some results of services provided by PEIEX have been presented by authors such as [17-20]. There is consensus about the importance of the existence of initiatives such as PEIEX for the promotion of exports by Brazilian companies. Based on information from ApexBrasil, PEIEX and on some works that have addressed the program, such as those already mentioned here, it is possible to establish that PEIEX can favor the increase in exports of companies that participate in the program.

Since the program’s actions, which include the preparation of the Operational Centers in which the program’s activities are carried out, the training of technicians in charge of assisting companies, the evaluation of companies and the proposed plans with suggestions to be applied in each company, have the potential to materialize this increase in exports in the participating companies. Thus, the following hypothesis is established: PEIEX positively impacts the exports of companies participating in the program. In statistical terms: H0: the impact or difference between the outcome of the treatment and control group = 0.

The population of interest for assessing the impact of PEIEX are Brazilian exporting companies. A sample must be obtained from this population of interest to configure the treatment (companies participating in the program) and control (companies not participating in the program) groups necessary for the procedures for the analysis of program’s impact.

It is important to point out that PEIEX is a program that has already been implemented. Therefore, the sample that will be obtained, both for the treatment group, that is, the one that corresponds to the companies that have already taken part in the application of this initiative, and those that did not participate in this program - control group -, must ideally fit over a period of -2/+1 years to allow identifying the impact of the program. With this information in mind, treatment and control groups will have to be formed. Table 2 below shows a Sample Plan.

Table 2: Sampling plan.

As mentioned in the Sampling Plan, treatment and control groups must be formed to assess the impact of PEIEX. In the case of the treatment group, this could be obtained through the program itself – without disregarding the possible restrictions arising from the General Data Protection Law (GPDL - Law No. 13,709/2018). The control group, on the other hand, may present an additional challenge for its configuration: the possible difficulty of accessing, through the program, data from these companies, since they did not participate in PEIEX.

It should be mentioned that in case companies are unable to form groups, an analysis at the Brazilian state level (aggregate) and not at the company level (individual) could be chosen. In this case, it would be necessary to resort to a source that provides the necessary data for the configuration of the groups and the counterfactual necessary for the impact assessment, one possibility being the Comex Stat portal of the Ministry of Industry, Foreign Trade and Services.

Subsequently, having already made the selection of the two groups, considering an analysis at the company level, they will have to be asked to respond to a questionnaire prepared based on the PEIEX objectives and actions described in the TC. It is worth remembering that the objectives and actions outlined by the program also serve as a basis for the construction of the TC formulated for the evaluation of the program’s impact. The period for data collection should consider the proposed Sampling Plan. Therefore, it will be possible that the results of the application of the program can be observed in the participating companies. However, considering that other situations may have influenced the results, situations that should be controlled (this point is addressed in what follows).

For data analysis, it is proposed to use the DD approach [27, 24, 16] which compares changes in outcomes over time between a population that participates in a program, i.e., the treatment group, and a population that does not participate in the program, the control group. It should be noted that what is estimated in the DD method is the counterfactual of the change in outcomes for the treatment group, where an estimate of this counterfactual is the change in outcomes for the control group [16].

A characteristic of the DD method is that it is advisable when the program eligibility criteria are not known with certainty. When criteria is known and clear, discontinuous regression is the most used approach [16]. In the case of PEIEX, the program has already been implemented and for this study there is no certainty of its eligibility criteria or of its actions.

A key point that should be highlighted before applying the DD method is related to the parallel trend assumption [28,27,24,16], fundamental assumption for a valid estimate of the counterfactual that the DD method can deliver to avoid incurring errors in the processing of regressions. In addition to the parallel trend, it is necessary to pay attention to another assumption in the DD method, namely: the correct specification of the regression model that includes control variables that consider possible effects on the studied variable [28].

In the case of the parallel trend assumption, one must assume that there are no time-varying differences between the treatment and control groups. That is, time must be equal between treatment and control groups, and group-specific unobservable factors must be constant over time. Since it is not possible to prove the equality of trends over time, there is the possibility of being tested. [16] mention a form of verification that consists of observing the trend of the dependent variable at different times, before the program intervention (at least 2 times), in the control and treatment groups.

Thus, the sample should be divided into four groups: (1) control group before the change; (2) control group after change; (3) prechange treatment group; and (4) treatment group after the change. The model for DD regression can be expressed as:

Y = β₀ + β₁.DG + β₂.DT + δ(DG.DT) + μ

Wherein:

Y = dependent variable

DG = dummy variable for groups (treatment and control)

DT = dummy variable for time (before and after treatment)

DG.DT = interaction term (i.e., treatment impact)

β0 = intercept

β₁ = coefficient of the groups

β₂ = coefficient of the time

δ = coefficient of the interaction term

μ = random error

(Figure 2) In this sense, to assist in the formation of treatment and control groups, the Propensity Score Matching (PSM) technique can also be used, a technique that seeks the propensity of individuals (observations) to belong to one group or another, matching participants and non-participants [27].

It should be remembered that access to data is essential for choosing the analysis to be used in the impact assessment. When obtaining data from companies to form the treatment and control groups, the analysis will be at the company level. On the other hand, in the impossibility of obtaining these data, one way of configuring the counterfactual and the two groups is to use secondary data from sources such as the aforementioned portal and taking into account the information in Table 1, which contains information on the states where PEIEX is applied.

As a way to validate the results, a placebo test can be applied. For this purpose, a false control group can be chosen in which, unlike the technique mentioned above, which evaluates the dependent variable at different times, a group that we know was not affected by the program chosen, in which the result of the comparison should indicate impact zero, otherwise we are facing some variable that is not being measured, and therefore, we cannot claim equality in the trend over time between the groups. Other forms of placebo may also be applied; that of sham control and treatment groups; and different comparison groups [16]. In other words, if the placebo test result is statistically insignificant (p-value ≥ α), there will be robust evidence of control group adequacy. Finally, a drawing of the proposed analytical scheme is presented (Figure 3).

This work aimed to deliver general guidelines for assessing the impact of PEIEX, which considered the delivery of a Theory of Change, as well as guidelines for assessing its impact. The guidelines were made using the approach that was deemed appropriate for that, the PSM and the DD. It should be mentioned that the reason for choosing these methods was mainly due to the complementarity between them and the lack of information necessary to suggest another method for impact assessment, such as, for example, random selection or discontinuous regression [29-32].

These two methods are possible to apply when the rules for selecting a program are clear, explicit and transparent, thus providing equal conditions for people or companies to participate in a program, in the case of an initiative such as PEIEX. This information allows good results to be expected in terms of a valid counterfactual to make a statistical inference of the impact of a program [33-35].

The application of the DD method must first consider the correct application of fundamental procedures and considered assumptions of this method: on the one hand, the clear specification of the regression model considering all the necessary control variables with factors that can influence the variable of interest and that may negatively affect the assessment of its impact. On the other hand, another assumption that must be met is that of a parallel trend. Therefore, one should consider measuring the variation of the treatment and control groups at least 2 times before the implementation of the program and once afterwards, thus ensuring that the time is the same for both groups and that there are no differences between them that could have been caused by another external factor.

Thus, the guidelines suggested for assessing the impact of the PEIEX program, may contribute to studies aimed at making statistical inferences about the program and thus contribute to its growth and improvement. This work was not free of limitations, one of which was the lack of possibility of obtaining more information about PEIEX was perhaps the most outstanding. This is perhaps due to the fact that this study was limited to a survey of published works with PEIEX results and a survey on the program and APEX websites. Information on impact assessment is of paramount importance, affecting its development at different stages, including the choice of method with which the impact assessment of a given program will be carried out.Notwithstanding this caveat, this work contributes both to studies focused on evaluating the impact of public policies and programs and to PEIEX. As future research, studies are suggested that apply the guidelines given here and that can inform the impact of this program, as well as studies that consider the application of interviews with the team responsible for PEIEX to obtain more detailed information that subsidizes its impact assessment.

Notwithstanding this caveat, this work contributes both to studies focused on evaluating the impact of public policies and programs and to PEIEX. As future research, studies are suggested that apply the guidelines given here and that can inform the impact of this program, as well as studies that consider the application of interviews with the team responsible for PEIEX to obtain more detailed information that subsidizes its impact assessment.

None.

No conflict of interest.