Because of its impact on economic growth mauro1995corruption ; hessami2014political , democracy stockemer2013bribes , and inequality gupta2002does , one of the biggest challenges that a government has to deal with is corruption. Transparency International defines corruption as the abuse of public power for private benefit, i.e., it assumes that corruption involves the participation of public officials. Considering this definition, a particular niche in which corruption can arise naturally is in public procurement; where public and private sectors interact through contracting, mostly to purchase goods or services. Corruption at the level of public-private contracting has high costs in many areas. For example, if buyers favor some suppliers over others through corrupt decisions, bribes or patronage (clientelism) , then market competition is affected north2009violence ; aidt2016rent . This lack of competition leads to a miss-allocation of resources, affecting areas such as budget composition mauro1995corruption , military and technological spending gupta2002does , social care aidt2016rent , and may even change the market structure and dynamics wachs2019social ; fazekas2020corruption . However, owing to the complexity of the contracts, the large sums of money involved, the number of participants, as well as the inherent complicity of public officials, this kind of corruption is difficult to identify, track, and prevent baldi2016bid ; oecd2007integrity ; oecd2015government .
A government turnover, i.e. a change in the individuals and/or parties in power, due to elections or otherwise, may generate significant changes in the way public contracting is done, and thus, in the types of corruption that can be involved clingermayer2003governmental . Broms et al. propose that frequent elections with uncertain outcomes may compel corrupt elites to pursue predatory strategies; however, if there is a well-established party system, regular electoral uncertainty may motivate corrupt elites to exercise restraint broms2019political . In this context, Fazekas et al. found in fazekas2020corruption that fair electoral contest and heterogeneous power-sharing may have the potential to mitigate corrupt market distortions, even in systematically corrupt places. On the other hand, while it seems to be true that government turnovers can diminish corruption in public procurement broms2019political ; fazekas2020corruption , there is also evidence that a change in government can maintain corrupt behaviours, only changing the favored suppliers david2019grand . An example of this was found by the Mexican Institute for Competitiveness (IMCO by its Spanish acronym, IMCO ), which analysed public procurement data from México, finding that in the change of government that took place between 2012 and 2013, the favored suppliers also changed, but not the amount of money and contracts given to the new suppliers. This phenomena was called “the compadres’ change” IMCOcompadres .
For years México has occupied low levels in the score of corruption perception index given by Transparency International TICorruption ; maria2015anatomia . Mexican citizens consider that corruption is one of the biggest problems in the country, only behind violence and insecurity maria2015anatomia . For years, government and corporate structures have been created, maintained, and adapted to obtain private benefits from public resources, which has carried huge consequences to the economic growth and human well-being in the country meyer2019poder . In 2018, México lived the largest electoral contest of its history, with more than 52.3 million voters, representing 62.62% of citizen participation INE . In this election, a “leftist” candidate, won the presidency for the first time. He was the most voted winner in Mexico’s history, obtaining 30.1 million votes (representing 53.1% of the voters), and his party won 388 congress positions of the 554 available (70% of the seats) INE . This event not only marked a government turnover but an ideological and structural change in government’s goals and methods meyer2019poder ; munoztriunfo ; hanrahan2019reflections . To study how this government change affected the mechanisms of public contracting and the forms in which corruption takes place, we analyzed data from more than 1.5 million contracts between the many agencies that conform the Mexican government and private suppliers, corresponding to the period 2013-2020, which includes the first two years of the new government (2019-2020) compranet .
Many states have taken advantage of the new
technology era to record their administrative
activities. This data, which can be used to analyze the projects undertaken by an administration, their successes and shortfalls, mainly describes the state’s procurement practices kim2014big . This large amount of administrative data collected by governments allows studying corruption from a new perspective radermacher2018official . A common approach to quantify corruption is by building risk factors from contracts’ data fazekas2013anatomy ; fazekas2013corruption ; fazekas2016objective ; fazekas2017corruption ; fazekas2018institutional ; fazekas2020corruption ; david2019grand ; wachs2019social ; wachs2020corruption . For example, single-bidder contracts have been shown to be effective to identify and predict corruption risk wachs2020corruption ; wachs2019social ; klavsnja2015corruption ; charron2017careers in various contexts; including studies on the relationship between corruption and political incumbency klavsnja2015corruption , and the effect of campaign contributions on corruption charron2017careers . Other approaches include the application of network theory to measure the impact of single-bidder contracts on local corruption wachs2019social . Network theory was also used to identify corruption risk distribution among countries. For these studies, the actors (buyers and suppliers) were represented by network nodes; these nodes were linked if contracts were celebrated between them, and the weight of the link was the fraction of single-bidder contracts between each pair wachs2020corruption .
Here we propose a somewhat different approach, taking advantage of specific public data that lists suppliers that were investigated and have been identified as having incurred in corrupt practices. Since 2013 México’s government collects a list of companies that have been caught providing invoices for simulated operations (or EFOS for its spanish acronym “Empresas que Facturan Operaciones Simuladas”) EFOS . These companies sell fake receipts to buyers who use them to avoid taxes or to cover acts of embezzlement. There is also, since 2013, a list of all the suppliers that have been caught doing one or several types of corrupt activities when participating in a public contract. For example, presenting fake documentation in order to win a contract, overcharging their services, breaching contract, or diverting resources. These companies are labeled as “sanctioned suppliers and contractors” (or PCS by their Spanish acronym “Proveedores y Contratistas Sancionados”) PCS .
It should be noted that to appear in these lists, these contractors were subject to a legal investigation, and some of the contractors are appealing the decision. Thus, these lists may change slightly as time goes by as a result of companies winning their appeals, which removes them from the lists, as well as due to conclusions of long lasting investigations, which may add new companies. Nevertheless, with the information we have, we identified in the available records of the government’s administrative activities, those contracts in which these companies, which have been suspect of corrupt activities, participated, and we classified the contracts accordingly. Also, we characterized each contract on the record following the risk factors proposed by Fazekas’ group infazekas2013anatomy ; fazekas2013corruption ; wachs2020corruption , adjusted to the data we have at hand. Once each contract is described by this set of quantitative variables, we propose a statistical framework to analyze whether there were changes in public contracting practices as a consequence of the government turnover in México, and identify in which variables this change occurred.
The results shown here were based solely on the analytical study of the open data, trying to avoid any political bias, and without use of any prior knowledge about the contract’s participants.
The lists of public contracts from 2013 to 2020 were taken from two different sources: One given by the electronic Mexican system of public governmental information on public procurement CompraNet compranet , and the other one given at the official site of the Mexican Government (Open Data) Datos Abiertos datoscont . In order to have the most complete data, we construct one list per year taking the lists from both sources, comparing them, and de-duplicating them. Each contract on these lists has a default set of variables that defines them, shown in Table 1.
The list of companies identified as EFOS is available on the site of the Mexican tax agency (SAT for “Secretaría de Administración Tributaria”) EFOS 111For this study we use the lists of both definitive and presumed EFOS, and the list of PCS contractors is available on the official Mexican open data site PCS . The only variable of interest in these lists is the suppliers’ name, so we can identify in which contracts these companies participated.
|i)||Buyer||N/A||Governments agency that made the contract|
|with a private company (the supplier)|
|Supplier||N/A||Private company that was awarded a contract|
|by a government agency (the buyer)|
|Year||N/A||Year in which the contract began|
|Begining week||N/A||Week of the year in which the contract began|
|Endging week||N/A||Week of the year in which the contract ended444The corresponding year on this variable is not necessarily the same in which the contract began|
|Ending-Beginning weeks||EBWeeks||Weeks that the contract lasted|
|Spending||N/A||Amount of money spent by the buyer (in USD)555Since most of the Spending is reported in mexican currency (MXN), we converted the amounts to USD using the equivalences given in federalreserve13 ; federalreserve1417 ; federalreserve1720|
2.1.1 Contract classes
We first identify on the contract lists those in which companies labeled as having been involved in corrupt activities participated. Since these companies have gone through a process to be labeled as either an EFOS or a PCS, we assume that they may be suspect of having incurred in corrupt behaviour in all the contracts they participated in. Thus, the contracts in which the supplier is an EFOS or a PCS, are classified as possibly corrupt and we put the corresponding label EFOS or PCS on them. Following the principle of presumption of innocence, we label as NC (for Non Corrupt) all the other contracts in the data set in which the supplier is free of official corruption charges. Thus, we have three classes of contracts labeled EFOS, PCS and NC, respectively. Of course we are aware that it is probable that corrupt contracts went undetected and end up in our NC class, however we expect that they will have little statistical weight in this class that represents the vast majority of the contracts.
|ii)||Total Contracts||T.Cont||Total number of contracts celebrated|
|between a buyer and supplier|
|Total Spending||T.Spending||Total amount of money paid to a supplier by a buyer|
|Total single-bidder contracts||T.AD||Total number of single-bidder contracts|
|assigned to a supplier by a buyer|
|Active Weeks||N/A||Different number of weeks of the year in which a supplier|
|was assigned contracts by a buyer|
|Contracts per active week||CPW||
|Spending per active week||SPW||
|Fraction of single-bidder contracts||RAD||
|Maximum Spending by a buyer||T.Spending.Max||
2.1.2 Contract description variables
Once we have identified the class to which each contract belongs, we use the information given in the data base to build a set of variables that characterize the participants in each contract, as well as the nature of the contracts themselves. To do this we took the risk corruption factor framework proposed by Fazekas et al. in fazekas2013anatomy ; fazekas2013corruption and retaken by IMCO in IMCOmapeando . However, since the available data is not detailed enough to build exactly the same risk factors developed in fazekas2013corruption ; fazekas2013anatomy ; IMCOmapeando we propose approximate versions of the factors that give similar information about the contract features and the characteristics of the participants. There are three types of items in the data sets:
Items that describe the features of the contracts. For example, the procedure by which the supplier won the contract (single-bidder, public contest, etc.), the amount allocated to the contract, or the week in which the contract began (Table 1).
Items that give information about the relationship between the buyers and suppliers. Examples of these are the number of contracts carried out between a buyer and a supplier, or the total amount of money spent by a buyer with the same supplier (Table 2).
Items that describe the features of the buyers. These include the maximum amount spent by a buyer with a supplier, or the maximum number of contracts carried out by a buyer with a single supplier (Table 2).
2.2 Statistical analysis
The main goal of this work is to identify if the government turnover carried along a methodological change in public procurement, particularly when contracting companies that have since been identified as being suspect of having incurred in corrupt practices. To achieve this goal we first need to analyze whether, within the same government period, there exist significant differences between the three contract classes. Specifically, we aim to verify that each contract class presents statistical differences in their characteristic variables, when compared with the other classes. If so, this will provide further justification for our decision to separate contracts into these three classes and will help associate each class with certain characteristic description variables.
The next step is to compare each class between the different governments to study if indeed there were significant changes in public procurement practices within the classes.
2.2.1 Binomial-Test and Kolmogorov-Smirnov-Test
Since each contract has two kinds of descriptive variables, dummy variables and non-dummy variables, we use different tests to compare each kind of variable between classes and periods. For the dummy variables set we used the Binomial Test (B-Test) conover98 ; hollander13 , and for the non-dummy variables set we use the Kolmogorov-Smirnov Test (KS-Test) smirnov1939estimate . The specific steps we follow to do this were:
We separate the data corresponding to the two different government periods. The 1st period covers from 2013 to 2018, the 2nd period from 2019 to 2020.
For each contract class we extract the data for each variable over all the contracts belonging to the class in each period.
We compare classes by pairs in the same period performing the two-sample B-Test or the two sample KS-Test for each variable666To compute the B-Test we follow the standard procedure to compute the and the given in conover98 ; hollander13 ; oakley21 and for KS-Test we use the R function ks.test from the dgof package R ; dgof ..
We consider that there are significant statistical differences between contract classes in those variables for which:
The B-Test results in a p-value and where the difference between fractions of the dummy variables is .
The KS-Test results in a statistic and a p-value .
With these values we insure that at least 10% of the contracts of one class present, in these variables, a different behaviour from the contracts of the other class.
The results of these comparisons are shown in sub-section 3.2.
2.2.2 Measuring differences between government periods
To measure the differences between government periods we take a slightly different path than above, since there is a natural variability in the distribution function within a period, and we seek to detect differences beyond this variability. Thus we perform the comparison as follows:
We separate the data of each government period by year as before.
For each contract class we compute:
For each dummy variable: the fraction of contracts in which the variable is present, over all the contracts belonging to the class in every year.
Then, for non-dummy variables, we compute the confidence interval (CI) at 99%888We considered this value for the confidence interval because of the small amount of data we have to compute it. For the computation of the CI we use the CI R function from the Rmisc package R ; Rmisc for each distribution, using the data from each year of the 1st period. For dummy variables we use a boxplot of the data from 2013-2018 for comparison999To generate the boxplot we use the R function boxplot from the Stats package R ..
Now, using the data from the contracts under the new government, those dummy variables whose fractions are outside from the minimum and maximum ranges of the boxplot for the corresponding data from the first period, and those non-dummy variables with at least 25% of their cumulative distribution curve lies outside of the CI of the corresponding data of the first period, are considered to have significant statistical differences beyond the natural variability of the distributions. Conversely, those dummy variables whose fractions are inside the corresponding boxplots’ ranges, or the non-dummy variables for which at least 75% of their cumulative distribution curves are within the corresponding CI, will be considered statistically equivalent, meaning that the behavior corresponding to this variable is similar between periods.
This method provides a qualitative tool to identify the differences (and similarities) of the different contract classes between periods,
At the same time, these comparisons may provide a context to check whether corruption is, or is not, being eradicated as promised by the new government amlocorrupcion ; obrador2018salida ; obrador2018new .
|Federal Budget (FB)||3.10E11||3.36E11||2.96E11||2.55E11||2.59E11||2.75E11||3.03E11||2.83E11|
|Total Spending (TS)||3.97E10||5.42E10||4.12E10||3.07E10||4.10E10||2.63E10||1.46E10||1.77E10|
|Total Contracts (TC)||173,253||190,995||215,260||225,955||224,723||188,826||182,483||138,821|
|TC by SB||117,750||128,896||157,395||169,015||173,021||144,435||146,219||113,610|
|TS on SB||1.11E10||1.20E10||1.16E10||8.92E9||1.04E10||8.92E9||6.57E9||7.87E9|
|%TC by SB||67.96||67.48||73.11||74.80||76.99||76.49||80.12||81.79|
|%TS on SB||28.06||22.27||28.20||29.08||25.50||33.88||44.92||44.37|
|TC by SB||178||1,925||115,647||209||2.397||126,290||331||2,810||154,254||271||3,185||165,559|
|TS on SB||1.23E7||1.49E9||9.65E9||1.17E7||7.20E8||1.13E10||1.12E7||8.63E8||1.07E10||6.45E6||9.07E8||8.01E9|
|% by SB||53.61||56.13||68.23||41.38||62.50||67.65||61.07||65.57||73.30||62.58||68.79||75.95|
|% on SB||19.63||34.23||27.31||8.92||17.61||22.69||17.36||27.59||28.27||13.43||34.89||28.56|
|TC by SB||223||4,680||168,118||145||2,758||141,532||31||2,831||143,357||7||1,155||112,448|
|TS on SB||6.98E6||1.07E9||9.36E19||6.25E6||1.02E9||7.89E9||2.15E6||4.16E8||6.15E9||3.64E5||2.30E8||7.64E9|
|% by SB||71.01||77.75||76.98||66.82||71.78||76.60||56.36||77.88||80.18||53.85||81.51||81.80|
|% on SB||15.35||36.96||24.64||37.63||35.17||33.72||38.15||31.06||46.32||8.68||72.85||43.86|
3.1 Context for comparison
To make a fair comparison between government periods and between different contract classes, we present how many resources were spent in each class, as well
as how many contracts belonged to each class per year. Table 3 shows the
approved federal budget (FB) per year fb13 ; fb14 ; fb15 ; fb16 ; fb17 ; fb18 ; fb19 ; fb20 (in USD), the Total Spending (TS) on public procurement (in USD), the Total Contracts (TC) made in each year, and the ratio TS/FB. We observe that the ratio TS/FB varies from 0.10 to 0.16 in the 1st government period (2013-2018, Fig. 1 - Top, blue dashed line, green circles), but after the
change of government, the ratio fell to 0.05 and 0.06 for 2019 and 2020 (Fig. 1 - Top, blue dashed line, purple circles),
i.e., even when the FB increased, the spending in public procurement fell by approximately a
half. Fig. 1 - Top (orange dashed line) shows that the number of contracts also fell
at the end of the 1st period, then continued to fall at the beginning of the 2nd period, and
remained low during the second year of 2nd period.
Considering these data, we compared the number of contracts and the amount of money
that ended in each contract class for every year (Table 4 - Rows 2 and 3, and Fig. 1- Center). It is noticeable that the number of contracts in the EFOS class in 2019 and 2020, and the amount of money spent on them, fell by an order of magnitude. In contrast, the number of contracts and the total spending in contracts in the PCS class remained roughly in the same scale over several years, and only suffered a significant decrease in 2020. For contracts in the NC class, the numbers are similar between periods.
We normalized these absolute numbers by TC and TS to compare the percentage of the Total Contracts (%TC) and of the Total Spending (%TI) in each contract class (Table 4 - Rows 4 and 5, and Fig. 1 - Bottom). We see that the NC class represents approximately 90% of the Total Spending and Total Contracts for all years. Something similar occurs for contracts in the PCS class: there were no major changes between years for %TC and %TS until 2020, where %TC decayed roughly by half to 1.02% and %TS fell to 1.78%. However, for the contracts in the EFOS class, there was a large decay in the fraction of contracts assigned, and a corresponding decrease in resources spent when the government changed.
As mentioned above, single-bidder contracts have been shown to be effective identifiers of corruption risk wachs2020corruption . In this regards, Table 5 and Fig. 2 - Top show the total number of contracts by single-bidder, the total spending made on them, and the percentage of these quantities normalized respect to TC and TS for each year. Table 6 and Fig. 2 - Center and Bottom show the total number of contracts and spending assigned to single-bidders in each contract class, and the percentage relative to TC and TS in each class (%TC and %TS). We can observe that even when the absolute number of total contracts and spending on single-bidders decayed in the government turnover (Fig. 2 - Top, blue and orange dashed lines), the percentage of contracts and spending made on single-bidder increased after the government changed (Fig. 2 - Top, blue and orange dashed lines). This means that even when the new government made less contracts by single-bidder and spent less money on them, these represent more than 80% of the total contracts and more than 40% of the total spending on all contracts. Analysing the single-bidder contracts and spending per contract class, we observe that even when the absolute numbers show a decay between periods (Fig. 2 - Center), the relative numbers either remain at the same values or even increase, specially for contracts in the PCS and NC classes (Fig. 2 - Bottom). It should be stressed that single bidder contracts are not, in themselves, illegal. However, our results do imply that this practice is more widespread in the new government, and thus, represents a relatively larger risk of corruption than in the previous government. In what follows, we must keep in mind these absolute and relative data to fairly size the differences and similarities in contract behaviors that we discuss later.
3.2 Comparison between contract classes
To verify whether the three different contract classes have intrinsic, statistically significant, procedural differences between them, we computed a B-Test (for the set of dummy variables) and a KS-Test (for the non-dummy variables) comparing by pairs each variable that describes the contracts. This comparison was made between all the contract classes in the same government period.
First we compared the EFOS class vs. the PCS class. For the 1st period, Table SI and Fig. S1 of Supplemental Material (SM) show that there were 21 variables for which the statistical tests showed significant differences. These variables were of type i) and ii). For example, for the variable “large supplier” (S.NOM, which is a variable of type i)), we obtained that almost 40% of the contracts in the PCS class were for large companies, while for contracts in the EFOS class, less than 5% of the contracts were carried out with this kind of supplier (Table SI). This indicates that, in this period, contracts in the EFOS class were mostly done with micro, small, or medium companies101010 To test if this difference in the type of companies contracted was beyond the null hypothesis of randomly choosing a contractor, we analyzed the distribution of each company size in each class, finding that the methods to pick a supplier indeed favored one particular company size for each different class.
To test if this difference in the type of companies contracted was beyond the null hypothesis of randomly choosing a contractor, we analyzed the distribution of each company size in each class, finding that the methods to pick a supplier indeed favored one particular company size for each different class.. Fig. 3 show a subset of those variables for which contracts carried out with labeled as EFOS and PCS were significantly different in the 1st government period (2013-2018). For the variable called Spending (which is also type i)), we found that 80% of the contracts in the EFOS class (red circles solid line) were for more than 5K USD, while only 60% of the contracts in PCS class (blue triangles dashed line) exceeded this amount (Fig. 3 - Top). This is interesting because we can also see in Fig. 3 - Bottom that companies identified as EFOS obtain a maximum of 20 contracts with the same buyer (T.Cont, which is type ii)), while companies labeled as PCS obtained up to 100 contracts with a same buyer. These two quantities — Spending and the Total number of Contracts (T.Cont) — tell us that even when most companies labeled as EFOS obtained contracts for more money than most of the PCS companies, PCS were more frequently contracted by the same buyer than EFOS. It is also important to notice that in this comparison, the fraction of single-bidder contracts awarded by a buyer to a supplier (RAD, also a type ii variable) - wachs2020corruption ) also showed significant differences. In Fig. 3 - Center Right we can see that more than 50% of EFOS exceed the threshold of RAD=0.5, while only 40% of PCS do so. Finally, Fig. 3 - Center Left shows us that the 1st period government made contracts with EFOS mainly during the 2nd half of the year (BeginningWeek type i)), while companies labeled as PCS were contracted mostly during the 1st half of the year.
For the 2nd government period, the comparison EFOS vs. PCS showed 13 variables with significant differences, all of them of type i) (Table SII and Fig. S2 of SM). Interestingly, 12 of these variables were also present in the set of variables that displayed differences in the 1st government period. Variables such as S.NOM and Spending appear with similar behavior as in the 1st period. Remarkably, the contracts’ duration variable (EBWeeks, which is a type i) variable) showed that 90% of the contracts in the EFOS class lasted more than three weeks, against only 60% for the contracts in PCS class. However, PCS contracts reached duration times of up to two years ( 0.1%), versus one year for the longest EFOS contracts (Fig. S2 of SM). The remaining variables with significant differences are related to procedure type (PT), procedure character (PC), or contract type (CT), all type i) variables. For example, 90% of the contracts in the PCS class were for acquisitions, while only 40% of EFOS were this type of contract (CT.ADQ) - Table S2 of SM). Also, the distribution of the variable for procedure type by single-bidder (PT.AD) shows that 80% of the PCS contracts were assigned by this procedure, while this happened for only 60% of the EFOS contracts (Table S2 of SM).
Regarding the comparison between EFOS and NC contracts, 14 variables presented significant differences for the 1st period, these variables were of all three types (Table SIII and Fig. S3 of SM). It is noticeable that the two variables that showed the most prominent differences were those that characterize buyer’s features (type iii)); the maximum number of contracts awarded by a buyer to a supplier (T.Cont.Max), and the maximum amount of money spent by a buyer in contracts with a supplier( T.Spending.Max). Variable T.Cont.Max shows that approximately 80% of the buyers that contracted EFOS were those whose contract activity was greater than five contracts to a single supplier per year (EFOS or not), while only 30% of the buyers that contracted NC companies had this characteristic (Fig. S3 of SM). For T.Spending.Max we have that 70% of buyers that contracted EFOS companies had spending with a single company for more than 950K USD, against 25% of buyers that contracted NC companies (Fig. S3 of SM). For the 2nd period the statistical tests gave 15 variables (again of the three types) with significant differences (Table SIV and Fig. S4 of SM). Here, also variables of type iii) (T.Cont.Max and T.Spending.Max) were those that showed the most prominent differences, the behavior is similar to the 1st government period: The T.Cont.Max variable shows that 85% of the buyers that contracted EFOS had activity of more than five contracts per year with a single company, against 50% of the buyers that contracted NC companies with more than five contracts per year with a single supplier. The T.Spending.Max variable shows that more than 80% of the buyers in contracts in the EFOS class spent more than 1.75M USD per year with the same company, against only 40% of the buyers of contracts in the NC class that spent that amount with a single supplier.
The PCS vs. NC contract classes’ comparison for the 1st government period (Table SV and Fig. S5 of SM) showed significant differences in 14 variables, again distributed among the variable three types. For example, the variable for national procedure character (PC.N, a type i) variable) shows that 90% of NC contracts were made under national regulations, i.e. the contracting protocols followed Mexican laws, while this procedure was followed by only 50% of PCS contracts. The variable for large suppliers (S.NOM) shows that 45% of contracts in the PCS class were made with large companies, while only 10% of the NC contracts were won by this kind of supplier.
The variables for buyers’ features (T.Cont.Max and T.Spending.Max) presented differences similar to those discussed before: almost 75% of the buyers that contracted PCS had activity of more than 5 contracts with the same supplier, while only 25% of the buyers that contracted NC companies had this contract activity. On the other hand, 70% of the buyers of PCS contracts spent more than 900K USD with the same company, while only 30% of the buyers of NC contracts had this amount of spending with the same supplier. For the 2nd government period, the KS-Test showed 19 variables with significant differences, again within the three variable types (Table SVI and Fig. S6 of SM). For example, the variable for large suppliers (S.NOM) has a similar behaviour as in the 1st government period: 63% of the contracts in the PCS class were for large companies, against only 14% of the NC contracts were by this kind of supplier. Finally, Fig. S6 of SM shows that 40% of the PCS contracts lasted more than 20 weeks (EBWeeks), while only 30% of the NC contracts had this duration, the PCS contracts tended to last more than the contracts in the NC class.
Thus, we see that our three contract classes do present significant differences when compared among each other in the same government period, and that these differences are similar between periods. These results justify the separation of the contracts in these three classes, and also support our hypothesis that the corruption that occurs through EFOS and PCS have different procedural patterns. At the same time, these two contract classes do have differences with those contracts that we cannot label as corrupt (NC). The next step now is to identify whether each class presents differences between government periods.
3.3 Comparison between government periods
As mentioned before, to identify differences and similarities in public procurement between government periods, we compared the variables that define the contracts either by computing the confidence interval (CI) generated by the years of the 1st period (2013-2018) and verifying whether or not the curves corresponding to the 2nd period (2019-2020) lie within the CI, or, for the dummy variables, chequing if the results for the second period fall inside the box plot of the data of the first period. Since the main goal is to try to determine whether the government transition that occurred in México at the end of 2018 brought a change in the levels of corruption and corruption schemes, we compared the data for each period in the three different contract classes. Our analysis of the data sets produced the following results.
3.3.1 Main differences between both periods
Fig. 4 shows the most significant differences between both periods for the three contract classes: EFOS, PCS, and NC. First, we observe that in the 1st period, 20% (on average) of the suppliers classified as EFOS were micro-companies (Fig.4 - Top Left). In contrast, in the 2nd period, this fraction grew to 50% in 2019 and to 100% in 2020. This signals an important change in the way interactions with EFOS were carried out between both periods. Another important difference in this class was the contract duration (Fig.4 - Bottom Left). In the 1st period, only 20% (on average) of the contracts with companies labeled as EFOS lasted more than ten weeks, while for the 2nd period, 40% of these contracts had a duration of more than ten weeks in 2019, and in 2020 this fraction grew to almost 90%. Even when the government of 2nd period invested less money on EFOS, and granted them fewer contracts (Fig. 1), the majority of these contracts were of long duration. Fig. S7 of SM shows the remaining variables in which there were significant differences between periods. For example, we notice that the new government tended to contract EFOS at the beginning of the year (BeginningWeek), while the previous government did so mostly during the second half of the year. Also, in the 1st period, 40% of contracts in the EFOS class were acquisitions (CT.ADQ), this fraction grew to almost 60% in 2019, and then fell to 20% in 2020.
For PCS class, we observe that in the 1st government period, the PCS contracts were mostly (60%) with small and medium companies (Fig.4 - Top Center). In contrast, in the 2nd government period, the majority of these contracts passed to large suppliers. On the other hand, for the 1st period only 55% of the relationships buyer-supplier in PCS class exceeded the threshold of 0.5 in the fraction of single-bidder contracts (Fig. 4 - Bottom Center). In contrast, in 2019, this fraction grew to 60%, and in 2020, it reached more than 75%.
Fig. S8 of the SM shows the remaining variables for which PCS presented significant differences between periods. For example, with the new government, PCS contracting increased in the middle of the year; i.e. the 2nd period government had a slightly higher tendency to contract PCS between weeks 20 and 40 than the 1st period government (BeginningWeek).
Next, for the NC class, we notice that the 2nd period had an increase of 15% in the number of contracts made under the rules and regulations of the North American Free Trade Agreement (NAFTA) (Fig.4 - Top Right). On the other hand, the variable for single-bidder procedure type (PT.AD) shows that there was also a slight increase in the percentage of contracts won through this kind of procedural contract from 75% to 80% (Fig. S9 SM). We emphasize that even when NC contracts have not been labeled as corrupt, we do find changes in variables, specifically the fraction of single-bidder contracts awarded by a buyer to a supplier (RAD), that are associated with an increase in the risk of corruption. Fig.4 - Bottom Right shows that the RAD variable had a change between government periods. In wachs2020corruption it was proposed that a RAD0.5 was a red flag for corruption. Considering this, we notice that the percentage of the relationships buyer-supplier, which had a high risk of being corrupt, increased from 70% to 80% after the government turnover. The IMCO proposed in IMCOmapeando that a Fav0.9 is also a strong red flag for corruption. In Fig. S9 of SM, we can observe that while the number of relationships between buyers and suppliers with high values of Fav decreased markedly with the change of government, the difference in the fraction of those exceeding 0.9 is minimal.
3.3.2 Main similarities between both periods
Having identified the main differences between the contract classes in both government periods, it may also be helpful to identify the similarities, i.e., identify those variables that did not suffer significant changes due to government transition. This will give an idea of which aspects of corruption may have remained. Besides, making this analysis in the NC class will provide a hint as to whether the risk factors for corruption changed or not.
Fig. 5 shows the main similarities between governments for the three contract classes. We notice, for example, that for the EFOS class, the percentage of contracts assigned to single-bidders (PT.AD) did not suffer significant changes between periods, staying within the range of 50% to 60% (Fig. 5 Top Left). Also the total number of single-bidder contracts assigned to a supplier by a buyer (T.AD, a type ii) variable) presents similarities between periods: only 20% of the relationships buyer-supplier had more than one contract as single-bidder (Fig. 5 Bottom Left). There are also similarities in the percentage of contracts for leases (CT.AR, Fig. S10 of SM).
For the PCS class, we found similarities in the Spending variable, for which 80% of the contracts were for less than 500K USD in both periods (Fig. 5 - Top Center). It is also noticeable that most of the PCS received less than ten contracts per active week (CW) from a single buyer and received less than 1M USD per active week (SPW) (Fig. 5 - Bottom Center and Fig. S11 of SM), but there are, in both periods, companies labeled as PCS that reached up to 30 contracts per active week and more than 2M USD per active week.
For the NC class, we found similarities in the procedure character, in which 7-11% of the contracts were made under international rules for both periods (Fig. 5 - Top Right). We can observe that almost all buyers contracted a company in less than ten different weeks of the year (ActiveWeeks); these companies obtained less than ten contracts per active week (CPW) and less than 1M USD per active week (SPW) (Fig. 5 Bottom Right and Fig. S12 of SM). For the rest of the companies, that obtained more contracts and more money per active week, the behavior between governments was the same, which implies that these indicators, which are also red flags for increased risk of corruption IMCOmapeando , did not change with the government transition.
3.3.3 Mixed cases
We also found some variables which laid inside the corresponding CI or box plot one year of the 2nd period, and the other year they laid outside. We call this set of variables mixed cases. Fig. 6 shows examples of these variables for each class. Contracts in the EFOS class presented this particular behavior in some variables of type i), such as the fraction of contracts carried out with “medium” size companies (S.MED) (Fig. 6 - Top Left) which, in 2019 was similar to the 1st period (the value lies inside of the boxplot). However, in 2020 this fraction falls below the minimum value of 1st period. The fraction of single-bidder contracts awarded by a buyer to a supplier (RAD) is also a mixed case in the EFOS class (Fig. 6 - Bottom Left). Here the CDF of 2019 lies inside of CI, actually very near the mean line (orange dashed line), but the CDF for 2020 is out of the CI. During the 1st period 55% of the relationships buyer-supplier exceed the threshold of RAD 0.5 single-bidder contract fraction, in 2019 this percentage was quite similar (50%), but in 2020 it fell to 20%. The remaining mixed case variables for the EFOS class are in Fig. S13 of SM.
For the contracts in the PCS class, there were 14 variables with mixed behavior. For example, (Fig. 6 - Top Center) shows the fraction of PCS contracts made under rules and regulations specified in the North American Free Trade Agreement (NAFTA), we can observe that in 2019 this fraction lies beyond the 75% percentile of the 1st period data, while it diminished its value to the 3th interquartile during 2020.
For the favoritism of a buyer for a supplier (Fav) (Fig. 6 - Bottom Center) we can notice that only a few relationships buyer-supplier exceeded the threshold of Fav0.9 in both periods, but even when nearly 90% of the relationships buyer-supplier for PCS suppliers had low favoritism, in 2020 the percentage of relationships that had Fav0.5 increased. For the maximum number of contracts awarded by a buyer to a supplier (T.Cont.Max) (Fig. S14) we observe that, in the 1st period, 85% (on average) of the buyers who contracted PCS made less than 20 contracts with the same company, this value decreased to 80% in 2019 but stayed very close to the CI. However, in 2020 the percentage of the buyers with less than 20 contracts with a single company fell to 70%, significantly outside of the CI and showing a quite different behaviour to that presented in 1st period and 2019. The remaining mixed case variables for the PCS class are shown in Fig. S14 of SM.
Finally, the NC class presented only two variables with mixed behavior, both of type i). One related to the fraction of contracts made for leases (CT.AR, Fig. 6 - Top Right), and other related to the week in which the contracts ended (EndingWeek, Fig. 6 - Bottom Right). In both cases 2020 was the atypical year.
México lived a historical government turnover in 2018 when, for the first time, a leftist candidate won the presidency, being the most voted candidate in México’s democratic history. The new government appeared to represent a complete change of regime, credibly promising to break with the widespread corrupt practices of previous governments. In this work we present a statistical framework to identify to what extent public procurement practices suffered significant changes due to the government transition, focusing on those contracts related to companies labeled due to having incurred in some kind of corrupt behavior. To do this, we classified each contract in one of the three different contract classes; EFOS or PCS if it had been carried out with a company identified as incurring in corrupt practices by the SAT, or NC if it had not been identified as such. Once classified, we characterize each contract class by a set of 33 variables. Each variable gives information about: the contract itself (i.e. the contract type, the company’s size, or the amount spent, etc); the relationship between buyer and supplier (i.e. the total number of contracts between the supplier with the same buyer, the total amount spent by the buyer with the same supplier, or the degree of favoritism); or the buyer’s features (for example the maximum number of contracts given by a buyer to a company in a year).
Before discussing the detailed comparison between the two governments, a first important difference between them, concerns the number of presumably corrupt contracts per year carried out during each government, as well as the total amount of resources spent in these contracts. In the first year of the new government, the number of EFOS contracts fell by a factor of four from the last year of the previous government (or a factor of eight with respect to the average of the six years of the previous government), and another factor of four, to a total of only 13 cases, in the second year. This reduction in the number of EFOS contracts represented a reduction by a factor of three with respect to the previous year (or a factor of approximately 11 with respect to the average of the previous government) in the amount of resources spent in this kind of contracts.
Also, and representing many more resources, the number of contracts carried out with companies identified as PCS during the first year of the new government was comparable to those of the previous government, however, the amount spent on these contracts during the first year of the new government
fell by a factor of two with respect to the last year of the previous government. In the second year, the number of contracts fell by a factor larger than 2, and the resources spent fell by another factor of 4. Thus, the data shows that there has been a significant reduction in both the number of corrupt contracts, and the corresponding resources spent in such contracts. Whether this reduction is due to an effective crackdown on corruption, a consequence of the austerity program undertaken by the new government, or some other reason, is a question the data cannot answer.
However, our purpose in this work was to try to go beyond the mere analysis of the total amounts of resources spent in contracts that are suspect of being corrupt. We attempt to establish whether the practices and warning flags regarding suspect contracts actually changed from one government period to the other. To do so, using the variables characterizing each contract, we began by verifying that each contract class presented statistically significant differences between them in the same government period. Thus, we can assign a rough statistical profile to each class of contract in each government period. Next, we analyze whether, as a consequence of the government transition, these variables suffered changes within each class.
Regarding contracts with EFOS, we identified that the new government has tended to favor micro-suppliers more than the older government. Companies this size obtained only 20% of EFOS contracts (on average) in the 1st goverenment period. This percentage grew to 50% in 2019 and to almost 100% in 2020. Also, EFOS contracts duration suffered a significant change; while in the 1st period only 20% of the contracts with EFOS lasted more than ten weeks, in 2019 this fraction grew to 40%, and in 2020 reached 90%. Even when there were significant differences in some variables, there were also similarities between periods for this class of contracts. For example, in the first government period, the percentage of the contracts assigned to single-bidders was between 55%-65%, while for the first two years of the 2nd period the percentages were 55% and 57% respectively. Also, most of these contracts ( 95%) were for less than 2M USD in both periods. This implies that the similarity in this case is due to the “cheaper” contracts.
The picture that emerges is that there appears to be been a strong progressive reduction in corruption related to EFOS contracts with the new government, with only a few instances detected in 2020. However, as we mentioned at the beginning of this work, the lists of corrupt companies may change, and there may still be undetected EFOS that are currently under investigation. The cases that were identified were carried out increasingly with micro-companies. One possible reason for this fact is that contracts with micro-companies may be harder to detect or represent smaller expenditures. It is interesting to note that the percentage of contracts in the EFOS class assigned to single bidders did not change from one government to the other, remaining between 50 and 60%. Thus, if we assume that a buyer that assigns a contract to a single bidder EFOS company is in collusion with that company, these results suggest that for both government periods, over half of these contracts may not be the result of fraud from the contracted EFOS companies, but rather, may have been carried out by corrupt officials in collusion with the companies.
In regards to contracts in the PCS class, these showed differences in the kind of suppliers that were favored, exhibiting a relative increase in the fraction of large companies that won contracts. In the 1st government period, only 40% of the PCS contracts were won by these companies, while in the 2nd period, the percentage reached 65%. Moreover, the fraction of single-bidders in this class also increased from one government to another. In the first government period, only 45% of the relationships buyer-supplier exceeded the RAD 0.5 (i.e., more than half of the contracts in these relationships were by single-bidder), while in the new government this fraction increased to 60% in 2019 and 75% in 2020.
On the other hand, it is also noteworthy that the favoritism in buyer-supplier relationships with companies labeled as PCS, as defined in this work, did not present significant changes between periods. This suggests that even if the PCS suppliers may have changed, the new suppliers have been favored in the same proportion as in the previous government, a similar phenomena as the Compadres’ change found by the IMCO IMCOcompadres .
The NC class presented differences mainly in the contract procedures and the in the fraction of single-bidder contracts. In the 2nd government period, there was an increase of 20% of contracts made under NAFTA procedures compared to the 1st period. There was also an increase in the RAD variable, suggesting that the strategy for contracting in the new government favored single-bidders. There is no legal evidence of corruption for companies in the NC class, however, this increase in the percentage of buyer-supplier relationships (RAD), that exceedes the 0.5 threshold, suggests a risk of undetected corruption in the new government.
Our framework to analyze a set of contracts through certain characteristic variables allowed us to identify specific changes and similarities between contract classes and between government periods. Most of these variables, such as RAD (fraction of single-bidder contracts), Fav (favoritism), CPW (contracts per active week) or SPW (spending per active week), were inspired by previously proposed risk factors of corruption. These variables can help red flag those contracts or companies that exceed a certain threshold. For example, a buyer-supplier relation should be red flagged as risky if RAD0.5 wachs2020corruption , or if Fav0.9 IMCOmapeando . Our study shows that for the EFOS and PCS classes, the risk factor Fav had little accuracy since it identified less than 5% of companies labeled as having participated in corrupt activieties. Something similar occurs for the variables CPW and SPW, since most of the relationships between buyers and companies labeled as EFOS or PCS, had only 2-5 active weeks per year, two contracts per active week, and less than 500K USD per active week. Thus we were unable to identify the corruption scheme in which a buyer assigns many small contracts in a short period of time to the same supplier IMCOmapeando .
While this failure may be a consequence of our approximation, since we do not have access to all the necessary data to compute these risk factors precisely as proposed in wachs2020corruption ; IMCOmapeando , it may also suggest an important limitation of the effectiveness of these variables to predict corruption. On the other hand, the risk factor RAD was accurate in identifying (on average) more than 50% of the companies labeled as EFOS and PCS in both government periods.
Also, even when this risk factor decreased in the EFOS class for 2020, it increased by 15-30% in the PCS class, and by 10% in the NC class during the second government period. This suggests, as we mentioned above, that there is a relative increase in practices that carry a high risk of corruption, or even in undetected corruption in the new government.
AFC thanks PostDoctoral Scholarship DGAPA-UNAM for financial support. AFC thanks to D Cervantes-Filoteo and JR Nicolás-Carlock for fruitful discussion.
- (1) P. Mauro, “Corruption and growth,” The quarterly journal of economics, vol. 110, no. 3, pp. 681–712, 1995.
- (2) Z. Hessami, “Political corruption, public procurement, and budget composition: Theory and evidence from oecd countries,” European Journal of political economy, vol. 34, pp. 372–389, 2014.
- (3) D. Stockemer, B. LaMontagne, and L. Scruggs, “Bribes and ballots: The impact of corruption on voter turnout in democracies,” International political science review, vol. 34, no. 1, pp. 74–90, 2013.
- (4) S. Gupta, H. Davoodi, and R. Alonso-Terme, “Does corruption affect income inequality and poverty?,” Economics of governance, vol. 3, no. 1, pp. 23–45, 2002.
- (5) D. C. North, J. J. Wallis, B. R. Weingast, et al., Violence and social orders: A conceptual framework for interpreting recorded human history. Cambridge University Press, 2009.
- (6) T. S. Aidt, “Rent seeking and the economics of corruption,” Constitutional Political Economy, vol. 27, no. 2, pp. 142–157, 2016.
- (7) J. Wachs, T. Yasseri, B. Lengyel, and J. Kertész, “Social capital predicts corruption risk in towns,” Royal Society open science, vol. 6, no. 4, p. 182103, 2019.
- (8) M. Fazekas and J. Wachs, “Corruption and the network structure of public contracting markets across government change,” Politics and Governance, vol. 8, no. 2, pp. 153–166, 2020.
- (9) S. Baldi, A. Bottasso, M. Conti, and C. Piccardo, “To bid or not to bid: That is the question: Public procurement, project complexity and corruption,” European Journal of Political Economy, vol. 43, pp. 89–106, 2016.
- (10) G. OECD., Integrity in public procurement: Good practice from A to Z. Organization For Economic, Paris, 2007.
- (11) G. OECD., Government at a Glance. Organization For Economic, Paris, 2015.
- (12) J. C. Clingermayer, R. C. Feiock, and C. Stream, “Governmental uncertainty and leadership turnover: Influences on contracting and sector choice for local services,” State and Local Government Review, vol. 35, no. 3, pp. 150–160, 2003.
- (13) R. Broms, C. Dahlström, and M. Fazekas, “Political competition and public procurement outcomes,” Comparative Political Studies, vol. 52, no. 9, pp. 1259–1292, 2019.
- (14) E. Dávid-Barrett and M. Fazekas, “Grand corruption and government change: an analysis of partisan favoritism in public procurement,” European Journal on Criminal Policy and Research, pp. 1–20, 2019.
- (15) “IMCO - Instituto Mexicano para la Competitividad A.C..” https://imco.org.mx, 2021. Accessed: 2021-04-12.
- (16) IMCO, “Índice de riesgos de corrupción: El sistema mexicano de contrataciones públicas.” https://imco.org.mx/indice-riesgos-corrupcion-sistema-mexicano-contrataciones-publicas/, 2017. Accesed: 2021-04-12.
- (17) “Transparency International - Corruption Perception Index.” https://www.transparency.org/en/cpi/2020/index/nzl, 2013-2020. Accessed: 2021-04-12.
- (18) A. C. María, Anatomía de la Corrupción. México: CIDE Instituto, 2015.
- (19) L. Meyer, El poder vacío: El agotamiento de un régimen sin legitimidad. DEBATE, México, 2019.
- (20) “INE - Instituto Nacional Electoral.” https://computos2018.ine.mx/#/presidencia/nacional/1/1/1/1, 2018. Accessed: 2021-04-12.
- (21) A. Muñoz Armenta, M. Hernández García, G. Gómez Romo de Vivar, D. A. Mares Sánchez, C. S. Muñoz Canto, I. R. Álvarez Olivas, M. Díaz Sandoval, A. Espejel Espinoza, V. H. Martínez González, G. Corona Armenta, et al., El triunfo de la izquierda en las elecciones de 2018?‘ Ideología o pragmatismo? Universidad de Guanajuato, Grañén Porrúa, México, 2020.
- (22) B. Hanrahan and P. Aroch Fugellie, “Reflections on the transformation in mexico,” Journal of Latin American Cultural Studies, vol. 28, no. 1, pp. 113–137, 2019.
- (23) “CompraNet - Contratos Públicos.” https://www.gob.mx/compranet/documentos/datos-abiertos-250375, 2021. Accessed: 2021-03-06.
- (24) G.-H. Kim, S. Trimi, and J.-H. Chung, “Big-data applications in the government sector,” Communications of the ACM, vol. 57, no. 3, pp. 78–85, 2014.
W. J. Radermacher, “Official statistics in the era of big data opportunities
International Journal of Data Science and Analytics, vol. 6, no. 3, pp. 225–231, 2018.
- (26) M. Fazekas, I. J. Tóth, and L. P. King, “Anatomy of grand corruption: A composite corruption risk index based on objective data,” Corruption Research Center Budapest Working Papers No. CRCB-WP/2013, vol. 2, 2013.
- (27) M. Fazekas, I. J. Tóth, and L. P. King, “Corruption manual for beginners:’corruption techniques’ in public procurement with examples from hungary,” Corruption Research Center Budapest Working Paper No. CRCB-WP/2013, vol. 1, 2013.
- (28) M. Fazekas, I. J. Tóth, and L. P. King, “An objective corruption risk index using public procurement data,” European Journal on Criminal Policy and Research, vol. 22, no. 3, pp. 369–397, 2016.
- (29) M. Fazekas, J. Skuhrovec, and J. Wachs, “Corruption, government turnover, and public contracting market structure–insights using network analysis and objective corruption proxies,” GTI Working Paper Series, 2017.
- (30) M. Fazekas, R. Ferrali, and J. Wachs, “Institutional quality, campaign contributions, and favouritism in us federal government contracting,” tech. rep., Working Paper series: GTI-WP/2018: 01. Government Transparency Institute, 2018.
- (31) J. Wachs, M. Fazekas, and J. Kertész, “Corruption risk in contracting markets: a network science perspective,” International Journal of Data Science and Analytics, pp. 1–16, 2020.
- (32) M. Klašnja, “Corruption and the incumbency disadvantage: Theory and evidence,” The Journal of Politics, vol. 77, no. 4, pp. 928–942, 2015.
- (33) N. Charron, C. Dahlström, M. Fazekas, and V. Lapuente, “Careers, connections, and corruption risks: Investigating the impact of bureaucratic meritocracy on public procurement processes,” The Journal of Politics, vol. 79, no. 1, pp. 89–104, 2017.
- (34) “Secretaría de Atención Tributaria - Empresa que Factura Operaciones Simuladas.” http://omawww.sat.gob.mx/cifras_sat/Paginas/datos/vinculo.html?page=ListCompleta69B.html, 2021. Accessed: 2021-03-06.
- (35) “Datos Abiertos - Proveedores y Contratistas Sancionados.” https://datos.gob.mx/busca/dataset/proveedores-y-contratistas-sancionados, 2021. Accessed: 2021-03-06.
- (36) “Datos Abiertos de México - Contratos Públicos.” https://datos.gob.mx/busca/dataset?q=Contratos+Ingresados+a+CompraNet&sort=score+desc%2C+metadata_modified+desc, 2021. Accessed: 2021-03-06.
- (37) “Secretaría de Economía - Clasificación de empresas.” http://www.2006-2012.economia.gob.mx/mexico-emprende/empresas, 2021. Accessed: 2021-05-24.
- (38) “Federal Reserve - Foreign Exchange Rates.” https://www.federalreserve.gov/releases/g5a/20140102/default.htm, 2013. Accessed: 2021-03-06.
- (39) “Federal Reserve - Foreign Exchange Rates.” https://www.federalreserve.gov/releases/g5a/20180111/, 2014-2017. Accessed: 2021-03-06.
- (40) “Federal Reserve - Foreign Exchange Rates.” https://www.federalreserve.gov/releases/g5a/current/, 2017-2020. Accessed: 2021-03-06.
- (41) IMCO, “Mapeando la corrupción.” https://mapeandolacorrupcion.mx/Anexo_Metodologico.pdf, 2018. Accesed: 2021-04-12.
- (42) W. J. Conover, Practical nonparametric statistics, vol. 350. John Wiley & Sons, 1998.
- (43) M. Hollander, D. A. Wolfe, and E. Chicken, Nonparametric statistical methods, vol. 751. John Wiley & Sons, 2013.
N. V. Smirnov, “Estimate of deviation between empirical distribution functions in two independent samples,”Bulletin Moscow University, vol. 2, no. 2, pp. 3–16, 1939.
- (45) J. Oakley, “Lecture notes in data science - mas113 part2: Data science.” http://www.jeremy-oakley.staff.shef.ac.uk/mas113/notes/index.html, February 2021. Accessed: 2021-06-30.
- (46) R Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2020.
- (47) T. A. Arnold and J. W. Emerson, “Nonparametric goodness-of-fit tests for discrete null distributions,” The R Journal, vol. 3, no. 2, pp. 34–39, 2011.
- (48) R. M. Hope, Rmisc: Rmisc: Ryan Miscellaneous, 2013. R package version 1.5.
- (49) “Proyecto alternativo de nación 2018-2024.” https://repositoriodocumental.ine.mx/xmlui/bitstream/handle/123456789/94367/CG2ex201712-22-rp-5-2-a2.pdf, 2017. Accessed: 2021-07-13.
- (50) A. M. L. Obrador, La Salida: Decadencia y renacimiento de México. Planeta, 2017.
- (51) A. M. L. Obrador, A new hope for Mexico. OR Books, LLC, 2018.
- (52) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2013.” https://www.dof.gob.mx/nota_detalle.php?codigo=5283490&fecha=27/12/2012, 2012. Accessed: 2021-03-06.
- (53) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2014.” https://www.dof.gob.mx/nota_detalle.php?codigo=5324132&fecha=03/12/2013, 2013. Accessed: 2021-03-06.
- (54) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2015.” http://www.dof.gob.mx/nota_detalle_popup.php?codigo=5374053, 2014. Accessed: 2021-03-06.
- (55) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2016.” http://dof.gob.mx/nota_detalle.php?codigo=5417699&fecha=27/11/2015, 2015. Accessed: 2021-03-06.
- (56) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2017.” https://www.dof.gob.mx/nota_detalle.php?codigo=5463184&fecha=30/11/2016, 2016. Accessed: 2021-03-06.
- (57) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2018.” http://www.dof.gob.mx/nota_detalle.php?codigo=5506080&fecha=29/11/2017, 2017. Accessed: 2021-03-06.
- (58) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2019.” http://dof.gob.mx/nota_detalle.php?codigo=5547479&fecha=28/12/2018, 2018. Accessed: 2021-03-06.
- (59) “Presupuesto de Egresos de la Federación para el Ejercicio Fiscal 2020.” https://www.dof.gob.mx/nota_detalle.php?codigo=5581629&fecha=11/12/2019, 2019. Accessed: 2021-03-06.