International roaming is an important feature of cellular networks, allowing subscribers to use their devices anywhere in the world as if at home. Under the IP Packet Exchange (IPX) model (ipx-gsm-whitepaper; ir34), MNOs contract the services of third party providers – the IPX Provider (IPX-P) – to offer their customers access to mobile services in any foreign country. No IPX-P on its own is able to provide connections on a global basis (e.g., single-handily interworking with all MNOs). Thus, IPX-Ps peer to other IPX-Ps (apnic17peering) to expand their geographical footprint. The resulting IPX Network, is an isolated network that bypases the public Internet (manynet), to ensure secure, SLA-compliant services, from video streaming and AR/VR to IoT verticals, such as connected cars.
Recent years have brought a rapid growth in the number of participants in the IPX ecosystem and the volume of traffic they exchanged. The growing number of international travelers, reaching 1.4 billion in 2018 (doi:10.18111/9789284421152) and the “flat-rating” or elimination of international roaming charges (lh_regulation; roaming_charge; sheehy:nerdwallet) has led to an exponential growth in roaming traffic, expected to increase 32-times by 2022. At the same time, users’ QoE expectations – when using VoIP or posting videos – has forced content and service providers to peer close to their users, wherever they may roam, thus adding to a growing interconnection ecosystem.
Despite its rapid growth and increased importance as the core of the mobile Internet, the IPX Network and its associated ecosystem (Figure 1) has received little to no attention by our community, due in part to its intrinsic opacity and separation from the public Internet. In this paper, we present the first characterization of the IPX ecosystem and a first-of-its-kind, detailed study of a large operational IPX Provider.
We contribute the first topology analysis of the IPX ecosystem. We analyze a private BGP routing table snapshot from an operational router that is part of the IPX Network to map the interconnection between IPX-Ps and Service Providers for the data roaming service (§ 3). We build the exhaustive list of 29 active IPX-Ps and detail their approach for peering using three major peering exchange points (AMS-IX Amsterdam, Equinix Ashburn and Equinix Singapore). 111IPX is not the same as IXP, though the acronyms are similar. IPX-Ps may rely on IXPs (such as AMS-IX) for peering with other IPX-Ps. We capture the breadth and full-mesh peering fabric of the IPX Network for data roaming, which enables inter-working between all MNOs currently active world-wide.
We present a first-of-its-kind detailed analysis of a large operational IPX-P system, and provide the insider view into the otherwise inaccessible IPX Network. Our study is based on real-world traffic records for its main service (radio signaling for data roaming (§ 4)) over a two-week period in December 2019. We showcase the operational IPX-P’s signaling and data roaming infrastructures (§ 4.2), with notable presence in Europe and the Americas. We study signaling traffic patterns (for different radio access technology) and data communications from over 22 million mobile devices roaming in the world.
2. The IP eXchange Model
In this section, we provide a detailed description of IPX and use data roaming, one of the main services offered, to illustrate the main IPX players and their interactions.
2.1. IPX Connectivity for SPs
At a high-level, the IPX ecosystem (Figure 1) includes Service Providers (SP) and networked IPX-Providers (IPX-Ps). IPX-Ps are third-party interconnection providers to SPs (e.g., MNOs, IoT providers). IPX-Ps peer with other IPX-Ps to extend their footprint worldwide.
While IP-based, the resulting IPX Network is a private network, separate from the public Internet, that meshes together the infrastructures of the IPX-Ps. It guarantees traffic separation between IPX services from the rest of the Internet. The IPX Network enables the transport of global roaming data between networks, with interoperability of different implementations and standards.
SPs require a single connection and agreement with one IPX-P in order to connect to the IPX Network, and interconnect with partner SPs world-wide.222Although direct interconnection between SPs through leased lines or Virtual Private Network (VPN) is possible, it is outside the scope of our analysis. For instance, to enable data roaming, two MNOs must each have an agreement with an IPX-P in order to interconnect. For redundancy, a SP could establish connections to more than one IPX-P. Depending on the footprint of the IPX-P’s infrastructure, SPs can use one or more Point of Presences of the IPX-P.
2.2. Study Case: Data Roaming over IPX
To establish roaming, roaming partner MNOs must have a functioning commercial agreement, implement their roaming technical solutions, establish inter-working and deploy their billing function. We use data roaming to illustrate the main IPX ecosystem players and their interactions.
In terms of business agreement solutions, the legacy option for MNOs is a standard bilateral agreement where the two parties involved define terms and conditions of their cooperation. These bilateral roaming agreements for roaming and inter-working are costly and generally of lower value today, something that has served as additional motivation for MNOs to adopt the IPX model.
Under the IPX model, operators connect to an IPX-P to gain access to many roaming partners world-wide, externalizing the inter-working establishment to the IPX-P offering the service. IPX-Ps are then peering with each-other to expand their international footprint through the IPX Network. This IPX hubbing solution does not preclude the existence of bilateral agreements between MNOs, which can be viewed as a complementary roaming model.
Once a commercial agreement has been created, the IPX-P sets up the technical roaming solution, including coordination over the signaling platform, and establishes the IPX-P interconnectivity. After MNOs establish roaming inter-working, they deploy the billing service, which is key to recovering roaming revenue. The roaming partners must each record the activity of roaming users in a given Visited Mobile Network Operator (VMNO). Then, by exchanging and comparing these records, the VMNO can claim revenue from the partner Home Mobile Network Operator (HMNO).
When a mobile device is at home, the subscriber’s traffic will take a short path inside the network to reach a suitable Packet Data Network Gateway (PGW) to the Internet. When inter-working exists between two MNOs, there are several network configurations the IPX Network supports to enable roaming. The IPX-P’s main function is to build the communication tunnel between the Serving Gateway (SGW) and the PGW, enabling traffic to flow to and from the roaming mobile device. The traffic of a roaming mobile device is directed to an egress PGW whose location depends on the roaming configuration. Different configurations for roaming over the IPX Network are available – home-routed roaming (HR), local breakout (LBO) and IPX hub breakout (IHBO). Prior work found that the default roaming configuration majority MNOs currently use in Europe is the HR roaming (mandalari2018experience). In the case of HR, the mobile device receives the IP address from its home MNO and the roaming traffic is then routed over a tunnel between the SGW in the VMNO and a PGW in the HMNO.
3. IPX Ecosystem Diversity
The core players of the IPX ecosystem are IPX-Ps and SPs. IPX-Ps provide the interconnection between SPs directly through their network or via peering with other IPX-Ps (see Figure 1). The resulting ecosystem has a layered topology: a core of tightly interconnected IPX-Ps in a full mesh (the IPX Network) and the edge of diverse SPs that interconnect through the IPX Network, either through a single IPX-P or multi-homed through multiple IPX-Ps. For this, SPs may use their own access network (e.g., fixed and mobile network operators) or use a local provider to connect to the PoP of the IPX-P (e.g., Application Service Providers (ASP)). SPs originate and/or terminate traffic for one or several services; they do not transport traffic.
3.1. IPX-Ps Interconnection and Peering
IPX-Ps establish interconnection either through private bilateral interconnections or through an Internet Exchange Point (IXP) (Figure 1). The benefits of peering are well known among Internet Service Providers and Content Delivery Networks, particularly when it comes to public peering via an IXP (ager2012anatomy). In recent years, we have seen more efforts to expand the peering culture to the mobile ecosystem (apnic17peering), by establishing new mobile peering infrastructure world-wide. This is (slowly) propagating to the mobile industry, and the interconnection of IPX-Ps via mobile peering at specific IXPs, which offer this service, is becoming commonplace. Currently, the two major IXPs offering the mobile peering service are AMS-IX and Equinix, with three locations overall (Amsterdam, Ashburn and Singapore).
IPX-Ps dynamically exchange routing information with other IPX-Ps using the BGP routing protocol. According to GSMA reccomendations (ir34), IPX-Ps should not act as a transit IPX-P (i.e., there can only be a maximum of two IPX-Ps between two partner SPs). Therefore, when an IPX-P has a customer SP who requires a connection to a customer SP of another IPX-P, the two IPX-Ps should peer, either through (direct) private peering or peering points. For example, in Figure 1, the path between MNO1 and MNO2 should only traverse two IPx-Ps (i.e., MNO1 -¿ IPX-P1 -¿ IPX-P2 -¿ MNO2). Also, IPX-P3 should never transit traffic for neither of its two peers, IPX-P1 and IPX-P2. To ensure this, network routes IPX-P3 receives either over private peering or over a peering point should not be re-advertised to other IPX-P peering partners. These recommendations result in a tightly interconnected IPX Network, with a theoretical diameter of two entities between any pair of SPs.
3.2. IPX Topology
Given the opaque nature of the IPX ecosystem, we cannot capture its characteristics from the public Internet, not by using public Internet routing data nor with active end-to-end measurements (e.g., traceroute). In order to shed light on the interconnection fabric between IPX-Ps and SPs in the IPX ecosystem, we analyze three different datasets: (i) routing dataset: a private BGP routing table snapshot for data roaming that one of the largest operational IPX-Ps provided us; (ii) peering dataset: the list of AMS-IX members that connect to the mobile peering services for data roaming service only, together with the full internal list of IPX-P peers from the operational IPX-P providing us the routing dataset; (iii) survey dataset: market surveys and reports (rocco; hot_telecom) from third parties. In the routing data, we capture one snapshot of the routing table on the 30th of January 2020, which provides one view of the IPX-P’s relationships for the data roaming service. We note that this information, though descriptive of the ecosystem, might be incomplete (i.e., there might be information not included here, but present in snapshots at other vantage points).
Specifically, the IPX-P routing table snapshot includes reachability information for a total of 10,418 IPv4 prefixes advertised by 59 different entities (neighbors), which include the peer IPX-Ps and the customer MNOs of the IPX-P we analyze. By checking the originating AS of the prefixes in the routing dataset, we find that within the IPX ecosystem there are a total of 824 different service providers for data roaming (i.e., MNOs, MVNOs or M2M platforms). This number is consistent with the total number of MNOs active world-wide that register with the GSMA. These are public IPv4 prefixes that IPX-Ps do not announce in the global BGP routing tables. Hence, they are not reachable from the public Internet. From Figure 1(a), we note that the median prefix length in the routing dataset is /29, and there is a large number of /32 prefixes that MNOs originate. These likely represent specific elements (e.g., Home Location Registry (HLR) or Mobility Management Entity (MME)) within the MNO infrastructure, which are involved in procedures for data roaming.
IPX Network: We merge and corroborate the information we extract from the above-mentioned datasets to build a list of IPX-Ps that currently form the IPX Network. In Annex B, we detail the full list of these providers and the methodology we found we compile this list. We specifically note that, in light of the growing popularity and worldwide footprint of IPX services, many heavyweight telecoms are participating in the IPX environment, leveraging their underlying extensive infrastructure.
We verify that these IPX-Ps appear in the routing dataset as active peers of the IPX-P providing us the routing table snapshot. The total number of entities advertising reachability information to the IPX-P is equal to 59 different ASes. Out of these, we separate the ones that advertise prefixes with an AS-Path length longer than one (i.e., they advertise their SPs). We check the overlap with the list of IPX-Ps we built, and find a set of 23 different ASes.333The remaining 36 ASes observed in the BGP routing table are SPs that are customers of the analyzed IPX-P.
IPX Interconnections: The set of 29 IPX-Ps we found must provide interconnection services to the over 800+ MNOs currently active world-wide (GSMA-MNOs). The number of PoPs indicates the number of locations where an IPX-P can cross-connect with SPs, giving insight into its world-wide geographical footprint. For example, large players such as Telia Sonera, TATA Communications, Orange, Vodafone, Telecom Italia, Telefonica or Telekom Austria offer an infrastructure with more than 100 PoPs world-wide each. The average number of PoPs among the 18 IPXs publicly disclosing this information is 116, hinting the breadth of the system we capture.
We further use the routing dataset in order to characterize the interconnection between IPX-Ps and SPs in the ecosystem. For the IPX-Ps present in the routing dataset, we verify the different number of SPs for which they advertise reachability information (Figure 2(a)).
We find that four major players within the IPX Network (namely, Syniverse, BICS, Orange and Comfone) together give services to a total of more than 600 MNOs (out of the total 800). Inversely, we also verify the popularity of multi-homing among SPs. In other words, we quantify the number of IPX-Ps that advertise reachability information for the same SP. Figure 2(b) shows that 80% of SPs are single-homed (i.e., they only connect to one IPX-P), while for the rest we observe up to seven different IPX-Ps. In particular, we note that multiple M2M platform providers use at least four different IPX-Ps, which is intuitive due to their reliance on roaming.
Finally, in order to characterize the interconnection between IPX-Ps, we analyze the AS-Path of the prefixes in the routing dataset. To comply with the recommendation of a fully connected IPX Network, there can be no more than two different IPX-Ps involved in the communication between two different SPs. Figure 1(b) shows the distribution of prefixes on AS-Path length. Note that we eliminated AS-Path prepending (used by IPX-Ps and SPs for traffic engineering).
We show that, indeed, majority prefixes advertised by peer IPX-Ps have a path length equal to two, confirming the tight interconnection required in the IPX Network. The paths longer than two ASes represent the result of MNOs working together with their parent networks (e.g., national MNOs connecting to their parent carrier) or third party network providers (e.g., for MVNOs, M2M platforms) to achieve a broader geographical footprint. Thus, sibling ASes that belong to the same organization appear in the same AS-Path.
We also observe that ASes in the IPX network use standard techniques for traffic engineering. In particular, we observe a heavy use of AS prepending. AS path prepending makes a route less preferred to receive traffic by making the AS path length for the route artificially long by repeating ASes in the AS path attribute. We observe 1,331 routes from 131 different origin ASes where prepending was used. We also observe extensive use of Multi Exit Discriminator (MED), a BGP attribute that serves to express preference between different links between two ASes. We observed 1,583 prefixes that contained the MED attribute.
4. A Large IPX Provider
We continue our analysis of the IPX ecosystem and zoom into the operations of one of the largest IPX-Ps in the ecosystem. The IPX-P we dissect is a Tier-1 Internet Service Provider operating one of the largest backbone networks world-wide. As part of its interconnectivity products, the carrier operates an IPX infrastructure that runs on top of its vast Multiprotocol Label Switching (MPLS) network.444An IPX-P requires access to an underlying backbone network. The IPX-P may own its own MPLS network or alternatively, it might lease capacity on MPLS networks on which they deployed the infrastructure needed to deliver and manage inter-operable cross-network services. The IPX-P infrastructure integrates more than 100 PoPs in 40+ countries with a particularly strong presence in America and Europe.
In terms of network connectivity, the IPX-P offers two types of interfaces, namely the IPX Access for clients (service providers) and the IPX Exchange for peering with other IPX-Ps. The main mobile peering points the IPX-P uses are those in Singapore, Ashburn and Amsterdam. By peering with other large Tier-1 carriers, the IPX-P extends its footprint world-wide to geographic regions where it does not own infrastructure (§ 3.2). The IPX-P serves clients in multiple countries in Europe (Germany, Spain, UK) and the Americas (including US, Mexico, Brazil, Argentina, Colombia, Peru, Chile, and Ecuador).
4.1. IPX-P Infrastructure and Monitoring
We now describe the IPX-P infrastructure, the monitoring methodology and the datasets we collected to characterize its operational system and services. We monitor the IPX-P infrastructure corresponding to the two main services – SCCP Global Signaling, LTE Diameter Exchange – for two weeks, from December 1st to December 14th 2019.
Overall, the total number of IMSIs we capture is of more than 22M active daily in 2G/3G and more than 2M active daily in 4G/LTE.555Note that there might be an overlap between these two sets. However, we aim to show here the load on the two different signaling infrastructures. They correspond to 215 home countries and 210 visited countries. The IPX-P’s customers are active within 19 countries and include MNOs, IoT/M2M connectivity providers and cloud service providers.
SCCP Global Signaling:
This service provides access to the IPX-P’s SS7 signaling network, satisfying the 2G/3G interconnection needs for international roaming of MNOs. The SCCP Signaling network of this particular IPX-P has a redundant configuration with four international Signaling Transfer Points located in North America (Miami, Puerto Rico) and Europe (Frankfurt, Madrid) as depicted in Figure 4.
To capture clients’ activity across this signaling platform, we monitor the Mobile Application Protocol (MAP) protocol, which supports end-user mobility and is used by devices to communicate with the major network elements, including the HLR, Visiting Location Registry (VLR) or the Mobile Switching Center (MSC). We collect traffic corresponding to the following procedures of each device belonging to one of the IPX-P’s clients (outbound roaming) or to foreign devices that connect to the network of one of the IPX-P’s clients (inbound roaming): i) location management (update location, update GPRS location, cancel location, purge mobile device); ii) authentication and security (send authentication information); iii) fault recovery.
LTE Diameter Exchange:
This service provides the Diameter signaling capabilities necessary to enable 4G roaming for customers. The infrastructure of this particular IPX-P includes four Diameter Routing Agents meant to forward Diameter messages and simplify interworking between different network elements. The LTE Diameter service integrates value added services, including Welcome SMS, Steering of Roaming or Sponsored Roaming.
To monitor the activity of the IPX-P’s customers across this platform, we monitor traffic across the geo-redundant signaling network with four DRAs located two in Europe (Frankfurt, Madrid) and two in North America (Miami, Boca Raton). The infrastructure is similar to the one in Fig. 4.
4.2. Signaling Traffic Trends
Figure 5 shows signaling activity of roaming mobile subscribers during the observation period of December 2019. We look at both MAP and Diameter signaling procedures. MAP is the most important application protocol in the Signaling System No. 7 (SS7) stack, and handles the roamers’ mobility between countries. Although this is still the most used protocol for mobile interconnection application messages, the more recent Diameter (rfc6733) signaling protocol has been growing with the adoption of LTE. Figure 4(a) focuses on the total signaling traffic and on the number of different mobile subscriber devices that generate this traffic.
We find that the number of devices using 2G/3G infrastructure (extracted from MAP traffic) is an order of magnitude higher than those using 4G infrastructure (based on the Diameter traffic). The volume of signaling traffic in the SCCP infrastructure is, correspondingly, more significant in terms of total volume than in the Diameter infrastructure. We also note the typical daily and weekly traffic patterns on mobile subscribers’ activities. For instance, December 1st was a Sunday, showing the expected decreasing trend in signaling traffic activity which can be seen again the following weekend (December 7-8th).
Each record in this dataset represents a signaling procedure that a network element triggers, corresponding to different standard routines. For instance, from the MAP interface we capture mobility management routines, including location management and authentication. Figure 4(b) shows the time series of signaling traffic broken down by type of signaling procedure, including Update Location (UL), Cancel Location (CL) and Send Authentication Information (SAI) messages. The latter, SAI, represents the highest fraction of MAP signaling traffic. Indeed, according to the GSM standard definition, the Serving GPRS Support Node (SGSN) in the visited network triggers the authentication of subscriber procedure upon IMSI attach, location update or before starting data communication, thus explaining the large volume of SAI messages.
Figure 4(c) shows the average number of records per IMSI calculated over all the IMSIs we observe in each one-hour interval (continuous line) during the observation period, as well as the standard deviation of the number of records per IMSI calculated over all the IMSIs active in the same one hour interval (shaded area). We observe both the MAP procedures for 2G/3G (red color) and the similar Diameter procedures for 4G/LTE (green color). While Diameter and MAP are different protocols, the underlying functional requirements (e.g., authenticating the user to set up a data communication) have many similarities in terms of the messages used for Diameter and the SS7 MAP protocol implementation. We note that the load in terms of average signaling records per IMSI is in the same order of magnitude (the continuous lines on the plot), regardless of the infrastructure the devices use; yet, there are significantly more messages for MAP, as Diameter is a more efficient protocol than MAP (rfc6733; rfc7075).
We further investigate the traffic load on the signaling infrastructure, both for the SCCP (with STPs in Frankfurt, Madrid, Miami and Puerto Rico) and the Diameter signaling infrastructure (using DRAs in Frankfurt, Madrid, Miami and Boca Raton). Figure 6 shows the aggregated signaling traffic (number of messages) per infrastructure point per day over the same observation period. We note that the redundant deployment of infrastructure points in each geographical area allows the IPX-P to load-balance the signaling traffic across them (e.g., for 4G between Frankfurt and Madrid in Europe and between Miami and Boca Raton in America). Signaling traffic in Europe through the signaling points in Frankfurt and Madrid is considerably higher than that flowing through the signaling points in America (Miami and Boca Raton/Puerto Rico). This is true for both the SCCP infrastructure (Fig. 5(a)) and the Diameter infrastructure (Fig. 5(b)). When comparing the two signaling services (i.e., SSCP and Diameter), we also note that the traffic volume in the SCCP signaling infrastructure is approximately twice larger than the traffic volume flowing through the Diameter signaling infrastructure. This proves the continued popularity of 2G/3G services in the region served by the IPX-P.
5. Related work
The growing demand for global, mobile broadband access and a shift to all-IP-based services (from broadband last mile to VoIP) have brought new impetus to the old idea of the IPX, first proposed by the GSMA in 2007 to replace the traditional, bilateral-agreement model for international roaming (ir34). Despite the continuous technical development by IPX-Ps and the related parties (US20140169286A1; US20150222554; US20120218924A1) there has been few academic works on the topic. Takaaki (takaaki:survey) provides an early survey of IPX and its technical requirements, but there have been no in-depth analysis of IPX since due, in part, to its closed nature. Our work presents the first in-depth analysis of the IPX Network and the associated ecosystem.
In this paper, we provided both a qualitative and quantitative description of the IPX ecosystem using information collected from the one of the largest IPX providers. We believe that understanding the IPX network is cornerstone to understand and evolve the mobile Internet, and that it will become more relevant as new services emerge. For example, relying on IPX services, novel technologies such as eSIM that allow remote provisioning of mobile devices permit Global Mobile Network providers (e.g., Truphone) to emerge and offer novel connectivity options to end-users.
Appendix A Ethical considerations
Data collection and retention at network middle-boxes are in accordance with the terms and conditions of the IPX-P and the local regulations, and only with the specific purpose of providing and managing the IPX service. The terms also include data processing for monitoring and reporting as allowed usages of collected data. Data processing only extracts aggregated information and we do we not have access to any personally identifiable information. We nevertheless consulted with the Institutional Review Board (IRB) office at our institution who confirmed that no IRB review was necessary as the study relies on the analysis of de-identified data.
Appendix B IPX Providers
We compiled a list of 29 IPX-Ps that, at the time of writing, interconnect to form the IPX Network. Table 1 lists the IPX-Ps we identified as active in the IPX Network, their (public) peering policy, number of PoPs and the type of Autonomous System (AS) number (private/public). We built the list of IPX-Ps and the information we show in the table by manually exploring the peering dataset. Specifically, at AMS-IX we found 27 customers for the mobile peering service with IPX/GRX tags (amsix). In addition, we found 24 of the total 27 IPX-Ps peering at AMS-IX also present at Equinix IXPs. According to GSMA, the majority of IPX-Ps connect to these two IXPs (ir34). Additionally, we checked the top 10 largest IXPs and some global IXPs and found no additional IXPs offering the mobile peering service. We complete this list with two additional IPX-Ps we observe in the internal list of peers from the operational IPX-P, thus bringing the total number of players in the IPX Network to 29, which we list in Table 1. We confirm this list and the IPX-Ps’ identities by checking several commercial market surveys and reports about IPX-Ps from diverse parties (hot_telecom; rocco; nanog51cowie).
We also find that among the set of currently active IPX-Ps, there are several which focus on interconnecting SPs within a specific region (e.g., Telin Indonesia, SAP). We do not include these in our analysis, but instead focus on the list we show in Table 1.
The number of PoPs in Table 1 indicates the number of locations where an IPX-P can cross-connect with SPs, giving insight into its world-wide geographical footprint. The average number of PoPs among the 18 IPXs publicly disclosing this information is 116.
|IPX-P Name||BGP||Peering Policy||PoPs|
|C & W||X||closed|