Over the past few years the field of high performance computing (HPC) has received attention from different vendors, which led to a steep rise in the number of chip architectures. All of these chips have different performance-power-price points, and thus different performance characteristics. This trend is believed to continue in the future with more vendors such as Marvell, Huawei, and Arm entering HPC and related fields with new designs. Benchmarking the architectures to understand their characteristics is pivotal for informed decision making and targeted code optimization. However, with hardware becoming more diverse, proper benchmarking is challenging and error-prone due to wide variety of available but often badly documented tuning knobs and settings.
In this paper we explore two modern Intel server processors, Cascade Lake SP and Broadwell EP, using carefully developed micro-architectural benchmarks, then show how these simple microbenchmark codes become relevant in application scenarios. During the process we demonstrate the different aspects of proper benchmarking like the importance of appropriate tools, the danger of black-box benchmark code, and the influence of different hardware and system settings. We also show how simple performance models can help to draw correct conclusions from the data.
Our microbenchmarking results highlight the changes from the Broadwell to the Cascade Lake architecture and their impact on the performance of HPC applications. Probably the biggest modification in this respect was the introduction of a new L3 cache design.
This paper makes the following relevant contributions:
We show how proper microarchitectural benchmarking can be used to reveal the cache performance characteristics of modern Intel processors. We compare the performance features of two recent Intel processor generations and resolve inconsistencies in published data.
We analyze the performance impact of the change in the L3 cache design from Broadwell EP to Skylake/Cascade Lake SP and investigate potential implications for HPC applications (effective L3 size, prefetcher, scalability).
We use standard kernels and benchmarks from selected application areas to investigate the characteristic performance change for these fields:
For DGEMM we show the impact of varying core and Uncore clock speed, problem size, and sub-NUMA clustering on Cascade Lake SP.
For a series of sparse matrix-vector multiplications we show the consequence of the nonscalable L3 cache and the benefit of the enhanced effective L3 size on Cascade Lake SP.
To understand the performance characteristics of the HPCG benchmark, we construct and validate the roofline model for all its components and the full solver for the first time. Using the model we identify an MPI desynchronization mechanism in the implementation that causes erratic performance of one solver component.
This paper is organized as follows. After describing the benchmark systems setup in Section 2, microarchitectural analysis using microbenchmarks (e.g., load and copy kernels and STREAM) is performed in Sections 3 to 5. In Section 6 we then revisit the findings and see how they affect code from realistic applications. Section 7 concludes the paper.
1.0.1 Related work
Molka et al.  used their BenchIT microbenchmarking framework to thoroughly analyze latency and bandwidth across the full memory hierarchy of Intel Sandy Bridge and AMD Bulldozer processors, but no application analysis or modeling was done. Hofmann et al. [6, 7] presented microbenchmark results for several Intel server CPUs. We extend their methodology towards Cascade Lake SP and also focus on application-near scenarios. Saini et al. [15, 14] compared a range of Intel server processors using diverse microbenchmarks, proxy apps and application codes. They did not, however, provide a thorough interpretation of the data in terms of the hardware architectures. McIntosh-Smith et al.  compared the Marvell ThunderX2 CPU with Intel Broadwell and Skylake using STREAM, proxy apps and full applications, but without mapping architectural features to microbenchmark experiments. Recently, Hammond et al. [5, 4] performed a benchmark analysis of the Intel Skylake and Marvell ThunderX2 CPUs, presenting data that was partly in contradiction to known microarchitectural features. A deeper investigation of these issues formed the seed for the present paper.
2 Testbed and environment
All experiments were carried out on one socket each of Intel’s Broadwell-EP (BDW) and Cascade Lake-SP (CLX) CPUs. These represent the previous and current generation CPUs in the Intel line of architectures which represents more than 85% of the top500 list published in November 2019. Table 1 summarizes the key specifications of the testbed.
The Broadwell-EP architecture has a traditional Intel design with a three-level inclusive cache hierarchy. The L1 and L2 caches are private to each core and the L3 is shared. BDW supports the AVX2 instruction set, which is capable of 256-bit wide SIMD. The Cascade Lake-SP architecture has a shared non-inclusive victim L3 cache. The particular model in our testbed supports the AVX-512 instruction set and has 512-bit wide SIMD. Both chips support “Cluster on Die [CoD]” (BDW) or “Sub-NUMA Clustering [SNC]” (CLX) feature, by which the chip can be logically split in two ccNUMA domains.
Both systems ran Ubuntu version 18.04.3 (Kernel 4.15.0). The Intel compiler version 19.0 update 2 with the highest optimization flag (-O3) was used throughout. Unless otherwise stated, we added architecture-specific flags -xAVX (-xCORE-AVX512 -qopt-zmm-usage=high) for BDW (CLX). The LIKWID tool suite in version 4.3 was used for performance counter measurements (likwid-perfctr tool) and benchmarking (likwid-bench tool). Note that likwid-bench generates assembly kernels automatically, providing full control over the finally executed code.
|Microarchitecture||Broadwell-EP (BDW)||Cascade Lake-SP (CLX)|
|Chip Model||Xeon E5-2697 v4||Xeon Gold 6248|
|Supported core freqs||1.2–3.6 GHz||1.2–3.9 GHz|
|Supported Uncore freqs||1.2–2.8 GHz||1.0–2.4 GHz|
|Latest SIMD extension||AVX2 / FMA||AVX-512|
|L1 cache capacity||1832 kB||2032 kB|
|L2 cache capacity||18256 kB||201 MB|
|L3 cache capacity||45 MB (182.5 MB)||27.5 MB (201.375 MB)|
|Memory Configuration||4 ch. DDR4-2400||6 ch. DDR4-2933|
|LD/ST throughput||2 LD, 1 ST (AVX)||2 LD, 1 ST (AVX512)|
|L1 - L2 bandwidth||64 B/cy||64 B/cy|
|L2 - L3 bandwidth||32 B/cy||16 B/cy + 16 B/cy|
|Theor. Mem. Bandwidth||76.8 GB/s||140.8 GB/s|
|Operating system||Ubuntu 18.04.3||Ubuntu 18.04.3|
|Compiler||Intel 19.0 update 2||Intel 19.0 update 2|
2.0.1 Influence of machine and environment settings
The machine and environment settings are a commonly neglected aspect of benchmarking, although they can have a decisive impact on performance. One should therefore take care to document all available settings. Figure 1(a) shows the influence of different operating system (OS) settings on a serial load-only benchmark running at 1.6 GHz on CLX for different data-set sizes in L3 and main memory. With the default OS setting (NUMA balancing on and transparent huge pages (THP) set to “madvise”), we can see a 2 hit in performance for big data sets. This behavior also strongly depends on the OS version. We observed it with Ubuntu 18.04.3 (see Table 1). Consequently, we use the setting that gives highest performance, i.e., NUMA balancing off and THP set to “always,” for all subsequent experiments.
Modern systems have an increasing number of knobs to tune on system startup. Figure 1(b) shows the consequences of the sub-NUMA clustering (SNC) feature on CLX for the load-only benchmark. With SNC active the single core has local access to only one sub-NUMA domain causing the shared L3 size to be halved. For accesses from main memory, disabling SNC slightly reduces the single core performance by 4% as seen in the inset of Figure 1(b).
3 Single-core bandwidth analysis
Single-core bandwidth analysis is critical to understand the machine characteristics and capability for a wide range of applications, but it requires great care especially when measuring cache bandwidths since any extra cycle will directly change the result. To show this we choose the popular bandwidth measurement tool lmbench . Figure 2 shows the load-only (full-read or frd) bandwidth obtained by lmbench as a function of data set size on CLX at 1.6 GHz. Ten runs per size are presented in a box-and-whisker plot.
Theoretically, one core is capable of two AVX-512 loads per cycle
for an L1 bandwidth of 128 byte/cy (204.8 Gbyte/s @ 1.6 GHz).
However, with the compiler option
-O2 (default setting) it
deviates by a huge factor of eight (25.5 Gbyte/s)
from the theoretical limit. The characteristic strong
performance gap between L1 and L2 is also missing.
Therefore, we tested different compiler
flags and compilers to see the effect (see Figure 2
) and observed a large span of performance values. Oddly, increasing the level of optimization (
-O3) dramatically decreases the performance. The highest bandwidth was attained for
-O2with the architecture-specific flags mentioned in Section 2, but even then the run-to-run variability is high, especially in L2.
It is impossible to draw any profound conclusions about the machine characteristic from such measurements, thus lmbench results for frd (e.g., [15, 14, 5, 4]) should be interpreted with due care. However, employing proper tools one can attain bandwidths close to the limits. This is demonstrated by the AVX-512 load-only bandwidth results obtained using likwid-bench . As seen in Figure 2 with likwid-bench we get 88% of the theoretical limit in L1, the expected drops at the respective cache sizes, and much less run-to-run fluctuation.
Figure 3 shows application bandwidths111Application bandwidth refers to the bandwidth as seen by the application without the inclusion of hidden data traffic like write-allocate transfers. from different memory hierarchy levels of BDW and CLX (load-only and copy kernels). The core clock frequency was fixed at 1.6 and 2 GHz for CLX and BDW, respectively, with SNC/CoD switched on. The bandwidth is shown in byte/cy, which makes it independent of core clock speed for L1 and L2 caches. Conversion to Gbyte/s is done by multiplying the byte/cy value with the clock frequency in GHz. The effect of single-core L1 bandwidth for scalar and different SIMD width is also shown in Figure 3(a) for CLX. It can be seen that the bandwidth reduces by 2 as expected when the SIMD width is halved each time.
4 Intel’s new shared L3 victim cache
From BDW to CLX there are no major observable changes to the behavior of L1 and L2 caches, expect that the L2 cache size has been significantly extended in CLX. However, starting from Skylake (SKX) the L3 cache has been redesigned. In the following we study the effect of this newly designed noninclusive victim L3 cache.
4.1 L3 cache replacement policy
A significant change with respect to the L3 cache concerns its replacement policy. Prior to SKX, Intel processors’ L3 caches implemented a so-called pseudo-LRU cache-replacement policy . This policy was fixed in the sense that the same pseudo-LRU policy was applied to all workloads. Starting with SKX, this situation changed. Empirical data indicates that SKX’s L3 cache uses an adaptive replacement policy . This means the processor implements a set of different replacement policies. During execution the constantly processor selects a suitable policy depending on the current workload in an attempt to maximize the L3-cache hit rate.
Experimental analysis suggests that the replacement policy selected by the processor for streaming-access patterns involves placing new cache lines only in one of the eleven ways of each cache set—the same strategy that is used when prefetching data using the prefetchnta instruction (cf. Sect. 188.8.131.52 in ). Consequently, data in the remaining ten ways of the sets will not be preempted and can later be reused.
Figure 4a demonstrates the benefit of this replacement policy optimized for streaming-access patterns by comparing it to the pseudo-LRU policies of previous generations’ L3 caches. The figure shows the L3-cache hit rate222Based on performance-counter data for the MEM_LOAD_RETIRED_L3_HIT and MISS events. for different data-set sizes on different processors for a load-only data-access pattern. To put the focus on the impact of the replacement policies on the cache hit rate, hardware prefetchers were disabled during measurements. Moreover, data-set sizes are normalized to compensate the processors’ different L3-cache capacities. The data indicates that older generations L3 caches offer no data reuse for data sets two times the cache capacity, whereas CLX’s L3 delivers hit rates of 20% even for data sets almost four times its capacity. Even for data more than ten times the L3 cache’s size can reuse be detected on CLX.
The fact that this improvement can also be observed in practice is demonstrated in Figure 4b, which shows measured bandwidth for the same load-only data-access pattern on CLX. For this measurement, all hardware prefetchers were enabled. The data indicates that the L3-cache hit-rate improvements directly translate into higher-than-memory bandwidths for data sets well exceeding the L3 cache’s capacity.
4.2 L3 scalability
Starting from Intel’s Sandy Bridge architecture (created in 2011) the shared L3 cache of all the Intel architectures up to Broadwell is known to scale very well with the number of cores . However, with SKX onwards the L3 cache architecture has changed from the usual ring bus architecture to a mesh architecture. Therefore in this section we test the scalability of this new L3 cache.
In order to test the L3 scalability we use again the likwid-bench tool and run the benchmark with increasing number of cores. The data-set size was carefully chosen to be 2 MB per core to ensure that the size is sufficiently bigger than the L2 cache however small enough such that no significant data traffic is incurred from the main memory.
The application bandwidths of the three basic kernels load-only, copy and update are shown in Figure 5 for CLX333The result on SKX is exactly the same as CLX and BDW. As the update kernel has equal number of loads and stores it shows the maximum attainable performance on both architectures. Note that also within cache hierarchies write-allocate transfers occur leading to lower copy application bandwidth. The striking difference between CLX and BDW for load-only bandwidth can finally be explained by the bi-directional L2-L3 link on CLX which only has half the load-only bandwidth as BDW (see Table 1).
In terms of scalability we find that the BDW scales almost linearly and attains an efficiency within 90%, proving that the BDW has almost purely scalable L3 cache. However, with CLX this behavior has changed drastically and the L3 cache saturates at higher core counts both with and without SNC enabled, yielding an efficiency of about 70%. Due to this for the applications that employ L3 cache blocking it might be worthwhile to investigate the impact of switching to pure L2 blocking on SKX and CLX architectures. In case of applications that use the shared property of L3 cache like some of the temporal blocking schemes  a similar saturation effect as seen in Figure 5 might be visible.
The effect of SNC/COD mode is also shown in Figure 5, with dotted lines corresponding to SNC off mode and solid to SNC on mode. For CLX with SNC off mode the bandwidth attained at half of the socket (ten threads) is higher than SNC on mode. This is due to the availability of more L3 tiles and controllers with SNC off mode.
5 Multi-core scaling with STREAM
The STREAM benchmark  has the purpose of measuring
the achievable memory bandwidth of a processor. Although the code
comprises four different loops, their performance is generally
similar and usually only the triad (
A(:)=B(:)+s*C(:)) is reported.
The benchmark output is a bandwidth number in Mbyte/s, assuming 24 byte of data traffic per iteration. The rules state that the working set size should
be at least four times the LLC size of the CPU. In the light of the new
LLC replacement policies (see Section 4.1), this appears too small
and we chose a 2 GB working set for our experiments.
Since the target array
A causes write misses, the assumption
of the benchmark about the code balance is wrong if write-back caches
are used and write-allocate transfers cannot be avoided. X86 processors
feature nontemporal store instructions (also known as
streaming stores), which bypass the normal cache hierarchy
and store into separate write-combine buffers. If a full cache line
is to be written, the write-allocate transfer can thus be avoided.
Nontemporal stores are only available in SIMD variants on Intel processors,
so if the compiler chooses not to use them (or is forced to by a directive
or a command line option), write-allocates will occur and the memory bandwidth
available to the application is reduced. This is why vectorization
appears to be linked with better STREAM bandwidth, while it is actually
the nontemporal store that cannot be applied for scalar code.
Note that a careful investigation of the impact of write-allocate
policies is also required on other modern processors
such as AMD- or ARM-based systems.
Figure 6 shows the bandwidth reported by the STREAM triad benchmark on BDW and CLX with (a) and without (b) CoD or SNC enabled. There are three data sets in each graph: full vectorization with the widest supported SIMD instruction set and standard stores (noNT), scalar code, and full vectorization with nontemporal stores (NT). Note that the scalar and “noNT” variants have very similar bandwidth, which is not surprising since they both cause write-allocate transfers for an overall code balance of 32 byte/it. The reported saturated bandwidth of the “NT” variant is a factor of higher because the memory interface delivers the same bandwidth but the code balance is only 24 byte/it.444Note that earlier Intel processors like Ivy Bridge and Sandy Bridge could not attain the same memory bandwidth with NT stores as without. The difference was small enough, however, to still warrant the use of NT stores in performance optimization.
The peculiar shape of the scaling curve with CoD or SNC enabled is a consequence of the “compact” pinning used for the OpenMP threads (filling the physical cores of the socket from left to right) and the static scheduling employed by the OpenMP runtime for the STREAM triad loop. If only part of the second ccNUMA domain is utilized, each of its active cores will have the same workload as each core on the other (strongly saturated) domain but more memory bandwidth available. Due to the implicit barrier at the end of the parallel region, these “fast” cores must wait for the slower cores on the other domain. Hence, each new additional core on the second domain can only run at the average performance of a core on the first domain, which leads to the linear scaling. A “scattered” pinning strategy would show only one saturation curve, of course. Note that the the available saturated memory bandwidth is independent of the CoD/SNC setting for the two CPUs under consideration.
6 Implications for real-world applications
In the previous sections we discussed on micro-benchmark analysis of the two Intel architectures. In the following we demonstrate how these analysis reflect in real applications by investigating important kernels such as DGEMM, sparse matrix-power-vector multiplication and HPCG code. In order to reflect the practical settings used in HPC runs we use turbo frequency and switch-off SNC for the following runs. Measurements which do not follow these settings are clearly mentioned. Statistical variations are shown whenever the fluctuations are bigger than 5%.
6.1 DGEMM—Double-precision general matrix-matrix multiplication
When implemented correctly, DGEMM is compute-bound on Intel processors. Each CLX core is capable of executing 32 floating-point operations (flops) per cycle (8 DP numbers per AVX-512 register, 16 flops per fused multiply-add (FMA) instruction, 32 flops using both AVX-512 FMA units). Running DGEMM on all twenty cores, the processor specimen from the testbed managed to sustain a frequency of 2.09 GHz. The upper limit to DGEMM performance is thus 1337.6 GFlop/s.
Figure 7a compares measured full-chip DGEMM performance on CLX in Turbo mode (black line) to theoretical peak performance (dashed red line). The data indicates that small values of are not suited to produce meaningful results. In addition to resulting in sub-optimal performance, values of
below 10,000 lead to significant variance in measurements, as demonstrated forusing a box-plot representation (and reproducing the results from ).
Figure 7b shows measured DGEMM performance subject to the number of active cores. When the frequency is fixed (in this instance at 1.6 GHz, which is the frequency the processor guarantees to attain when running AVX-512 enabled code on all its cores), DGEMM performance scales practically perfectly with the number of active cores (black line). Consequently, the change of slope in Turbo mode stems solely from a reduction in frequency when increasing the number of active cores. Moreover, data shows that SNC mode is slightly detrimental to performance (blue vs. green line).
On Haswell-based processors, a sensitivity of DGEMM performance to the Uncore frequency could be observed : When running cores in Turbo mode, increasing the Uncore frequency resulted in a decreasing the share of the processor’s TDP available to the cores, which lead them to decrease their frequency. On CLX this is no longer the case. When running DGEMM on all cores in Turbo mode results in a clock frequency of 2.09 GHz—independent of the Uncore frequency. Analysis using hardware events suggest the Uncore clock is subordinated to the core one: Using the appropriate MSR (0x620), the Uncore clock can only be increased up to 1.5 GHz. There are, however, no negative consequences of that limitation. Traffic analysis in the memory hierarchy indicates that DGEMM is blocked for the L2 cache, so the Uncore clock (which influences L3 and memory bandwidth) plays no significant role for DGEMM.
6.2 SpMPV – Sparse Matrix-Power-Vector Multiplication
The SpMPV benchmark (see Algorithm 1) computes , where
is a sparse matrix, as a sequence of sparse matrix-vector products. The SpMPV kernel is used in a wide range of numerical algorithms like Chebyshev filter diagonalization for eigenvalue solvers16], and numerical time propagation .
The sparse matrix is stored in the compressed row storage (CRS) format using double precision, and we choose in our experiments. For the basic sparse matrix vector (SpMV) kernel we use the implementation in Intel MKL 19.0.2. The benchmark is repeated multiple times to ensure that it runs for at least one second, so we report the average performance over many runs.
For this benchmark we selected five matrices from the publicly available SuiteSparse Matrix Collection . The choice of matrices was motivated by some of the hardware properties (in particular L3 features) as investigated in previous sections via microbenchmarks. The details of the chosen matrices are tabulated in Table 2. The matrices were pre-processed with reverse Cuthill-McKee (RCM) to attain better data locality; however, all performance measurements use the pure SpMPV execution time, ignoring the time taken for reordering.
|Index||Matrix name||size (MB)|
6.2.1 L3 scalability
Figure (a)a shows the scaling performance of the ct20stif matrix on CLX and BDW. This matrix is just 30 MB in size and fits easily into the caches of both processors.555Note that even though the L3 cache of CLX is just 27.5 MB in size, it is an exclusive victim cache. The applicable cache size for applications using all cores is thus 47.5 MB, which the aggregate L2/L3 cache size. The L3 bandwidth saturation of CLX as shown in Section 4.2 is reflected by the performance saturation in the SpMPV benchmark. For this matrix, BDW performs better than CLX since the sparse matrix kernel is predominantly load bound and limited by the bandwidth of the load-only micro-benchmark (see Figure (a)a).
Despite its advantage over CLX, the in-cache SpMPV scaling on BDW is not linear (parallel efficiency at all cores), which stands in contrast with the microbenchmark results in Figure (a)a. The main reason is that the SpMPV benchmark was run in “Turbo Mode,” causing the clock frequency to drop by 25% when using all the cores (3.6 GHZ at single core to 2.7 GHz at full socket).
6.2.2 L3 cache replacement policy
We have seen in Section 4.1 that CLX has a more sophisticated adaptive L3 cache replacement policy, which allows it to extend the caching effect for working sets as big as ten times the cache size. Here we show that SpMPV can profit from this as well. We choose three matrices that are within five times the L3 cache size (index 2,3, and 4 in table 2) and a moderately large matrix that is 36 times bigger than the L3 cache (index 5 in table 2).
Figure (b)b shows the full-socket performance and memory transfer volume for the four matrices. Theoretically, with a least-recently used (LRU) policy the benchmark requires a minimum memory data transfer volume of bytes per non-zero entry of the matrix . This lower limit is shown in Figure (b)b (right panel) with dashed lines. We can observe that in some cases the actual memory traffic is lower than the theoretical minimum, because the L3 cache can satisfy some of the cacheline requests. Even though CLX and BDW have almost the same amount of cache, the effect is more prominent on CLX. On BDW it is visible only for the boneS01 matrix, which is bigger than its L3 cache, while on CLX it can be observed even for larger matrices. This is compatible with the micro-benchmark results in Section 4.1. For some matrices the transfer volume is well below 12 bytes per entry, which indicates that not just the vectors but also some fraction of the matrix stays in cache.
As shown in the left panel of Figure (b)b, the decrease in memory traffic directly leads to higher performance. For two matrices on CLX the performance is higher than the maximum predicted by the Roofline model (dashed line) even when using the highest attainable memory bandwidth (load-only). This is in line with data presented in .
6.3 HPCG – High Performance Conjugate Gradient
HPCG666http://www.hpcg-benchmark.org/ (High Performance Conjugate Gradient) is a popular memory-bound proxy application which mimics the behavior of many realistic sparse iterative algorithms. However, there has been little work to date on analytic performance modeling of this benchmark. In this section we analyze HPCG using the Roofline approach.
The HPCG benchmark implements a preconditioned conjugate gradient (CG) algorithm with a multi-grid (MG) preconditioner. The linear system is derived from a 27-point stencil discretization, but the corresponding sparse matrix is explicitly stored. The benchmark uses the two BLAS-1 kernels DOT and WAXPBY and two kernels (SpMV and MG) involving the sparse matrix. The chip-level performance of HPCG should thus be governed by the memory bandwidth of the processor. Since the benchmark prints the Gflop/s performance of all kernels after a run, this should be straightforward to corroborate. However, the bandwidths derived from different kernels in HPCG vary a lot (see Table 3): For the WAXPBY kernel (w[i]=a*x[i]+y[i]), which has a code balance of 12 byte/flop777The plain WAXPBY kernel has a code balance of 16 byte/flop if a write-allocate transfer must be accounted for; however, in HPCG it is called with w and x being the same array, so no write-allocate applies., the reported performance is 5.14 Gflop/s on a full socket of BDW. On the other hand, for the DOT kernel with a reported code balance of 8 byte/flop the benchmark reports a performance of 10.16 Gflop/s. According to the Roofline model this translates into memory bandwidths of 61.7 Gbyte/s and 81.3 Gbyte/s, respectively. The latter value is substantially higher than any STREAM value presented for BDW in Figure 6. In the following, we use performance analysis and measurements to explore the cause of this discrepancy, and to check whether the HPCG kernel bandwidths are in line with the microbenchmark analysis.
For this analysis we use the recent reference variant of HPCG (version 3.1), which is a straightforward implementation using hybrid MPI+OpenMP parallelization. However, the local symmetric Gauss-Seidel (symGS) smoother used in MG has a distance-1 dependency and is not shared-memory parallel. The main loop of the benchmark is shown in Algorithm 2, where is the sparse matrix stored in CRS format.
As the symGS kernel consumes more than 80% of the entire runtime, the benchmark is run with pure MPI using one process per core. The code implements weak scaling across MPI processes; we choose a local problem size of for a working set of about 1.2 GB per process. The maximum number of CG iteration was set at 25, the highest compiler optimization flag was used (see Table 1), and the contiguous storage of sparse matrix data structures was enabled (-DHPCG_CONTIGUOUS_ARRAYS).
Performance analysis of kernels
We use the roofline model to model each of the four kernels separately. Due to their strongly memory-bound characteristics, an upper performance limit is given by , where is the full-socket (saturated) memory bandwidth and is the code balance of the kernel . As we have a mixture of BLAS-1 ( iterations) and sparse ( iterations) kernels, is computed in terms of bytes required and work done per row of the matrix.
The reference implementation contains three DOT kernels (see Algorithm 2). Two of them require two input vectors to be loaded (lines 4 and 8 in Algorithm 2) and the other requires just one (norm computation in line 12), resulting in a total average code balance of . All three WAXPBY kernels require one vector to be loaded and one vector to be both loaded and stored, resulting in . For sparse kernels the total data transferred for the inner iterations has to be considered. As shown in Section 6.2, the optimal code balance for SpMV is
bytes per non-zero matrix entry, i.e.,. For the MG preconditioner we consider only the finest grid since the coarse grids do not substantially contribute to the overall runtime. Therefore the MG consists mainly of one symGS pre-smoothing step followed by one SpMV and one symGS post-smoothing step. The symGS comprises a forward sweep (
0:nrows) followed by a backward sweep (
nrows:0). Both have the same optimal code balance as SpMV, which means that the entire MG operation has a code balance of five times that of SpMV: .
The correctness of the predicted code balance can be verified using performance counters. We use the likwid-perfctr tool to count the number of main memory data transfers for each of the kernels.888See https://github.com/RRZE-HPC/likwid/wiki/TestAccuracy for validation of the data groups. Table 3 summarizes the predicted and measured code balance values for full-socket execution along with the reported performance and number of flops per row for the four kernels in HPCG. Except for DDOT, the deviation between predicted and measured code balance is less than 10%.
Surprisingly, DDOT has a measured code balance that is lower than the model, pointing towards caching effects. However, a single input vector for DDOT has a size of 560 MB, which is more than ten times the available cache size. As shown in Section 4.1, even CLX is not able to show any significant caching effect with such working sets. Closer investigation revealed desynchronization of MPI processes to be the reason for the low code balance: In Algorithm 2 we can see that some of the DOT kernel can reuse data from previous kernels. For example, the last DOT (line 12) reuses the vector form the preceding WAXPBY. Therefore, if MPI processes desynchronize such that only some of them are already in DOT while the others are in different kernels (like WAXPBY), then the processes in DOT can reuse the data, while the others just need to stream data as there is no reuse. This is possible since there are no explicit synchronization points between previous kernels and DOT.
This desynchronization effect is not predictable and will vary between runs and machine as can be observed by the significant performance fluctuation of DOT in Figure 9. To verify our assumption we added barriers before the DOT kernels, which caused the measured to go up to 13.3 byte/row, matching the expected value. The desynchronization effect clearly shows the importance of analyzing statistical fluctuations and deviations from performance models. Ignoring them can easily lead to false conclusions about hardware characteristics and code behavior.
|Arch||Kernels||Code-balance ()||Performance ()||Flops||Calls||HPCG perf.|
Combining kernel predictions
Once the performance predictions for individual kernels are in place, we can combine them to get a prediction of the entire HPCG. This is done by using a time-based formulation of the roofline model and linearly combining the predicted kernel runtimes based on their call counts. If is the number of flops per row and the number of times the kernel is invoked, the final prediction is
Table 3 gives an overview of , , and for different kernels and compares the predicted and measured performance on a full socket. The prediction is consistently higher than the model because we used the highest attainable bandwidth for the roofline model prediction. For Intel processors this is the load-only bandwidth (68 Gbyte/s) for CLX (BDW), which is approximately 10% higher than the STREAM values (cf. Section 5). Figure 9 shows the scaling performance of the different kernels in HPCG. The typical saturation pattern of memory-bound code can be observed on both architectures.
7 Conclusions and Outlook
Two recent, state of the art generations of Intel architectures have been analyzed: Broadwell EP and Cascade Lake SP. We started with a basic microarchitectural study concentrating on data access. The analysis showed that our benchmarks were able to obtain 85% of the theoretical limits and for the first time demonstrated the performance effect of the Intel’s newly designed shared L3 victim cache. During the process of microbenchmarking we also identified the importance of selecting proper benchmark tools, the impact of various hardware, software, and OS settings, thereby demonstrating the necessity for detailed documentation. We further demonstrated that the observations made in microbenchmark analysis are well reflected in real-world application scenarios. To this extent we investigated the performance characteristics of DGEMM, sparse matrix-vector multiplication, and HPCG. For the first time a roofline model of HPCG and its components was established and successfully validated for both architectures. Performance modeling was used as a guiding tool throughout this work to get deeper insight and explain anomalies.
Future work will include investigation of benchmarks for random and latency-bound codes along with the development of suitable performance models. The existing and further upcoming wide range of architectures will bring more parameters and benchmarking challenges, which will be very interesting and worthwhile to investigate.
We are indebted to Thomas Zeiser and Michael Meier (RRZE) for providing a reliable benchmarking environment. This work was partially funded by the ESSEX project in the DFG priority programme 1648 (SPPEXA).
-  Intel 64 and IA-32 Architectures Optimization Reference Manual. Intel Press (June 2016), http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
-  Alappat, C.L., Hager, G., Schenk, O., Thies, J., Basermann, A., Bishop, A.R., Fehske, H., Wellein, G.: A Recursive Algebraic Coloring Technique for Hardware-Efficient Symmetric Sparse Matrix-Vector Multiplication (2019), https://arxiv.org/abs/1907.06487
-  Davis, T.A., Hu, Y.: The University of Florida Sparse Matrix Collection. ACM Trans. Math. Softw. 38(1), 1:1–1:25 (Dec 2011), http://doi.acm.org/10.1145/2049662.2049663
-  Hammond, S., Hughes, C., M.Levenhagen, Vaughan, C., A.Younge, B.Schwaller, M.Aguilar, K.Pedretti, J.Laros: Evaluating the Marvell ThunderX2 Server Processor for HPC Workloads. In: The 6th Special Session on High-Performance Computing Benchmarking and Optimization (HPBench 2019) (July 2019)
-  Hammond, S., Vaughan, C., Hughes, C.: Evaluating the Intel Skylake Xeon Processor for HPC Workloads. In: 2018 International Conference on High Performance Computing Simulation (HPCS). pp. 342–349 (July 2018)
-  Hofmann, J., Fey, D., Eitzinger, J., Hager, G., Wellein, G.: Analysis of Intel’s Haswell microarchitecture using the ECM model and microbenchmarks. In: Hannig, F., Cardoso, J.M.P., Pionteck, T., Fey, D., Schröder-Preikschat, W., Teich, J. (eds.) Architecture of Computing Systems – ARCS 2016: 29th International Conference, Nuremberg, Germany, April 4–7, 2016, Proceedings. pp. 210–222. Springer International Publishing, Cham (2016), http://dx.doi.org/10.1007/978-3-319-30695-7_16
-  Hofmann, J., Hager, G., Wellein, G., Fey, D.: An analysis of core- and chip-level architectural features in four generations of Intel server processors. In: Kunkel, J.M., Yokota, R., Balaji, P., Keyes, D. (eds.) High Performance Computing: 32nd International Conference, ISC High Performance 2017, Frankfurt, Germany, June 18–22, 2017, Proceedings. pp. 294–314. Springer International Publishing, Cham (2017), https://doi.org/10.1007/978-3-319-58667-0_16
-  McCalpin, J.D.: Memory Bandwidth and Machine Balance in Current High Performance Computers. IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter pp. 19–25 (Dec 1995)
-  McIntosh-Smith, S., Price, J., Deakin, T., Poenaru, A.: A performance analysis of the first generation of HPC-optimized Arm processors. Concurrency and Computation: Practice and Experience 31(16), e5110 (2019), https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.5110, e5110 cpe.5110
-  McVoy, L., Staelin, C.: Lmbench: Portable tools for performance analysis. In: Proceedings of the 1996 Annual Conference on USENIX Annual Technical Conference. pp. 23–23. ATEC ’96, USENIX Association, Berkeley, CA, USA (1996), http://dl.acm.org/citation.cfm?id=1268299.1268322
-  Molka, D., Hackenberg, D., Schöne, R.: Main memory and cache performance of intel sandy bridge and amd bulldozer. In: Proceedings of the Workshop on Memory Systems Performance and Correctness. pp. 4:1–4:10. MSPC ’14, ACM, New York, NY, USA (2014), http://doi.acm.org/10.1145/2618128.2618129
-  Pieper, A., Kreutzer, M., Alvermann, A., Galgon, M., Fehske, H., Hager, G., Lang, B., Wellein, G.: High-performance implementation of Chebyshev filter diagonalization for interior eigenvalue computations. Journal of Computational Physics 325, 226 – 243 (2016), http://www.sciencedirect.com/science/article/pii/S0021999116303837
-  Qureshi, M.K., Jaleel, A., Patt, Y.N., Steely, S.C., Emer, J.: Adaptive Insertion Policies for High Performance Caching. In: Proceedings of the 34th Annual International Symposium on Computer Architecture. pp. 381–391. ISCA ’07, ACM, New York, NY, USA (2007), http://doi.acm.org/10.1145/1250662.1250709
-  Saini, S., Hood, R.: Performance Evaluation of Intel Broadwell Nodes Based Supercomputer Using Computational Fluid Dynamics and Climate Applications. In: 2017 IEEE 19th International Conference on High Performance Computing and Communications Workshops (HPCCWS). pp. 58–65 (Dec 2017)
Saini, S., Hood, R., Chang, J., Baron, J.: Performance Evaluation of an Intel Haswell-and Ivy Bridge-Based Supercomputer Using Scientific and Engineering Applications. In: 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS). pp. 1196–1203 (Dec 2016)
-  Staar, P.W.J., Barkoutsos, P.K., Istrate, R., Malossi, A.C.I., Tavernelli, I., Moll, N., Giefers, H., Hagleitner, C., Bekas, C., Curioni, A.: Stochastic Matrix-Function Estimators: Scalable Big-Data Kernels with High Performance. In: 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS). pp. 812–821 (May 2016)
-  Suhov, A.Y.: An Accurate Polynomial Approximation of Exponential Integrators. Journal of Scientific Computing 60(3), 684–698 (Sep 2014), https://doi.org/10.1007/s10915-013-9813-x
-  Treibig, J., Hager, G., Wellein, G.: likwid-bench: An Extensible Microbenchmarking Platform for x86 Multicore Compute Nodes. In: Parallel Tools Workshop. pp. 27–36 (2011)
-  Wellein, G., Hager, G., Zeiser, T., Wittmann, M., Fehske, H.: Efficient Temporal Blocking for Stencil Computations by Multicore-Aware Wavefront Parallelization. In: 2009 33rd Annual IEEE International Computer Software and Applications Conference. vol. 1, pp. 579–586 (July 2009)