1 Introduction
Across many domains, numerical computations specified over the reals are actually implemented using floatingpoint arithmetic. Due to their finite nature, operations on floatingpoint numbers cannot be calculated exactly and accumulate roundoff errors. In addition, realvalued identities such as associativity no longer hold, making manual reasoning and optimization challenging. To address these challenges, new automated tools have recently been developed which build on advances in program rewriting and verification techniques to enable even nonexperts to analyze and optimize their floating point code.
Some of these tools use sound techniques to statically bound roundoff errors of straightline floatingpoint programs [8, 11, 21, 14, 15, 7] and partially automate complex analysis tasks [17, 9]. Other such tools use dynamic techniques to find inputs that suffer from large rounding errors [3, 22]. Yet other tools perform rewritingbased optimization [4, 16, 19, 7] and mixedprecision tuning [2, 6] to improve the accuracy and performance of floatingpoint programs.
Since these tools are typically complementary, each focusing on a distinct aspect of numerical reliability, users will need to compose several to meet their development needs. This makes ease of composition essential, and some first steps in this regard have been taken by the FPBench project [5], which provides a common specification language for inputs to floatingpoint analysis tools similar to the one provided by the SMTLIB standard [1]. However, no literature yet exists on the actual use of FPBench to compose tools and on the challenges that stand in the way of combining different floatingpoint tools, such as differing notions of error and different sets of supported functions.
In this paper we report on our experience implementing the first combination of two complementary floatingpoint analysis tools using FPBench: Herbie [16] and Daisy [7]. Herbie optimizes the accuracy of straightline floatingpoint expressions, but employs a dynamic roundoff error analysis and thus cannot provide sound guarantees on the results. In contrast, Daisy performs static analysis of straightline expressions, which is sound w.r.t. IEEE754 floatingpoint semantics [12]. Our combination of the tools is implemented as a script in the FPBench repository (https://github.com/FPBench/FPBench).
We see this combination of a heuristic and a sound technique as particularly interesting; Daisy can act as a backend for validating Herbie’s optimizations. Daisy computes improved worstcase roundoff error bounds for many (but not all) expressions optimized by Herbie. On others it raises an alarm, discovering divisionbyzero errors introduced by Herbie. We also improved the precision of Daisy’s analysis of special functions as we found that some were sound but not accurate enough. Thus, the combination was also useful in uncovering limitations of both tools.
Daisy additionally implements a sound genetic programmingbased accuracy optimization procedure. Our combination of Daisy and Herbie allows us to compare it to Herbie’s unsound procedure based on greedy search. We discover important differences between the two procedures, suggesting that the techniques are not competitive but in fact complementary and best used in combination.
Some of the challenges we encountered, such as differing supported functions and different error measures, are likely to be encountered by other researchers or even end users combining floatingpoint tools, and our experience shows how these challenges can be surmounted. Our evaluation on benchmarks from the FPBench suite also shows that tool composition can provide endtoend results not achievable by either tool in isolation and suggests that further connections with other tools should be investigated.
2 Implementation
The highlevel goal of our combination is to use Daisy as a verification backend to Herbie to obtain a sound upper bound on the roundoff error of the expression returned by Herbie. By also evaluating the roundoff error of Herbie’s output and of the input expression, we can obtain additional validation of the improvement. It should be noted, however, that Daisy cannot verify whether the actual worstcase or average roundoff error has decreased—a decrease in the computed upper bound can be due to an actual decrease or simply due to a stronger static bound. In many cases, however, such as in safetycritical systems, just proving a smaller static bound is already useful.
We have implemented the combination in a script, which we sketch in Figure 1. For each straightline input program f, we first run Herbie to compute an optimized version f. Both the optimized and unoptimized version are translated into Daisy’s input format (using FPCore2Scala), and Daisy is run on both versions to compute error bounds.
Daisy supports several different types of error analysis, and we run Daisy in a portfolio style, where the tightest bound computed by any of the analyses is used. In particular, we use the interval analysis with subdivisions mode and the SMT solver mode (with Z3 [10] as the solver).^{1}^{1}1We found that neither interval analysis without subdivision nor alternate SMT solvers provided tighter bounds. Since each analysis is sound, this provides the tightest error bound that Daisy can prove.
To implement the toolchain from Figure 1, we had to address two major differences between Herbie and Daisy: Herbie and Daisy use different input (and output) formats, and Daisy requires domain bounds on all input variables, whereas Herbie allows unbounded inputs.
Formats
To avoid having to add new frontends, we implemented a translator from FPBench’s FPCore format to Daisy’s Scalabased input language. As Herbie produces optimized expressions in FPCore, this translator allows us to run Daisy on both the benchmarks and on Herbie’s optimized expressions. This translator is now part of the FPBench toolchain and can be used by other researchers and by users to integrate Daisy with other tools developed as part of the FPBench project.
Preconditions
Both Daisy and Herbie allow preconditions for restricting the valid inputs to a floatingpoint computation. For Herbie, these preconditions are optional. In contrast, Daisy requires input ranges for performing a forward dataflow analysis to compute sound absolute roundoff error bounds. Several of the benchmarks in FPBench did not have a specified precondition. For our experiments, we manually added a meaningful precondition to these programs, with preconditions chosen to focus on input values with significant rounding errors. To avoid biasing the results, the preconditions were simple orderof magnitude ranges for each variable, with the endpoints of these ranges chosen from , , or and their inverses and negations.
Improvements in Daisy and Herbie
Connecting Daisy and Herbie and running each on several previously unseen benchmarks uncovered numerous possibilities for improvements in Herbie and Daisy.
In Herbie, several bugs were discovered by our efforts: an incorrect typechecking rule for let statements (which would reject some valid programs); incorrect handling of duplicate fields (which allowed one field to improperly override another); and an infinite loop in the sampling code (triggered by some preconditions). Real users running older versions of Herbie have since also reported these bugs, suggesting that issues addressed during tool composition helped improve user experience generally.
In Daisy, we discovered that the analysis of elementary functions was unnecessarily conservative and improved the rational approximations used. Error handling in both tools was also improved such that issues like (potential) divisions by zero or timeouts are now accurately reported. This more precise feedback significantly improves user friendliness and reduces debugging time.
3 Experimental Results
We perform two evaluations of our combination of Daisy and Herbie: we first use Daisy as a verification backend for Herbie and then we compare Daisy’s and Herbie’s rewriting algorithms. Both experiments use all supported benchmarks from the FPBench suite. A table with all the evaluation data can be found in the appendix. The experiments were run on a machine with an i74790K CPU and 32GB of memory. For each benchmark we give both Daisy and Herbie a timeout of 2 minutes.
Composing Daisy and Herbie
Our first experiment considers Daisy solely as a tool for computing floatingpoint error bounds. Herbie is then used to attempt to improve the benchmark’s accuracy.
Of the 105 benchmarks, Herbie times out on 24 of them. Of the remaining 48 benchmarks, Daisy raises an alarm^{2}^{2}2Indicating that it could not prove the absence of invalid operations, such as divisions by zero. on 34 and can prove a bound on 48. Of the 48 benchmarks where Daisy can prove an error bound, Daisy’s roundoff error analysis can prove a tighter worstcase error bound for 23 of Herbie’s outputs, an equal bound for 14, and a looser bound for 11. These results are summarized in the leftmost graph in Figure 2.
Of the 34 benchmarks where Daisy raises an alarm, for 24 Daisy raises an alarm on the original input program and for 27 the alarm is raised on the output program Herbie produced. Some of these alarms are true positives while others are spurious. For example, in some benchmarks Herbie had introduced a possible division by 0, which Daisy was able to detect. In others, Herbie’s output contained expressions like , where Daisy is unable to prove that no division by zero occurs.
We see this as good evidence that Daisy can be used as a verification backend for Herbie. One challenge is that Daisy’s error analysis can only show that Herbie’s output has a smaller error bound, not that it is more accurate at any particular point on the original program. Additionally, it is difficult to determine which of Daisy’s alarms are spurious. Despite these challenges, the combination of Daisy and Herbie was able to produce large, verified improvements in rounding error in many benchmarks.
Comparing Daisy’s and Herbie’s Error Measures
One topic of particular interest in combining Daisy and Herbie is their different measures of floatingpoint error. Herbie measures the error in terms units in the last place, or ULPs. To compute this error, Herbie randomly samples input values and takes the average of error across those inputs. It thus provides a dynamic, unsound measure of average ULPs of error. Daisy, meanwhile, uses a mathematical abstraction based on the definition of rounding and IEEE754 operator semantics to provide static, sound bounds on the maximum absolute error. The relationship between these two measures of error is central to using Herbie and Daisy together.
Despite these stark differences between Herbie’s and Daisy’s error measures, our evaluation data shows that Daisy and Herbie can be fruitfully used together: Daisy verifies that Herbie’s improved program is no less accurate for 37/48 of the benchmarks. This suggests that, though Daisy and Herbie use very different means to measure error, both are successfully measuring the same underlying notion of error. The fact that Daisy’s and Herbie’s error measures are suited to their particular approaches (static analysis and program search) suggests that future tools should focus not on measuring error “correctly” but on finding an error measure well suited to their technical approach.
Comparing Daisy and Herbie
Our second experiment compares Daisy’s and Herbie’s rewriting algorithms. Daisy uses genetic programming to search for a program with a tighter bound. Herbie, by contrast, uses a greedy search over a suite of different rewriting steps. We compare Herbie’s rewriting algorithm, Daisy’s rewriting algorithm, and Daisy’s rewriting algorithm applied to the results of Herbie’s rewriting algorithm. Figure 2 summarizes the accuracy improvements.
Of the 105 benchmarks, at least one of the rewriting algorithms succeeds on 71. Of the 71, Daisy’s rewriting algorithm tightens the worstcase bound for 33 benchmarks; Herbie’s for 23 benchmarks; and the combination for 30 benchmarks. Furthermore, Herbie’s rewriting algorithm loosens the worstcase bound for 11 benchmarks, a consequence of its unsound error measurement technique or differing notion of error, while the combination does so for only 1.
Not only the number but also size of the error improvement matters. Daisy’s rewriting algorithm was able to reduce the error bound by a factor of 1.2 (0.08 orders of magnitude) on average; Herbie’s by a factor of 10.14 (1.01 orders of magnitude); and the combination by a factor of 12.79 (1.11 orders of magnitude). The combination clearly provided the greatest reduction in error bounds; furthermore, Daisy’s algorithm provides larger benefits when applied to Herbie’s optimized program than when applied to the benchmark directly.
It seems that Daisy’s rewriting algorithm provides a fairly consistent but small tightening of error bounds, while Herbie’s algorithm can suggest dramatic and unexpected changes in the expression. However, these large changes sometimes have significantly looser error bounds. In those cases, combining Herbie’s and Daisy’s rewriting algorithms provides a tighter error bound, reaping the benefits of Herbie’s rewriting algorithm without the large increases in error bounds that it sometimes causes.
4 Discussion
This paper reports on the combination of Daisy and Herbie and illustrates the benefits of composing complementary floatingpoint tools to achieve results neither tool provides in isolation. This case study serves as a representative example: similar combinations could be constructed for other tools using this paper’s approach. Combinations of Gappa [9], Fluctuat [11], FPTaylor [21], or other verification tools [8, 14, 15] with Herbie could also allow validating Herbie’s optimizations. Verification tools could also be used to validate the output of other unsound tools, such as Precimonious [18] and STOKE [20]. Comparisons with sound optimization tools such as Salsa [4] and FPTuner [2] could also be explored.
In our experiments we have seen that there is no correlation between the error measures of Daisy and Herbie. Experiments with other tools and other error measurements could help clarify the relationships between different error measures. Particularly interesting is whether other uncorrelated error measures nonetheless evaluate changes to programs in correlated ways, as Daisy’s and Herbie’s error measures do. Tools that use error measures related in this way can be composed, even if they use different error measures
Ultimately, we envision using the combination of Daisy and Herbie within larger developments such as VCFloat [17]. VCFloat provides partial automation for reasoning about floatingpoint computations in CompCert Clight [13] programs. In this context, our toolchain could provide an optimization tactic, that could be applied to (provably) increase accuracy for floatingpoint segments of Clight programs.
References
 [1] Barrett, C., Stump, A., Tinelli, C.: The SMTLIB Standard: Version 2.0. In: Gupta, A., Kroening, D. (eds.) International Workshop on Satisfiability Modulo Theories (Edinburgh, UK) (2010)
 [2] Chiang, W.F., Baranowski, M., Briggs, I., Solovyev, A., Gopalakrishnan, G., Rakamarić, Z.: Rigorous FloatingPoint MixedPrecision Tuning. In: Symposium on Principles of Programming Languages (POPL). pp. 300–315. ACM (2017)
 [3] Chiang, W.F., Gopalakrishnan, G., Rakamaric, Z., Solovyev, A.: Efficient Search for Inputs Causing High Floatingpoint Errors. In: Symposium on Principles and Practice of Parallel Programming (PPoPP). vol. 49, pp. 43–52. ACM (2014)
 [4] Damouche, N., Martel, M., Chapoutot, A.: Intraprocedural optimization of the numerical accuracy of programs. In: Formal Methods for Industrial Critical Systems. pp. 31–46. Springer (2015)
 [5] Damouche, N., Martel, M., Panchekha, P., Qiu, C., SanchezStern, A., Tatlock, Z.: Toward a Standard Benchmark Format and Suite for FloatingPoint Analysis. In: International Workshop on Numerical Software Verification. pp. 63–77. Springer (2016)
 [6] Darulova, E., Horn, E., Sharma, S.: Sound MixedPrecision Optimization with Rewriting. In: International Conference on CyberPhysical Systems (ICCPS) (2018)
 [7] Darulova, E., Izycheva, A., Nasir, F., Ritter, F., Becker, H., Bastian, R.: Daisy  Framework for Analysis and Optimization of Numerical Programs. In: International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS) (2018)
 [8] Darulova, E., Kuncak, V.: Towards a Compiler for Reals. ACM Transactions on Programming Languages and Systems (TOPLAS) 39(2), 8 (2017)
 [9] De Dinechin, F., Lauter, C.Q., Melquiond, G.: Assisted Verification of Elementary Functions using Gappa. In: ACM Symposium on Applied Computing. pp. 1318–1322. ACM (2006)
 [10] De Moura, L., Bjørner, N.: Z3: An efficient SMT solver. TACAS’08/ETAPS’08 (2008), http://dl.acm.org/citation.cfm?id=1792734.1792766
 [11] Goubault, E., Putot, S.: Robustness Analysis of Finite Precision Implementations. In: Asian Symposium on Programming Languages and Systems (APLAS). pp. 50–57. Springer (2013)
 [12] IEEE, C.S.: IEEE Standard for FloatingPoint Arithmetic. IEEE Std 7542008 (2008)
 [13] Leroy, X.: Formal Verification of a Realistic Compiler. Communications of the ACM 52(7) (2009)
 [14] Magron, V., Constantinides, G., Donaldson, A.: Certified Roundoff Error Bounds Using Semidefinite Programming. ACM Transactions on Mathematical Software 43(4), 1–34 (2017)
 [15] Moscato, M., Titolo, L., Dutle, A., Munoz, C.A.: Automatic Estimation of Verified FloatingPoint RoundOff Errors via Static Analysis. In: International Conference on Computer Safety, Reliability, and Security. pp. 213–229. Springer (2017)
 [16] Panchekha, P., SanchezStern, A., Wilcox, J.R., Tatlock, Z.: Automatically Improving Accuracy for Floating Point Expressions. In: Conference on Programming Language Design and Implementation (PLDI) (2015)
 [17] Ramananandro, T., Mountcastle, P., Meister, B., Lethin, R.: A Unified Coq Framework for Verifying C Programs with FloatingPoint Computations. In: Certified Programs and Proofs (CPP). pp. 15–26. ACM (2016)
 [18] RubioGonzález, C., Nguyen, C., Nguyen, H.D., Demmel, J., Kahan, W., Sen, K., Bailey, D.H., Iancu, C., Hough, D.: Precimonious: Tuning assistant for floatingpoint precision. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. pp. 27:1–27:12. SC ’13, ACM (2013)
 [19] SanchezStern, A., Panchekha, P., Lerner, S., Tatlock, Z.: Finding Root Causes of Floating Point Error with Herbgrind. arXiv preprint arXiv:1705.10416 (2017)
 [20] Schkufza, E., Sharma, R., Aiken, A.: Stochastic optimization of floatingpoint programs with tunable precision. In: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation. pp. 53–64. PLDI ’14, ACM (2014)
 [21] Solovyev, A., Jacobsen, C., Rakamarić, Z., Gopalakrishnan, G.: Rigorous Estimation of FloatingPoint Roundoff Errors with Symbolic Taylor Expansions. In: International Symposium on Formal Methods (FM). pp. 532–550. Springer (2015)

[22]
Zou, D., Wang, R., Xiong, Y., Zhang, L., Su, Z., Mei, H.: A Genetic Algorithm for Detecting Significant Floatingpoint Inaccuracies. In: IEEE International Conference on Software Engineering (ICSE). vol. 1, pp. 529–539. IEEE (2015)
Appendix 0.A Experimental Data
In this section we give the experimental results on which all conclusions from Section 3 are based. The experiments were run on a machine with an i74790K CPU and 32GB of memory. For each benchmark we give both Daisy and Herbie a timeout of 2 minutes. To obtain our data we have run the script from Figure 1.
Table 1 and Table 2 contain all benchmarks for which we can compute an endtoend result, i.e. at least one run of Daisy succeeds on both Herbies input and output, and Herbie does not time out.
Table 3 and Table 4 contain all benchmarks for which at least one of our rewritings applied in Section 3 succeds, i.e. either rewriting with only Daisy, only Herbie or with both does not time out and does not raise an alarm. In the table, TO refers to either Herbie or Daisy not computing a result within 2 minutes. SQRTNEG and DIV0 refer to Daisy finding a possible square root of a negative number, respectively division by zero. POW refers to Daisy encountering a power expression where is not an integer value. COND and CBRT refer to Daisy failing because the input contains a conditional, respectively a cube root, as both are currently not implemented in Daisy.
In the following we list the benchmarks for which we did not obtain an endtoend result, sorted by the reason for the failure:

Herbie timeout:
floudas1, hartman6, kepler1, kepler2, triangle7, triangle8, triangle9, triangle10, triangle11, triangle12, odometryX1, odometryY1, odometryX5, pid3, pid5, leadLag3, leadLag5, trapezoid3, trapezoid5, train4_out1, train4_state1, train4_state2, train4_state3, train4_state4, train4_state5, train4_state7, train4_state8, train4_state9 
Daisy reports possible division by zero:
sine, pid1, NMSE problem 3.3.2, NMSE problem 3.3.3, NMSE problem 3.3.6, NMSE p42 positive, NMSE p42 negative, NMSE problem 3.2.1 positive, NMSE problem 3.2.1 negative, NMSE example 3.9, NMSE example 3.10, NMSE problem 3.4.1, NMSE problem 3.4.2, NMSE problem 3.4.4, NMSE problem 3.4.5, NMSE section 3.11 
Herbie introduces cuberoot which is unhandled by Daisy:
arcLength, jetEngine, odometryX3, NMSE example 3.6, NMSE example 3.8 
Daisy reports possible square root of a negative number:
triangle2, triangle3, triangle4, triangle5, triangle6 
Use of noninteger power, unhandled by Daisy:
NMSE problem 3.3.1 
Internal exception by Daisy due to a conditional introduced by Herbie:
NMSE problem 3.3.7
Name  Herbie  Daisy IA  Daisy IAsubdiv  Daisy dReal  Daisy Z3  Daisy Best  Error improvement  

src error  res error  src error  res error  src error  res error  src error  res error  src error  res error  src err  res err  res / src  
logexp  0.0289  0.0289  3.33e12  3.33e12  2.18e13  2.18e13  3.33e12  3.33e12  3.33e12  3.33e12  2.18e13  2.18e13  1.00 
sphere  0.0105  0.0105  1.20e14  1.20e14  1.20e14  1.20e14  1.20e14  1.20e14  1.20e14  1.20e14  1.20e14  1.20e14  1.00 
floudas2  0  0  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15  1.00 
floudas3  0.136  0.136  1.58e14  1.58e14  1.49e14  1.49e14  1.09e14  1.09e14  1.09e14  1.09e14  1.09e14  1.09e14  1.00 
hartman3  0.128  0.12  6.52e10  6.52e10  6.88e14  6.88e14  1.69e13  1.69e13  1.70e13  1.70e13  6.88e14  6.88e14  1.00 
kepler0  0.576  0.423  1.04e13  6.54e14  8.94e14  5.42e14  9.20e14  6.54e14  9.06e14  6.54e14  8.94e14  5.42e14  6.06e01 
doppler1  0.434  0.434  4.19e13  4.19e13  2.38e13  2.38e13  4.19e13  4.19e13  4.19e13  4.19e13  2.38e13  2.38e13  1.00 
doppler2  0.424  0.0415  1.05e12  4.25e13  5.08e13  3.43e13  1.03e12  3.63e13  1.03e12  3.63e13  5.08e13  3.43e13  6.75e01 
doppler3  0.423  0.0459  1.68e13  1.06e13  1.03e13  9.55e14  1.68e13  9.85e14  1.68e13  9.85e14  1.03e13  9.55e14  9.27e01 
rigidBody1  0.00502  0.00577  3.22e13  2.77e13  3.22e13  2.77e13  3.22e13  2.77e13  3.22e13  2.77e13  3.22e13  2.77e13  8.60e01 
rigidBody2  0.0658  0.0658  3.65e11  5.75e13  3.65e11  5.75e13  3.65e11  5.75e13  3.65e11  5.75e13  3.65e11  5.75e13  1.58e02 
turbine1  0.429  0.511  9.49e14  1.58e13  4.24e14  5.84e14  8.87e14  1.34e13  8.87e14  1.34e13  4.24e14  5.84e14  1.38 
turbine2  0.554  0.66  1.39e13  1.12e10  4.58e14  9.35e13  1.23e13  8.83e11  1.23e13  8.83e11  4.58e14  9.35e13  2.04e+01 
turbine3  0.335  0.219  7.07e14  3.26e11  1.86e14  6.96e14  6.27e14  3.26e11  6.27e14  3.26e11  1.86e14  6.96e14  3.74 
verhulst  0.336  0.385  4.74e16  5.40e16  3.59e16  4.26e16  4.18e16  4.85e16  4.18e16  4.85e16  3.59e16  4.26e16  1.19 
predatorPrey  0.412  0.409  2.08e16  2.39e16  1.97e16  2.28e16  2.08e16  2.39e16  2.08e16  2.39e16  1.97e16  2.28e16  1.16 
carbonGas  0.44  0.426  3.91e08  1.66e07  1.40e08  2.54e08  3.35e08  1.60e07  3.35e08  1.60e07  1.40e08  2.54e08  1.81 
sqroot  0.008  0.00275  6.47e16  4.54e16  6.16e16  3.58e16  6.47e16  4.23e16  6.47e16  4.22e16  6.16e16  3.58e16  5.81e01 
sineOrder3  0.00745  0.00745  1.45e15  1.45e15  1.01e15  1.01e15  1.23e15  1.23e15  1.23e15  1.23e15  1.01e15  1.01e15  1.00 
triangle  2.55  0.455  6.53e14  2.52e14  5.58e14  2.15e14  6.05e14  2.44e14  6.04e14  2.44e14  5.58e14  2.15e14  3.85e01 
triangle1  1.73  0.404  SQRT  SQRT  SQRT  SQRT  2.12e11  1.11e11  4.04e10  4.88e11  2.12e11  1.11e11  5.24e01 
bspline3  0.116  0.0891  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16  1.00 
Name  Herbie  Daisy IA  Daisy IAsubdiv  Daisy dReal  Daisy Z3  Daisy Best  Error improvement  

src error  res error  src error  res error  src error  res error  src error  res error  src error  res error  src err  res err  res / src  
pendulum1  0.0194  0.0194  4.61e16  4.61e16  4.61e16  4.61e16  4.61e16  4.61e16  4.61e16  4.61e16  4.61e16  4.61e16  1.00 
pendulum2  0.137  0.137  9.42e16  9.38e16  9.37e16  9.33e16  9.42e16  9.38e16  9.42e16  9.38e16  9.37e16  9.33e16  9.96e01 
analysis1  0.0184  0.0184  1.67e15  1.67e15  9.41e16  9.41e16  1.30e15  1.30e15  1.67e15  1.67e15  9.41e16  9.41e16  1.00 
analysis2  0.0139  0.0139  6.08e14  6.08e14  2.07e15  2.07e15  3.11e15  3.11e15  6.08e14  6.08e14  2.07e15  2.07e15  1.00 
odometryY3  0.548  0.591  1.99e14  1.91e14  1.98e14  1.89e14  1.99e14  1.91e14  1.99e14  1.91e14  1.98e14  1.89e14  9.55e01 
odometryY5  0.602  0.585  4.29e14  3.88e14  4.18e14  3.70e14  4.26e14  3.76e14  4.26e14  3.76e14  4.18e14  3.70e14  8.85e01 
leadLag1  0.48  0  9.75e12  3.20e12  8.84e12  3.20e12  8.84e12  3.20e12  8.84e12  3.20e12  8.84e12  3.20e12  3.62e01 
trapezoid1  5.49  0.365  6.58e09  7.74e11  6.58e09  7.74e11  6.58e09  7.74e11  6.58e09  7.74e11  6.58e09  7.74e11  1.18e02 
introexample  0.022  0.022  1.14e10  1.14e10  2.84e11  2.84e11  1.14e10  1.14e10  1.14e10  1.14e10  2.84e11  2.84e11  1.00 
sec4example  0.404  0.305  2.49e09  4.99e16  1.43e10  2.86e16  2.49e09  4.99e16  2.49e09  4.99e16  1.43e10  2.86e16  2.00e06 
test01_sum3  0.361  0.144  4.00e15  1.89e15  3.55e15  1.83e15  3.55e15  1.89e15  3.55e15  1.89e15  3.55e15  1.83e15  5.15e01 
test02_sum8  0.416  0.349  7.99e15  7.55e15  7.99e15  7.55e15  7.99e15  7.55e15  7.99e15  7.55e15  7.99e15  7.55e15  9.45e01 
test03_nonlin2  0.0362  0.0415  4.75e14  2.23e11  1.28e14  7.56e13  4.67e14  2.23e11  4.67e14  2.23e11  1.28e14  7.56e13  5.91e+01 
test04_dqmom9  0.628  0.499  2  4.21e+04  1.55  1.20e+04  2  4.21e+04  2  4.21e+04  1.55  1.20e+04  7.74e+03 
test05_nonlin1 r4  0.776  0.343  3.89e06  1.67e16  6.32e07  1.67e16  3.89e06  1.67e16  3.89e06  1.67e16  6.32e07  1.67e16  2.64e10 
test05_nonlin1 test2  0.325  0.325  1.39e16  1.39e16  1.39e16  1.39e16  1.39e16  1.39e16  1.39e16  1.39e16  1.39e16  1.39e16  1.00 
test06_sums4 sum1  0.00425  0.00425  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.00 
test06_sums4 sum2  0.00363  0.00363  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15  1.00 
NMSE example 3.1  50.1  0.435  0.041  6.49e10  0.0102  3.09e10  0.041  6.49e10  0.041  6.49e10  102e02  3.09e10  3.03e08 
NMSE example 3.3  28.9  0.00575  1.33e15  1.50e25  7.77e16  1.50e25  1.11e15  1.50e25  1.33e15  1.50e25  7.77e16  1.50e25  1.93e10 
NMSE example 3.4  61.9  0  2.22e+04  1.29e26  2.22e+04  1.29e26  2.22e+04  1.29e26  2.22e+04  1.29e26  2.22e+04  1.29e26  5.81e31 
NMSE problem 3.3.5  38.9  0.0404  8.88e16  1.50e25  7.77e16  1.50e25  6.67e16  1.50e25  8.88e16  1.50e25  6.67e16  1.50e25  2.25e10 
NMSE example 3.7  60  0  2.22e16  1.29e26  2.22e16  1.29e26  2.22e16  1.29e26  2.22e16  1.29e26  2.22e16  1.29e26  5.81e11 
NMSE problem 3.4.3  60  0  3.33e16  5.17e26  3.33e16  5.17e26  3.33e16  5.17e26  3.33e16  5.17e26  3.33e16  5.17e26  1.55e10 
NMSE section 3.5  29.4  29.4  2.22e16  2.22e16  2.22e16  2.22e16  2.22e16  2.22e16  2.22e16  2.22e16  2.22e16  2.22e16  1.00 
Name  Baseline Daisy without rewriting  Different Rewritings applied  

Daisy  Herbie  Both  minimum  
logexp  3.33e12  2.18e13  3.33e12  2.18e13  2.18e13 
sphere  1.20e14  1.17e14  1.20e14  1.17e14  1.17e14 
floudas1  3.01e13  3.38e13  TO  TO  3.38e13 
floudas2  1.11e15  1.11e15  1.11e15  1.11e15  1.11e15 
floudas3  1.09e14  1.09e14  1.09e14  1.31e14  1.09e14 
hartman3  1.70e13  TO  1.70e13  6.92e14  6.92e14 
kepler0  9.06e14  5.93e14  TO  TO  5.93e14 
kepler1  4.25e13  2.70e13  TO  TO  2.70e13 
kepler2  2.19e12  1.54e12  TO  TO  1.54e12 
doppler1  4.19e13  1.62e13  1.91e13  1.47e13  1.47e13 
doppler2  1.03e12  3.03e13  3.63e13  2.84e13  2.84e13 
doppler3  1.68e13  7.97e14  9.85e14  7.17e14  7.17e14 
rigidBody1  3.22e13  2.24e13  TO  TO  2.24e13 
rigidBody2  3.65e11  2.94e11  3.38e11  3.35e11  2.94e11 
turbine1  8.87e14  4.02e14  9.02e14  3.97e14  3.97e14 
turbine2  1.23e13  4.51e14  1.22e13  4.50e14  4.50e14 
turbine3  6.27e14  1.73e14  3.27e11  1.66e14  1.66e14 
verhulst  4.18e16  3.59e16  4.85e16  4.00e16  3.59e16 
predatorPrey  2.08e16  1.87e16  5.92e16  1.87e16  1.87e16 
carbonGas  3.35e08  1.20e08  3.12e08  1.11e08  1.11e08 
sine  6.95e16  4.86e16  DIV0  DIV0  4.86e16 
sqroot  6.47e16  2.35e16  4.22e16  2.35e16  2.35e16 
sineOrder3  1.23e15  1.00e15  1.23e15  1.00e15  1.00e15 
triangle  6.04e14  5.58e14  2.44e14  2.08e14  2.08e14 
triangle1  4.04e10  SQRTNEG  1.01e06  DIV0  1.01e06 
triangle2  SQRTNEG  SQRTNEG  6.31e10  SQRTNEG  6.31e10 
bspline3  1.06e16  1.06e16  1.06e16  1.06e16  1.06e16 
pendulum1  4.61e16  4.61e16  4.61e16  4.60e16  4.60e16 
pendulum2  9.42e16  9.36e16  9.46e16  9.37e16  9.36e16 
analysis1  1.67e15  9.41e16  1.67e15  9.41e16  9.41e16 
analysis2  6.08e14  2.07e15  6.57e14  TO  2.07e15 
odometryX1  6.40e15  TO  4.97e15  TO  4.97e15 
odometryY1  2.56e15  TO  2.03e15  TO  2.03e15 
odometryX3  2.23e14  TO  2.24e14  TO  2.24e14 
odometryY3  1.99e14  TO  2.15e14  TO  2.15e14 
pid1  2.17e15  2.17e15  FN/cbrt  FN/cbrt  2.17e15 
leadLag1  8.84e12  8.84e12  3.20e12  3.20e12  3.20e12 
trapezoid1  6.58e09  TO  5.14e11  3.68e11  3.68e11 
trapezoid5  6.64e09  TO  1.26e10  7.15e11  7.15e11 
introexample  1.14e10  2.84e11  DIV0  DIV0  2.84e11 
Name  Baseline Daisy without rewriting  Different Rewritings applied  

Daisy  Herbie  Both  minimum  
sec4example  2.49e09  1.43e10  2.49e09  1.48e10  1.43e10 
test01_sum3  3.55e15  2.28e15  1.89e15  1.83e15  1.83e15 
test02_sum8  7.99e15  6.66e15  7.55e15  6.66e15  6.66e15 
test03_nonlin2  4.67e14  1.28e14  4.67e14  1.28e14  1.28e14 
test04_dqmom9  2  1.55  4.21e+04  6.66e01  6.66e01 
test05_nonlin1 r4  3.89e06  6.32e07  1.67e16  1.39e16  1.39e16 
test05_nonlin1 test2  1.39e16  1.39e16  2.78e16  2.22e16  1.39e16 
test06_sums4 sum1  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15 
test06_sums4 sum2  1.33e15  1.33e15  1.33e15  1.33e15  1.33e15 
NMSE example 3.1  0.041  1.02e02  3.07e12  2.30e12  2.30e12 
NMSE example 3.3  1.33e15  7.77e16  1.50e25  1.39e25  1.39e25 
NMSE example 3.4  22200  2.22e+04  1.29e26  9.69e27  9.69e27 
NMSE example 3.6  4.10e12  1.02e12  DIV0  DIV0  1.02e12 
NMSE problem 3.3.1  8.19e17  2.05e17  POW  POW  2.05e17 
NMSE problem 3.3.2  DIV0  DIV0  1.29e26  1.29e26  1.29e26 
NMSE problem 3.3.3  1.64e16  4.10e17  DIV0  DIV0  4.10e17 
NMSE problem 3.3.5  8.88e16  7.77e16  1.50e25  1.39e25  1.39e25 
NMSE problem 3.3.6  8.19e07  2.05e07  DIV0  DIV0  2.05e07 
NMSE problem 3.3.7  9.12e16  9.66e16  COND  COND  9.66e16 
NMSE example 3.7  2.22e16  2.22e16  1.29e26  1.29e26  1.29e26 
NMSE example 3.8  1.64e+14  6.83e+12  FN/cbrt  FN/cbrt  6.83e+12 
NMSE example 3.9  DIV0  DIV0  1.14e26  1.14e26  1.14e26 
NMSE example 3.10  DIV0  DIV0  2.22e16  1.11e16  1.11e16 
NMSE problem 3.4.3  3.33e16  3.33e16  5.17e26  3.88e26  3.88e26 
NMSE problem 3.4.4  DIV0  DIV0  3.47e16  2.68e16  2.68e16 
NMSE problem 3.4.5  DIV0  DIV0  1.11e16  5.55e17  5.55e17 
NMSE section 3.5  2.22e16  2.22e16  3.55e36  2.80e36  2.80e36 
NMSE section 3.11  DIV0  DIV0  6.46e+13  6.46e+13  6.46e+13 
train4_out1  4.28e10  3.19e10  4.28e10  3.19e10  3.19e10 
train4_state1  6.66e15  1.33e15  TO  TO  1.33e15 
train4_state2  7.11e15  8.90e16  TO  TO  8.90e16 
train4_state3  6.66e15  8.90e16  TO  TO  8.90e16 
train4_state4  6.22e15  1.33e15  TO  TO  1.33e15 
train4_state5  1.23e14  4.40e15  TO  TO  4.40e15 
train4_state6  1.13e14  4.22e15  TO  TO  4.22e15 
train4_state7  1.04e14  4.22e15  TO  TO  4.22e15 
train4_state8  9.55e15  4.22e15  TO  TO  4.22e15 
train4_state9  8.66e15  3.33e15  TO  TO  3.33e15 
Comments
There are no comments yet.