There is an increasing demand to support materials research in the development of novel or improved applications with advanced strategies. Unfortunately, nowadays, manufacturing processes in a vast range of applications in areas such as automotive, aerospace, microengineering, telecommunications, biotechnologies, microfluidics, photovoltaics, is still performed using expensive trial and error approaches Chitale and Gupta (2013). Thus, conventional manufacturing strategies are expected to lead to financial risks and inhibit competitiveness. Although, technological advances and software engineering have been extensively used by manufacturing companies to provide predictive tools in various fields of engineering (i.e. aerospace DorMohammdi et al. (2017), automotive Del Re et al. (2010), etc.), such instruments for advanced processing of materials has not yet been developed.
A very promising and high-resolution material machining process is performed via using lasers, which are proving to be ideal tools for controlling the energy deposition and respective modifications on the surface, or volume, of a material. In particular, material processing with femtosecond (fs) pulsed lasers has received considerable attention due to the fact that it is related to a high precision, rapid energy delivery and minimisation of the heat affected area. Direct fs-laser surface micro-and nano-patterning has been demonstrated in many types of materials including semiconductors, metals, dielectrics, ceramics, and polymers. Therefore, a fs-based technology is used to an abundance of diverse applications ranging from micro-device fabrication to optoelectronics, microfluidics and biomedicine Vorobyev and Guo (2013); Zorba et al. (2008); Bäuerle (2013); Papadopoulou et al. (2010); Simitzi et al. (2015); Papadopoulos et al. (2019); Stratakis (2012); Stratakis et al. (2020). These applications require a thorough knowledge of the fundamentals of laser interaction with the target material for enhanced controllability of the resulting modification of the target relief. Physical mechanisms that lead to surface modification have been extensively explored both theoretically and experimentally Tsibidis et al. (2012, 2015a, 2014); Derrien et al. (2013); Skolski et al. (2012); Bonse et al. (2012); Garrelie et al. (2011); Wang and Guo (2005); Huang et al. (2009); Bonse and Krüger (2010); Tsibidis et al. (2016); Shimotsuma et al. (2003); Chichkov et al. (1996); Tsibidis and Stratakis (2017, 2020).
Materials modelling has been a powerful tool that provides key information for tailoring and designing materials or even identifying new materials, providing a cost-effective method and minimising the use of trial and error approaches aiming to reduce the need for an increasing number of experiments. The use of materials modelling in industries is very versatile and it can offer a solution towards controlling the output procedure through a systematic exploration of the, usually, complex physical (multiscale) processes that occur during the manufacturing processes that occur during the manufacturing process Tsibidis et al. (2012, 2015a); Tsibidis and Stratakis (2017); Rethfeld et al. (2017); Ivanov et al. (2010); Lin et al. (2008). In principle, the optimum fabrication conditions can be identified via the computational execution of self-consistent virtual experiments. Nevertheless, despite the significance originated from the use of materials modelling, the complexities due to the need of a large number of simulated experiments in which multiscale physical models are involved downgrade the benefit of the methodology as it leads to slow decision making. Therefore, towards improving the decision making process and reduce the production time, further tools are required, based on machine learning-based methods.
A number of predictive modelling approaches based on Machine Learning (ML) techniques were used for material processing. In Raccuglia et al. Raccuglia et al. (2016), an ML model was trained on datasets of ‘failed’ experiments (data that were archived in notebooks stemming from unsuccessful experiments) to predict reaction outcomes for the crystallization of templated Vanadium selenites. Oliynyk et al. Oliynyk et al. (2016) used ML models to study Heusler compounds and properties. In another study, ML methodology was used in laser-based manufacturing to improve geometric accuracy of the fabricated parts Francis and Bian (2019)
. On the other hand, an increase of the accuracy and need for prediction of distortion quantification in the fabricated materials was further facilitated with the employment of a Deep Learning approach. Moreover, Tani and KobayashiTani and Kobayashi (2018) performed a big-data analysis to describe how surface morphology affects the laser ablation process. More specifically, a comparison of a produced 3D depth profile before and after single-shot ablation from thousands of data for various materials, they observed and modeled hysteresis behavior. In another work by Mills et al. Mills et al. (2018); Heath et al. (2018)
, used a neural network-based approach to explore the morphology features of an induced 3D surface profile of a substrate after being laser machined with a single laser pulse, for random laser spatial intensity profiles. On the other hand, Agrawal et al.Agrawal et al. (2014) predicted the fatigue strength of steels. Both physics-based and data-driven approaches were used to correlate properties of alloys and manufacturing process parameters. In that study, data-driven models through extrapolation were able to sample extreme value properties, where the current state-of-the-art physics-based models suffer from severe limitations .
Inspired by the above studies and challenges, we propose to complement laser manufacturing processing and discovery with data-driven analysis. Existing corpora of experimental and simulated measurements constitute a valuable collection of information which remains mostly unexploited when new materials are investigated. Predictive modelling through the utilization of statistical and machine learning models offers an efficient approach to encode the accumulated experience and knowledge from previous experiments into a mathematical model and, subsequently, be able to extrapolate into unexplored conditions. ML models are trainable parametric models that aim to perform a learning task such as classification or regressionDuda et al. (2001); Bishop (2007). An ML model can be abstractly regarded as a function or a mapping that transforms the input to an output. In order to accurately learn the mapping between the input variables (e.g., laser fabrication conditions) and the output property (e.g., the observed material structure as labeled by the experts), an optimization procedure is defined and solved.
ML has already revolutionised research fields such as computer vision, speech recognition and natural language processing. The recent success of ML and especially of artificial neural networks stems from their ability to produce super-human performance on tasks where labeled data are abundantGoodfellow et al. (2016). Unfortunately, a large portion of datasets in various scientific and engineering fields have relatively small sizes making the use of data-hungry approaches challenging if not prohibitive due to high generalization errors. As already mentioned, collecting experimental data and to some extent simulated data in laser manufacturing is costly both in time and budget hindering the effort to efficiently and automatically morph the desired structural material properties. Furthermore, in order to reliably extrapolate the existing knowledge to unknown parameter regimes or novel materials, it is important to develop models that are robust. In principle, simpler ML models are expected to be more robust and they often transfer to unseen parameter regimes. In contrast, complex ML models with no induced constraints not only require more data to be sufficiently trained without being overfitted but also they tend to produce completely off forecasts when applied to an unseen parameter regime.
In this paper, we aim to train and evaluate a series of predictive models with different levels of complexity and expression. Following the Occam’s razor principle, we search for the simplest ML models which do not compromise in terms of predictive performance. The studied ML models are trained and evaluated on three materials (two metals and one semi-conductor) where both experimental and simulated data are available. Performance results indicate that simpler predictive models with a curated and well-educated preprocessing step enjoyed the highest accuracy. This result is a consequence of the amount of training data, which is a crucial factor. In a second stage, the experimental data are further augmented with simulation data extracted from multiscale modeling on the physical processes upon laser-material interaction. This possibility further enabled the study of the effect of sample size on predictive performance, upon retraining the ML models. We observe that the use of a larger sample size, especially, at or close to the transition boundaries, significantly benefits the predictive models’ accuracy. Finally, we propose to quantify the uncertainty regions where the predictive models are not certain about the outcome. Since the studied ML models generate a probability distribution of the structures for each sample, uncertainty quantification is calculated using the information entropy functionShannon (1948); Cover and Thomas (2006). The inverse of the information entropy is used as a measure of certainty.
This paper is organised as follows: in Section II, a basic description of the fundamental physical processes that induce surface modification on solids upon irradiation with fs laser pulses is presented. Section III presents the datasets, the preprocessing steps applied to the input and it briefly introduces and discusses the main characteristics of the predictive models trained and evaluated in this study. The obtained results are demonstrated, analysed and discussed in Section IV while conclusions are summarised in Section V.
Ii Materials Modeling
ii.1 Laser-induced periodic surface structures
The employment of ultra-short pulsed laser sources for material processing has received considerable attention over the past decades due to the important technological applications, particularly in industry and medicine Vorobyev and Guo (2013); Zorba et al. (2008); Bäuerle (2013); Papadopoulou et al. (2010); Simitzi et al. (2015); Papadopoulos et al. (2019); Stratakis (2012)
. There is a plethora of surface structures generated by laser pulses while the so-called laser-induced periodic surface structures (LIPSS) on solids that have been studied extensively are related to those applications. A range of LIPSS types have been produced based on the laser parameters and the irradiated material. According to the morphological features of the induced surface structures such as their periodicity and orientation, LIPSS can be classified in: (a) High Spatial Frequency LIPSS (HSFL), (b) Low Spatial Frequency LIPSS (LSFL), (c) Grooves, (d) Spikes, and (e) complex ones. The LIPSS fabrication technique as well as the associated laser driven physical phenomena have been the topic of an extensive investigation. This is due to the fact that the technique constitutes a precise, single-step and scalable method to fabricate highly ordered, multi-directional and complex surface structures that mimic the unique morphological features of certain species found in nature, an approach which is usually coined as biomimetics. A thorough knowledge of the fundamental mechanisms that lead to the LIPSS formation provides the possibility of generating numerous and unique surface biomimetic structuresZorba et al. (2008); Papadopoulos et al. (2019); Skoulas et al. (2017); Kirner et al. (2017); Müller et al. (2016); Baron et al. (2018) for a range of applications including tribology, tissue engineering, advanced optics (for a review on LIPSS and potential applications, see Refs.Bonse et al. (2016); Stratakis et al. (2020)).
ii.2 Modelling LIPSS
Various mechanisms have been proposed to account for the development of LIPSS Bonse et al. (2016); Stratakis et al. (2020): interference of the incident wave with an induced scattered far-field wave, or with a surface plasmon wave (SPW), or due to self-organisation mechanisms. Laser irradiation of solids with ultrashort-pulses involves a series of multiscale processes while the type of structures that are induced is dependent on the laser energy and the energy dose (i.e. number of pulses, ). To describe surface modification for semiconductors and metals, which is the scope of the present study, it is noted that the first pulse leads to the formation of a crater with a rimmed region at the edges for =1 as a result of possible mass removal (i.e. ablation) and mass displacement Tsibidis et al. (2012). To evaluate the role of the laser parameters in the surface modification processes, a multiscale modelling approach is required. In principle, modules that account for (i) Laser energy absorption, (ii) Electron excitation, (iii) Heat transfer, (iv) Phase transformation, (v) Resolidification have to be incorporated in the model. These processes occur at different timescales and therefore, to describe, appropriately, the system dynamics, the use of current or revised theoretical models need to be employed. More specifically, the laser energy absorption can be described by approximate solutions Tsibidis et al. (2012) or through a more accurate description of the propagation of the electromagnetic wave of the laser beam Rudenko et al. (2016). Electron excitation and dynamics can be modelled through the employment of models based on Boltzmann transport equations Rethfeld et al. (2017) or through the use of density functional theory Tsibidis et al. (2020); Marini et al. (2009) while relaxation processes and transfer of the energy of the electron subsystem to the material lattice are described through well-established two temperature models Anisimov et al. (1974). Finally, phase transformation which includes either a transition from solid to the liquid phase and versa or elastic/plastic deformation of a part of the material can be described either through Navier-Stokes Tsibidis et al. (2012) or elastodynamics equations Tsibidis and Stratakis (2017); Tsibidis et al. (2018), respectively, or from more advanced methods based on Molecular dynamics Ivanov et al. (2010) (for a detailed description of the various modules of the multiscale model see Tsibidis et al. (2012)). Nevertheless, while the aforementioned models are aimed to provide consistent solutions, the picture becomes more complex due to potential limitations of the validity of each particular model. Thus, the use of particular models is especially sensitive to the laser parameter values (i.e. fluence, pulse duration, laser wavelength, polarisation state, material) while coupling of individual modules on each temporal regime, again, is possible to lead to inconsistent and incorrect results.
While the multiscale model presented above is, in principle, capable to provide the mechanism for surface patterning, appropriate modifications are required to describe accurately morphological changes at increasing (or fluence). More specifically, irradiation of a non-flat profile as a result of irradiation with =1, leads to an interference of the incident beam with a scattered wave (it could involve the excitation of Surface Plasmon wave, SP) resulting, in turn, into a spatially modulated energy deposition on the surface of the material Tsibidis et al. (2012). The spatially modulated form of the energy distribution yields a periodic shape (with periodicity equal to the SP wavelength Tsibidis et al. (2012); Derrien et al. (2013); Bonse et al. (2009)) that is projected firstly, onto the excited electron dynamics, electron temperature and lattice temperature and finally to the longer timescale effects related to melting and resolidification of the irradiated material Bonse et al. (2016); Stratakis et al. (2020)
. As a result, a surface profile covered with LIPSS of periodicity of the size of the wavelength will be created . These structures are orientated perpendicularly to the polarisation vector of the laser beam and are termed as LSFL (for the sake of simplicity, in this report, they will be calledTsibidis et al. (2012)). As increases or at higher energies (i.e. fluences, ), the profile becomes deeper and the height of the ripple crest increases; various theoretical models have been developed to account for the experimentally observed decrease of the periodicity of LIPSS Huang et al. (2009); Tsibidis and Stratakis (2017). A saturation point is reached when SP excitation ceases after a large number of ; afterwords, melting of the material leads to a fluid transport along the well of the rippled zone leading to another type of structures, the Tsibidis et al. (2015a). These structures are orientated parallel to the orientation of the laser beam while their periodicity is larger than the laser wavelength. Further irradiation with larger is expected to lead to pointed structures, that are termed Tsibidis et al. (2015a). Theoretical results based on multiscale modelling show the fluid transport that leads to the formation of ripples (Fig. 1(a)) Tsibidis et al. (2012, 2015a) while convention roll development and movement account for the formation of grooves (Fig. 1(b)) Tsibidis et al. (2015a, 2018). Finally, spike formation is also explained through the employment of hydrodynamical models (Fig. 1(c)) Tsibidis et al. (2015a).
ii.3 Experimental setup
Laser irradiation is performed with the use of an Yb:KGW Pharos – SP laser system from Light Conversion that emits linearly polarized IR pulses at 1026 nm central wavelength at 1 kHz repetition rate of 170 fs pulse width. The samples used in the experimements were specimen of stainless steel ( 1.7131), alpha-beta Titanium alloy (Ti6Al4V) and crystalline Silicon (Si) (-doped). The laser beam was guided using silver mirrors and focused on the sample surface with an focal length
=200 mm plano convex lens. The spot size was characterized with a CCD camera close to the focal plane and was estimated aroundand consistent with a Gaussian intensity profile. Irradiation was performed within the Rayleigh range of the focal position and the number of pulses receptive to the sample for static irradiation were controlled with an external mechanical shutter. The laser power was modulated from the laser amplifier settings and all irradiation experiments were performed at normal incidence. Fig.2 shows Scanning Electron Microscopy (SEM) images for the surface patterns obtained for stainless steel (left column) and Silicon (right column) in which formation of ripples, grooves or spikes in the central region is visible. To emphasise on the role of and in the formation of various types of structures, the impact of both the fluence and energy dose is illustrated in Fig.2(a) (=2, =1.5 J/cm), Fig.2(b) (=40, =1.5 J/cm), Fig.2(c) (=80, =1.5 J/cm), Fig.2(d) (=80, =0.4 J/cm), Fig.2(e) (=80, =0.7 J/cm), and Fig.2(f) (=80, =1.5 J/cm),
ii.4 Materials modelling simulation approach
A common approach followed to solve the set of equations constituting the multiscale model (describing laser energy absorption, electron excitation, heat transfer and relaxation processes, hydrodynamics, resolidification and elastoplasticity) Tsibidis et al. (2015a, 2012) is the employment of a staggered grid finite difference method which is found to be effective in suppressing numerical oscillations Tsibidis et al. (2015a, 2012, b). Unlike the conventional finite difference method, temperatures (heat transfer equations), electron densities, and pressure are computed at the centre of each element while time derivatives of the displacements and first-order spatial derivative terms are evaluated at locations midway between consecutive grid points. For time-dependent flows, a common technique to solve the Navier-Stokes equations (for fluid transport) is the projection method and the velocity and pressure fields are calculated on a staggered grid using fully implicit formulations Tsibidis et al. (2015a, 2012, b) . On the other hand, the horizontal and vertical velocities are defined in the centres of the horizontal and vertical cells faces, respectively. A multiple pulse irradiation scheme is required to derive the surface relief Tsibidis et al. (2015a, 2012, b) and therefore after each , the induced profile is used to compute the energy absorption and dynamics when the next pulse irradiates the material. To simulate the multiscale process, a numerical scheme based on the use of a finite difference method is followed while the discretisation of time and space has been chosen to satisfy the Neumann stability criterion. It is assumed that on the boundaries, von Neumann boundary conditions are satisfied and heat losses at the front and back surfaces of the material are negligible. The initial conditions are both the electron and lattice temperatures are at room temperature. To simulate mass removal, it is assumed that it occurs if the material is heated above a critical temperature. The boiling temperature of the material is selected as the critical temperature. Simulations were conducted for all four materials. The simulated surface patterns for =10 (ripples) and =100 (grooves) for Silicon are illustrated in Fig. 3(a) and Fig. 3(c), respectively, for =1.4 J/cm while at lower (i.e. =60) the distinction between groove formation and ripple suppression in the central region is not clear (Fig. 3(b)). The latter structures resemble the early stage of groove formation. Similar results have been produced for the three other materials. Finally, Fig. 3(d) illustrates spike formation for =400.
Iii Predictive Modeling
In this section, we describe the annotated datasets and their statistical properties as well as the proposed preprocessing steps. We also present several standard ML models along with their advantages and disadvantages.
iii.1 Material Datasets
A library of structure types and samples is acquired as a function of and . Figs. 4 and 5 show the distributions of the annotated (i.e., labeled with the structure type) experimental and simulated data, respectively, for three materials. The dependence of the structure types on the laser conditions is clear in all cases. Moreover, a comparison of the structure maps produced with simulations and experiments show that more types of classification are observed during experiments (i.e. termed as ’roughness’ or ’no structures’) which are not predicted in simulations. Furthermore, the maps indicate that the onset of one type of structure does not occur for the same combination of the fluence and the energy dose. These effects can be attributed to the fact that the physical models that are used to describe the underlying processes aim to approximately account for the physical mechanisms that take place; the validity of those models are, thus, characterised by limitations and revised theoretical frameworks might be required. On the other hand, laser patterning conditions in an experimental protocol cannot be easily specified to agree with the values selected in the theoretical models. This ambiguity is, thus, expected to influence the energy deposition and the laser absorbed energy which is critical for the production of the structure types.
It is also evident that a significant number of experiments (or simulations) is required for the accurate determination of the structures. Tables 1 & 2 report the number of samples per structural category per material for the experimental and simulation datasets, respectively. The small number of experimental samples cannot be ignored during the selection and training of predictive models. Finally, it is worth-noting that special attention is required around the boundary regions where the transition from one structure to another occurs (i.e. these hybrid states that are described in the previous sections are illustrated as white zones between regions). Hence, an improved precision requires a larger number of time consuming simulations and/or experiments. Therefore, the development of an alternative, systematic, less computational expensive and reliable predictive tool is needed to determine the structure regions.
iii.2 Feature Construction
Despite the development of several general purpose ML models during the last decades, not all models are appropriate for all data collections since there are trade-offs between dataset size and complexity as well as between accuracy and robustness or transferability to new data. Especially, in applications where data are relatively scarce and expensive to generate either less complex models have to be selected or experimental data must be supplemented with simulated data. Another important yet usually hidden aspect of ML success is the proper preparation and preprocessing of the data. Even though ML models are built to work for any type of data distributions, the results are often significantly improved by simple transformation and/or normalization of the data. For instance, various control parameters in many physical/chemical/material systems take values that range in several orders of magnitude. The utilisation of nonlinear transformations such as the power transformation or the application of the logarithmic function regularises the statistical properties of the parameters towards a Gaussian distribution assisting the performance of predictive models. In this paper, we apply the logarithmic transformation to the number of pulses,, while we keep unchanged the fluence parameter, .
Remembering that a predictive model is a complex mapping between the input and the output, this mapping describes the nonlinear relationships and interactions between the input variables and the output. On the other hand, it is often beneficial to reduce the complexity of the mapping, hence, of the model whenever this is possible by applying feature construction. A standard approach to construct new features from existing ones is by taking the product or the ratio between two or more features. In this work, we take the product between two and three features and construct the quadratic as well as the cubic terms. Given that the initial feature vector has two elements (logarithm of and ), there are three quadratic and four cubic terms resulting in three datasets per material with two, five and nine features, respectively.
iii.3 Machine Learning Models
We will describe, briefly, the features of some representative predictive models and their characteristic properties. Typically, a predictive model has a trainable set of coefficients (or parameters),
, and it approximates the posterior probability distributionof each label, , given the sample, Murphy (2013). The label corresponds to the material structure while the sample corresponds to the fabrication conditions (or configuration). After training, the structure prediction for a new configuration, , is obtained from
where is the learned coefficient vector. We start the demonstration with the simpler models and then proceed with more advanced ones aiming to clarify which models are appropriate based on their ability to learn from small sample sizes, their complexity and interpretability.
Nearest Neighbors (-NN). -NN is an instance-based classification method where a new instance (i.e., sample) is compared with existing instances already available in the training set Duda et al. (2001); Bishop (2007). -NN decides the label of new samples from the labels’ occurrence frequency of the closest neighboring samples. There is no training involved in -NN and the user has to specify the number of neighbors, , and define the distance (or similarity metric) between the samples. The distance must take into account the type of the features (discrete, ordinal, continuous, etc.). The performance of
-NN is sensitive to the chosen distance and it might be heavily deteriorated by an poorly-behaved distance. For instance, features with high variance could dominate the distance value resulting in bad performance. Thus, data preprocessing techniques such as feature normalization is typically required.-NN serves as a standard baseline model when the number of samples is low, however, it does not scale well with the feature dimension or for large datasets where the inference is proportional to the sample size making -NN very slow.
. Under this strong independence assumption, the posterior probability is straightforwardly calculated using the Bayes theorem. GNB additionally assumes that the conditional univariate random variables follow the normal distribution parametrized by the mean and the variance. GNB’s parameter estimation, which is fast, is typically performed using the maximum likelihood method. The simplicity of GNB model often results in inferior results however it has better generalization performance when the independence assumption stands true. GNB is also utilized as a baseline model that offers a point of reference for more sophisticated ML approaches.
. It belongs to the family of generalized linear models where the output is a non-linear function of a linear combination of the inputs. The non-linear function known as the logit function (from logistic unit) is statistics outputs the probability of the input to belong to one of the two classes. The regression coefficients are estimated using maximum likelihood estimation. There are no hyper-parameters for tuning making LRM easier to train. The LRM formulation can be straightforwardly generalized to multi-labeled classification problems.
Moreover, LRM is interpretable in the sense that the regression coefficient contain information about the importance of the respective feature. This is an important advantage over most ML models which are treated as black boxes since it can lead to knowledge discovery. Enhanced interpretability is achieved when maximum likelihood estimation is supplemented with a regularization term that favors parsimonious models Tibshirani (1996); Zou and Hastie (2003). Sparse models can reveal the dominant and relevant features that correlate with the outcome.
Support Vector Machines (SVM).
An SVM classifier aims to separate the data points into two classes by constructing a maximum-distance hyperplaneCortes and Vapnik (1995); Schölkopf and Smola (2002); Cristianini and Shawe-Taylor (2000). The data points that participate in the construction of the hyperplane are called support vectors. Utilizing the kernel method, data are transformed in high or even infinite dimensions where the optimal boundary is searched for. The optimization problem is solved using quadratic programming Schölkopf and Smola (2002). The user has to specify which kernel is most appropriate for the task at hand as well as the soft margin parameter that controls how much misclassification error is allowed. SVM training scales as a function of sample size with an order somewhere between quadratic and cubic making them computationally expensive for large datasets. Nevertheless, SVMs frequently produce state-of-the-art results for low sample size datasets Caruana and Niculescu-Mizil (2006).
Gradient Boosting Classifier (GBC).Gradient Boosting is an ML technique that iteratively and gradually learns the classification task from many weak classifiers Freund and Schapire (1995); Friedman (2000). Typically, the weak classifier is a classification and regression tree Breiman et al. (1984)
which represents a decision making process. GBC builds a bag of trees one at a time, where each new tree aims to eliminate errors made by the already trained trees and at the end it produces an ensemble model determined by the weak classifiers. Gradient boosting is an optimization problem in the sense that a loss function is calculated and optimised in each stepChen and Guestrin (2016). The typical loss corresponds to a function of the residual errors (the difference between actual value and predicted value) which is iteratively minimized. GBC is sensitive to noise in the data and relatively harder to tune, however, it has generated excellent results in a wide spectrum of ML tasks Caruana and Niculescu-Mizil (2006) and it is one of the most popular ML techniques.
Additional ML Models. The resurrection of artificial neural networks in the recent years has been attributed to their excellent performance both as feature constructors and as powerful classifiers Goodfellow et al. (2016). The potential of neural nets in laser manufacturing is unquestionable especially when the target is to classify material properties from raw data such as microscope images. However, the learning task we study here does not require models with such large learning capacity as neural nets. Nevertheless, we present a series of results with neural nets in Appendix showing that they do learn the studied classification task given that we provide sufficiently many samples for training.
Finally, another popular ensemble model forth mentioning is random forestsBreiman (2001)
whose building blocks are decision trees as with GBC. However, there are significant differences between GBC and random forests in terms of how the trees are built as well as on how the results of each tree are combined. Despite producing state-of-the-art results in several classification tasks over the years, random forests often perform worse than GBC when the hyper-parameters of GBC are carefully tunedCaruana and Niculescu-Mizil (2006).
The performance assessment of ML models on their ability to correctly predict the material structure given a laser configuration is presented in this Section. We will show the importance of feature construction in both experimental and simulated data as well as the fact that measuring and annotating sufficient amounts of data plays a crucial role for excellent predictive performance.
iv.1 Performance metrics and evaluation protocol
We use as a metric of performance the Area Under the ROC111ROC: Receiving Operating Characteristics from detection theory Egan (1975). Curve (AUC) which is insensitive to the number of instances per category. Since the classification tasks are multiclass and not binary classification tasks, we choose to convert them into “One vs Rest” binary classification problems where the first class corresponds to data from one label while we merge the remaining data to obtain the second class. Then, we compute the true positive and false positive rates for each class. In order to construct the overall ROC curve for a material, we compute the average for all classes.
We follow the standard cross validation (CV) protocol and split the data into training and testing datasets. CV is necessary in order to obtain unbiased results. We apply -fold CV where each dataset is split into subsets and of them are used for training and the remaining for performance assessment. The procedure is repeated times –one time for each subset– and the average performance is calculated and reported. Due to the fact that the experimental data are limited, we set where is the sample size of the dataset, a technique known as Leave One Out CV (LOOCV). We apply 6-fold CV for the simulated datasets. Finally, we performed limited hyper-parameter tuning and the optimal hyper-parameter values are shown in Table 3.
iv.2 Predictive performance of constructed features
We first assess the performance of the constructed features in terms of average AUC for the LR model. The use of logistic regression is preferred over nonlinear models such as SVMs because we want to avoid imperil with model biases the predictive power of the additional features (see also Supplementary Materials ). Fig. 6 shows the ROC curves for linear (magenta lines), linear+quadratic (green lines) and linear+quadratic+cubic (blue lines) features for Silicon. The dotted orange line corresponds to the random classifier. Both experimental (left panel) and simulated (right panel) datasets are considered. In both cases, the constructed features assisted the predictive model to increase its accuracy. The average AUC value in experimental datasets is increased by 7% when the quadratic features are added as well as when both quadratic and cubic features are added as input. On simulated data, LR model achieved the highest average AUC value which is an excellent result showing that the use of nonlinear features along with larger datasets results in better predictive performance. We also observe qualitatively similar results for the other materials of this study.
iv.3 Performance of predictive models on experimental data
Fig. 7 presents the ROC curves as well as the average AUC values on experimental data for the optimized predictive models for the three materials tested. Given the superior performance of the constructed features, all training and evaluation have been performed on the extended linear+quadratic+cubic feature set. For each material, there are at least two models that achieve AUC above 0.9 implying that the classification tasks are successfully learned. The comparison between predictive models reveals that the LR model (green lines) has the best performance in terms of AUC in two out of three test materials. Moreover, SVM classifier (SVC) and LR model are the two predictive models with AUC above 0.9 in all materials. Interestingly, the high-capacity GBC performed worse than GNB. Two potential reasons for such behavior are the sample size of the datasets is not adequate for robust learning of GBC as well as GBC is sensitive to hyper-parameter tuning.
These is also a clear separation between materials in terms of how accurate the predictive models are. All predictive models for Ti6Al4V produce average AUC values that are above 0.95 while for the other two materials (i.e., Si and Steel 1.7131) the average AUC is below 0.92 yet above 0.84 for all predictive models. Overall, the ability of the tested predictive models to accurately learn the mapping between the laser parameters and the observed structure is high. Interestingly, simpler models perform better in terms of AUC mainly due to the small number of experimental data for which simpler models generalize in a more robust manner. We explore in more details the effect of sample size in the next section.
iv.4 Effect of sample size on predictive models
We examine whether or not the observed performance difference between experimental and simulated datasets in Fig. 6 can be attributed to the sample size of each dataset. Since the transition from one structure to another occurs rapidly, it is beneficial to collect as much samples as possible from the boundary regions and the resolution of the sampling may play an important role in accurately determining the boundaries. Thus, we quantify the effect of dataset size in the performance of predictive models and particularly of logistic regression model. Fig. 8 shows the ROC curves for linear features (left panel), linear+quadratic+cubic features (right panel) and various sample size percentages. In both cases, the AUC is increased as a function of the number of data instances.
Given the limited amount of experimental data, it is desirable to use the simulated data and increase the predictive performance. Unfortunately, the direct augmentation of simulated data to the experimental data will not improve the results due to the fact that they are not perfectly aligned. As explained earlier, the reason for the misalignment is that not all physical phenomena are taken into account by the materials modelling. The proposed data-driven solution is the introduction of an affine transformation which adjusts the domain of the simulated data to the domain of the experimental data. The parameters of the affine transformation are estimated with the Cognitive-based Adaptive Optimization (CAO) algorithm Renzaglia et al. (2010). CAO is a derivative-free stochastic optimization method. Fig. 9 and Table 4 present the ROC curves, the AUC values as well as the relative improvement of the augmentation with the adjusted simulated data. Evidently, all predictive models trained with linear features benefited from the augmented data with the improvement being around 10% for four out of five models. Particularly, SVC with an average AUC value of 0.97 becomes the best performing ML model.
|only exp.||exp. & sim. aug.||improvement (%)|
iv.5 Uncertainty Regions
Lastly, we quantify the uncertainty of the predictive models on the parametric space where the risk of a misclassification error is high. Knowing the regions were models are not certain is helpful in designing the next experiments and extract now knowledge since the uncertain regimes contain relatively more information. On the opposite side, knowing where predictive models are confident about the structure is also useful for production purposes because clear, unequivocal structures are of practical merit. We propose to quantify the uncertainty with Shannon’s information entropy function Shannon (1948); Cover and Thomas (2006) on the probability distribution generated by the predictive model. Thus, uncertainty is defined through information entropy for each point, , as
is the estimated predictive distribution of the labels. Uncertainty takes the highest value when all structures have equal probabilities (i.e., uniform distribution) while the minimum value is attained when the probability for one structure is unity. Fig.10 presents the uncertainty as a function of the input variables for the three studied materials. Evidently, the transition boundaries have higher uncertainty value (yellow color). In contrast, low uncertainty (dark blue color) are attained close to the center of each structure’s parameter region.
V Concluding Remarks and Future Directions
In this work, we presented a detailed ML-based approach to estimate laser parameters for fabrication of surface patterns with fs laser beams. Departing from the traditional selection of optimal input laser parameters for a given output, usually done manually through a trial-error method, we implemented ML techniques to make calibration of laser parameters more automatic, faster and easier than the existing practices. The ML forecasting model shows very good accuracy and capability towards predicting the occurrence of all three types of self-assembled structures efficiently. The approach achieved a successful quantification of the uncertainty of each region related to particular structures and each material while automatically estimated the most uncertain points. A future extension to the model could be related to a fully automated process of structure type identification through image processing tools and structure labelling to efficiently explore the parameter space. There is no doubt that the approach requires further validation and possibly more development, however, the predictive design is expected to transform surface patterning technique by making it more data-driven while providing routes for optimization of low-cost fabrication of products with desired properties.
Despite the impressive performance of the ML-based methodology to predict the laser parameters to produce structures with particular features, there are still several questions that arise about the analysis and the realistic application of the results. More specifically, the investigation was focused, for the sake of simplicity, on three types of structures (i.e.ripples, grooves, spikes) in the absence of regions with questionable structure type classification. Although some preliminary predictive modelling was performed to analyse those structures, a more conclusive analysis requires further investigation.
It should finally be emphasized that the precision of the predictive modelling approach and information of critical decision points can further be enhanced by the set up of more reliable and accurate experimental protocols as well as the development of more advanced theoretical physics models. Nevertheless, the predictive model presented in this work is aimed to set the basis for a systematic fabrication methodology by reducing the number of expensive trial and error techniques and time consuming simulated experiments. Results manifested that the combination of predictive and material modelling tools are capable to reduce the time and cost required to move materials from discovery to application. Therefore, ML-based models are expected to enhance the innovation capacity of laser manufacturing companies as it constitutes a powerful tool designed for simulating and testing new techniques and methods, developing new advanced materials and products, and exploring new directions in the field of laser materials processing and manufacturing.
M-C.V. acknowledges financial support from (MIS 5031029) under Greece - Israel Call for Proposals for Joint R&D Projects, co-financed by the national funding through the Operational Program Competitiveness, Entrepreneurship and Innovation, and the European Regional Development Fund of the European Union. G.D.T and E.St. acknowledge funding from project (MIS 5002735), implemented under the “Action for Strengthening Research and Innovation Infrastructures” funded by the Operational Programme “Competitiveness, Entrepreneurship and Innovation” and co-financed by Greece and the EU (European Regional Development Fund). G.D.T, A.M, E.Sk. and E.St. acknowledge support by the European Union’s Horizon 2020 research and innovation program through the project (grant agreement No. 862016). Y.P. acknowledges partial support by the project “Innovative Actions in Environmental Research and Development (PErAn)” (MIS 5002358) funded by the Operational Programme “Competitiveness, Entrepreneurship and Innovation” (NSRF 2014-2020).
The data that support the findings of this study are available within the article.
Exploration of data science techniques to predict fatigue strength of steel from composition and processing parameters. Integrating Materials and Manufacturing Innovation 3 (1), pp. 8. Cited by: §I.
- Electron emission from metal surfaces exposed to ultrashort laser pulses. Zhurnal Eksperimentalnoi i Teoreticheskoi Fiziki 66, pp. 776–781. Cited by: §II.2.
- Biomimetic surface structures in steel fabricated with femtosecond laser pulses: influence of laser rescanning on morphology and wettability. Beilstein journal of nanotechnology 9 (1), pp. 2802–2812. Cited by: §II.1.
- Laser processing and chemistry. Springer Science & Business Media. Cited by: §I, §II.1.
- Pattern recognition and machine learning (information science and statistics). Springer. Cited by: §I, §III.3, §III.3.
- Laser-induced periodic surface structures—a scientific evergreen. IEEE Journal of selected topics in quantum electronics 23 (3). Cited by: §II.1, §II.2, §II.2.
- Femtosecond laser-induced periodic surface structures. J. Laser Appl 24 (4), pp. 042006. Cited by: §I.
- Pulse number dependence of laser-induced periodic surface structures for femtosecond laser irradiation of silicon. Journal of Applied Physics 108 (3), pp. 034903. Cited by: §I.
- On the role of surface plasmon polaritons in the formation of laser-induced periodic surface structures upon irradiation of silicon by femtosecond-laser pulses. Journal of Applied Physics 106 (10), pp. 104910. Cited by: §II.2.
- Classification and regression trees. Wadsworth and Brooks, Monterey, CA. Cited by: §III.3.
- Random forests. Machine Learning 45 (1), pp. 5–32. External Links: Cited by: §III.3.
An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd International Conference on Machine Learning, Vol. 148, pp. 161–168. Cited by: §III.3, §III.3, §III.3.
- XGBoost: a scalable tree boosting system.. In KDD, pp. 785–794. Cited by: §III.3.
- Femtosecond, picosecond and nanosecond laser ablation of solids. Applied physics A 63 (2), pp. 109–115. Cited by: §I.
- Product Design and Manufacturing. PHI Learning. External Links: Cited by: §I.
- Support vector networks. Machine Learning 20, pp. 273–297. Cited by: §III.3.
- Elements of information theory 2nd edition. Wiley-Interscience. External Links: Cited by: §I, §IV.5.
- An introduction to support vector machines and other kernel-based learning methods. 1 edition, Cambridge University Press. Cited by: §III.3.
- Automotive model predictive control: models, methods and applications. Vol. 402, Springer. Cited by: §I.
- Possible surface plasmon polariton excitation under femtosecond laser irradiation of silicon. Journal of Applied Physics 114 (8), pp. 083104. Cited by: §I, §II.2.
- Damage-tolerant composite design principles for aircraft components under fatigue service loading using multi-scale progressive failure analysis. Journal of Composite Materials 51 (15), pp. 2181–2202. Cited by: §I.
- Pattern classification. 2 edition, Wiley, New York. Cited by: §I, §III.3, §III.3.
- Signal detection theory and ROC analysis. Series in Cognition and Perception, Academic Press, New York, NY. Cited by: footnote 1.
- Deep learning for distortion prediction in laser-based additive manufacturing using big data. Manufacturing Letters 20, pp. 10–14. Cited by: §I.
A Decision Theoretic Generalization of On-Line Learning and an Application to Boosting.
Second European Conference on Computational Learning Theory (EuroCOLT-95), pp. 23–37. Cited by: §III.3.
- Greedy function approximation: a gradient boosting machine. Annals of Statistics 29, pp. 1189–1232. Cited by: §III.3.
- Evidence of surface plasmon resonance in ultrafast laser-induced ripples. Optics Express 19 (10), pp. 9035–9043. Cited by: §I.
- Deep learning. MIT Press. External Links: Cited by: §I, §III.3.
- The elements of statistical learning. Springer Series in Statistics, Springer New York Inc., New York, NY, USA. Cited by: §III.3.
- Machine learning for 3d simulated visualization of laser machining. Optics Express 26 (17), pp. 21574–21584. Cited by: §I.
- Origin of laser-induced near-subwavelength ripples: interference between surface plasmons and incident laser. ACS nano 3 (12), pp. 4062–4070. Cited by: §I, §II.2.
- Nanocrystalline structure of nanobump generated by localized photoexcitation of metal film. Journal of Applied Physics 107 (1), pp. 013519. Cited by: §I, §II.2.
- Mimicking bug-like surface structures and their fluid transport produced by ultrashort laser pulse irradiation of steel. Applied Physics A 123 (12), pp. 754. Cited by: §II.1.
- Electron-phonon coupling and electron heat capacity of metals under conditions of strong electron-phonon nonequilibrium. Physical Review B 77 (7), pp. 075133. Cited by: §I.
- Yambo: an ab initio tool for excited state calculations. Computer Physics Communications 180 (8), pp. 1392–1403. Cited by: §II.2.
- Predictive capabilities for laser machining via a neural network. Optics express 26 (13), pp. 17245–17253. Cited by: §I.
- Bio-inspired functional surfaces based on laser-induced periodic surface structures. Materials 9 (6), pp. 476. Cited by: §II.1.
- Machine learning : a probabilistic perspective. MIT Press, Cambridge, Mass. [u.a.]. Cited by: §III.3.
- High-throughput machine-learning-driven synthesis of full-heusler compounds. Chemistry of Materials 28 (20), pp. 7324–7331. Cited by: §I.
- Biomimetic omnidirectional antireflective glass via direct ultrafast laser nanostructuring. Advanced Materials 31 (32), pp. 1901123. Cited by: §I, §II.1.
Silicon scaffolds promoting three-dimensional neuronal web of cytoplasmic processes. Tissue Engineering Part C: Methods 16 (3), pp. 497–502. Cited by: §I, §II.1.
- Machine-learning-assisted materials discovery using failed experiments. Nature 533 (7601), pp. 73–76. Cited by: §I.
- Cognitive-based adaptive control for cooperative multi-robot coverage. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3314–3320. Cited by: §IV.4.
- Modelling ultrafast laser ablation. Journal of Physics D: Applied Physics 50 (19), pp. 193001. Cited by: §I, §II.2.
- From random inhomogeneities to periodic nanostructures induced in bulk silica by ultrashort laser. Physical Review B 93 (7), pp. 075427. Cited by: §II.2.
- Learning with kernels : support vector machines, regularization, optimization, and beyond. Adaptive computation and machine learning, MIT Press. Cited by: §III.3.
- A mathematical theory of communication.. Bell Syst. Tech. J. 27 (3), pp. 379–423. Cited by: §I, §IV.5.
- Self-organized nanogratings in glass irradiated by ultrashort light pulses. Physical review letters 91 (24), pp. 247405. Cited by: §I.
- Laser fabricated discontinuous anisotropic microconical substrates as a new model scaffold to control the directionality of neuronal network outgrowth. Biomaterials 67, pp. 115–128. Cited by: §I, §II.1.
- Laser-induced periodic surface structures: fingerprints of light localization. Physical review B 85 (7), pp. 075320. Cited by: §I.
- Biomimetic surface structuring using cylindrical vector femtosecond laser beams. Scientific Reports 7 (1), pp. 1–11. Cited by: §II.1.
- Laser engineering of biomimetic surfaces. Materials Science and Engineering: R: Reports (), pp. (). Cited by: §I, §II.1, §II.2, §II.2.
- Nanomaterials by ultrafast laser processing of surfaces. Science of Advanced Materials 4 (3-4), pp. 407–431. Cited by: §I, §II.1.
- Pulse-by-pulse depth profile measurement of femtosecond laser ablation on copper. Applied Physics A 124 (3), pp. 265. Cited by: §I.
- Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B) 58, pp. 267–288. Cited by: §III.3.
- Dynamics of ripple formation on silicon surfaces by ultrashort laser pulses in subablation conditions. Physical Review B 86 (11), pp. 115316. Cited by: §I, §I, §II.2, §II.2, §II.4.
- From ripples to spikes: a hydrodynamical mechanism to interpret femtosecond laser-induced self-assembled structures. Physical Review B 92 (4), pp. 041405. Cited by: §I, §I, Figure 1, §II.2, §II.4.
- Modelling periodic structure formation on 100cr6 steel after irradiation with femtosecond-pulsed laser beams. Applied Physics A 124 (1), pp. 27. Cited by: §II.2, §II.2.
- Modeling ultrafast out-of-equilibrium carrier dynamics and relaxation processes upon irradiation of hexagonal silicon carbide with femtosecond laser pulses. Physical Review B 101 (7), pp. 075207. Cited by: §II.2.
- Convection roll-driven generation of supra-wavelength periodic surface structures on dielectrics upon irradiation with femtosecond pulsed lasers. Physical Review B 94 (8), pp. 081305. Cited by: §I.
- Ripple formation on nickel irradiated with radially polarized femtosecond beams. Optics letters 40 (22), pp. 5172–5175. Cited by: §II.4.
- Controlled ultrashort-pulse laser-induced ripple formation on semiconductors. Applied Physics A 114 (1), pp. 57–68. Cited by: §I.
- Ripple formation on silver after irradiation with radially polarised ultrashort-pulsed lasers. Journal of Applied Physics 121 (16), pp. 163106. Cited by: §I, §I, §II.2, §II.2.
- Ionisation processes and laser induced periodic surface structures in dielectrics with mid-infrared femtosecond laser pulses. Scientific Reports 10 (1), pp. 8675. Cited by: §I.
- Direct femtosecond laser surface nano/microstructuring and its applications. Laser & Photonics Reviews 7 (3), pp. 385–407. Cited by: §I, §II.1.
- Ultrafast dynamics of femtosecond laser-induced periodic surface pattern formation on metals. Applied Physics Letters 87 (25), pp. 251914. Cited by: §I.
- Biomimetic artificial surfaces quantitatively reproduce the water repellency of a lotus leaf. Advanced materials 20 (21), pp. 4049–4054. Cited by: §I, §II.1.
- Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67 (2), pp. 301–320. Cited by: §III.3.
Appendix A Predictive performance of SVC on constructed features
Fig. 11 illustrates the effect of constructed features on the predictive performance using the SVC model. It is evident that SVC did not benefit from the additional features and the performance is more or less unchanged. This is partially explained by the fact that SVC is a non-linear model capable of intrinsically constructing its own features.
Appendix B Predictive performance of augmented data on Si and Steel
In this section we present the ROC curves for Si (Fig. 12) and Steel (1.7131) (Fig. 13). Results for Si show that, the augmentation of experimental data with simulated ones improve considerably the predictions, as in Ti while the performance is unchanged for Steel (1.7131). Tables 5 and 6 quantify the improvement of the predictive models seen in the figures for Si and Steel (1.7131), respectively. Evidently, the performance improvement of the augmented dataset for a material depends on the success and accuracy of the domain transformation.
|only exp.||exp. & sim. aug.||improvement (%)|
|only exp.||exp. & sim. aug.||improvement (%)|
Appendix C Predictive performance with neural networks
In this section we present the predictive performance of neural networks. We first demonstrate the performance of two neural networks with one hidden layer and 50 units (Fig 14) and 10 units (Fig 15) on simulated data. We observe that the neural network with more units enjoy higher AUC. Moreover, the constructed features assisted the performance of the neural networks. We also remark that the number of samples used for the training has a significant effect on the performance which deteriorates considerably. The deterioration is even higher when the training is performed on experimental data as shown in Fig. 16. The lack of sufficient experimental data makes the training of neural networks infeasible.
Appendix D Uncertainty regions for other predictive models
Figs. 17 and 18 present the estimated uncertainty using k-NN and random forest, respectively. Both predictive models are trained on the experimental data. Uncertainty regions are qualitatively similar. Nevertheless, uncertainty estimate using random forest are more focused around the transition boundaries.