Complex Number Artificial Neural Net (CANN) is a natural consequence of using ANN for complex number, while complex number appear in many problems. The complex number problems include (Fourier, Laplace, Z, etc) transform based methods (Haberman, 2013) , steady-state problems (Banerjee, 1994) , mapping 2D problem to complex plane (Greenberg, 1998) , and electromagnetic signals such as MRI image (Virtue et al., 2017)
. Due to such necessity, CANN appears from 1990’s even long before the days of deep learning, as discussed in(Reichert and Serre, 2013)
. As the result of various efforts, building blocks of CANN has been developed; Convolution, Fully-connection, Back-propagation, Batch-normalization, Initialization, Pooling, and Activation. Among them, Complex Convolution and Fully-connection are essentially multiplication and addition, which are clearly established for complex number in mathematics(Trabelsi et al., 2017) . Back-propagation for complex number is developed for general complex number function (Georgiou and Koutsougeras, 1992; Nitta, 1997), and for holomorphic function (La Corte and Zou, 2014) . Complex Batch-normalization is also developed following the idea of real Batch-normalization (Trabelsi et al., 2017) . The complex number Initialization is suggested to be done in polar-form (Trabelsi et al., 2017)
, not Cartesian. Max Pooling for complex number is another necessity, but the authors cannot see any research. Complex Activation is in active development(Arjovsky et al., 2015; Guberman, 2016; Virtue et al., 2017; Georgiou and Koutsougeras, 1992); see (La Corte and Zou, 2014) for brief review about various types of complex activation functions.
For complex activation, holomorphic (a.k.a. analytic) function has been an important topic (Hitzer, 2013; Jalab and Ibrahim, 2011; Vitagliano et al., 2003; Kim and Adali, 2001; Mandic and Goh, 2009; Tripathi and Kalra, 2010) , since it makes the back-propagation simpler thus training becomes faster (Amin et al., 2011; La Corte, 2014)
. The importance of Complex Phase Angle of CANN has been suggested, including similarity to biological neuron(Reichert and Serre, 2013). The importance leads to the idea of phase-preserving complex activation function (Georgiou and Koutsougeras, 1992; Virtue et al., 2017). In contrast, (Hirose, 2013; Kim and Adalı, 2003) suggested that phase-preservation makes training more difficult.
We focus on the complex activation, and propose a methods to generalize partitioned activation such as LReLU and SELU for complex number. There are 4 variations in the method to accommodate various cases; 1 of them is potentially holomorphic and alter phase, 1 is not holomorphic and alter phase while guarantees interaction between real and imaginary parts, and 2 are not holomorphic and potentially keep complex phase angle.
2 Review of Current Complex Activation Functions
Certain complex-argument activation functions are derived from real-argument counterparts, using 3 ways depending on the characteristic of each function; No change, Modification, and generalization. We will briefly discuss them with particular weight on the generalization, since our approach is to generalize.
2.1 No change
Certain no-partitioned activation functions can be used for complex-argument without any change. Such functions include logistic, , , etc. These are already defined for complex-argument, and we can just use them. But they suffer from poles near the origin (La Corte and Zou, 2014) .
Partitioned activation functions are not straightforward for complex-argument, due to the partition points on the real-axis. So approach of “just take the idea of real activation, and create a corresponding complex activation” is tried. We call it modification. One example is modReLU (Arjovsky et al., 2015) as
where , is the phase angle of , and
is a trainable parameter. The modReLU takes the idea of inactive region from ReLU, and forms region around origin. Unfortunately, the modReLU is not holomorphic, while keeps the complex phase.
We call a complex activation Generalized , if the values on real-axis coincide with the real-argument counter-part. The easiest generalization of partitioned activation function would be to separately apply activation function to real and imaginary parts (Nitta, 1997; Faijul Amin and Murase, 2009) . One example is Separate Complex ReLU (SCReLU) as
where . Another simple generalization of ReLU is to activate only when both real and imaginary parts are positive, which is called zReLU (Guberman, 2016)
Both SCReLU and ReLU are holomorphic, and are essentially not complex but 2 separate real activations (La Corte and Zou, 2014) . Another generalization of ReLU is Complex Cardioid (CC) (Virtue et al., 2017)
which keeps the complex angle, but is not holomorphic. Our approach is inspired by the CC.
3 New Generalization Method
Consider a partitioned activation function for real number, which has the typical form
where and are local functions for positive and negative regions. Activation functions like LReLU and SELU are examples of the above case. Now, replace partitions with Heaviside unit-step function , and we can rewrite as
Next, select a complex-argument function, whose real axis values coincide with the and . We pick
where is the phase angle of complex number , and (is an integer) is a parameter. Then, we can get a generalized complex-argument function
This approach can be easily extended to more complicated cases, like S-shaped ReLU (SReLU) which has 3 partitions. The typical form
is rewritten as
, then generalized to
where denotes the phase angle of complex number , and is the location of partition boundary p.
In case real valued scale is preferred (eg: to keep the complex angle), we made simple modification for making replacement of real valued. The modifications are
and generalization using them are
Another approach is to approximate on real axis with a complex-argument function. A few well known such functions are based on (hyperbolic tangent) , (arc tangent), (sine integral), and (error function) functions. Among them, we pick
based Sigmoid function as below, sinceand has poles on imaginary axis, and is oscillatory on real axis.
where is a just a parameter such that smaller more closely approximates on the real axis. Then approximately generalized functions are
respectively for (15) and (16) cases. An important advantage of this approximate generalization is that the is holomorphic , if all the are holomorphic. The reason is simple; a) logistic function is holomorphic, b) product and sum of holomorphic functions are also holomorphic. Then, each terms in (15) and (16) are holomorphic which are products of holomorphic, and is holomorphic which is sum of holomorphic. With using normalization (eg: Batch-Renormalization (Ioffe, 2017)) is suggested to prevent exploding feature values toward .
3.1 Example 1: Leaky Rectified Linear Unit (LReLU)
The LReLU (Maas et al., 2013) is the one of the most popular activation function (The ReLU is just a LReLU with ). The real-argument LReLU is
and is generalized using (8) with n=0 to “Complex LReLU” (CLReLU) as
which we call “cos LReLU” (cLReLU) and “abs LReLU” (aLReLU), respectively. Meanwhile, a “Holomorphic LReLU” (HLReLU) can be derived using (15) as
which is holomorphic, since both the and are polynomials which are all holomorphic. Please note that the HLReLU alters phase angle of argument, since the argument z is multiplied with a complex number.
3.2 Example 2: Scaled Exponential Linear Unit (SELU)
The SELU (Klambauer et al., 2017) was recently developed, and rapidly becomes popular due to its self-normalizing feature (The ELU is just a SELU with ). The real-argument SELU is
Meanwhile, a Holomorphic “SELU” (HSELU) can be derived using (15) as
which is also holomorphic, since both the scaled shifted exponential and polynomial are holomorphic.
4 Concluding Remark
A generalization of partitioned real number activation function to complex number has been proposed. The generalization process has 2 steps; a) Replace partition on activation with , b) Generalize with a complex number function. The generalization has 4 variations, and 1 of them is potentially holomorphic. The generalization scheme has been demonstrated using 2 popular partitioned activations; LReLU and SELU. The properties of generalized complex activations are summarized on table (1) .
|Original Real Activation||Generalized Complex Activation||Holomorphic||Real-Complex Interaction||Phase Preverving|
|LReLU (17)||CLReLU (18)||X||O||X|
|SELU (21)||CSELU (22a)||X||O||X|
Furthermore, the method can be used for various partitioned real activation to make them into complex activation. He hope that this humble research adds another building block for complex ANN, which is important for complex number problems.
Amin et al. 
Md Faijul Amin, Muhammad Ilias Amin, A. Y. H. Al-Nuaimi, and Kazuyuki Murase.
Wirtinger Calculus Based Gradient Descent and Levenberg-Marquardt Learning Algorithms in Complex-Valued Neural Networks.In Neural Information Processing, Lecture Notes in Computer Science, pages 550–559. Springer, Berlin, Heidelberg, November 2011. ISBN 978-3-642-24954-9 978-3-642-24955-6. doi: 10.1007/978-3-642-24955-6_66. URL https://link.springer.com/chapter/10.1007/978-3-642-24955-6_66.
- Arjovsky et al.  Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary Evolution Recurrent Neural Networks. arXiv:1511.06464 [cs, stat], November 2015. URL http://arxiv.org/abs/1511.06464. arXiv: 1511.06464.
- Banerjee  Prasanta Kumar Banerjee. The Boundary Element Methods in Engineering. Mcgraw-Hill College, London ; New York, rev sub edition edition, January 1994. ISBN 978-0-07-707769-3.
- Faijul Amin and Murase  Md. Faijul Amin and Kazuyuki Murase. Single-layered complex-valued neural network for real-valued classification problems. Neurocomputing, 72(4):945–955, January 2009. ISSN 0925-2312. doi: 10.1016/j.neucom.2008.04.006. URL http://www.sciencedirect.com/science/article/pii/S0925231208002439.
Georgiou and Koutsougeras 
G. M. Georgiou and C. Koutsougeras.
Complex domain backpropagation.IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 39(5):330–334, May 1992. ISSN 1057-7130. doi: 10.1109/82.142037. URL http://ieeexplore.ieee.org/document/142037/.
- Greenberg  Michael Greenberg. Advanced Engineering Mathematics. Prentice Hall, Upper Saddle River, N.J, 2 edition edition, January 1998. ISBN 978-0-13-321431-4. Google-Books-ID: CpoUngEACAAJ.
- Guberman  Nitzan Guberman. On Complex Valued Convolutional Neural Networks. arXiv:1602.09046 [cs], February 2016. URL http://arxiv.org/abs/1602.09046. arXiv: 1602.09046.
Applied Partial Differential Equations: With Fourier Series and Boundary Value Problems. PEARSON, 2013. ISBN 978-0-321-79705-6. Google-Books-ID: hGNwLgEACAAJ.
- Hirose  Akira Hirose, editor. Complex-valued neural networks: advances and applications. IEEE Press series on computational intelligence. John Wiley & Sons Inc, Hoboken, N.J, 2013. ISBN 978-1-118-34460-6. OCLC: ocn812254892.
- Hitzer  Eckhard Hitzer. Non-constant bounded holomorphic functions of hyperbolic numbers - Candidates for hyperbolic activation functions. arXiv:1306.1653 [cs, math], June 2013. URL http://arxiv.org/abs/1306.1653. arXiv: 1306.1653.
- Ioffe  Sergey Ioffe. Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models. arXiv:1702.03275 [cs], February 2017. URL http://arxiv.org/abs/1702.03275. arXiv: 1702.03275.
- Jalab and Ibrahim  Hamid A. Jalab and Rabha W. Ibrahim. New activation functions for complex-valued neural network. International Journal of Physical Sciences, 6(7):1766–1772, 2011.
- Kim and Adali  Taehwan Kim and Tülay Adali. Complex backpropagation neural network using elementary transcendental activation functions. In Acoustics, Speech, and Signal Processing, 2001. Proceedings.(ICASSP’01). 2001 IEEE International Conference on, volume 2, pages 1281–1284. IEEE, 2001.
Kim and Adalı 
Taehwan Kim and Tülay Adalı.
Approximation by fully complex multilayer perceptrons.Neural computation, 15(7):1641–1666, 2003. URL https://www.mitpressjournals.org/doi/abs/10.1162/089976603321891846.
- Klambauer et al.  Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-Normalizing Neural Networks. arXiv:1706.02515 [cs, stat], June 2017. URL http://arxiv.org/abs/1706.02515. arXiv: 1706.02515.
- La Corte  Diana Thomson La Corte. Newton’s Method Backpropagation for Complex-Valued Holomorphic Neural Networks: Algebraic and Analytic Properties. Theses and Dissertations, August 2014. URL https://dc.uwm.edu/etd/565.
- La Corte and Zou  Diana Thomson La Corte and Yi Ming Zou. Newton’s Method Backpropagation for Complex-Valued Holomorphic Multilayer Perceptrons. 2014 International Joint Conference on Neural Networks (IJCNN), pages 2854–2861, June 2014. doi: 10.1109/IJCNN.2014.6889384. URL http://arxiv.org/abs/1406.5254. arXiv: 1406.5254.
- Maas et al.  Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3, 2013.
- Mandic and Goh  Danilo P. Mandic and Vanessa Su Lee Goh. Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models. John Wiley & Sons, April 2009. ISBN 978-0-470-74263-1. Google-Books-ID: MaW8MaIkztUC.
- Nitta  Tohru Nitta. An Extension of the Back-Propagation Algorithm to Complex Numbers. Neural Networks, 10(8):1391–1415, November 1997. ISSN 0893-6080. doi: 10.1016/S0893-6080(97)00036-1. URL http://www.sciencedirect.com/science/article/pii/S0893608097000361.
- Reichert and Serre  David P. Reichert and Thomas Serre. Neuronal Synchrony in Complex-Valued Deep Networks. arXiv:1312.6115 [cs, q-bio, stat], December 2013. URL http://arxiv.org/abs/1312.6115. arXiv: 1312.6115.
- Trabelsi et al.  Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J. Pal. Deep Complex Networks. arXiv:1705.09792 [cs], May 2017. URL http://arxiv.org/abs/1705.09792. arXiv: 1705.09792.
- Tripathi and Kalra  B. K. Tripathi and P. K. Kalra. High Dimensional Neural Networks and Applications. In Intelligent Autonomous Systems, Studies in Computational Intelligence, pages 215–233. Springer, Berlin, Heidelberg, 2010. ISBN 978-3-642-11675-9 978-3-642-11676-6. URL https://link.springer.com/chapter/10.1007/978-3-642-11676-6_10. DOI: 10.1007/978-3-642-11676-6_10.
- Virtue et al.  Patrick Virtue, Stella X. Yu, and Michael Lustig. Better than Real: Complex-valued Neural Nets for MRI Fingerprinting. arXiv:1707.00070 [cs], June 2017. URL http://arxiv.org/abs/1707.00070. arXiv: 1707.00070.
- Vitagliano et al.  Francesca Vitagliano, Raffaele Parisi, and Aurelio Uncini. Generalized splitting 2d flexible activation function. In Italian Workshop on Neural Nets, pages 85–95. Springer, 2003.