The study of animal collective behaviour involves the search for the relevant signals and mechanisms used by the animals for social interactions [33, 26]. Robots can help ethologists to test various hypothesis on the nature of these signals by inducing specific and controlled stimuli to assess animal response.
Autonomous robots are capable to interact with animals and can serve as tools to study social dynamics . This approach has already been used in studies to analyse the behaviour of ducks , drosophila , cockroaches , fish [10, 27, 28, 25, 24, 6, 2], bees [20, 29, 38] and birds [23, 14, 19].
Here, we socially integrate a behavioural biomimetic robotic lure into a group of four zebrafish (Danio rerio) moving in a structured environment and validate its acceptance by the animals. This problem is difficult because the robotic lure must be designed to be perceived as a social companion by the animals: it must, to a certain extent look like a fish, behave like a fish, be able to respond appropriately to environmental and social cues to close the loop of social interactions with the fish. Closing the loop of social interactions requires real-time individual perception and a decision-making algorithm to control the robot behaviours .
These aspects were investigated in [10, 9] through the use of biomimetic robotic fish lures driven by a calibrated biomimetic model to make the robot mimics expected fish behaviour. An evolutionary algorithm (NSGA-II ) was used to optimise the parameters of this model so that the resulting collective dynamics corresponded to those observed in biological experiments. This type of controller allowed the robot to be a real group-member making its own decisions rather than a passive follower.
However, the model calibration was done off-line and not during the ongoing experiments. As such, it could not take into account the changes in animal behaviour across experiments and the intrinsic behavioural differences between groups used in experiments.
Here, we tackle this problem by continuously refining and calibrating the biomimetic model driving the robot behaviours in real-time during the experiment by using on-line evolutionary algorithm (NSGA-II ). This task is computationally-intensive and requires three computers to deal with agent real-time tracking, robot control, real-time data-analysis, and model calibration. We test this methodology in a set of 10 experiments with four fish and one robot. In each case, the robot closed-loop behaviour becomes progressively socially integrated into the group of fish. This is the first step towards evolving mixed-group of animals and robot .
Ii Materials and Methods
Ii-a Experimental set-up
We use the experimental set-up from [9, 4, 35, 10, 12] (Fig. LABEL:fig:workflow, ”Control & tracking” part) with a white plexiglass arena (Fig. LABEL:fig:setupA) of mm composed of two rooms linked by a corridor. We use the FishBot robot [3, 5, 4], powered by two conductive plates under the aquarium, to interact with fish. An overhead camera captures frames (15 FPS, px), that are then tracked to find the fish positions. A complementary fish-eye camera (15 FPS, px) placed under the fish tank is used to track the position of the robot.
We used 10 groups of 4 adults wild-type AB zebrafish (Danio rerio) in ten 30-minutes experiments as in [10, 9, 35]: 30 minutes is sufficient to capture the behaviour and dynamics of groups of 4 zebrafish. Fish are released in the aquarium after the lure is placed in the aquarium.
To ensure real-time adaptation, our methodology is computationally intensive, and uses three networked 32-core computers (Fig. LABEL:fig:workflow). Computer 1 is used to track the agents in real-time and control the robot according to the behavioural model of Sec. II-B. Computer 2 performs every
data-analysis on the tracked positions of agents from Computer 1, and estimates the biomimetism of robot behaviour (which, in our case, can be viewed as a metric of social integration as defined in). Computer 3 re-calibrates every the behavioural model to correspond as close as possible to the behaviour of experimental fish (measured by Computer 2). The resulting calibrated parameter set is then sent to Computer 1 to serve as parameters of the robot controller model. It allows the robot to progressively mimics the behaviour of the fish and be socially accepted.
The experiments performed in this study were conducted under the authorisation of the Buffon Ethical Committee (registered to the French National Ethical Committee for Animal Experiments #40) after submission to the French state ethical board for animal experiments.
Ii-B Behavioural model
We use the multi-level model from [9, 10] (inspired from ) that describes the individual and collective behaviours of fish (Fig. LABEL:fig:setupB). This model takes into account both social interactions and environmental cues (i.e. walls and structure of the tanks). It is stochastic, multi-level and context-dependent.
Fish behave differently depending on their spatial position. Namely, this model identify three zones of the structured set-up with different fish behaviours (Fig. LABEL:fig:setupA): when they are close to the walls, when they are in the centre of the rooms, and when they pass through the corridor. Near the walls, fish perform mainly thigmotactism (wall following) while in room centre they exhibit exploratory behaviour. In the corridor, they tend to go in a straight line with increased speed to reach the subsequent room. Fish also react to social cues leading to collective behaviour such as collective departures from the rooms . Very few models of fish collective behaviours take into account the presence of walls [11, 8].
The agents update their position vectorwith a velocity vector :
with the linear speed of the agent and its orientation. The linear speed of the agent is randomly drawn from the experimentally measured instantaneous speed distribution.
is drawn from probability density function (PDF) computed as a mixture distribution of von Mises distributions centred on the stimuli perceived by the focal agent. It takes into account the influence of other agents and of the walls of the experimental arena. The resulting PDF is composed of the weighted sum of (i) a PDF taking into account the effect of the walls and (ii) a PDF describing the response to other agents. The parameter, used as a multiplicative term of the final PDF, modulates the attraction of agents towards target zones.
We numerically compute the cumulative distribution function (CDF) corresponding to this final PDF by performing a cumulative trapezoidal numerical integration of the PDF in the interval. Then, the model draws a random direction in this distribution by inverse transform sampling. The position of the fish is then updated according to this direction and his velocity.
Ii-C Communication between computer nodes
We connect the three computers (Fig. LABEL:fig:workflow) using the ZeroMQ distributed messaging protocol : computers receiving messages act as ZeroMQ subscribers, and computers sending messages act as ZeroMQ publishers.
The tracked agent trajectories are compiled in the form of trajectory files, and sent every from Computer 1 to Computer 2 through the
rsync  command line application (a process which usually only need to that is sufficient because the parameter update is every ). Then, Computer 1 send a ZeroMQ message to Computer 2 to acknowledge that the transfer is completed. Data-analysis scores from Computer 2 to Computer 3, and model parameters from Computer 3 to Computer 1 are sent every through ZeroMQ messages.
Ii-D Real-time tracking
We use the CATS framework  to track agents (fish and robot) in real-time, on Computer 1. Fish are tracked (but not identified) by using frames captured by the overhead camera (Fig. LABEL:fig:workflow) through the Shi-Tomasi method  implemented in the OpenCV library . In parallel, the robot is tracked through the video frames from the camera below the fish-tank by colour and contours detection . Every the positions of the agents are sent to Computer 2 for data-analysis.
Every , Computer 2 calculates the behavioural statistics using the tracked positions of agents (from computer 1) over the last of the running experiment, for all three zones of the arena. For a zone , these statistics are: the distribution of inter-individual distances between agents (), the distribution of distances of agents to their nearest wall (), the distribution of zones occupation (), the transition probabilities from zone to others ().
These statistics are computed either only on fish agents (Control case: ) or on fish and robotic agents (Robot Social Integration case: ). We define a similarity score (ranging from to ) to measure the biomimetism of robot behaviour compared to the Control case:
The function is defined as such: . The function is the Hellinger distance between two histograms . It is defined as: where and are the bin frequencies.
Cazenille et al. [10, 9] demonstrated that robotic lures with biomimetic morphology and behaviour are be more socially integrated into the group of fish than non-biomimetic lures. As such, the biomimetism score defined earlier corresponds to the social acceptatation of the robot by the fish.
When this statistics and scores are computed, they are dispatched to Computer 3 (by the ZeroMQ system described in Sec. II-C) to guide the optimisation process.
Ii-F Real-time optimisation of model parameters
We design a calibration methodology (Fig. LABEL:fig:workflow) capable of optimising in real-time the parameters of the behavioural model from Sec. II-B to mimic as close as possible to the behaviour of experimental fish. The behavioural similarity is quantified as described in Sec. II-E.
It is inspired from the off-line calibration methodology in  and uses the NSGA-II  multi-objective global optimiser (population of 60 individuals, 300 generations) with three objectives to maximise. We define a fitness with three objectives: the first objective is a performance objective corresponding to the function. Two other objectives are considered to guide the evolutionary process: one that promotes genotypic diversity  (defined by the mean euclidean distance of the genome of an individual to the genomes of the other individuals of the current population), the other encouraging behavioural diversity (defined by the euclidean distance between the , , and scores of an individual).
This process is performed and restarted every on Computer 3 (starting after the beginning of the experiment to gather data, Fig. LABEL:fig:workflow) using data gathered during the last
. Every restart of the evolutionary algorithm keeps the last generation of individuals evolved during the previous round of evolution to bootstrap the current round of evolution, a system akin to transfer learning. On our 32-core computer, one generation is computed approximately every, so around generations are computed at every evolutionary round.
We do not optimise the linear speed of the agents. It is randomly drawn from the experimental speed distribution. We use the NSGA-II implementation provided by the DEAP python library .
Ii-G Robot implementation and control
The robot is driven by the model presented in Sec. II-B thanks to the CATS framework . The model is calibrated every using the methodology of Sec. II-F, in experiments involving four fish and one robot. Every , the tracked positions of the four fish are integrated into the model to compute the target position of a fifth agent. The robot is programmed to follow this target position by using the biomimetic movement patterns as in [5, 10].
We assessed the evolution of the similarity scores (defined in Sec. II-E) between robot behaviour and fish behaviour, across sliding windows of intervals of a set of 10 trials each one lasting minutes, starting in each trial from the second time interval ( to ) to gather enough experimental data. These scores are compiled in Fig. LABEL:fig:plotScores
. The variance for the 10 trials is plotted as a grey area around the curves and remains rather small.
From its initial value of about (second time interval: to ), the average fitness (mean scores) fastly converges to values around starting from the fourth time interval ( on to ). This is also observed for similarity scores of transitions and of distances to nearest wall. This shows that both of these behavioural features can be effectively optimised through an online evolutionary algorithm process, and remain stables during the experiment.
The similarity score of zone occupation is particularly high at the beginning of the experiment, and is only slightly improved by our calibration methodology; this would suggests that room occupation is only slightly dependent of model parameters. This could be explained by a strong effect of room geometry (room size, and the general room configuration of the arena) over room occupation: rooms cover a larger area than the corridor. This could also be an effect of the aggregative behaviour exhibited by fish and by the model: the robot would follow the fish, which would tend to follow walls, thus explaining the relative invariance of the occupation score with respect to parameter values.
The variations in similarity score of inter-individual distances suggests changes of fish aggregative behaviour during the experiment. This could be explained by the fact that, while zebrafish tend to remain cohesive most of the time, they have the tendency of forming short-lived (a few seconds to a few minutes) sub-groups, especially when confronted to a fragmented environment .
Iv Discussion and Conclusion
Animal-robot interaction studies employ simple robot behavioural model that are not adaptive or updated during the experiments. Often they are not biomimetic and do not close the interaction loop between the animals and the robots . Here we present a methodology to calibrate in real-time a multi-level context-dependent biomimetic model of fish behaviour to drive the behaviour of a biomimetic robot into a group of zebrafish. The model parameters are continuously refined to accurately correspond to the collective dynamics exhibited by fish during the experiments. The real-time nature of this calibration process allows the robot to react to changes in observed fish dynamics and cope with uncertainties.
Animals can present significant inter-individual behavioural differences. They can present significant differences in terms of personalities typically bold and shy types . In most of the experiments individuals are selected randomly from a stock. Consequently, each group trial present differences depending on the characteristics of the individuals. Currently, the models are calibrated on a set of averaged experimental data and are not optimised to take into account inter-individuals differences. We present here a method to adapt in real-time the models and that is thus capable to cope with this issue. This method can reduce significantly the number of experimental trials necessary to calibrate the model.
Our approach is computationally intensive and use three networked computers to handle in real-time the tracking, the robot control, the data analysis and the model calibration tasks.
Our methodology builds on the work presented in  by adding real-time capabilities to the calibration process. However, it also suffers from the same limitations. Namely, the model we calibrate must still be structurally defined empirically (i.e. defining behavioural attractors, zones of the environment, etc) with ethological a-priori knowledge about fish dynamics. The calibration process could also still be improved by taking into account additional behavioural metrics in the computation of similarity scores, either in term of collective dynamics (e.g. agent groups aspects, residence time in a zone), individual behaviours (e.g. agent trajectory aspects, curvature of trajectories). This could possibly be bypassed through the use of a calibration process without explicit similarity measure (e.g. GAN  or Turing Learning ). Our behavioural model could be revised to account for collective departures of agents from one room to the other, as described in biological studies .
Additionally, our methodology could make use of global optimisation techniques designed to minimise the number of evaluations before reaching convergence, like Bayesian Optimisation [37, 13]. This would reduce calibration computation costs, and possibly reduce the time needed to accurately calibrate the models.
This work was funded by EU-ICT project ’ASSISIbf’, no 601074.
-  rsync(1) Linux User’s Manual
-  Bierbach, D., Landgraf, T., Romanczuk, P., Lukas, J., Nguyen, H., Wolf, M., Krause, J.: Using a robotic fish to investigate individual differences in social responsiveness in the guppy. bioRxiv p. 304501 (2018)
-  Bonnet, F., Binder, S., de Oliveria, M., Halloy, J., Mondada, F.: A miniature mobile robot developed to be socially integrated with species of small fish. In: Robotics and Biomimetics (ROBIO), 2014 IEEE International Conference on. pp. 747–752. IEEE (2014)
-  Bonnet, F., Cazenille, L., Gribovskiy, A., Halloy, J., Mondada, F.: Multi-robots control and tracking framework for bio-hybrid systems with closed-loop interaction. In: Robotics and Automation (ICRA), 2017 IEEE International Conference on. IEEE (2017)
-  Bonnet, F., Cazenille, L., Seguret, A., Gribovskiy, A., Collignon, B., Halloy, J., Mondada, F.: Design of a modular robotic system that mimics small fish locomotion and body movements for ethological studies. International Journal of Advanced Robotic Systems 14(3), 1729881417706628 (2017)
-  Bonnet, F., Gribovskiy, A., Halloy, J., Mondada, F.: Closed-loop interactions between a shoal of zebrafish and a group of robotic fish in a circular corridor. Swarm Intelligence pp. 1–18 (2018)
-  Bradski, G.: The opencv library. Dr. Dobb’s Journal of Software Tools (2000)
-  Calovi, D.S., Litchinko, A., Lecheval, V., Lopez, U., Escudero, A.P., Chaté, H., Sire, C., Theraulaz, G.: Disentangling and modeling interactions in fish with burst-and-coast swimming reveal distinct alignment and attraction behaviors. PLoS computational biology 14(1), e1005933 (2018)
-  Cazenille, L., Chemtob, Y., Bonnet, F., Gribovskiy, A., Mondada, F., Bredeche, N., Halloy, J.: Automated calibration of a biomimetic space-dependent model for zebrafish and robot collective behaviour in a structured environment. In: Conference on biomimetic and biohybrid systems. pp. 107–118. Springer (2017)
-  Cazenille, L., Collignon, B., Bonnet, F., Gribovskiy, A., Mondada, F., Bredeche, N., Halloy, J.: How mimetic should a robotic fish be to socially integrate into zebrafish groups ? Bioinspiration & biomimetics (2017)
-  Collignon, B., Séguret, A., Halloy, J.: A stochastic vision-based model inspired by zebrafish collective behaviour in heterogeneous environments. Royal Society Open Science 3(1), 150473 (2016)
-  Collignon, B., Séguret, A., Chemtob, Y., Cazenille, L., Halloy, J.: Collective departures in zebrafish: profiling the initiators. arXiv preprint arXiv:1701.03611 (2017)
-  Cully, A., Clune, J., Tarapore, D., Mouret, J.B.: Robots that can adapt like animals. Nature 521(7553), 503 (2015)
-  De Margerie, E., Lumineau, S., Houdelier, C., Yris, M.R.: Influence of a mobile robot on the spatial behaviour of quail chicks. Bioinspiration & Biomimetics 6(3), 034001 (2011)
-  Deza, M., Deza, E.: Dictionary of distances. Elsevier (2006)
Fortin, F.A., Rainville, F.M.D., Gardner, M.A., Parizeau, M., Gagné, C.: Deap: Evolutionary algorithms made easy. Journal of Machine Learning Research 13(Jul), 2171–2175 (2012)
-  Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
-  Gribovskiy, A., Halloy, J., Deneubourg, J., Mondada, F.: Designing a socially integrated mobile robot for ethological research. Robotics and Autonomous Systems 103, 42–55 (2018)
-  Griparić, K., Haus, T., Miklić, D., Polić, M., Bogdan, S.: A robotic system for researching social integration in honeybees. PloS one 12(8), e0181977 (2017)
-  Halloy, J., Sempo, G., Caprari, G., Rivault, C., Asadpour, M., Tâche, F., Said, I., Durier, V., Canonge, S., Amé, J.: Social integration of robots into groups of cockroaches to control self-organized choices. Science 318(5853), 1155–1158 (2007)
-  Hintjens, P.: ZeroMQ: messaging for many applications. ” O’Reilly Media, Inc.” (2013)
-  Jolly, L., Pittet, F., Caudal, J.P., Mouret, J.B., Houdelier, C., Lumineau, S., De Margerie, E.: Animal-to-robot social attachment: initial requisites in a gallinaceous bird. Bioinspiration & biomimetics 11(1), 016007 (2016)
-  Katzschmann, R.K., DelPreto, J., MacCurdy, R., Rus, D.: Exploration of underwater life with an acoustically controlled soft robotic fish. Science Robotics 3(16) (2018), http://robotics.sciencemag.org/content/3/16/eaar3449
-  Kim, C., Ruberto, T., Phamduy, P., Porfiri, M.: Closed-loop control of zebrafish behaviour in three dimensions using a robotic stimulus. Scientific reports 8(1), 657 (2018)
-  Knight, J.: Animal behaviour: When robots go wild. Nature 434(7036), 954–955 (2005)
-  Landgraf, T., Nguyen, H., Schröer, J., Szengel, A., Clément, R., Bierbach, D., Krause, J.: Blending in with the shoal: Robotic fish swarms for investigating strategies of group formation in guppies. In: Biomimetic and Biohybrid Systems, Living Machines 2014, pp. 178–189. Lecture Notes in Computer Science, Springer (2014)
-  Landgraf, T., Bierbach, D., Nguyen, H., Muggelberg, N., Romanczuk, P., Krause, J.: Robofish: increased acceptance of interactive robotic fish with realistic eyes and natural motion patterns by live trinidadian guppies. Bioinspiration & biomimetics 11(1), 015001 (2016)
-  Landgraf, T., Oertel, M., Kirbach, A., Menzel, R., Rojas, R.: Imitation of the honeybee dance communication system by means of a biomimetic robot. In: Conference on Biomimetic and Biohybrid Systems. pp. 132–143. Springer (2012)
-  Li, W., Gauci, M., Groß, R.: Turing learning: a metric-free approach to inferring behavior and its application to swarms. Swarm Intelligence 10(3), 211–243 (2016)
-  Mondada, F., Halloy, J., Martinoli, A., Correll, N., Gribovskiy, A., Sempo, G., Siegwart, R., Deneubourg, J.: A general methodology for the control of mixed natural-artificial societies. In: Kernbach, S. (ed.) Handbook of collective robotics: fundamentals and challenges, chap. 15, pp. 547–585. Pan Stanford (2013)
-  Mouret, J., Doncieux, S.: Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary computation 20(1), 91–133 (2012)
-  Patricelli, G.: Robotics in the study of animal behavior. In: Breed, M., Moore, J. (eds.) Encyclopedia of Animal Behavior, pp. 91–99. Greenwood Press Westport, CT (2010)
-  Séguret, A., Collignon, B., Halloy, J.: Strain differences in the collective behaviour of zebrafish (danio rerio) in heterogeneous environment. Royal Society open science 3(10), 160451 (2016)
-  Séguret, A., Collignon, B., Cazenille, L., Chemtob, Y., Halloy, J.: Loose social organisation of ab strain zebrafish groups in a two-patch environment. arXiv preprint arXiv:1701.02572 (2017)
-  Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems. pp. 2951–2959 (2012)
-  Stefanec, M., Szopek, M., Schmickl, T., Mills, R.: Governing the swarm: Controlling a bio-hybrid society of bees & robots with computational feedback loops. In: Computational Intelligence (SSCI), 2017 IEEE Symposium Series on. pp. 1–8. IEEE (2017)
-  Suzuki, S., be, K.: Topological structural analysis of digitized binary images by border following. Computer Vision, Graphics, and Image Processing 30(1), 32–46 (1985)
-  Toms, C.N., Echevarria, D.J.: Back to basics: searching for a comprehensive framework for exploring individual differences in zebrafish (danio rerio) behavior. Zebrafish 11(4), 325–340 (2014)
-  Vaughan, R., Sumpter, N., Henderson, J., Frost, A., Cameron, S.: Experiments in automatic flock control. Robotics and Autonomous Systems 31(1), 109–117 (2000)
-  Zabala, F., Polidoro, P., Robie, A., Branson, K., Perona, P., Dickinson, M.: A simple strategy for detecting moving objects during locomotion revealed by animal-robot interactions. Current Biology 22(14), 1344–1350 (2012)