A Mosquito Pick-and-Place System for PfSPZ-based Malaria Vaccine Production

04/12/2020
by   Henry Phalen, et al.
0

The treatment of malaria is a global health challenge that stands to benefit from the widespread introduction of a vaccine for the disease. A method has been developed to create a live organism vaccine using the sporozoites (SPZ) of the parasite Plasmodium falciparum (Pf), which are concentrated in the salivary glands of infected mosquitoes. Current manual dissection methods to obtain these PfSPZ are not optimally efficient for large-scale vaccine production. We propose an improved dissection procedure and a mechanical fixture that increases the rate of mosquito dissection and helps to deskill this stage of the production process. We further demonstrate the automation of a key step in this production process, the picking and placing of mosquitoes from a staging apparatus into a dissection assembly. This unit test of a robotic mosquito pick-and-place system is performed using a custom-designed micro-gripper attached to a four degree of freedom (4-DOF) robot under the guidance of a computer vision system. Mosquitoes are autonomously grasped and pulled to a pair of notched dissection blades to remove the head of the mosquito, allowing access to the salivary glands. Placement into these blades is adapted based on output from computer vision to accommodate for the unique anatomy and orientation of each grasped mosquito. In this pilot test of the system on 50 mosquitoes, we demonstrate a 100 placing the mosquito with its neck within the blade notches such that the head can be removed. This is a promising result for this difficult and non-standard pick-and-place task.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 3

page 4

page 5

page 6

page 7

page 9

page 12

06/12/2015

CloudCV: Large Scale Distributed Computer Vision as a Cloud Service

We are witnessing a proliferation of massive visual data. Unfortunately ...
03/05/2019

An Efficient Production Process for Extracting Salivary Glands from Mosquitoes

Malaria is the one of the leading causes of morbidity and mortality in m...
04/27/2018

Routing Driverless Transport Vehicles in Car Assembly with Answer Set Programming

Automated storage and retrieval systems are principal components of mode...
05/16/2014

Computer Vision Approach for Low Cost, High Precision Measurement of Grapevine Trunk Diameter in Outdoor Conditions

Trunk diameter is a variable of agricultural interest, used mainly in th...
10/10/2021

Nano Version Control and Robots of Robots: Data Driven, Regenerative Production Code

A reflection of the Corona pandemic highlights the need for more sustain...
03/23/2018

Welfare Without Taxation - Autonomous production revenues for Universal Basic Income

In the face of shifting means of production from manual human labor to l...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Malaria presents a tremendous public health burden. The World Health Organization estimates 228 million individuals worldwide were infected with malaria in 2018, and with an estimated 405,000 deaths, the disease is among the top 20 leading causes of death globally among both adults and infants

[WHOreport2019, owidcausesofdeath]. With increasing drug and insecticide resistance, it has become difficult for current treatments to maintain efficacy in reducing the prevalence of malaria worldwide [WHOstrat]. Development of malarial vaccines present a promising way forward in the global effort for malaria eradication [WHOstrat].

Humans become infected with malaria-causing parasites when Anopheles mosquitoes inoculate the sporozoite (SPZ) developmental stage of the parasite. SPZ reside in mosquito salivary glands immediately prior to transmission during feeding. Progress has been made in the development of the Sanaria Plasmodium falciparum (Pf) sporozoite-based vaccine (Sanaria® PfSPZ Vaccine), an effective vaccine manufactured from PfSPZ extracted from the salivary glands of female Anopheles mosquitoes [meta, ishizuka, epstein, sissoko2017safety, lyke2017attenuated, mordmuller2017sterile]. Such a vaccine may reduce the burden of the disease by providing immunity against Pf, the most common malarial parasite, which was estimated to account for greater than 95% of deaths caused by malaria in 2017 [WHOreport2018]. These vaccines have shown a high level of protective efficacy against controlled human malaria infection and malaria transmitted in the field, making them ideal for large-scale malaria elimination campaigns in geographically defined malarious regions. However, a limiting step in the manufacture of PfSPZ-based vaccines has been the extraction of the salivary glands and isolation of sporozoites from very large numbers of infected mosquitoes to meet expected demand for the vaccine.

The process of salivary gland dissection has only been demonstrated with training-intensive manual processes. In these traditional methods, technicians are presented with freshly-sacrificed, lab-grown mosquitoes and process them under a microscope, one at a time. To gain access to the glands, a technician first removes the mosquito head using the beveled edge of a hypodermic needle as a knife. Next, the technician gently squeezes the mosquito body to remove from the thorax a volume of exudate that includes the PfSPZ-laden salivary glands. The exudate from mosquitoes is collected and processed for the isolation of PfSPZ. Technicians require extensive training to be proficient at dissection and have a throughput of around 4-5 mosquitoes per minute on average. In order be certified for manual dissection at Sanaria, untrained personnel go through a rigorous training procedure involving 1-3 one-hour sessions every week. Each trainee is graded on metrics including cleanliness of the exudate, throughput, and SPZ yields from infected mosquitoes. Training is considered complete if the trainee’s throughput is approximately 250 mosquitoes dissected per hour (Mdph) and the sporozoite yields are within 25% of a certified dissector. The time to complete this training has varied tremendously between operators depending on dexterity and hand-eye coordination skills for successful micromanipulation of mosquitoes under a stereomicroscope. Although further entrainment does occur for every operator as they continue participation in dissection, the gestation period prior to qualification is too variable and long, averaging around 16 weeks.

The automation of salivary gland harvesting from mosquitoes has been attempted, to our knowledge, only one time in the past. This included an effort to fully automate the salivary gland extraction production process by means of a robot. Although there is very little technical material publicly available about this system beyond a YouTube video included as part of a fundraising effort [youtube] and news articles [Lap, bor], the proposed system evidently included a computer vision subsystem to locate mosquitoes in a dish and a Cartesian (XYZ) robot to position an end effector to grasp the mosquito and feed each one sequentially into a tube for further processing. However, no literature supports the success of any such process at this time.

Our group proposes a novel system, automated end-to-end, to perform the mosquito dissection for PfSPZ vaccine production. This is a complex system with several components, each targeting a stage of the dissection process. The main components include a feeder mechanism for sorting and presenting mosquito bodies, a dissector which automatically removes the mosquito head and squeezes and harvests the exudate, and a pick-and-place robot that manipulates mosquito bodies from the feeder to the dissector. Further, several computer vision implementations are used in the control and verification of these components’ functions.

To help define formal design requirements and clarify the design feasibility and priority for components within this larger system, we first designed [patent] a mechanical fixture device to improve the performance of manual dissection. This manual mosquito micro-dissection fixture (3MDF) is a simple, modular fixturing system that allows several time-consuming steps of the process to be performed concurrently on multiple mosquitoes, while also greatly simplifying the remaining per-mosquito actions performed by the human technician. We show that the time required to train an operator to perform the dissection procedure is reduced using the 3MDF, while average throughput is also increased.

Though a demonstrable improvement over traditional manual methods, the 3MDF was developed only as a first step towards an end-to-end automated dissection system to enable world-wide vaccination efforts. The decision to first pursue the 3MDF proved essential to developing effective automation techniques. As a result of the 3MDF, the design of the automated system was able to become largely centered on simply automating the steps within this fixturing process: identify a mosquito, move the mosquito to a favorable position, grab the mosquito by its proboscis, place the neck in cutting blades, cut to remove the head, squeeze the exudate, and harvest the exudate. Parallel processing is a key feature of our proposed automation system for mosquito dissection. Many of these steps can be performed concurrently by a machine whereas a single technician is limited by performing them sequentially. Thus, our approach develops component systems to achieve these individual subtasks. Based on preliminary testing, we identified the rate-limiting step (as well as an absolute necessity for full automation) to be the pick-and-place of mosquito bodies. This demands accurate visual perception paired with physical precision in order to recognize a mosquito, analyze it, and best align it to remove the head. We report our work to overcome these challenges through the development of a vision-guided, robust pick-and-place robotic system for loading mosquitoes into a 3MDF-based device. As our research group works toward development of an end-to-end automated mosquito salivary gland dissection system, the demonstration of a robust pick-and-place apparatus is a key milestone in realization of that goal.

Ii Manual Mosquito Micro-dissection Fixture

Ii-a Design

In developing the 3MDF approach, we recognized that the fundamental problem was to align each mosquito so that the decapitation and gland extraction steps could be achieved without needless complications from extraneous mosquito parts such as legs and wings. Further, this could be accomplished with relatively simple fixturing enabling the technician to load batches of mosquitoes into cartridges aligning them so that subsequent steps could be performed in parallel.

Our fixture design (shown in Fig. 1) consists of several modular components, including sorting cartridges, blade assembly, squeezer, and staging area. Each sorting cartridge has 20 slots allowing for the dissection of 20 mosquitoes at a time. A slot is wide, making it slightly wider than a mosquito body so that the mosquito can easily be placed in the slot but still held in position during the subsequent squeezing and harvesting stages. The slot length and depth are and respectively. Because the surface over which the mosquitoes are dragged must be smooth, each cartridge has a staging area made of acrylic. The staging area is by . About 30 to 40 mosquitoes can be spread out over this area which is sufficient to efficiently fill the cartridges. The sorting cartridge is removable so that when the salivary glands are extracted, they do not become trapped behind the blades.

Figure 1: Design of mosquito gland extraction apparatus, including the sorting cartridge (blue), the blade assembly (pink), squeezer (brown) and staging area (tan).

The blades have notches approximately wide in which the mosquito neck sits. Like hair clippers, the blades slide past one another, cutting off each mosquito head simultaneously. Because the mosquito’s neck is contacted by a sharp edge on both sides, this blade design ensures a clean cut. The blades are spring loaded so that they sit flush against the sorter cartridge. Both blades are removable, allowing for them to be easily cleaned or simply replaced. The blades are made of thick stainless steel and are thin enough that sharpening is not necessary.

The squeezer is comb-like, with 20 rectangular teeth that fit into each of the slots on the sorter. To ensure the glands are extracted, the squeezer must contact each mosquito in the thorax region. Therefore, the squeezer has two round pegs that fit into matching holes in the sorter to guarantee perfect alignment. The teeth fit in the slots with very little clearance to ensure that the glands are extruded forward and do not become trapped between the side of the slot and the squeezer tooth.

Ii-B 3MDF Methods

The workflow proceeds as follows: First, a cartridge is inserted into the apparatus so that the slots align with the blade openings, and a clump of mosquitoes is placed onto the staging area. A small amount of an aqueous medium is also placed onto the staging area. Using tweezers, the technician grasps mosquitoes one-at-a-time by the proboscis and drags the mosquito into a cartridge slot and places it so that the mosquito neck is between the clipper blades, as shown in Fig. 2. This process causes the legs and wings of the mosquito to fold back along the mosquito body, where they are constrained by the cartridge slot. This process is repeated until all cartridge slots are filled.

Figure 2: Sorting mosquitoes into cartridge slots. The technician grasps each mosquito by its proboscis and drags it into a slot.

Once the necks are aligned between the blades, the blades are actuated manually via a button on the side of the device (labeled in Fig. 1), enabling them to slide past one another and sever the neck of each mosquito. Next, the sorting cartridge is removed, and another empty sorting cartridge can be inserted.

Finally, the glands are extracted. To do this, the squeezer comb is aligned with the cartridge by placing its aligning posts into the corresponding holes in the cartridge so that the teeth on the comb rest on the mosquito thoraces. The technician then presses down to squeeze the glands out of the thoraces. The glands are ejected from the mosquito onto the flat surface in front of the blades where they can be collected by a pipettor and placed into collection tubes.

In this study, eight untrained operators were tasked to complete the 3MDF procedure. Task completion time was tracked to determine mosquito throughput for comparison with other dissection methods.

Iii Automated System Design Concept

Iii-a System Overview

The tools and methods used in the 3MDF served as a blueprint for the design of an automated mosquito dissection system, being better suited for automation compared to the traditional dissection methods. Here, we focus primarily on the robotic pick-and-place component of the fully automated mosquito micro dissection system. The larger system will ultimately take freshly sacrificed mosquitoes suspended in aqueous culture media and output a collection of mosquito exudate including PfSPZ-laden salivary glands. A concept of this dissection system and a prototype realization are provided in Fig. 3. We briefly describe this system to clarify the context of the robotic pick-and-place subsystem, which will be the middle of three primary system components.

Figure 3: Automated mosquito dissection system. (a) Concept image for system. (b) Prototype realization of automated system. Background has been removed for visualization.

First, a staging apparatus will separate mosquitoes and present them one at a time to the robot. Freshly sacrificed mosquitoes sit in a basin of media beneath the system. A spinning rotor in the basin creates a vortex that will carry mosquitoes in solution to the top of a separation cone. This cone has channels in one sector down which media will flow onto a ring of orientable mesh-bottomed cups. This ring will rotationally index around the cone so that, by controlling the vortex speed and concentration of mosquitoes in the basin, the cups will on average have one mosquito on them once they pass beyond the sector of the cone with channels. At an index beyond the channel, a camera will image a single cup and a computer vision algorithm will determine if a mosquito is present. If so, at the next indexed position, the cup will be rotated to orient the mosquito so that the mosquito’s proboscis will point radially outward from the ring. Finally, the ring will be rotated to an index such that the cup is aligned with a tangent linear stage that will comprise the third subsystem, a dissection assembly line. The development of the staging apparatus is described in detail in [mengdi].

The pick-and-place robot will be positioned on the other side of the linear stage and will reach over to the cup, grasp the mosquito by its proboscis and drag it onto a 3MDF-resembling cartridge attached to a linear stage. Similar to how a human technician would use the 3MDF, the robot will drag the mosquito into a slot and place the mosquito’s neck into notches cut in two parallel dissection blades. An overhead camera will be used to provide computer vision feedback of this process. The blades will be actuated, cutting the head. After disposing of the mosquito’s head, the robot will return to the ring which will have rotated to present a new mosquito. The linear stage will index laterally immediately after the mosquito is cut. As additional mosquito bodies are positioned on the cartridge, the linear stage will translate and expose mosquitoes to stations where a comb structure, like that in the 3MDF, will be lowered onto the mosquito to squeeze out the exudate and allow for salivary glands collection. Work on the dissection assembly line is currently ongoing within our research group.

Iii-B Requirements

We focus here on the robotic pick-and-place system, along with its difficult and important task of picking up mosquitoes presented on a mesh surface, and precisely placing them so that only the neck lies within the blades. In order to extract the salivary glands of the mosquito, the dissection point has to be precisely at the intersection of the head and body. If the mosquito is not placed far enough into the blade, the cut will occur on the head, leaving no passage for the exudate to be squeezed out, effectively wasting the mosquito and PfSPZ within. Because the salivary glands are located just behind the mosquito’s head, placing the mosquito too far into the blade would result in some of the gland being lost in the cut, also decreasing PfSPZ yield. The mosquito neck is approximately in length, and the blades each have a thickness of , leaving only about for error. Moreover, grasping must occur only on the proboscis to prevent any damage to the body that might ruin the salivary glands or create an alternative opening from which exudate may squeeze out. The proboscis is on average long, with a diameter of approximately .

In addition to size challenges, this procedure presents multiple difficulties not typically faced in a standard pick-and-place procedure. One of the main challenges is the mosquito-to-mosquito variation. Some of this is anatomical in origin. Each mosquito varies somewhat in size and is not axis-symmetric, meaning the alignment of the neck relative to the body depends on which side the mosquito body lies on. Further, mosquitoes are very flexible. By grabbing and pulling the mosquito from the proboscis, the body tends to straighten out in time for placement, but first, the mosquito must be identified and grasped from a variety of twisted, compressed, or otherwise contorted orientations. While upstream processes are expected to align the mosquito’s proboscis within 15 degrees of an ideal orientation for the robot to grab, the mosquito can still be located anywhere on the cup and must be grasped accordingly. Because of its length, the proboscis can still be grabbed even if there is some error in the robot positioning, or computer vision targeting. However, this, combined with the general variability in proboscis lengths, means that the offset between the robot’s grasp point and the mosquito’s neck is not consistent trial-to-trial. As such, it is not enough to program a sequence of robot movements; these challenges necessitate adaptive automation. Based on visualization of the mosquito’s anatomy and its grasping location, the robot must perform customized movements to successfully grasp and place the mosquito.

Iii-C Experimental Setup

The robot used in this procedure is a 4-DOF, linear stage robot by New England Affiliated Technologies, Lawrence, MA (Fig. 4). A dual-axis X-Y table is used as the base for the robot, onto which a Z axis is mounted orthogonally (NEAT: XYR-6060 and NEAT: LM-400 respectively). The robot also has a rotary axis which is not used in this study. Each axis is driven by a 12V DC servo motor, with a leadscrew, has a travel of , and is coupled with an incremental encoder. The positioning resolution of these axes was measured with a dial indicator to be approximately . The entire assembly is mounted to an optical table. Robot motion is driven by a Galil controller (DMC-4143), interfaced to a Linux computer by ethernet connection. Attached to the robot is a custom-designed micro-gripper mechanism visualized in Fig. 5. A cam mechanism attached to a HexTronik HXT900 servo motor drives the rail of a linear guide within its carriage, causing the tooltip to open and close. The tooltip of the micro-gripper is adapted from an Alcon Grieshaber retinal surgical forceps. Movement of the linear guide rail extends or retracts a sleeve over normally-open gripper jaws. The micro-gripper is controlled by sending position commands to the servo motor via USB serial communication from the computer to an Arduino Uno microprocessor.

Figure 4: Experimental Setup. A close-up image of dissection blades and cartridge is inset with length scale for reference.
Figure 5: Custom-designed micro-gripper used to grasp mosquitoes.

Mosquitoes are staged for dissection on a modified 3MDF device that is also mounted to an optical table. The 3MDF cartridge is modified to have a hole away from the blades in which is placed a diameter cup that matches the one used in the upstream staging apparatus. This cup is covered with a nylon mesh that is used for media drainage in that apparatus. The mosquito is dragged into a slot in the 3MDF cartridge and placed into the 3MDF blades. These are two thick stainless steel blades with wide by deep notches cut in them to match the midpoint of the slots. The closest blade to the cartridge is stationary while the further blade can be manually actuated by pressing a button on the side of the device. This action causes the mosquito neck to be caught between the two blades and cut.

The setup also includes three cameras (Fig. 4). An overhead microscopic camera (OptixCam Summit D3K2-5) with an Omano OM-10K zoom lens is used to capture a complete view of the workspace and is used by the computer vision to identify a mosquito’s presence and approximate location. A second camera (Plugable USB Microscope Camera), is mounted on the robot and is used to identify the location of the mosquito’s body parts for accurate picking and placing. We refer to these as the overhead and onboard cameras respectively. A third camera (Opti-Tekscope USB Microscope Camera) is placed to the side and rear of the setup so that its visual field is in line with the blades. This camera is not necessary for system operation and is only used to visualize placement to evaluate trial success.

The automated procedure uses the overhead and onboard cameras to guide the robot’s motion. The procedure consists of three stages. In the first stage, an image of the entire workspace is captured using the overhead camera. This image is converted to HSV space and the mosquito is segmented out. Next, a bounding box is fit to this region and a weighted centroid is calculated for the mosquito, as shown in Fig. 6(a). The tooltip is moved near the mosquito, to a position where the onboard camera, due to its close proximity, is able to capture an image of the mosquito with more features and details.

In the second stage, a computer vision algorithm identifies the mosquito’s proboscis in the detailed onboard camera image shown in Fig. 6(b). The tooltip is moved to a point above the centroid of the proboscis (Fig. 6(c)), which is used as the grasp location for the mosquito (Fig. 6(d)). Finally, the robot drags the mosquito to an empty slot on the cartridge near the blades.

In the third stage, the onboard camera captures a final image shown in Fig. 6(e) with the tooltip in view to detect the mosquito head-to-tooltip offset. The robot uses this offset value to position the mosquito with its neck between the dissection blades (Fig. 6(f)). Our group is also investigating the use of a keypoint-based computer vision approach [hongtao], which was not used in this study.

Figure 6: Side view of robot path and related representative images captured by the vision system. (a) Image captured from overhead camera showing bounding box of detected mosquito. (b) Image captured from onboard camera to determine proboscis centroid. (c) Image captured from onboard camera before grasping. (d) Image captured from onboard camera immediately after grasping. (e) Image of the mosquito taken used to calculate head-to-tooltip offset. (f) Image after aligning the mosquito neck with the blades.

Iii-D Calibration

To relate the robot and camera coordinate systems, we use a two stage calibration process. In the first stage, the tooltip of the micro-gripper is located in the overhead camera frame. The tool is segmented from the background in HSV space using Otsu’s binarization

[otsu], and contour identification is used to detect the tool. The lowest point of the tool contour is then used as the tooltip. In the second stage of calibration, the robot is moved through the camera space across a grid of points. The tooltip is detected and recorded at each position. The resulting grid of points from both coordinate systems are used as the inputs for a Bernstein polynomial fitting routine as performed in [bernstein]. In this routine, two fourth degree polynomials are fit to create a mapping from the camera coordinates to the robot’s encoder coordinates. Such a polynomial fitting method also compensates for radial and aspherical lens distortions.

The tooltip does not move with respect to the overhead camera, so the polynomial fitting method described above could not be used to calibrate this camera. Instead, a pre-calibrated grid of a resolution x was placed in the background. The robot was then moved by a known distance along each axis, and images were captured before and after motion. The pixel motion of the grid was calibrated to the corresponding change in robot encoder counts.

The location of the cartridge and blades in robot coordinates were determined using a shim. The robot was slowly advanced until the robot held the shim firmly to the surface of interest and encoder counts at this location were used as a reference.

Iv Mosquito Localization and Segmentation

The vision system consists of two cameras: the overhead camera and the onboard camera, as shown in Fig. 4. The two cameras assist in the task of localizing the mosquito position and orientation within the robot workspace and then identifying the proboscis grasp location. This is done in multiple stages to pick and place the mosquito accurately, as shown in Fig. 6.

Iv-a Approximate localization using overhead camera

The first stage consists of locating the mosquito in the overhead camera frame, captured at a high resolution (2560x1922 px). This is done with a series of operations on the obtained image:

  1. Gamma Correction is performed to remove high intensity details of image, and to increase the contrast of the lower intensity mosquito region in the image.

  2. The image is then cropped to remove blades and the fixture at static locations in the image.

  3. The image is then converted to HSV space, and the saturation channel is selected as the most representative image for segmentation.

  4. Gaussian Blur () is then applied to the saturation channel to reduce noise and false edges within the image.

  5. Next, Otsu’s [otsu]

    binarization is used to obtain optimal upper and lower bounds for thresholding, maximizing the variance between groups. This leads to an approximate binary mask representing the mosquito within the image.

  6. Morphological opening operation is then performed on the binary mask to remove smaller objects from the foreground.

  7. Gaussian blurring () is then applied to further smoothen the image, and remove any noise in the mask.

  8. Morphological erosion () is then performed to particularly smoothen the contours of the mask.

  9. Connected component analysis is then used to select the largest regions within the mask image, above an area threshold (5% of image area).

  10. The largest regions of the mask obtained and its bounding boxes are then used as approximate locations of mosquitoes from the overhead camera. The bounding box and centroid of the mosquito is as shown in Fig. 6 (a) in green.

The bounding box obtained is used for pick procedure to move the gripper to initial location.

Iv-B Accurate location using onboard camera

In the second stage, the onboard camera image is used. The mosquito needs to be grasped on its proboscis for a successful grasp so no damage is made to the body or the salivary glands within the thorax. To dissect the mosquito, the neck needs to be accurately placed within the 3MDF device. For this purpose, a deep convolutional neural network is used to identify different anatomical regions of the mosquito such as proboscis, head, or body from a given image as a dense pixelwise mask. This mask is then used to infer the grasp point and the dissection point accurately in stage 2 and 3.

Figure 7: Network architecture for segmentation along with overlaid segmentation mask on image showing the detected proboscis, head, body and the background.

To train the network on such a novel task, a dataset of 185 images (1600x1200 px) of mosquitoes in various positions and orientations were captured from the onboard camera. This dataset was then annotated manually with dense pixel labels for the three regions: proboscis, head, and body. The images and labels were then downsampled to a resolution of 300x400 px for faster training. To leverage the power of previous work in image classification in computer vision, a SegNet model [segnet] pretrained with VGG-16[vgg16]

weights was used. This enables the initial layers of the network to function as generalizable feature detectors, whereas the final layers are fine-tuned by transfer learning specifically for this task.

To generalize the network weights and generate a larger training dataset, rigorous data augmentation was used during the network training phase. Since the mosquitoes can present in any orientation or position, random rotation between was applied along with random X-Y reflections. The image-label pairs were then randomly translated along both X and Y axes by 100px. Finally the pairs were also scaled randomly between scaling factors to compensate for variability in sizes of mosquitoes. Both Generalized dice loss [generalDice]

and multi-class cross entropy loss were tested as the loss functions, however weighted multi-class cross entropy loss yielded better results. Since the proboscis occupies a much smaller region within the image in comparison to the background, each class was weighted by the inverse of area of each region in training labels. The loss function was modified accordingly.

The model weights were then fine-tuned on this dataset for 1500 epochs, after which the model converged and no significant changes in accuracy were observed. During inference, the onboard camera only encounters cases where one mosquito is in the image at any instance. To enforce this prior knowledge, using connected component analysis, the largest regions within each label class were then selected as the mask. All the other regions were assigned as background class. Another prior condition enforced during inference was the edge connectivity between proboscis to head, and head to body. This was done by dilating both relevant classes and using intersection of images as the edge. The centroid of this edge was then used as the dissection point between head and body.

V Experimental Methods

V-a Study Design

Testing was performed to investigate the efficacy of the designed system to pick up a mosquito and place that mosquito within blades that can remove the insect’s head. The process was performed on 50 non-infected Anopheles mosquitoes. Prior to testing, the mosquitoes were kept in an airtight, refrigerated container of phosphate-buffered saline solution (PBS) following sacrifice one day prior. Functioning as a unit test for this subsystem within the eventual automated mosquito dissection system, only the grasping and subsequent positioning of the mosquitoes by the robot were considered for trial success or failure. All actions of the system during the test were performed autonomously with feedback from computer vision, and a manual cut was performed at the end of each trial to facilitate determination of trial success.

V-B Pick Procedure

The experimental procedure is demonstrated in Fig. 6. A mosquito is removed from the PBS solution by its proboscis with tweezers and placed anywhere on a circular mesh cup of radius with its center placed away from the blades as measured on the central axis of the cartridge slot. The mosquito was placed so that the proboscis was positioned forward toward the blades, pointing within 45 degrees this line. One such placement is shown in Fig. 6(a). These conditions were chosen to mimic the worst case expected from the upstream mosquito-staging apparatus that this process will later be integrated with. To further match the expected results of this upstream process, no further attempt at standardization of mosquito starting position were made (e.g. what side the mosquito was lying on, relative straightness of legs). The micro-gripper tooltip begins the trial at a location away from the cup and above the cartridge surface.

A bounding box around the mosquito is identified by computer vision in an image from the overhead camera, and the robot moves to a point in front of the centroid of that region (Fig. 6(b)). This brings the mosquito into view of the onboard camera without placing the gripper over top of the mosquito body. By lowering toward the mesh surface, the mosquito is brought into focus. The centroid of the proboscis region is identified and the robot moves the gripper to a location above this this point (Fig 6(c)), and then drops down to the mesh surface and the gripper is closed to grab the proboscis (Fig 6(d)).

The robot lifts up and drags the mosquito to a position from the blades (Fig. 6(e)). Here, an image from the onboard camera is again analyzed by the computer vision system. This task serves two functions, to confirm successful grasping of the mosquito, and to determine more accurately where on the proboscis the gripper has grabbed. The trial is considered a successful demonstration of grasping if the mosquito is visualized as being held within the micro-gripper at this point.

V-C Place Procedure

The vision system provides the location of the proximal end of the proboscis, where it attaches to the mosquito’s head. This location is transformed into robot coordinates and a head-to-tooltip offset is determined by subtracting it from the current encoder values. Only the offset in line with the cartridge grooves (a horizontal offset in Fig. 6(e)) is considered. The robot then executes another set of programmed movements. The robot raises the mosquito head and moves forward a nominal distance to clear the blades plus the offset, such that the mosquito’s neck should be right above the blades (Fig. 6(f)). Then the tooltip moves down , placing the neck within the notch of the blades if properly aligned. At this point, another subsystem of the automated mosquito dissection system would actuate the blades to cut the head and further process the mosquito. In this unit test, the blade is manually actuated. The test is considered a successful placement if the mosquito’s neck is placed into the notch of the dissection blades such that the head could be removed. As a final step of the process, the robot pulls away from the blade, moving the head, if still in its grasp, to a location where it can be cleaned off with a modest jet of air or fluid that does not disturb the tooltip calibration. Video footage from all three cameras is recorded throughout and saved for analysis. The commanded speed of each robot axis was , chosen somewhat conservatively compared to the robot’s top speed to minimize overshoot. The speed was decreased to when lowering the mosquito neck into the blades, reducing the inertia of the mosquito and thus the tendency to pivot or flip over the blades rather than settle between them.

Vi Results

Vi-a Training and Performance Comparisons

Using the 3MDF, untrained operators demonstrated high production throughput rates. Even in their very first trials, operator times ranged from 338-649 mosquitoes per hour (Table I). The average throughput was 470 mosquitoes dissected per hour (Mdph). After this test, trainees continued their course over the next several days. Entrainment, as defined by Sanaria’s procedures, occurred over as little as 3 trials. The number of sessions required to complete training was fairly consistent, taking one day (3–10 trials over one hour) to 3 days, compared to the manual process which averaged 16 weeks with 1-3 one-hour sessions per week. This represents at least a five-fold reduction in average training time. Every trainee was able to achieve a total mosquito alignment and decapitation time for a 20 mosquito cartridge of less than 1.5 minutes and gland collection time of 0.5-1.0 minute, similar to operator 6, (Table I) and a total output of 600 mosquitoes per hour. In comparison, the dissection rate by the traditional manual methods performed by trained operators averaged around 290 Mdph with a wide range in individual operator capabilities (260 to 430 Mdph) as shown in Fig. 8.

The automated pick-and-place task, which our preliminary tests indicated would be the rate-limiting step of the end-to-end automated dissection system, took on average 7.5 seconds to complete one cycle in this test (Fig. 8

). This consisted of 7.34 seconds for robot movement and 0.16 seconds for image processing on average. This corresponds to a rate of approximately 470 Mdph, with a small variability (standard deviation of 11 Mdph) due to variable placement resulting in different distances over which the mosquito was dragged in a given cycle. There remains room for substantial optimization of the robot trajectory in future design iterations. The right-most box in Fig.

8 represents the theoretical throughput associated with removing 2.5 seconds from the robot movement for each cycle, which we predict would be possible without mechanical changes (see Section VII).

Figure 8: Mosquito throughput based on dissection method. Shown are throughput rates in mosquitoes dissected per hour (Mdph). The boxes represent to percentiles of data, and the red lines represent median values. The traditional manual process is from internal production data from Sanaria with trained operators. The 3MDF data represents the results in Table I. The pick-and-place throughput data is estimated based on cycle times of the robotic system in this test. The projected automated system throughput is added to show the estimated throughput with optimized robotic movement, calculated by removing 2.5 seconds from each cycle time in this test (see section VII).
1 2 3 4 5 6 7 8 Avg.
(A) 2.4 2.5 2.3 2.5 1.5 1.3 1.1 1.2 1.9
(B) 0.7 0.8 0.5 1.1 1.2 0.7 1.1 0.7 0.8
Total 3.1 3.3 2.8 3.6 2.7 2.0 2.2 1.9 2.7
Rate 393 364 429 338 444 600 545 649 470

(A) Time taken (in min.) to align 20 mosquitoes; (B) Time taken (in min.) for gland extrusion and collection. Total time for 20 mosquitoes (in min.), and rate is mosquito throughput per hour. Eight untrained operators were tested as a part of this study.

Table I: Production Rates for Untrained Operators using 3MDF

Vi-B Mosquito Localization and Segmentation

The segmentation network was trained for 1500 epochs and both training and validation losses converged as shown in Fig. 9. The network accuracy was measured using multiple metrics as defined and described in [segmmetrics]

, including pixelwise accuracy and weighted intersection-of-union(w-IoU). Since the classes are highly imbalanced with proboscis and the head being most important, w-IoU is selected as a metric to combine accuracy and class weight. Note that the validation accuracy is always higher than the training accuracy since we use rigorous data augmentation during training phase of the network. The global pixelwise accuracy across all classes was 88.3%, however this includes the background as a class. Only the proboscis and head are of particular interest and the accuracy obtained for both were 0.7719 and 0.8408 respectively. The confusion matrix for class accuracy is as shown in Fig.

10. The prediction improved further after performing the edge-connectivity filtering and post-processing steps as described in Section IV-B. An example of segmentation predicted by the network is as shown in Fig. 7, as the output of network.

Figure 9: Network training accuracy and losses. The network converged after approx. 1500 epochs taking 195 mins, with a training accuracy of 84.0% and validation accuracy of 91.8%.
Figure 10: Normalized pixelwise classification accuracy, highlighted according to classification accuracy. The overall network accuracy across all classes is 88.3%.

Vi-C Automated Pick-and-Place Results

Throughout the automated experiment, there were no issues moving to a mosquito’s location, grasping it by the proboscis, or dragging it on the surface of the cartridge. All 50 (100%) of the mosquitoes were observed with the proboscis grasped by the micro-gripper during the second vision check (Fig. 6(e)). Of these 50 mosquitoes, 45 (90%) were placed such that their necks were aligned correctly within the blades. Placement was considered successful if the alignment allowed the blades to cut the neck such that the head could be fully removed. The results were confirmed post-test with close-up video taken of the blades during the placement and cutting steps. An example of a mosquito being accurately placed is provided in Fig. 11(a).

The five mosquitoes that were not accurately placed exhibited similar behavior, flipping over the blades when pulled down by the robot. This action is demonstrated in Fig. 11(b). In these cases, the mosquitoes appear to collide with either the slot walls or the blades. That collision point acts as fulcrum, causing the downward motion of the robot to flip them over the blades, rather than pull the neck into the notches. We were unable to correlate this behavior with any other variable including initial mosquito orientation, grasp location of the proboscis, trial number, or a qualitative assessment of the computer vision’s head-to-gripper offset estimation.

Figure 11: A demonstration of mosquito placement. (a) A mosquito being accurately placed with neck between the blades. (b) Inaccurate placement of the mosquito, resulting in the body pivoting over the blades when the proboscis is pulled down by the gripper, rather than the neck settling between the blades.

Vii Discussion

The immediate goal was to develop production fixtures and methods that could significantly improve the existing manual process while providing the experience necessary to develop a fully automated mosquito dissection and gland extraction process. Accordingly, we have developed prototypes for robust mosquito alignment, decapitation and gland extraction, and demonstrated highly successful performance of a robotic pick-and-place subsystem.

Gland extraction from mosquito thoraces is vastly simplified and deskilled using the 3MDF as indicated by improved training capability. The training time to qualify individual operators for mosquito processing is radically reduced compared to manual dissection while simultaneously doubling throughput. While not as easily scaled or as consistent as an automated solution, the 3MDF appears to provide a substantial interim improvement, while simultaneously serving as the task template for an end-to-end automated system. This is evidenced by the successful development of an automated system to perform the pick-and-place task from the 3MDF method. The demonstration of a simplified dissection procedure, coupled with automation of a key component step, provides confidence in our approach as we work to automate all 3MDF procedure tasks and move towards fully automated mosquito dissection for malaria vaccine production.

Developing the 3MDF required considerable experimentation with alternative designs before we converged the embodiment described here, and we will discuss these experiences briefly. The approach of grasping each mosquito’s proboscis and dragging it across a surface lubricated with aqueous culture medium into an appropriate cartridge slot was very successful from the beginning. We explored several different decapitation methods and apparatus before settling on the clipper blades arrangement described above. This method reliably severs the necks with minimum displacement of the mosquito bodies, unlike single-blade methods. Also, technicians do not find it difficult to guide mosquitoes so that the necks are properly positioned between the blades. Similarly, we explored several alternative approaches for salivary gland extraction and collection before finally settling on our current comb squeezer and pipette collection approach. In particular, we found that having a fairly tight fit between the squeezer comb teeth and the cartridge slots ensures that the gland material is extruded to the front surface of the cartridge. At this point, the glands tend to stick to the front surface, and the suction device can easily gather them, as well as any that have slid down to the bottom of the cartridge.

The robotic pick-and-place subsystem demonstrated highly successful results in this unit test. With no failures in grasping or moving the mosquito, the micro-gripper and computer vision methods were shown to be adequately designed for the task. The ability of the system to achieve these promising results indicates that the computer vision system was effective at providing appropriate adaptations to robot movement. Although we are still working to improve the system, the 90% success rate from this study is very encouraging, especially considering the challenges presented by this non-standard pick-and-place task. It is not surprising that the placing task would prove more difficult than grasping the mosquitoes as it requires more accuracy. In order to move the mosquito, the robot can grasp anywhere on the length of the proboscis. Placing the mosquito’s neck between the blades requires more precision and any inconsistencies in grasping, deformation, and anatomy must be accommodated in this step. In the few cases where adequate placement was not achieved, the mosquitoes were observed to flip over the blades about a contact point with either the blades or the cartridge. This behavior occurred both when the neck appeared to be misaligned with the blade notches as well as in cases when the alignment appeared adequate. When there was neck misalignment, either the head or body of the mosquito, which are wider than the notch within the blades, contacted the top of the blades and caused the mosquito to flip over when the robot pulled the proboscis downward. Our work to better determine the tooltip-to-head offset should improve the accuracy of alignment. As the robot holds the proboscis above the cartridge surface, its length is foreshortened in the top-down view provided by the onboard camera. A better estimation of the offset may be obtained geometrically or from a side-view camera where the proboscis profile should not be distorted. We will also target further improvement through mechanical changes to the blade and cartridge geometries to better guide the mosquito neck into position even in cases of small errors in robot positioning. These modifications should also address the situations in which alignment appeared adequate by video observation.

The robot movements combined took 7.33 seconds on average in this test. The computer vision steps have been optimized to run on a GPU at a rate of 0.16 seconds on average per image. As the first vision step will happen in parallel with later robot movements this time can be excluded from the total cycle time count. Without hardware changes, we estimate up to 2.5 additional seconds could reasonably be reduced, leading to an average rate around 700 mosquitoes per hour. These time savings would be achieved by means of movement optimizations such a less conservative speed, both throughout and specifically while inserting the neck between the blades, and movement of the home point of the robot closer to the cup. The final system is also expected to use a smaller, lighter robot, which should achieve faster accelerations and may allow for further time improvement through increased nominal speed. Still, automation of the dissection process can be beneficial even before large increases in throughput can be realized as the consistencies associated with lower variability (e.g. in SPZ yield, cost) should ease downstream processes in vaccine production.

Viii Conclusion

We have presented the design and workflow for a manual mosquito micro-dissection system that addresses a significant problem in the production of a malaria vaccine, as well as demonstrates the automation of the challenging and non-standard pick-and-place task involved in mosquito dissection. Our 3MDF design is currently being refined to improve multi-user and long-term use and to ensure materials compliance for manufacturing under current good manufacturing practices (cGMPs) as specified by the FDA. We are in the process of implementing a cGMP-compliant version of the 3MDF to extract mosquito salivary glands in the production scheme for phase III clinical trials.

Aside from its near-term value in increasing productivity while reducing operator training time, the 3MDF also serves as a blueprint in the development of a fully automated dissection system. Since the system design is modular, and each component is shown to work manually, we can integrate additional automated components as they are designed. In the near term, we will focus on automating the downstream 3MDF processes (head removal, squeezing, and collection) and integrating them with our vision-guided robotic pick-and-place system, while also further refining our vision algorithms. We will then move on to development of an improved automatic feeding process. As we move forward, additional challenges will include optimizing throughput and implementing further error detection and recovery actions. The successful demonstration of the robotic pick-and-place component of this system represents a major milestone in our effort to automate the malaria vaccine production process. Although the system is highly specialized, it could be adapted to develop treatment for other mosquito-based diseases. Our systems approach and many of the basic techniques might also be applied to other image-guided automation applications for handling of small, delicate, or variable materials. We hope that our experience reinforces to the reader the key role that manual process refinement, such as the development of the 3MDF, plays in the development of automated technology.

Acknowledgments

This work was supported in part by NIH SBIR grants R43AI112165 and R44AI134500. Additionally, H. Phalen is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1746891.

References