A Flexible Job Shop Scheduling Representation of the Autonomous In-Space Assembly Task Assignment Problem

I Introduction

As autonomous robotic systems are deployed to support efforts in industry or exploration it is vital that these systems have the ability to autonomously respond to new environments and off-nominal conditions. As in-space exploration continues to increase, with humanity returning to the moon and journeying to Mars according to the current directive for NASA, the required infrastructures will grow in scale and complexity. Constructing and maintaining these infrastructures with an astronaut workforce raises circular prerequisite errors given these infrastructures will be required to support the presence of astronauts. A solution to this integrates the use of autonomous robotic systems where these autonomous units collaboratively assemble and maintain infrastructures such as living quarters and power acquisition facilities similar to those in Fig. 1, prior to the arrival of astronauts [belvin_-space_2016]. Due to the cost of sending equipment into space and the time lost delivering it to a location such as Mars, it is not feasible to deliver pre-assembled facilities. This will lead to a variety of tasks requiring autonomous robotic attention. To ensure that a single robot’s failure does not halt construction, there must be ability overlap across the different types of jobs that each robot is capable of completing. This overlap of ability between robotic units creates the possibility of many feasible assembly schemes. Solving for valid schemes that are also efficient will become more important as the number of assemblies increases. In addition to being solved before deployment, new schemes may need to be determined after work has already begun if an off-nominal occurrence causes the original scheme to be invalid.

Fig. 1: Concept of autonomous robot assembly for in-space facilities [noauthor_virginia_nodate].

This work seeks to provide a general problem formulation that describes the in-space assembly task assignment problem, facilitating the application of different solution methods that seek a valid and optimal assembly scheme. Additionally, two possible solution formulations were developed and evaluated. This general formulation takes the form of a flexible job shop scheduling problem (FJSP) which was then utilized in the investigation of two different solution methodologies, mixed integer programming (MIP) and reinforcement learning (RL). A MIP solution methodology was selected due to its extensive use in solving job shop scheduling problems (JSP) [OZGUVEN2012846, ku_mixed_2016] and its ability to find provably optimal solutions for these problems given enough computation time. An RL approach was chosen due to its inherent ability to learn from elements not explicitly defined in the solution formulation. This ability allows for increasing interactions and prerequisites between jobs without additional complexity to the state space [sutton_reinforcement_1998]. The rest of this paper is structured in the following way. Section II will discuss the general problem formulation as an FJSP. Section III will then define the realistic scenario used in the solution evaluations. Section IV will discuss the MIP solution formulation to the FJSP followed by section V which will discuss the RL solution formulation. Section VI will describe the simulation details for the MIP and RL formulations and section VII will discuss the results from both of these simulations. The final section, section VIII, will discuss the results and future research.

Ii General Problem Formulation

This paper proposes framing the autonomous assembly task assignment problem as a FJSP, an extension of the JSP, where each operation can be processed on any machine. In this formulation, a given job, $j \in J$ , will have a set of operations, $O_{j p}$ , where $p \in P$ is a processing plan defining the processing strategy. This processing strategy defines how the operations are processed by the machines (robotic units) represented as $m \in M$ . Depending on the the type of operation, different types of robots will have different completion efficiencies. If $r \in R$ represents a type of robot working on a specific project (a set of jobs required to complete the desired facility or structure) and $q \in Q$ represents a type of operation in $J$ then the completion efficiency for each $(r, q)$ can be represented as an $| R | \times | Q |$ efficiency matrix, $E$ . In many projects, the work on a given job cannot begin until a different job in the project has already been completed. To efficiently represent this precedence constraint a directed acyclic graph (DAG), $G (V, A)$ , is defined where the set of vertices, $V$ , represent the jobs and the arcs, $A$ , represent the directed precedence paths between jobs. Additionally, $H$ represents a set of these arcs that require machine-operation continuity. In an assembly scenario, it is important to include information for the distance between different jobs in the task assignment process. These distances can be represented as the edges, $E$ , of a completed graph, $G (V, E)$ , where the vertices are the jobs in the project.

Iii Experimental Scenario

To illustrate the implementation of this formulation and provide a realistic experimental example for simulation, consider a solar farm assembly scenario where robotic systems must autonomously assemble solar panels for an in-space application on the surface of the moon. For a mission such as this, it is beneficial to launch the solar panels as base components to maximize the volume usage in the delivery system. These components will then be deposited in a storage area on the edge of the project’s workspace. As mentioned in section I, robotic units with a range of abilities would be present to assemble these base components into solar panels. One such unit would be a large robot, capable of unloading the basic components from a lander or moving the components quickly across the workspace. NASA Langley Research Center (LaRC) has developed the Lightweight Surface Manipulation System (LSMS) [doggett_design_2008, dorsey_recent_2011] to accomplished this type of task. This assembly scenario will also require robots that are mobile and capable of utilizing grippers to: move assembly components, operate tools used for affixing components, and collaborate in the carrying of larger components. This type of robot will likely take a form akin to the rover and arm planned in the Mars 2020 mission [williford_chapter_2018] where a robotic arm is affixed to a rover chassis capable of traversing the terrain in the assembly workspace. The experimental robot fulfilling this role in the research presented here is the Mobile Assembly Robotic Collaborator (MARC) at the Field and Space Experimental Robotics (FASER) Laboratory [noauthor_virginia_nodate]. In autonomous assembly, precise manipulation is required to align components before they are permanently affixed into place. To accomplish this type of task, LaRC has developed the Assembler robot, a serialized parallel robot consisting of modularizable Stewart platforms [moser_reinforcement_2019]. Built units of these robots can be seen in Fig. 2. The robots used in the following simulations are modeled after these robots.

(a) LSMS [doggett_design_2008, dorsey_recent_2011]

As a representation of the solar panel assembly, this work will model the solar panel as three components; two for the frame and one for the solar cell sheet, shown in Fig. 2(a) as $1, 2, & 3$ . These three components will be attached via the connection points $A, B, C, & D$ shown in the same figure.

To assemble this solar array, the components will need to be moved, jigged (aligned), and then affixed (welded or attached) at the connection points. These three types of jobs, represented by $M, J, & A$ respectively, will have operations consisting of combinations of four operation types that constitutes the set $Q$ . The operation types and the set of operations for each job are given in Tables I $&$ II respectively. The completion efficiency for each robot type to each operation type is also given in Table I. To successfully assemble the solar panel, both of the frame components must first be moved to the assembly location. Following this, each component of the frame must be aligned into position with respect to the adjoining piece (for this representation only one component needs to be aligned to a stationary piece for each frame piece). Once aligned, it must be welded into place before the alignment can be released. When the frame is ready to receive the solar cell sheet, it is moved into place. One end is affixed to the frame and then the cell is unrolled before it is affixed to the final side of the frame ending the assembly sequence. The DAG representing these precedence constraints for assembling a solar panel is given in Fig. 2(c) and the holding constraints are between the jigging jobs and their respective affixing jobs since the components can not be disturbed between these jobs. This representation of the solar farm assembly problem will be used in the following sections demonstrating the solution formulations to solve for a valid assembly scheme.

q	Description	Assembler	MARC	LSMS
0	Hold frame link	1	5	10
1	Hold solar cell sheet	1	5	10
2	Weld connection point	5	1	10
3	Locomote	100	5	1

TABLE I: Operation types and completion efficiencies.

Jobs $J$	Process plans $P_{j} = {O_{j 0}, \dots, O_{j n_{j}}}$
$M_{1}$	${[3]}$
$M_{2}$	${[3]}$
$J_{A}$	${[0, 0]}$
$J_{B}$	${[0, 0]}$
$A_{A}$	${[0, 0, 2]}$
$A_{B}$	${[0, 0, 2]}$
$M_{3}$	${[3]}$
$J_{C}$	${[1]}$
$A_{C}$	${[1, 2]}$
$M_{D}$	${[3]}$
$J_{D}$	${[0, 1]}$
$A_{D}$	${[1, 2]}$

TABLE II: Jobs, Process plans, and operation lists. In this example, each process job has exactly one process plan. For example, job

A_{A}

has one process plan

O_{A_{a} 0}

with operations three operations of types

0, 0

, and

2

Iv Mixed Integer Programming Formulation

Mixed integer programming is a powerful tool for solving optimization problems and provides a way to compare solutions to calculated bounds on optimal solutions. In particular, the backbone of commercial solvers use Branch & Bound to intelligently explore the space of feasible solutions while producing tighter bounds on how good the optimal solution could be. As a result, a report on a measure of optimality gap (a ratio of how far the best found solution is from optimality) can be given. The formulation originally inspired by [OZGUVEN2012846] is written to allow multiple process plans per job, allowing for different types of operations to be done to complete a job, and hence potential for faster job completion.

The model is described in the following form. For shorthand, $\forall j, p, o, m$ means $\forall j \in J$ , $p \in P_{j}$ , $o \in O_{j p}$ , $m \in M$ , respectively. Furthermore, a specific process plane is referred to as $j p$ and operations as $j p o$ , to reference which job or plan they come from.

Sets
$A \subseteq J \times J$ arc set for precedence of jobs $H \subseteq J \times J$ operation continuity set

Parameters
$T_{j p o m}$ Operation time of $m$ to do $j p o$ $S_{j j^{'} m}$ Setup time of $m$ moving from $j$ to $j^{'}$

Continuous Variables (non-negative)
$s_{j}$ the start time of job $j$ $c_{j}$ completion time of job $j$ $c_{m a x}$ upper bound of completion times $c_{j p o}$ completion time of operation $j p o$ $c_{j m}$ completion time of machine $m$ on job $j$

Binary Variables
$x_{j p}$ plan $p$ is selected for job $j$ $x_{j m}$ machine $m$ is selected for job $j$ $x_{j j^{'} m}$ machine $m$ moves from $j$ to $j^{'} \neq j$ $x_{j p o m}$ machine $m$ assigned to operation $j p o$

Mixed Integer Programming Model

Objective function:

min

c_{max}

(1a)

Completion and start time constraints: The total makespan $c_{max}$ occurs when all jobs are finished, and jobs are finished when all operations for the job are finished. Furthermore, from the DAG, jobs cannot start until preceding jobs are completed:

	$c_{max}$	$\geq c_{j} \geq c_{j p o}$	$\forall j, p, o$		(1b)
	$s_{j}$	$\geq c_{j^{'}}$	$\forall (j^{'}, j) \in A$		(1c)

If $m$ moving from $j^{'}$ to $j$ , then $j$ must start after completion of $j^{'}$ and travel time of $m$ to $j$ : ( $L$ is a large enough number)

s_{j}

\geq c_{j^{'}} + (S_{j j^{'} m} + L) x_{j j^{'} m} - L

\forall j \neq j^{'}, m

(1d)

c_{j}

\geq s_{j} + T_{j p o m} x_{j p o m}

\forall j, p, o, m

(1e)

Assignment constraints: Exactly 1 process plan chosen for job $j$ :

\sum p \in P_{j} x_{j p}

= 1

\forall j

(1f)

If m assigned to operation $j p o$ , then process plan p must be chosen for job $j$ :

x_{j p o m}

\leq x_{j p}

\forall j, p, o, m

(1g)

If $j p$ is chosen, then each $o$ in $j p$ must be assigned exactly 1 machine:

x_{j p}

= \sum m \in M \sum o \in O_{j p} x_{j p o m}

\forall j, p

(1h)

If $m$ assigned to $j p o$ , then $p$ must be chosen for $j$ :

\sum p \in P_{j} \sum o \in O_{j p} x_{j p o m}

\leq x_{j m}

\forall j, p, m

(1i)

If machine $m$ chosen for job $j$ , then it must have exactly one edge entering and leave that job on its path:

	$x_{j m}$	$= x_{j m}^{end} + \sum j^{'} \in J ∖ {j} x_{j j^{'} m}$	$\forall j, m$		(1j)
	$x_{j m}$	$= x_{j m}^{start} + \sum j^{'} \in J ∖ {j} x_{j^{'} j m}$	$\forall j, m$		(1k)

$m$ must have path from starting to ending node:

\sum j \in J x_{j m}^{start} = \sum j \in J x_{j m}^{end}

= 1

\forall m

(1l)

Operation continuity: for specific arcs $(j, j^{'}) \in H$ , if machine $m$ does operation $¯ o$ for $j$ , then it must do operation $¯ o$ for $j^{'}$ and it must transition to job $j^{'}$ next

	$\sum p \in P_{j} x_{j p ¯ o m}$	$= \sum p^{'} \in P_{j} x_{j^{'} p^{'} ¯ o m}$	$\forall m, (j, j^{'}) \in H,$		(1m)
	$x_{j j^{'} m}$	$\geq \sum p \in P_{j} x_{j p ¯ o m}$	$\forall m, (j, j^{'}) \in H .$		(1n)

V Reinforcement Learning Formulation

As stated previously, a reinforcement learning method was chosen as a solution method in an attempt to implicitly learn some of the environment dynamics that were explicitly encoded in the mixed integer programming formulation. The chosen reinforcement learning algorithm for this evaluation was an off-policy temporal difference algorithm known as Q-learning. The Q-learning equation is defined by

Q (s_{t}, a_{t}) \leftarrow Q (s_{t}, a_{t}) + α [r_{t + 1} + γ max a Q (s_{t + 1}, a) - Q (s_{t}, a_{t})] .

(2)

This temporal difference method samples the environment, learning from experiences rather then a dynamics model, giving it similarities to a Monte Carlo Method while updating estimates based partially on previously learned estimates making it akin to a dynamics programming method

[sutton_reinforcement_1998]. This method updates

Q

(the action-value function) to approximate the optimal action-value function off-policy (i.e. independent of the current policy). The discount factor,

γ

, controls the weight of the future effect of the action chosen at time

t

. The learning rate,

α

, controls the weight at which the present value of the action-value entry is updated from the temporal difference evaluation. To evaluate if the RL was able to learn the precedent and holding constraints the action space was the same size as the state space, allowing the policy to choose any state as the next state. To minimize the total makespan a small penalty was incurred at every time step and a large penalty was given if the policy violated the precedent or holding constraints or if it tried to complete a job that had already been completed.

Vi Simulation

To evaluate the solution methodologies with the scenario described above, the completion efficiency values shown in Table I where chosen to reflect the general ability differences of the robots described above. The general spatial layout of this simulation environment is qualitatively shown in Fig. 4. In this workspace, the Assembler robot is used in its disassociated state. That is, each Stewart Platform is spatially separated from the others while still being treated as a single robot unit. This is represented by the four red diamonds. Additionally, the LSMS base and its end-effector are denoted by the red hexagons while the MARC units are represented by the red triangles. The blue circles represent the qualitative spatial location of the jobs. In the simulations the spatial distances between jobs are shown in Table III. The time it takes a robot to traverse this distance is the distance value multiplied by the specific robot’s ability to locomote divided by a constant value ( $30$ for all robots aside from MARC 2 which used a constant value of $25$ ).

Fig. 4: Project workspace used in the simulations.

Vi-a Mixed Integer Programming

For the mixed integer programming solution simulation, the setup time was the amount of time it took a robot to traverse the distance between jobs. The current model formulation allows robots to begin the simulation at any location, which is easily adjusted by changing the fix starting points. For this simulation only one process plan was used, which prescribed one machine per task for a given job. The operation time was the completion efficiency of the robot assigned to work on it given in Table I. Using these parameters and the formulation described in section IV this simulation was evaluated using the Gurobi solver [noauthor_gurobi_nodate]. The results from this simulation will be discussed in section VII.

Jobs	M1	M2	Ja	Jb	Aa	Ab	M3	Jc	Ac	Md	Jd	Ad
M1	0	0	43	50	38	55	0	36	35	0	56	55
M2	0	0	41	50	36	55	0	35	35	0	55	55
Ja	43	41	0	0	0	0	40	0	0	40	0	0
Jb	50	50	0	0	0	0	52	0	0	53	0	0
Aa	38	36	0	0	0	0	35	0	0	35	0	0
Ab	55	55	0	0	0	0	57	0	0	58	0	0
M3	0	0	40	52	35	57	0	35	35	0	55	55
Jc	36	35	0	0	0	0	35	0	0	35	0	0
Ac	35	35	0	0	0	0	35	0	0	35	0	0
Md	0	0	40	53	35	58	0	35	35	0	55	55
Jd	56	55	0	0	0	0	55	0	0	55	0	0
Ad	55	55	0	0	0	0	55	0	0	55	0	0

TABLE III: Spatial distances between jobs used in the simulations.

Vi-B Reinforcement Learning

The reinforcement learning simulation used the same setup and completion efficiency data as the MIP. However, due to the limitations stemming from the size of the state space, only a subset of the jobs, $M_{2}, J_{A}, A_{A}, J_{B}, A_{B}$ , were used. These five jobs were chosen because they reflect the same constraints seen in the simulation for the MIP formulation. The state space for this Q-learning consisted of the five different jobs multiplied by the 24 possible different permutations based off the process plan yielding a state space of 120 entries, where each state is a job and process plan assignment combination. This leads to a state-action $Q$ -table of the size $120 \times 120$ since the action space consists of the next job and process plan to proceed to from a given state. After a coarse grid search $γ = 0.4$ and $α = 0.5$ were chosen for the learning parameters. The penalty for violating a constraint or trying to complete a job already completed was $- 1 E 4$ whereas a the penalty for time unit spent was $- 1$ . This RL simulation was programmed using OpenAI’s Gym framework [noauthor_gym_nodate].

Vii Results

Vii-a Mixed Integer Programming

The mixed integer program was solved using Gurobi version 9.0 and its Python API [noauthor_gurobi_nodate]. Computations were done on a 2014 MacBook Air laptop running macOS Mojave with a 1.7 GHz Intel Core i7, and memory of 8 GB, 1600 MHz DDR3. For the single solar panel example, the MIP finds an optimal solution within 2 seconds and proves optimality in 5 seconds. A larger example with 2 solar panels was also tested where an optimal solution was found in 26 seconds, however, the solver had trouble proving optimality. The optimal schedule output from the MIP for both cases are shown in Figs. 5 and 6. Colored rectangles indicate intervals of time that a job is being worked on, while black lines indicate a machine traveling to a different job. Some jobs are located in the same place and do not require travel time between those jobs.

Fig. 5: Schedule of machines and jobs over time for one solar panel from MIP. Note that the choice of starting position allows for a solution where machines do not need to move anywhere.

Fig. 6: Schedule of machines and jobs over time with two solar panels from MIP. Distance between jobs on each solar panel was preserved, but the setup of the second solar panel is shifted a distance away from the first one.

Vii-B Reinforcement Learning

The number of jobs had to be reduced in order for the RL to converge to a viable, though nonoptimal, solution. This nonoptimality is highlighted by the fact that the LSMS was not chosen to complete $M_{2}$ despite it being the better choice as described in Fig. 7. Fig. 7(a) shows that the model has converged and with additional training time the chosen actions for the given states will not change. It is important to note that the RL did not always converge to a correct schedule. Fig. 8

shows the variance between state-action spaces. This is most likely due to the fact that the state-action space is very large compared to the number of correct state-actions it needs to learn for an optimal schedule.

Fig. 7: Schedule of machines and jobs over time for a reduced job set from RL. Note that the RL did not learn to use the LSMS to complete $M_{2}$ which would have been more efficient.

Viii Discussion & Conclusion

The work presented here describes a novel application of FJSP to frame the in-space autonomous assembly problem providing a general description that can then be utilized by different solution formulations. The proposed MIP solution efficiently solved to optimality for the test instances. This approach solves the deterministic version of the problem from an offline (predetermined solution) planning perspective. This solution is ideal since it guarantees the most efficient way to complete the project. However, in this formulation, all of the precedence and holding constraints had to be directly encoded in the constraint equations. As autonomous assembly scenarios become more complex, the interjob dynamics will become harder to explicitly define. In contrast to the MIP results, the RL approach did not successfully converge to an optimal schedule. While it did learn the interjob dynamics to create a policy of decisions based on interactions with the environment thus allowing for flexible scheduling based on unforeseen circumstances, it was limited by the state space formulation. Future research will evaluate ways to combine the strengths of these two methods along with understanding stochastic elements of uncertainty to give autonomous systems the ability to autonomously learn and solve for schedules to in-space assembly projects facilitating persistent space exploration.

A Flexible Job Shop Scheduling Representation of the Autonomous In-Space Assembly Task Assignment Problem

Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discovery

Decoupling Dynamics and Reward for Transfer Learning

Optimal Control of Industrial Assembly Lines

Safety-guaranteed Reinforcement Learning based on Multi-class Support Vector Machine

Optimal Job Scheduling and Bandwidth Augmentation in Hybrid Data Center Networks

Delayed Geometric Discounts: An Alternative Criterion for Reinforcement Learning

Autonomous Multirotor Landing on Landing Pads and Lava Flows

I Introduction

Ii General Problem Formulation

Iii Experimental Scenario

Iv Mixed Integer Programming Formulation

V Reinforcement Learning Formulation