The arrangement of objects into a desirable layout is an everyday problem that is nonetheless surprisingly complex. For example, to find a desirable furniture arrangement for a living-room, one must consider the visibility of the television, a suitable separation of sofas, and access to adjacent rooms, among other factors that differ according to taste and style. It is often difficult for people to solve layout problems, as is affirmed by the existence of professional interior layout designers and self-help resources.
The principal motivation in computer graphics for automatic or semi-automatic layout synthesis is the need to model realistic virtual worlds . Methods that can automatically synthesize realistic, large-scale virtual environments (Fig. 1) are useful for gaming, educational, and training purposes. Such methods are more useful in practice if they can generate larger, more complex environments and execute faster.
In recent years, researchers have proposed several methods for synthesizing layouts that pose layout synthesis as a highly non-convex optimization problem subject to numerous constraints. Due to the challenging nature of these problems, previous work applied stochastic optimization to sample viable layout candidates. Markov chain Monte Carlo (McMC) methods  are the preferred technique because the constraints are often difficult to express as differentiable functions. Unfortunately, these techniques become inefficient when dealing with large numbers of objects.
To overcome this problem, we introduce a continuous framework for layout synthesis. Our main observation is that there are commonalities between layout synthesis and the elastic simulation of deformable objects. Elasticity penalizes the deformation of an object—the energy increases proportionally to the magnitude of the deformation—and layout synthesis penalizes the magnitude of constraint violation. Both can be formulated as optimization problems, and both can be tackled using continuous optimization procedures. Our new, continuous approach enables the fast generation of large-scale, tightly-packed layouts that are intractable using previous approaches. However, like stochastic methods, our deterministic approach can synthesize multiple viable layouts for a given environment (Fig. 2, Fig. 3). To our knowledge, ours is the first physics-motivated approach to layout synthesis.
Our method takes as input an environment, a set of objects, and prescribed aesthetic and/or functional layout constraints that can be easily modified. Initially the positions and orientations of the objects are randomized, which is analogous to choosing a random initial guess in an iterative solver. Object positions and orientations are iteratively modified to achieve a viable layout. At each iteration, the objects are moved so as to satisfy competing constraints. Hard constraints such as observing layout boundaries and preventing collisions between layout objects are enforced by default. The procedure converges when all the prescribed constraints are adequately satisfied. We show that a diverse set of constraints, which have been applied in prior layout synthesis schemes, can be formulated within our framework. Layout objects can also be grouped, and each group assigned aesthetic or functional layout constraints. Groups are reusable in defining other layout groups, and they are also easily modifiable both in terms of the participating objects and the group layout constraints.
Our main contributions in this paper are as follows:
We propose a novel, physics-motivated approach to layout synthesis.
We formulate a set of common layout constraints within our framework.
We develop a novel, continuous and deterministic layout synthesis algorithm with significantly reduced computational cost, making large-scale layout synthesis problems tractable.
The remainder of the paper is organized as follows: Section 2 reviews relevant prior work. Section 3 presents the technical details of our approach. Section 4 describes our experiments and presents our results. In Section 5, we further discuss our framework, including its strengths and limitations, and suggest avenues for future work.
2 Related Work
2.1 Layout Synthesis
We focus on layout synthesis problems in which a set of objects is to be arranged in an open space. The objects are assumed to be rigid bodies. The goal of the layout problem is to position and orient the objects such that they satisfy several functional and aesthetic criteria. These criteria are encoded as the terms of a non-convex objective function. The main challenge stems from finding an arrangement that respects conflicting terms, resulting in a multitude of possible layout outcomes, some of which may be unsatisfactory. Relevant publications in this category include the following:
Yu et al.  and Merrell et al.  introduced an McMC-based approach to furniture arrangement. Yeh et al.  formulated layout constraints with factor graphs, allowing a variable number of elements in the synthesized layout. These McMC-based, stochastic sampling methods can synthesize scenes that respect a complex and conflicting set of constraints, but they have been shown to work only on layouts with a limited number of objects and relaxed spacing. The underlying inefficiency of these approaches stems from the fact that they do not employ local gradient information—they merely sample a new position of a furniture item by applying a shift move to the current position.
Fu et al.  synthesize layouts from object relation graphs learned from a database of floor plans. Feng et al.  optimized mid-scale layouts by stochastically optimizing an objective function derived from agent-based simulation. Fisher et al.  generated small, local layouts of objects in a scene, guided by exemplars—e.g., the layout of items on a desk. Layouts are generated by sampling probabilistically from an occurrence model distribution. Additionally, the authors report that since the layout generation is probabilistic, it cannot handle hard constraints such as rigid grid layouts or exact alignment relationships.
Peng et al.  introduced a method that creates layouts from deformable templates, which differs from prior work that assumed objects are rigid. They incorporated a continuous method in their approach, but use it only to deform objects, whereas we use ours to position objects. Recently, Wu et al. 
proposed a mixed integer-linear programming formulation for floor plan synthesis.
Layout research has also addressed contexts other than interior design. Majerowicz et al.  focused on adding objects to shelves in a 2D setting. Bao et al.  uses a combined stochastic and numerical optimization approach to explore and refine building layouts. Zhu et al.  synthesized layouts of mechanical components to control the motion of a toy. Cao et al. [14, 15] synthesized manga layouts. Reinert et al.  introduced an interactive layout generation method of arranging shapes according to aesthetic attributes, such as color and size. We focus on interior and exterior design layouts, but our method generalizes to other contexts.
2.2 Physics-Based vs Position-Based Methods
Physics-based modeling techniques have been used in various contexts, from animation [17, 18], to geometric design [19, 20], to architectural floor plan design [21, 22]. Position-Based Dynamics (PBD), independently introduced by Muller et al.  and by Stam , was originally proposed as a means of simulating physical models in situations, such as games, where speed and robustness takes priority over physical realism. Researchers have since applied the approach to a variety of simulations, from soft and rigid bodies , fluids , to crowd simulation . PBD is part of a larger family of simulation methods called position-based methods. Bender et al.  present a survey. The common characteristic of these methods is that they work directly with positions rather than with forces as does true Newtonian dynamics (so PBD should more properly be called Position-Based Kinematics). The method works by iteratively adjusting particle positions to satisfy a set of constraints. To our knowledge, we are the first to pursue this approach in the context of layout synthesis.
Starting from random initial layouts, our method explores sequences of possible object arrangements by iteratively solving user-prescribed layout constraints.
A layout is represented by a set of oriented particles, each of which denotes the position and orientation of an associated 3D object mesh. Each particle has three attributes:
a position ,
an orientation , and
a mass , and corresponding inverse mass ,
determined by the volume of the bounding box of the associated object.
Our algorithm modifies the position and orientation of each particle in order to satisfy a set of layout constraints, which restrict the positions and orientations of several layout items. A constraint comprises
a scalar constraint function ,
a stiffness parameter , and
a constraint type,
either an equality constraint or an inequality constraint , where .
Starting from a uniformly distributed random initial position for each object in the layout, our algorithm solves each constraint independently. The constraint’s spring-like stiffness determines the magnitude of the positional correction toward satisfying the constraint. The positional corrections are either processed sequentially, or averaged in a batch.
To measure the quality of a layout, we employ the energy function
where denotes constraint with respective weight .
Algorithm 1 overviews our method. Line 2 initializes object to a random location and orientation . In each iteration of the main loop, starting at Line 4, each constraint is calculated and immediately projected, so that the next constraint uses the updated result (details in Section 3.3). Collision constraints require special treatment, since they may change in each iteration, and are generated in Line 8 using a spatial hash (details in Section 3.3.9).
Each constraint has an associated stiffness parameter which is updated in each iteration (Line 5). Depending on the constraint type, the stiffness either decreases, increases or remains constant. For example, the pairwise distance constraint decreases over time, the collision constraint increases, and for hard constraints, like the layout boundary, the stiffness is constant. This is a type of numerical continuation method . Increasing the number of iterations results in more physically-plausible solutions. In addition, since some constraints are conflicting, we evaluate them in different orders, by interleaving them, similar to Stam’s proposal .
3.1 Constraint Projection
PBD satisfies a system of constraints by iteratively solving each constraint independently. According to Bender et al. , the correction is derived using the first-order approximation and restricting the correction to be in the direction of the constraint gradient: i.e., . This leads to the following formula for the positional correction to particle :
with scale factor
where the stiffness parameter determines the influence of the constraint. The stiffness at each iteration varies according to , where is the initial stiffness and determines the rate at which approaches zero.
We employ two different schemes for satisfying constraints. Each constraint is either solved independently and projected, the updated particle position immediately becoming visible to other constraints, or a subset of constraints is solved as a batch. In the batch case, we average the positional corrections of all the constraints affecting , with averaging coefficient 1.2, as suggested by Macklin et al. .
3.2 Parenting and Grouping
Layout objects can be grouped; for example, tiers of seats in a theater, or a table and chairs. In our framework, the group is also represented by an oriented particle. The dimensions and size of the group is approximated by a bounding box. Furthermore, we can hierarchically define layout constraints within the group.
Constraints internal to a group can be rigid or nonrigid. In the rigid case, we simply apply positional corrections only to the particle representing the group, such that the grouped objects remain fixed relative to each other. In the nonrigid case, the particle representing the group can move, but so can the grouped objects with respect to one another subject to the layout constraints internal to the group.
For example, Fig. 4 illustrates objects grouped along line segments and circular arcs, which enables us to design seating tiers and add pathways in the theater scenes of Fig. 3 while maintaining pairwise distances between the chairs. Segments are defined by the medial axis of the group’s bounding box. An arc is defined by its endpoints along with the center of the circle. When the particle representing the group moves, for each object in the group, a pairwise distance constraint (Section 3.3.1) is applied between a member object and the nearest point on the curve, which is represented by a particle with infinite mass. Since the curve is represented parametrically, the parametric ordering of the associated layout objects can be used to apply the pairwise distance and other group constraints, and the ordered application of constraints can improve convergence. We interleave the application of constraints, as proposed by Umetani et al. .
3.3 Constraint Types
The following sections discuss the constraints that we employ to produce layouts that are consistent with design standards .
3.3.1 Pairwise Distance Constraint
In interior design, two furniture objects and (e.g., a chair and a table) are often required to be at a certain distance from each other in order for the layout to be deemed comfortable. We impose a desired distance between the particles and representing the objects (Fig. 4(a)) with the constraint function
where , with gradients
where . Imposed as an equality constraint , the particle positional corrections are 
onto the vector starting at focal point.
3.3.2 Focal Point Distance Constraint
An object, such as the stage in a theater or a TV in a living-room, can be deemed a focal point for a group of objects [33, 34], and the objects may be constrained to be at a distance from the focal point. We enforce such constraints simply by adding a pairwise distance constraint (4) between the particle that represents the focal point object and each of the surrounding objects represented by particles (Fig. 4(b)). The focal point object can be prevented from correcting its position by setting its inverse mass to zero.
3.3.3 Traffic Lane Constraint
Objects should often be arranged to accommodate traffic lanes that introduce space between objects or groups of objects to allow easy access [33, 34]; e.g., walkways in a theater (Fig. 3). To this end, we enforce a clearance around a vector extending from a particle. This is implemented analogously to a pairwise distance constraint (4) between a particle and the closest point on a vector from another particle (Fig. 4(c)):
is the point along nearest to , and is the desired minimal distance of from ; i.e., (7) is enforced as an inequality constraint, . We treat as a ghost particle that is rigidly attached to . Hence, any positional correction of is applied to . The ghost particle has inverse mass , which may be set to 0 if need be to prevent the constraint from affecting the position of .
3.3.4 Heat Point Constraint
For an object or group of objects, the heat point is the position where the center of mass, with and , of the particles representing the objects is required to be. For example, a computer display, keyboard, and mouse should be located near the middle of the front of a tabletop for easy access. To this end, we define the constraint function
whose gradient for particle is
3.3.5 Focal Point Symmetry Constraint
We can constrain a group of objects to be positioned symmetrically around a vector directed away from the group’s focal point —e.g., two chairs positioned symmetrically in front of the television—by projecting the center of mass of their representative particles onto (Fig. 4(d)). We constrain to coincide with the projection using a constraint function analogous to (9):
3.3.6 Visual Balance Constraint
Placing all the furnishings at one end of a room would create an imbalanced, inharmonious ambiance, which prompted Merrell et al.  to propose a visual balance constraint. To create a sense of equilibrium, we want to arrange the furnishings such that the mean of the visual weights is close to the center of the room [33, 35]. Since larger objects have more visual weight, we define the visual weight of an object in accordance to its dimensions projected onto the ground plane. We implement a visual balance constraint analogously to (9) between the room’s centroid and the center of mass of particles representing the visual weights of furnishings (Fig. 6a):
where denotes the center of the room. Since the room’s centroid is static, particle is assigned zero inverse mass.
3.3.7 Layout Boundaries and Distance to Wall Constraint
Large furnishings usually work best when placed near a wall . For example, we usually avoid placing bookshelves in the center of a room. Also, in realistic use cases, furnishings should not collide with walls. Finally, all layout objects are constrained by layout boundaries.
We define both inequality and equality constraints for object constrained to be near a wall with distance (Fig. 6(a)) as follows:
where is the point on the wall nearest to the position of the particle representing object (in case of multiple nearest points, we simply choose the first point found). The positional corrections are analogous to (6), but note that since is fixed, corrections are applied only to .
3.3.8 Accessibility Constraint
Clearance between furniture items is essential for human comfort . For instance, a coffee table should be close, but not too close, to a sofa in a living-room. We employ a modified version of an accessibility constraint proposed by Yu et al. . Every object is associated with a bounding box, where the faces of the bounding box that are orthogonal to the ground plane are identified with accessibility centers , where correspond to the back, left, front, and right faces, respectively.
Our accessibility constraint is defined using a pairwise distance constraint function between particle representing object and , accessibility center of object (Fig. 6(b)):
Distance , where is the diagonal of the bounding box of the object, and is the diagonal of the accessibility center. The accessibility constraint is an inequality constraint , which is activated only if the respective cuboids defined by the accessibility distances intersect. The positional correction to is applied as if accessibility point is rigidly attached.
3.3.9 Collision Constraint
The collision constraint is complementary to the accessibility constraint. Simply put, to ensure a realistic layout, the bounding cuboids of the layout objects should not collide. To that end, collisions between objects are resolved by a pairwise distance inequality constraint between the bounding spheres (Fig. 7(a)). Let and represent objects and whose bounding sphere radii are and . Our inequality collision constraint function is
Since checking for all pairwise object collisions is computationally expensive, we employ a spatial hash in order to reduce the number of collision checks.
If during the constraint projection objects and are colliding, and both are constrained to be next to a wall (Section 3.3.7), we perform an additional collision constraint projection between their respective closest wall points (Fig. 7(b)). These wall points are considered rigidly attached ghost particles. This allows us to satisfy both the pairwise distance constraint and the distance to wall constraint. If there are multiple candidate wall points, we simply choose the first one found.
3.3.10 Pairwise Orientation Constraint
In layout design, the proper orientation of an object relative to another object in the same furnishing group can promote intimacy or improve functionality . For example, a sofa should face a TV, a coffee table should be parallel to a sofa, and a seat in a theater should face the stage.
Yu et al.  propose a pairwise orientation constraint between interacting layout objects. Similarly, we define an equality orientation constraint between interacting particles and . Let and be the current and desired orientation of particle relative to , and let and be the current and desired orientation of particle relative to . Then the pairwise orientation constraint functions are defined as
where we calculate the smallest angular difference between and , and similarly for particle . This rotational correction is then applied to the particles with corresponding stiffness . The corrected particle orientations are and . The particle positions and remain unchanged.
3.3.11 Orientation to Wall Constraint
Some furnishings work best when placed parallel to a wall (e.g., a table or TV shelf ). Following Yu et al. , we formulate an equality constraint between particle and the nearest wall using the constraint function
where is the orientation of the represented object with respect to the closest wall point and is its desired orientation. To satisfy this constraint, we proceed analogously to the previous section.
3.3.12 Vertical Stacking Constraint
In layout design, accessories serve either a functional or decorative purpose . Vertical stacking is a common way to arrange accessories. For example, books may be stacked in order to conserve space.
Objects to be stacked can be prespecified. If object is to be stacked on object , the vertical distance between the particles representing these objects should be equal to half the sum of the objects’ heights . We formulate the constraint function
involving the components of and , with axis normal to the ground plane. Hence, particle should be placed above particle (Fig. 6b). Additionally, we constrain the ground plane coordinates of particle to match those of particle :
4 Experiments and Results
We implemented our layout synthesis system in Python and Cython, and ran our experiments on a 2.5 GHz Intel i7 Macintosh system.
Fig. 9 shows examples of our experimental scenarios. In each experiment, the initial object locations and orientations were set randomly, as shown in Fig. 9a. Accessibility and collision constraints apply and are generated in all experiments. We ran our layout synthesis algorithm with the constraints described below (refer to Fig. 10). Since collision, accessibility, and wall constraints are treated as hard constraints, for weights in the energy function (1) we chose 150.0 for the collision and accessibility constraints, 20.0 for the wall constraints, and 1.0 for the remaining constraints. We determined these to be suitable weights experimentally. The iterative procedure terminates when there has been no improvement to the minimum layout energy for the previous 50 iterations.
|# Objects||Our Method (sec)||SA-McMC (sec)|
4.1 Layout Synthesis
We next describe the experimental scenarios of Fig. 9.
4.1.1 Theater Variations
We demonstrated the efficacy of our method by running our algorithm in a theater scene with various seating arrangements and two different constraint strategies:
Each chair is at a predefined distance to the stage due to a focal point constraint, and is constrained not to collide with other chairs. The chairs are not associated with tiers, and there are no pairwise distance constraints between chairs in the same tier (Fig. 9).
The stage is initially located at the front midpoint of the theater. There are up to traffic lanes for stage pathways.
With Approach 1, we allocated 200 chairs and further tested the scalability of our algorithm by running additional experiments with varying numbers of chairs. Fig. 13 plots the run-times. Distances are enforced through accessibility and collision constraints; even without distance constraints, the chairs maintain nearly regular intra-tier distributions.
The scenes synthesized with Approach 2 involve between 168 to 246 layout objects. Within tiers, the distribution of chairs is constrained to be around the center of the tier, using a heat point constraint. The traffic lanes conflict with the pairwise distance constraints of the chairs in each tier.
The picnic scene consists of 14 tables, 48 chairs, 8 trash cans, 6 BBQ grills, and a carousel. The main constraints are focal point constraints between each group of chairs and their table, distances between chairs around tables, distances between tables, and heat point constraints to position BBQ grills, trash cans, and picnic tables.
The living-room layout (Fig. 14) contains 2 chairs, 2 indoor plants, a sofa and armchairs, a coat rack, a door, a desk, an office chair, and a TV. The main constraints are focal point constraints between the TV, sofa, and armchairs, as well as wall constraints on the big furniture objects and plants.
We also demonstrated the performance of our algorithm for a desk with small objects, including 12 books, 3 pencils, a food plate, a binder, a photo frame, a potted plant, a laptop computer, and a mug. The main constraints are focal point constraints between certain objects, heat point constraints on different desk parts, and a stacking constraint for the books.
4.2 Layout Synthesis in Tightly-Packed Scenarios
Our method copes well with highly constrained and tightly-packed settings.
4.2.1 Tightly-Packed Bedroom
The tightly packed bedroom contains multiple beds and pieces of furniture. The beds are rigidly grouped together with army style accessories. The beds, bookcase and table are constrained to be next to the wall. The coat rack is constrained to be at a certain distance from the bookcase. We demonstrate that using different initial conditions results in different suggested layouts. Even though the space is tight, our method successfully synthesizes different layout suggestions (Fig. 15).
4.2.2 Tightly-Packed Picnic
This tightly-packed setting demonstrates our method’s ability to synthesize diverse layouts with different numbers and types of furniture objects. We synthesize a tightly-packed picnic scenario in two stages. In the first stage, we randomly vary the number of layout objects of each type, similar to Yeh et al.. The available layout objects are a superset of the previous picnic scenario, with an additional rectangular picnic table. For a more uniform layout, we rigidly attached 4 chairs to each round picnic table. In the second stage of the synthesis, we run our method for 270 iterations. Fig. 2 shows these synthesized layouts.
4.3 Comparison to Simulated Annealing
We compared the performance of our method to a baseline layout synthesis approach that applies simulated annealing with a Metropolis-Hastings McMC state-search step, which we denote SA-McMC. Our implementation is based on code used in . In this implementation, the proposal function shifts an attribute of one layout object in each state-search step. We employed the same energy function (1) to track the quality of the synthesized layouts and ran the comparison several times, with different conditions, such as varying constraint weights and temperature schedules for the SA-McMC algorithm. For all our experiments, we used linear, evenly-spaced schedules for about 20,000 iterations, with an additional stopping condition in case the energy function value did not improve by more than 0.1% during the previous 1,500 iterations.
Experimentally, we noticed that a baseline SA-McMC approach has trouble accommodating tightly-packed and constrained layouts, as in the tightly-packed bedroom and picnic scenarios described in the previous section. Superficially, synthesized layouts appeared satisfactory; however, upon closer inspection, they suffered from unresolved collisions (Fig. 15(a)). A possible explanation is that, unlike our approach, SA-McMC does not exploit local constraint gradient information when shifting between layout configurations. In theory, SA-McMC can escape such collisions through the choice of different parameter settings, using more complex hand-crafted SA-McMC shifts moves, and/or tuning the weights of different constraints in the energy. Unfortunately, none of the settings with which we experimented yielded collision-free layout suggestions for the tightly-packed scenarios.
Fig. 17 plots the energy as a function of iteration number for SA-McMC and our method for the tightly-packed picnic scenario shown in Fig. 1. Generally, we observed that SA-McMC is slower by at least an order of magnitude compared to our method (Table I). The computational cost of SA-McMC increases dramatically with the number of objects to be synthesized. Hence, our method is the one that is more suitable for interactive applications.
We have demonstrated that continuous, deterministic, point-based simulation methods can produce satisfactory layout synthesis results at low computational cost, at least an order of magnitude faster than previous methods, which makes it suitable for augmented reality applications (Appendix A). Layout synthesis objectives are represented by a set of positional and orientational constraints. Our approach satisfies the conflicting set of constraints iteratively, resulting in fast layout synthesis with quality similar to or better than that of previous stochastic, McMC-based alternatives.
Previous work in layout synthesis does not incorporate constraint gradient information, but instead uses manually crafted McMC moves that augment a layout configuration. While this type of method provides an easy mechanism to explore different layout variations, we achieve the same effect via different initializations. As observed in our comparison, the run-times of the baseline SA-McMC method increases significantly with the number of layout objects. In these approaches, a user can define a more relaxed set of constraints, which at least in principle could lead to a desired layout, but at much greater computational cost. In practice, we observed that these methods perform poorly in tightly-packed layouts, resulting in unrealistic collisions between layout items. By contrast, our method is dramatically faster due to its continuous nature. Notably, it requires only a few seconds of run-time for dozens of objects and it naturally scales to hundreds of objects with only moderately increasing computational cost.
Since layout synthesis poses a non-convex problem, it is difficult—and fortunately unnecessary—to find the global optimum. We experimented with nonlinear global optimization solvers (NLopt ), but the results were either poor in quality (i.e., unrealistic layouts with many colliding layout items), or the run-times were intractable. In contrast to these solvers, our method does not directly try to minimize the global energy, but rather iteratively satisfies individual constraints. This induces positional corrections to layout items that propagate throughout the layout, transforming an initially poor, high-energy layout to a satisfactory low-energy layout. The global energy quantifies the quality of layouts and enables our comparisons to prior work.
The intended workflow of layout synthesis methods is for the automated approach to synthesize a variety of viable layouts, from which the user can select one or more that they prefer. Our method supports this workflow. It can generate a variety of layouts by repeatedly running from different random initial conditions (Fig. 15). Ultimately, the quality of configurations is a subjective matter. Hence, at least from the user’s perspective, a global optimum is not definitive.
We observed that solving constraints sequentially, where the new positions are immediately visible to other constraints, makes quick progress at first, but the convergence rate slowly decreases as the iterations progress. For example, in the living-room experiment, the first few iterations yield a layout that is visually similar to the final one, whereas later iterations produce smaller refinements of the layout and resolve cuboid accessibility area intersections.
Like SA-McMC, the layout’s energy can increase from one iteration to the next, which allows to escape suboptimal local minima. Unlike SA-McMC, our method may not converge in the traditional optimization sense; however, our termination criterion is based on the satisfaction of most of the constraints, which is ultimately what matters. In practice, we never observed outcomes that failed to satisfy layout objectives. Most starting seeds lead to a satisfactory layout, all lead to a collision free layout. Although conceptually simple, our method produces impressive results.
5.2 Future Work
In the present study, we did not encode all the constraints that may be relevant in layout design; however, our method can easily be extended to a broader set of layout constraints. Incorporating GPU parallelization can further speed up the procedure, as could a hierarchical approach, where the layout synthesis problem is broken into stages. It will also be interesting to adjust the stiffness factors in a nonuniform manner in an effort to converge to better global solutions.
Due to the sequential, local constraint satisfaction approach of our position-based method, we may observe oscillations and collisions between objects. For example, when there is a collision between the accessibility areas of two objects, the constraint may be partially resolved by projecting one of these objects into a collision with a third object. In future work, we plan to design automatic schemes for detecting and resolving these conditions.
The layout objects and their relationship to their corresponding groups can be stochastically sampled from predefined distributions; e.g., from real-world scene datasets. This is similar to but faster than applying factor graphs .
Appendix A Augmented Reality Layout Synthesis
As another use-case example of our method, we demonstrate the fast synthesis of furniture layouts from and into 2D images of vacant spaces. After the user uploads an image of an indoor or outdoor space, selects furnishings, and specifies layout objectives, our system then automatically analyzes the space using scene understanding algorithms from computer vision, and finally renders into the original image optimal layouts of the selected furnishings satisfying the given objectives. The system works as follows:
Semantic scene segmentation: Our system employs SegNet , a state-of-the-art, pixel-wise semantic segmentation network, trained on the SUN-RGBD dataset  with common indoor scene objects, to output 37 categories of per-pixel semantic image labels. GrabCut  is then applied to segment the pixels labeled ’floor’. Fig. 18 shows examples of the segmentation.
3D scene estimation: We ask users to place a checkerboard calibration marker into the imaged scene, from which our system estimates the camera parameters, the orientation and scale of the floor/ground, and (using Holistically-Nested Edge detection) the height of the scene perpendicular to the estimated ground plane.
Layout synthesis and visualization: The system runs our layout synthesis method to generate optimal layouts, which it then renders into the image via a virtual camera with the aforementioned estimated camera parameters. Fig. 19 shows examples of the results.
Reference  provides additional details.
-  R. M. Smelik, T. Tutenel, R. Bidarra, and B. Benes, “A survey on procedural modelling for virtual worlds,” in Comp. Graphics Forum, vol. 33, no. 6, 2014, pp. 31–50.
-  S. Chib and E. Greenberg, “Understanding the Metropolis-Hastings algorithm,” The American Statistician, vol. 49, no. 4, pp. 327–335, 1995.
-  L.-F. Yu, S. K. Yeung, C.-K. Tang, D. Terzopoulos, T. F. Chan, and S. Osher, “Make it home: Automatic optimization of furniture arrangement,” ACM Trans. Graphics, vol. 30, no. 4, p. 86, 2011.
-  P. Merrell, E. Schkufza, Z. Li, M. Agrawala, and V. Koltun, “Interactive furniture layout using interior design guidelines,” in ACM Trans. Graphics, vol. 30, no. 4, 2011, p. 87.
-  Y.-T. Yeh, L. Yang, M. Watson, N. D. Goodman, and P. Hanrahan, “Synthesizing open worlds with constraints using locally annealed reversible jump McMC,” ACM Trans. Graphics, vol. 31, no. 4, p. 56, 2012.
-  Q. Fu, X. Chen, X. Wang, S. Wen, B. Zhou, and H. Fu, “Adaptive synthesis of indoor scenes via activity-associated object relation graphs,” ACM Trans. Graphics, vol. 36, no. 6, p. 201, 2017.
-  T. Feng, L.-F. Yu, S.-K. Yeung, K. Yin, and K. Zhou, “Crowd-driven mid-scale layout design,” ACM Trans. Graphics, vol. 35, no. 4, 2016.
-  M. Fisher, D. Ritchie, M. Savva, T. Funkhouser, and P. Hanrahan, “Example-based synthesis of 3D object arrangements,” ACM Trans. Graphics, vol. 31, no. 6, p. 135, 2012.
-  C.-H. Peng, Y.-L. Yang, and P. Wonka, “Computing layouts with deformable templates,” ACM Trans. Graphics, vol. 33, no. 4, p. 99, 2014.
-  W. Wu, L. Fan, L. Liu, and P. Wonka, “Miqp-based layout design for building interiors,” in Comp. Graphics Forum, 2018.
-  L. Majerowicz, A. Shamir, A. Sheffer, and H. H. Hoos, “Filling your shelves: Synthesizing diverse style-preserving artifact arrangements,” IEEE Trans. on Visualization and Comp. Graphics, vol. 20, no. 11, pp. 1507–1518, 2014.
-  F. Bao, D.-M. Yan, N. J. Mitra, and P. Wonka, “Generating and exploring good building layouts,” ACM Trans. Graphics, vol. 32, no. 4, 2013.
-  L. Zhu, W. Xu, J. Snyder, Y. Liu, G. Wang, and B. Guo, “Motion-guided mechanical toy modeling.” ACM Trans. Graphics, vol. 31, no. 6, p. 127, 2012.
-  Y. Cao, A. B. Chan, and R. W. Lau, “Automatic stylistic manga layout,” ACM Trans. Graphics, vol. 31, no. 6, p. 141, 2012.
-  Y. Cao, R. W. Lau, and A. B. Chan, “Look over here: Attention-directing composition of manga elements,” ACM Trans. Graphics, vol. 33, no. 4, p. 94, 2014.
-  B. Reinert, T. Ritschel, and H.-P. Seidel, “Interactive by-example design of artistic packing layouts,” ACM Trans. Graphics, vol. 32, no. 6, p. 218, 2013.
-  D. Terzopoulos, J. Platt, A. Barr, and K. Fleischer, “Elastically deformable models,” ACM Trans. Graphics, vol. 21, no. 4, pp. 205–214, 1987.
-  P.-L. Manteaux, C. Wojtan, R. Narain, S. Redon, F. Faure, and M.-P. Cani, “Adaptive physically based models in computer graphics,” in Comp. Graphics Forum, 2016.
-  H. Qin and D. Terzopoulos, “D-nurbs: a physics-based framework for geometric design,” IEEE Trans. on Visualization and Comp. Graphics, vol. 2, no. 1, pp. 85–96, 1996.
-  R. Attar, R. Aish, J. Stam, D. Brinsmead, A. Tessier, M. Glueck, and A. Khan, “Physics-based generative design,” in CAAD Futures Conf., 2009, pp. 231–244.
-  M. Harada, A. Witkin, and D. Baraff, “Interactive physically-based manipulation of discrete/continuous models,” ACM Trans. Graphics, pp. 199–208, 1995.
-  S. A. Arvin and D. H. House, “Modeling architectural design objectives in physically based space planning,” Automation in Construction, vol. 11, no. 2, pp. 213–225, 2002.
-  M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff, “Position based dynamics,” Virtual Reality Interactions and Physical Simulations (VRIPHYS), vol. 18, no. 2, pp. 109–118, 2007.
-  J. Stam, “Nucleus: Towards a unified dynamics solver for computer graphics,” in Proc. IEEE Int. Conf. on Comp.-Aided Design and Comp. Graphics, 2009, pp. 1–11.
-  C. Deul, P. Charrier, and J. Bender, “Position-based rigid-body dynamics,” Comp. Animation and Virtual Worlds, vol. 27, no. 2, pp. 103–112, 2016.
-  M. Macklin and M. Müller, “Position based fluids,” ACM Trans. Graphics, vol. 32, no. 4, p. 104, 2013.
-  T. Weiss, A. Litteneker, C. Jiang, and D. Terzopoulos, “Position-based multi-agent dynamics for real-time crowd simulation,” in Motion in Games, ser. MIG ’17. ACM, 2017, pp. 10:1–10:8.
-  J. Bender, M. Müller, M. A. Otaduy, M. Teschner, and M. Macklin, “A survey on position-based simulation methods in computer graphics,” in Comp. Graphics Forum, vol. 33, no. 6, 2014, pp. 228–251.
-  E. L. Algower and K. Georg, Introduction to Numerical Continuation Methods. SIAM, 2003, vol. 45.
-  M. Macklin, M. Müller, N. Chentanez, and T. Kim, “Unified particle physics for real-time applications,” ACM Trans Graph, vol. 33, no. 4, p. 104, 2014.
-  N. Umetani, R. Schmidt, and J. Stam, “Position-based elastic rods,” in Proc. of the ACM SIGGRAPH/Eurographics Symp. on Comp. Animation, 2014, pp. 21–30.
-  J. DeChiara, J. Panero, and M. Zelnik, Time-Saver Standards for Interior Design and Space Planning. McGraw-Hill, 2001.
-  L. M. Jones and P. S. Allen, Beginnings of Interior Environments. Pearson, 2014.
-  C. Deasy and T. E. Lasswell, Designing Places for People. Whitney, 1990.
-  S. Lok, S. Feiner, and G. Ngai, “Evaluation of visual balance for automated layout,” in Proc. of the 9th Int. Conf. on Intelligent user interfaces, 2004, pp. 101–108.
-  C. Talbott, M. Matthews, and C. Cosentino, Decorating for Good: A Step-by-step Guide to Rearranging What You Already Own. C. Potter, 1999.
-  S. G. Johnson, The NLopt nonlinear-optimization package, 2011. [Online]. Available: http://ab-initio.mit.edu/nlopt
-  V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” arXiv preprint arXiv:1511.00561, 2015.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database:
Large-scale scene recognition from abbey to zoo,” in
Comp. Vis. and Pattern Recognition (CVPR), 2010, pp. 3485–3492.
-  C. Rother, V. Kolmogorov, and A. Blake, “Grabcut: Interactive foreground extraction using iterated graph cuts,” ACM Trans. Graphics, vol. 23, no. 3, pp. 309–314, 2004.
-  S. Xie and Z. Tu, “Holistically-nested edge detection,” in Proc. of the Int. Conf. on Comp. Vis. (ICCV), 2015, pp. 1395–1403.
-  T. Weiss, M. Nakada, and D. Terzopoulos, “Automated layout synthesis and visualization from images of interior or exterior spaces,” in IEEE CVPR Workshop on Vis. Meets Cognition, 2017, pp. 41–47.