Let be a simple polygon with vertices, and let be a set of points or line segments inside . We develop efficient data structures for visibility counting queries in which we wish to report the number of objects from visible to some (constant-complexity) query object . An object in is visible from if there is a line segment connecting and contained in . We are mostly interested in the case when is a point or a line segment. Our aim is to obtain fast, , query times, using as little space as possible. Our work is motivated by problems in movement analysis where we have sets of entities, for example, an animal species and their predators, moving in an environment, and we wish to determine if there is mutual visibility between the entities of different sets. We also want to quantify ‘how much’ the sets can see each other. Assuming we have measurements at certain points in time, solving the mutual visibility problem between two such times reduces to counting visibility between line segments (for moving entities) and points (for static objects or entities).
Computing visibility is a classical problem in computational geometry [ghosh07, orourke87]. Algorithms for efficiently testing visibility between a pair of points, for computing visibility polygons [elgindy81, joe87, lee83], and for constructing visibility graphs [overmars88] have been a topic of study for over thirty years. There is even a host of work on computing visibility on terrains and in other three-dimensional environments [agarwal93rayshoot, berg94]. For many of these problems, the data structure version of the problem has also been considered. In these versions, the polygon is given up front, and the task is to store it so that we can efficiently query whether or not a pair of points is mutually visible [chazelle89, guibas87, hershberger95], or report the entire visibility polygon of [aronov02]. In particular, when is a simple polygon with vertices, the former type of queries can be answered optimally—in time using linear space [hershberger95]. Answering the latter type of queries can be done in time using space [aronov02]. The visibility polygon itself has complexity [elgindy81].
Computing the visibility polygon of a line segment has been considered, as well. When the polygon modelling the environment is simple, the visibility polygon, called a weak visibility polygon, denoted for a line segment , still has linear complexity, and can be computed in time [guibas87]. Chen and Wang [chen15weak] consider the data structure version of the problem: they describe a linear-space data structure that can be queried in time, and an -space data structure that can be queried in time.
Computing the visibility polygon of a line segment allows us to answer whether an entity moving along can see a particular fixed point , i.e. there is a time at which the moving entity can see if and only if lies inside . If the point may also move, it is not necessarily true that the entity can see if the trajectory of intersects . Eades et al. [eades20] present data structures that can answer such queries efficiently. In particular, they present data structures of size that can answer such a query in time . They present results even in case the polygon has holes. Aronov et al. [aronov02] show that we can also efficiently maintain the visibility polygon of an entity as it is moving.
Visibility counting queries have been studied before, as well. Bose et al. [bose02] studied the case where, for a simple polygon and a query point, the number of visible polygon edges is reported. The same problem has been considered for weak visibility from a query segment [bygi15]. For the case of a set of disjoint line segments and a query point, approximation algorithms exist [alipour15, gudmundsson10, suri86]. In contrast to these settings, we wish to count visible line segments with visibility obstructed by a simple polygon (other than the line segments). Closer to our setting is the problem of reporting all pairs of visible points within a simple polygon [ben-moshe04].
Results and organisation.
Our goal is to efficiently count the number of objects, in particular, line segments or points, in a set that are visible to a query object . We denote this number by . Given , , and , we can easily compute in optimal time (see Section 2). We are mostly interested in the data structure version of the problem, in which we are given the polygon and the set in advance, and we wish to compute efficiently once we are given the query object . We show that we can indeed answer such queries efficiently, that is, in polylogarithmic time in and . The exact query times and the space usage and preprocessing times depend on the type of the query object and the type of objects in . See Table 1 for an overview.
In Section 3, we consider the case where the query object is a point. We show how to answer queries efficiently using the arrangement of all (weak) visibility polygons, one for each object in . As Bose et al. [bose02, Section 6.2] argued, such an arrangement has complexity in the worst case.
 Let be a simple polygon with vertices, and let be a set of points or line segments inside . In time, we can build a data structure of size that can report the number of points or segments in visible from a query point in time.
We then show that if the objects in are points, we can significantly decrease the required space. We argue that we do not need to construct the visibility polygons of all points in , thus avoiding an term in the space and preprocessing time. We use a hierarchical decomposition of the polygon and the fact that the visibility of a point in a subpolygon into another subpolygon is described by a single constant-complexity cone. We then obtain the following result. Here and in the rest of the paper, is an arbitrarily small constant.
 Let be a simple polygon with vertices, and let be a set of points inside . In time, we can build a data structure of size that can report the number of points from visible from a query point in time.
In Section 4, we turn our attention to the case where the query object is a line segment and the objects in are points. One possible solution in this scenario would be to store the visibility polygons for the points in so that we can count the number of such polygons stabbed by the query segment. However, since these visibility polygons have total complexity and the query may have an arbitrary orientation, a solution achieving polylogarithmic query time will likely use at least space [agarwal93spacepart, agarwal96, gupta95]. So, we again use an approach that hierarchically decomposes the polygon to limit the space usage. Unfortunately, testing visibility between the points in and the query segment is more complicated in this case. Moreover, the segment can intersect multiple regions of the decomposition, so we have to avoid double counting. All of this makes the problem significantly harder. We manage to overcome these difficulties using careful geometric arguments and an inclusion–exclusion-style counting scheme. This leads to the following result, saving at least a linear factor compared to an approach based on stabbing visibility polygons:
 Let be a simple polygon with vertices, and let be a set of points inside . In time , we can build a data structure of size that can report the number of points from visible from a query segment in time.
In Section 5, we show that we can extend these arguments even further and solve the scenario where the objects in are also line segments. Somewhat surprisingly, this does not impact the space or time complexity of the data structure. (Note that in this setting, visibility between segments does not represent visibility between moving points; refer to Section 6 for a discussion of that problem.)
 Let be a simple polygon with vertices, and let be a set of segments inside . In time , we can build a data structure of size that can report the number of points from visible from a query segment in time.
Finally, in Section 6, we discuss some extensions of our results. We generalise the approach of Section 5 to the case where the objects in are simple polygons. We consider some query variations and show that we can compute the pairwise visibility of two sets of objects—that is, solve the problem that motivated this work—in time subquadratic in the number of objects.
In this Section, we review some basic terminology and tools we use to build our data structures.
Visibility in a simple polygon.
We refer to the parts of the polygon that can be seen from some point as its visibility polygon, denoted . The visibility polygon has complexity [elgindy81]. We can also construct a visibility polygon for a line segment , denoted , which is the union of the visibility polygons of all points on ; it is referred to as a weak visibility polygon. Such a polygon still has complexity [guibas87].
Let be a simple polygon with vertices, and let be a set of points or line segments inside . We can compute the number of objects from visible to a point or line segment in time .
If is a set of points, it suffices to compute the visibility polygon of and preprocess it for -time point location queries. Both preprocessing steps take linear time [guibas87, kirkpatrick83], and querying takes time in total. In case consists of line segments, we can similarly test if a segment of is visible when one of the endpoints is visible. We also need to count the number of visible segments whose endpoints lie outside of . We can also do this in time by computing a sufficiently large bounding box and constructing a -time ray shooting data structure on . This allows us to test if a segment intersects in time. Since has only a single hole, we can turn it into a simple polygon, build a ray shooting structure for simple polygons [hershberger95], and answer a query by ray shooting queries. ∎
Given a visibility polygon for some point and a line segment , if and intersect, their intersection is a single line segment.
Assume for contradiction that the intersection between and consists of multiple, possibly degenerate, line segments, for some . Take some points and on consecutive segments and . Consider the line segments and . By definition of the visibility polygon, these segments are inside . Since is inside , the segment is also inside . Since is simple, it must then hold that the interior of the triangle with vertices , , and is also inside . More precisely, cannot contain any of the boundary of . Now consider a line segment for a point between segments and on . Since its endpoint is outside , the line segment must cross the boundary of inside . This contradicts the previous claim that is empty; thus, it must be that the intersection between and is a line segment if they intersect. ∎
A cone is a subspace of the plane that is enclosed by two rays starting at some point , called the apex of the cone. Let be a diagonal of and let and be the two subpolygons we obtain when splitting with . Consider some point in . We define the visibility cone of through , denoted , as the collection of rays starting at that intersect and do not intersect the boundary of , except at (see Figure 1).
The intersection between the visibility cone and the diagonal is a single line segment or empty.
The visibility cone is clearly a subset of the visibility polygon . Since the intersection between and the visibility polygon is a single line segment (or empty) by Section 2, the same must hold for the visibility cone. ∎
Suppose we want to preprocess a set of lines in the plane so that given a query point , we can count the number of lines below the query point. Let be a parameter; then a -cutting of is a subdivision of the plane with the property that each cell is intersected by at most lines [chazelle93]. If lies in a certain cell of the cutting, we know, for all lines that do not cross the cell, whether they are above or below , and so we can store the count with the cell, or report the lines in a precomputed canonical subset; for the lines that cross the cell, we can recurse. The data structure that performs such a query is called a cutting tree; it can be constructed in time, uses space, and supports answering the queries in time , for any constant . Intuitively, the parameter here determines the trade-off between the height of the recursion tree and the number of nodes for which a certain line in is relevant. If we pick , the -cutting of is just the arrangement of . The bounds above are based on picking , so the height of the recursion tree is . This approach follows the work of Clarkson [clarkson87], with Chazelle [chazelle93] obtaining the bounds above by improving the cutting construction.
An obvious benefit of this approach over just constructing the arrangement on and doing point location in that arrangement is that using cuttings, we can obtain canonical subsets and perform nested queries on them without an explosion in storage required; the resulting data structure is called a multilevel cutting tree. Specifically, we can query with points and a direction associated with each point (above or below) and return the lines of that pass on the correct side (above or below) of all query points. If we pick and nest levels in a -level cutting tree, we get the same construction time and storage bounds as for a regular cutting tree; but the query time is now . Chazelle et al. [chazelle92fast] show that if we set , each level of a multilevel cutting tree is a constant-height tree, so the answer to the query can be represented using only canonical subsets. The space used and the preprocessing time remains .
Let be a set of lines and let be a constant. Suppose we want to answer the following query: given points and associated directions (above or below), or vertical rays, find the lines in that lie on the correct side of all points (or intersect all rays). In time , we can construct a data structure using storage that supports such queries. The lines are returned as canonical subsets, and the query time is .
Dualising the problem in the usual way, we can alternatively report or count points from the set that lie in a query half-plane; or in the intersection of several half-planes, using a multilevel cutting tree.
Let be a set of points and let be a constant. In time , we can construct a data structure using storage that returns canonical subsets with the points in that lie in the intersection of the query half-planes in time .
Let be a set of points, each of them an apex of a cone; and let the query point be , again an apex of a cone . In time , we can construct a data structure using storage that returns a representation of the points in , such that for any , lies in the cone of and . The points are returned as canonical subsets, and the query time is . Alternatively, the points can be counted.
We can construct a four-level cutting tree; the first two levels can select the nodes that represent points from lying in . Note that to select the points that lie in , we need to perform two consecutive half-plane queries, as is an intersection of two half-planes that meet at point . We can use Section 2 to handle these; note that every time we get a constant number of canonical subsets, so any new point location queries can be done in time on each level. After two levels, we get canonical subsets. The next two levels handle the other condition: select the points whose cones contain . This can be done by checking that lies below the upper boundaries of the cones and that lies above the lower boundaries of the cones. Again, we need to do point location queries on each level and for each canonical subset; we can use Section 2 to see that we still have a constant number of those. Overall, we do a constant number of point location queries and go down a four-level data structure, where every level is a constant-depth tree. Therefore, the query overall takes time. As stated previously, adding the levels does not increase the storage or the preprocessing time requirements. ∎
Let be a vertical line and let be a set of cones starting left of and whose top and bottom rays intersect . In time , we can construct two two-level cutting trees for of total size , so that for a query segment that is fully to the right of , we can count the number of cones that contain or intersect in time.
A cone partitions the space to the right of in three regions: the regions above and below and the region inside . Segment does not intersect if it is contained in either the top or the bottom region. This is exactly when either both points of are above the supporting line of the upper boundary of or when both are below the supporting line of the lower boundary of . Hence, if we store the supporting lines of the upper and lower boundaries of in two two-level cutting trees, similarly to Section 2, we can count the number of cones that are not visible for . By storing the total number of cones, we can now determine the number of visible cones. ∎
Let be a set of lines and a query line segment. We can store in a multilevel cutting tree, using space and preprocessing time, so that we can count the number of lines in intersected by in time .
For a simple polygon on vertices, Chazelle [chazelle82] shows that we can construct a balanced hierarchical decomposition of by recursively splitting the polygon into two subpolygons of approximately equal size. The polygon is split on diagonals between two vertices of the polygon. The recursion stops when reaching triangles. The decomposition can be computed in time and stored using space in a balanced binary tree.
Hourglasses and the shortest path data structure.
An hourglass for two segments and in a simple polygon is the union of geodesic shortest paths in from a point on to a point on [guibas89]. If the upper chain and lower chain of an hourglass share vertices, it is closed, otherwise it is open. A visibility glass is a subset of the hourglass consisting of all line segments between a point on and a point on [eades20].
Guibas and Hershberger [guibas89], with later improvements [hershberger91], describe a data structure to compute shortest paths in a simple polygon . They use the polygon decomposition by Chazelle [chazelle82] and also store hourglasses between the splitting diagonals of the decomposition. The data structure uses storage and preprocessing time and can answer the following queries in time:
- Shortest path query.
Given points , return the geodesic shortest path between and in as a set of nodes of the decomposition. The shortest path between and is a concatenation of the polygonal chains of the boundaries of the (open or closed) hourglasses in these nodes together with at most segments called bridges connecting two hourglass boundaries.
- Segment location query.
Given a segment , return the two leaf triangles containing and in the decomposition and the pairwise disjoint open hourglasses such that the two leaf triangles and hourglasses fully cover . We refer to the returned structure as the polygon cover of .
- Cone query.
Given a point and a line segment in , return the visibility cone from through . This can be done by getting the shortest paths from to and and taking the two segments closest to to extend them into a cone.
3 Point Queries
In this Section, given a set of points in a simple polygon on vertices, we count the points of that are in the visibility polygon of a query point . We present two solutions: (i) a simple arrangement-based approach that is also applicable to the case where contains line segments, which achieves a very fast query time at the cost of large storage and preprocessing time; and (ii) a cutting-tree-based approach with query times slower by a factor of , but with much better storage requirements and preprocessing time.
3.1 Point Location in an Arrangement
We first consider a straightforward approach that relies on the following observation: the number of objects in visible from a query point is equal to the number of (weak) visibility polygons of the objects in stabbed by . Hence, we can construct all (weak) visibility polygons of the objects in and compute the arrangement of the edges of these polygons. For each cell in the arrangement, we store the number of visibility polygons that contain . Then a point location query for yields the number of visible objects in .
Computing the visibility polygons takes time, and constructing the arrangement using an output-sensitive line segment intersection algorithm takes time [chazelle92intls], where is the number of vertices of . Building a point location structure on for -time point location queries takes time [kirkpatrick83]. The space usage is . As Bose et al. [bose02] show, the complexity of is in the worst case.
3.2 Hierarchical Decomposition
To design a data structure that uses less storage than that of Section 3.1, we observe that if we subdivide the polygon, we can count the number of visible objects by summing up the number of visible objects residing in the cells of the subdivision. To efficiently compute these counts, we use the polygon decomposition approach, as described in Section 2. With each split in our decomposition, we store data structures that can efficiently count the number of visible objects in the associated subpolygon.
Let us solve the following problem first. We are given a simple polygon and a (w.l.o.g.) vertical diagonal that splits it into two simple polygons and . Furthermore, we are given a set of points in . Given a query point in , we want to count the points in that see . We base our approach on the following observation.
Given a simple polygon split into two simple polygons and by a diagonal between two vertices and two points and , consider the visibility cones and , i.e. the cones from and through into the other subpolygons. Point sees in if and only if and .
First suppose that and . We need to show that and see each other, that is, that the line segment lies in . Observe that both and lie in , and is convex, so lies in ; symmetrically, lies in . Furthermore, note that since both cones are cones through , the segment must cross at some point . Then by construction of , the segment lies entirely in ; similarly, lies entirely in . As also the diagonal lies in , we conclude that lies in .
Now suppose that and see each other in . As they are on the opposite sides of the diagonal and the polygon is simple, the line segment must cross at some point . As lies inside , clearly, lies inside and lies inside . Then the visibility cone must include the ray from through , and so is in ; symmetrically, is in . ∎
Section 3.2 shows that to count the number of points of that see , it suffices to construct the cones from all points in through and the cone from through and count the number of points from satisfying the condition of Section 3.2 (see also Figure 3). The cones from all into can be precomputed, so only the cone from into needs to be computed at query time. The query of this type can be realised using a multilevel cutting tree, as explained in Section 2. We also still need to compute the cone at query time and precompute the cones for all ; we shall handle this later.
Let us return to the original problem. To solve it, we can use the balanced polygon decomposition [chazelle82], as discussed in Section 2. Following Guibas and Hershberger [guibas89, hershberger91], we represent it as a binary tree (see Figure 4). Observe that as long as there is some diagonal separating our query point from a subset of points of , we can use the approach above.
Every node of the tree is associated with a diagonal, and the two children correspond to the left and the right subpolygon. With each node, we store two data structures described above: one for the query point to the left of the diagonal and one for the query to the right.
The query then proceeds as follows. Suppose the polygon is triangulated, and the triangles correspond to the leaves in the decomposition. Given a query point , find the triangle it belongs to; then traverse the tree bottom up. In the leaf, can see all the points of that are in the same triangle, so we start with that count. As we proceed up the tree, we query the correct associated data structure—if is to the right of the diagonal, we want to count the points to the left of the diagonal in the current subpolygon that see . It is easy to see that this way we end up with the total number of points in that see , since the subsets of that we count are disjoint as we move up the tree and cover the entire set .
The correctness follows from the considerations above; it remains to analyse the time and storage requirements. For the query time, we do point location of the query point in the triangulation of and make a single pass up the decomposition tree, making queries in the associated multilevel cutting trees. Clearly, the height of the tree is . At every level of the decomposition tree, we need to construct the visibility cone from the query point to the current diagonal; this can be done in time using the shortest path data structure for [guibas89, hershberger91], as discussed in Section 2. Then we need to query the associated data structure, except at the leaf, where we simply fetch the count. The query then takes time
where the sum of all is at most . For the storage requirements, we need to store the associated data structures on in total points at every level of the tree, as well as a single copy of the shortest path data structure, yielding overall storage. Finally, we analyse the preprocessing time. Triangulating a simple polygon takes time [chazelle91]. Constructing the decomposition can be done in additional time [guibas89]. Constructing the associated data structures takes time per level, so overall, after determining the visibility cones for the points of to all the relevant diagonals, which can be done in time , as each point of occurs a constant number of times per level of the decomposition, and constructing the cone takes time. Overall we need time. ∎
4 Segment Queries
In this Section, given a simple polygon and a set of stationary entities (points) in , we construct a data structure that efficiently counts the points in visible from a query segment . We cannot reuse the approach of Section 3.1, since may intersect multiple arrangement cells, so the query time would depend on the number of visible entities, as we need to keep the list to avoid double counting; even if this issue were solved, we would need to sum up the values from the cells we might cross. Therefore, we construct a new data structure using the insights of the hierarchical decomposition of Section 3.2. That approach is also not directly usable, as discussed in Section 3.2.
We use the data structure by Guibas and Hershberger [guibas89] (abbreviated GHDS) on , discussed in Section 2, as the foundation. For a given query , the GHDS partitions into four types of regions (Figure 6): hourglasses (orange); triangles that contain or , denoted by and (blue); regions that have as a border the upper or the lower chain of an hourglass, referred to as side polygons (green); and regions that have as a border one of the edges of or , referred to as end polygons (red). The number of visible objects in is the sum of the counts of objects in at least one of the relevant hourglasses or triangles, plus the size of the set of objects contained in a side or an end polygon that are visible to . This allows us to subdivide the problem into tractable parts with strong assumptions.
Counting the visible objects inside the relevant hourglasses and triangles is easy, since all of them are visible. For the side polygons, we make an observation regarding the conditions for an object not to be visible, and we subtract that count from the overall count of points in the relevant side polygons. Finally, for the end polygons, we make a case distinction on the way the visibility cones of the objects cross the adjacent triangles, and use inclusion–exclusion-style arguments to obtain the correct count.
4.1 The Data Structure
In this Section, we describe what our data structure stores and how to compute it. We start with some helpful observations, leading to a helper data structure.
Let be an hourglass bounding a side polygon . Denote the left diagonal of by ; and denote the polygon bounded by by . Let be a set of visibility rays from objects in into that exit through . (See Figure 7.) In time , we can compute a data structure of size , so that given a query segment , we can count the number of rays in that are intersected by in time .
Suppose that some ray intersects . Since is a visibility ray into that intersects , it extends into and its apex is to the right of the supporting line of . Since , it follows that can only intersect left of the supporting line of .
Let be the subsegment of that is left of this line. We can compute in constant time. The segment intersects if and only if it intersects the supporting line of . Testing for the intersection between a preprocessed set of lines and a query segment can be done with a two-level cutting tree (Section 2), concluding the proof. ∎
Let be an hourglass bounding a side polygon . Denote the left diagonal of by ; and denote the polygon bounded by by . Let be a set of visibility rays from objects in into that exit through . Finally, denote the leftmost vertex of the convex chain separating from by . (See Figure 7.) Given a query point to the left of the supporting line of , whose shortest path to in forms an upwards convex chain, we wish to count the rays in that intersect this convex chain. In time , we can compute a data structure of size that can answer such queries in time .
The shortest path from to , together with , forms a convex polygon. A ray starting in (and thus outside this convex polygon) that intersects an edge of the convex polygon must intersect exactly two edges of the polygon.
We store in the data structure of Section 4.1. We also store a shortest path map on with as its root [guibas87], computed in time and consisting of segments. For every edge in the map, we query the data structure of Section 4.1 to obtain the number of rays in that intersect that edge. With every vertex in the shortest path map, we store the total number of intersections between the rays in and the path from to . Constructing this augmented data structure uses space and takes time .
Given this data structure, we answer a query as follows. That shortest path from to consists of a convex chain of segments of the shortest path map between and some vertex , followed by a segment . In time , we can identify the vertex . We query the vertex for the (stored) total number of intersections with the rays in in time. Then we query the data structure of Section 4.1 with segments and in time and add all three counts together. This way, we count all the intersections of the rays in with the boundary of the convex polygon defined by and the shortest path from to . For each ray, we now count both intersections; so we divide the total by two and return the result. This procedure is needed, since a ray can intersect the shortest path two times (and not intersect ). The query takes time . ∎
We now introduce our Segment Query Data Structure (SQDS). It is based on the GHDS, augmented with extra data for visibility queries. The GHDS decomposes the polygon into hourglasses and triangles; we describe the data structures we store with each class.
The data structure for hourglasses.
Consider an hourglass bounded by diagonals , , and the upper and the lower chains and in the GHDS (Figure 8a). Let be the side polygon of that is incident to the upper chain, and let be the visibility cones of entities in into . For the hourglass itself, we store the number of objects in that are contained in . For ease of exposition, we refer to the boundaries of a cone as the left and the right boundary, when viewed from the apex of the cone in the direction of the cone. For the upper chain of , we store in SQDS:
The number of non-empty visibility cones in .
The right cone boundaries of all cones in that fully exit through in the data structure of Section 4.1.
The left cone boundaries of all cones in that fully exit through in the data structure of Section 4.1.
We store symmetrical data structures for the bottom chain of (i.e. the left cone boundaries for the cones that exit through and the right cone boundaries for the cones that exit through ).
The data structure for triangles.
Let be a triangle in the polygon decomposition underlying the GHDS. For , we store the number of objects in that are contained in . In addition, consider an edge of . To introduce our data structure, we assume that is vertical. Denote the subpolygon (end polygon) adjacent to and bounded by by ; for ease of exposition, assume is to the right of . Let be the visibility cones into of objects in . Let be the set of subcones of the cones in that intersect ; similarly, define as the subcones intersecting . We store:
All cones in a data structure of Section 2.
Symmetric to T2, but for .
All the left cone boundaries that intersect in a cutting tree.
All the right cone boundaries that intersect in a cutting tree.
This list contains the data structures we store for ; we store analogous data structures for edges and of . See also Figure 8b.
The SQDS requires space and can be constructed in time .
During preprocessing, we construct the GHDS and explicitly store all the hourglasses, requiring space and time (Section 2). We augment the decomposition with our data structures. There are hourglasses and triangles, so we need (by Section 4.1) at most space.
For each of our data structures, we need the visibility cones to the diagonals of the triangles and hourglasses to compute the data structures. We compute these in per triangle or hourglass, giving a total time of . Constructing our data structures, given the visibility cones, takes additionally time. In total, we construct the segment query data structure in time . ∎
Next, we describe how to query the data structure, so that we can efficiently count visible entities. Given a query segment , we compute the number of visible entities in from using the SQDS. We obtain the start triangle containing , hourglasses, and an end triangle containing . Since all entities of inside the hourglasses and the two triangles are visible, we sum up their counts that we store in SQDS. It remains to count the objects in the end polygons bounded by the triangles and the two side polygons per hourglass.
4.2 Counting Entities in the End Polygons
Let be a triangle in our decomposition that contains . Assume without loss of generality that is vertical with to its right and above (see Figure 9a). Let be the end polygon adjacent to left of , and let be the set comprised, for each point , of visibility cones . To shorten the phrasing, when we say that a cone sees a segment , we mean that there is mutual visibility between the apex of that cone and in the complete polygon . A cone sees whenever intersects and there are no edges of that block visibility. First, consider the special case where is contained in . Since there are no edges of in , a cone sees the segment if and only if intersects it, and we conclude:
Using T1 stored per edge of the triangle , we can count the objects in all the end polygons of that see a query segment in time.
4.2.1 From Segments to Piercing Segments
Now suppose that and . We assume that pierces (the case when pierces is symmetrical). In this and the next Sections, we consider a case distinction on the types of the cones, based on how they pass through and some adjacent triangle. We argue that in all cases we can count the visible objects correctly.
We partition the set of cones into three classes using the vertex of (Figure 9):
blue cones pass completely above ;
red cones pass completely below ; and
purple cones contain .
Next, we argue that we can count the number of visible cones per class.
Counting visible red cones.
Consider a cone . Since only intersects and , and exits the triangle through by assumption, any part of that can see must be in (see Figure 9c). Hence, sees if and only if lies below the left ray of .
Using T4 (resp. T5), we can count the number of red cones in that see , assuming pierces (resp. ), in time.
A red cone sees if and only if it sees , which is contained in . Moreover, red cones originate to the left of the supporting line of , and is to the right of that line. Thus, we can employ the cutting tree we stored in T4 to count the cones that see . ∎
Counting visible purple cones.
Consider a cone ; we show the following Section. Let be bounded by the right and the left rays , , and let be the ray from the apex of through the vertex . The query segment sees if and only if it sees the cone bounded by and .
If is visible to , then it is visible to . It remains to show that if is visible to , then it must be visible to .
The point is either contained in , in the area of above , or in the area of below . If is contained in the area below , then, since pierces , the segment always intersects the ray from the apex of through , and the segment is thus always visible to (and ). If is contained in the area above , then intersects if and only if it enters through the left ray . If intersects in , then and (and ) are always mutually visible, since there are no polygon vertices in that may block the line of sight. If intersects outside of , then cannot be visible to . Finally, if is contained in , then the segment is always visible to (and ). ∎
Section 4.2.1 implies that whenever intersects , we can use instead of , where is the collection of all such cones. These cones in match the definition of cones in , and we store them together in T2.
Counting visible blue and top cones.
Consider a cone . Since may see outside triangle , we consider the triangle that is incident to . If is inside , we can use the data structure of Section 2 in T2 to count the number of visible blue cones, since visibility cannot be blocked in . In the next Section, we show that if is outside , we can count the visible blue cones in time using T2 and T3.
4.2.2 Counting Visible Cones in an Adjacent Triangle
We assume that pierces in the edge and then passes through some triangle . Let be the intersection point between and .
Abusing (or overloading) notation, we again partition the set of cones in three classes, but now using : the orange cones pass above , the grey cones pass below , and the green cones contain (Figure 10a). Since there are no polygon vertices in , all the cones in see . Next, we consider all the cones in —all the cones in are treated in a symmetrical fashion.
A cone sees if and only if it sees along the right cone boundary .
Let be a ray in such that sees along . Let be the intersection point of with . Since the apex of lies left of , the slope of and must be lower than that of the supporting line of . By definition of the right cone boundary (and our chosen orientation of ), the slope of is at least the slope of . Thus, if intersects , then intersects left of that point of intersection. Since realises mutual visibility between and , the area bounded by and cannot contain any polygon vertices, and thus must also realise mutual visibility between and . ∎
Let be the right rays of the cones of class , e.g. are the right rays of the cones in , and let be the left rays of the cones of class . Let be the number of cones of class that see . Similarly, define as the number of cones of class that see some segment . For some two points and and a class , denote the number of rays from along which we can see the segment in by ; and similarly, use for .
For the query segment , we have that .
Therefore, we can count visibility of orange cones separately for and . Let be the funnel from to .
Let be a ray in . Visibility of along is blocked if and only if intersects the top boundary of the funnel before intersecting