Arch. Math., Vol. 63, 182-190 (1994) 0003-889X/94/6302,0182 $ 3.30/0 �9 1994 Birkh/iuser Vertag, Basel

Staircase k-kernels for orthogonal polygons

By

MARILYN BREEN

1. Introduction. Let S be a nonempty set in IR 2. Set S is called an orthogonal polygon (rectilinear polygon) if S is a connected union of finitely many convex polygons (possibly degenerate) whose edges are parallel to the coordinate axes. Let 2 be a simple polygonal path in ~ 2 whose edges [v i_ 1, vl], I _< i _< k, are parallel to the coordinate axes. Such a path 2 is called a stawcase path if the associated vectors alternate in direction. That is, for i odd, the vectors vi_ 1 vl have the same direction, and for ~ even, the vectors ~_ 1 ~ have the same direction, i < i < k. Edge [v~_ 1, v~] will be called north, south, easL or wes~ according to the direction of vector v~_ 1 v~. If the staircase path ,~ is a union of at most k edges, then 2 is a staircase k-path. For points x and y in set S, we say x sees y via staircase k-paths if there is a staircase k-path in S containing both x and y. Set S is called staircase k-convex provided for every x, y in S, x sees y via staircase k-paths. Similarly, S is starshaped via staircase k-paths if for some point p in S, p sees each point orS via staircase k-paths, and the set of all such points p is the staircase k-kernel of S. denoted Ker~ S. Set S is horizontally convex if for each x, y in S with [x, y] horizontal, it follows that Ix, y] ~ S. Vertically convex is defined analogously. Finally, S is an orthogonally convex polygon if S is an orthogonal polygon which is both horizontally convex and vertically convex. Using [5, Lemma 1], an orthogonal polygon S is orthogonally convex if and only if it is staircase convex.

There are many interesting results in convexity which involve the notion of visibility via straight line segments. (See [2], [3], [4], [8], [9]). In work with orthogonal polygons, however, it is often useful to replace visibility via segments with the related idea of visibility via staircase paths in order to obtain analogues of familiar visibility results. (See [6], [5], [1].) For example, it is well-known that when S is starshaped via segments, the convex kernel of S is the intersection of all maximal convex subsets of S ([8]). Analogously, for S a simply connected orthogonat polygon which is starshaped via staircase paths, the staircase kernel of S is the intersection of all maximal orthogonally convex (staircase convex) polygons in S ([1]). Since the classical result has been extended to k th order kernels using polygonal k-paths ([7]), it is natural to ask if a similar resull holds in orthogonal polygons for staircase k-kernels. The purpose of this paper is to obtain such a result.

Throughout the paper, cl S and int S will be used to denote the closure and interior. respectively, for set S.

2. The results. The first theorem is an analogue of [1, Lemma 1].

Vol. 63, 1994 Staircase k-kernels for orthogonal polygons 183

Theorem 1. let S 4:0 be an orthogonal polygon, and let k be an integer, k > 2. Then S contains finitely many maximal staircase k-convex polygons. Furthermore, every staircase k-convex polygon in S lies in a maximal staircase k-convex polygon in S.

P r o o f. As in [1, Lemma 1], we let Y denote the family of lines determined by edges of S. Clearly 5r gives rise to a collection Y of nondegenerate closed rectangular regions such that each member T of Y is minimal and u {T: Tin J } = cl (int S). Let N be the family {int T: Tin Y} ~ {(s, t) : [s, t] an edge of T, T in Y-} u {(s, t) : [s, t] an edge of S and (s, t) n cl (int S) = 0}. Then u{cl B : B E N } = S.

Let N be any staircase k-convex polygon in S, and let b 0 e N c~ B for some B in N. We assert that N u cl B lies in a staircase k-convex polygon in S. Consider the collection cg of all horizontal and vertical segments [b, n], where b ecl B and hEN. Define N ' = u {C : C in cg} ~ N u cl B. By an argument from [1, Lemma t], set N ' is an ortho- gonally convex (staircase convex) polygon, and N ' ~ S.

We will use the following procedure to extend N ' (and hence N w cl B) to a staircase k-convex polygon: Examine the rays from B parallel to the y axis and directed south. If one of these rays meets set N at a point, then select such a point p~ of N having smallest y coordinate, and let B~ denote a minimal closed rectangular region containing p~ and B. Observe that B, __c S: Since p~, b o e N , there is a staircase path in N from G to b o. By the way B is defined, there are no vertical edges of S which lie south of points in B. It follows that B~ ~ S.

Repeat the argument above for rays from B directed north, east, and west to define sets B,, B~ and B w, respectively. Let

N" = N ' u B~u B, ,u B r B., =- N u B~u B. u B r Bw.

We will show that N" is a staircase k-convex polygon in S. Set N ' is both horizontally and vertically convex, and it is easy to see that N" preserves

this property: It suffices to consider points u, v where u ~ B s and v ~ N' . If [u, v] is vertical, then by the definition of N", [u, v] c= N ' . If [u, v] is horizontal, let 2 be a staircase path in N' from b o to p~. Then by our choice of Ps, 2 necessarily meets the line of u and v in some point w of Bs n N' , and since N ' is orthogonally convex, [v, w] __ N ' = N". Since [w, u] ~ B s c__ N", we have [v, w] u [w, u] c__ N ' , so [v, u] ~ N" (independent of the order of these points). It follows that N" is both vertically and horizontally convex, and therefore N" is orthogonally convex (staircase convex).

It remains to show that N" is staircase k-convex. Let a, cEN" , There are two possibil- ities for each point: either it is in N or it belongs to one of the sets Bs, B,, B e, B~. If both points are in N, they are joined by a staircase k-path in N. If both are in B~ w B, u B~ w B,~, then they are joined by a 2-path in N ' . We will examine the remaining possibility. The following intermediate lemma will be useful.

Lemma 1. For a e N and b~c l B, there is a staircase k-path in N" from a to b.

P r o o f. First observe that if a is directly south, north, east, or west of B, then the minimal closed rectangular region determined by B and a lies in N". Hence there are two

1 8 4 M . BREEN A R C H , MA'~L

distinct staircase 2-paths in N" from a to b. For the rest of the argument, without toss of generality we assume that a is southwest of B. Choose b o e B m S 4: 0, and let 2 be a staircase k-path in N from a to bo. A nonempty subset 2 ' o f 2 will lie either directty South or directly west of B, and we let v denote the last point of 2 ,,~ 2 ~ as we travel a long 2 f rom a to b o. (See Figure 1),

r . . . . . - I B

l o ~ b : L_ . . . . .5

t)

Figure I.

Then 2 from a to. v, followed by an appropriately chosen 2-path from v to be, wilt yield a staircase k-path from a to b in N". This path satisfies the lemma.

Returning tO the proof of Theorem 1, let a~N and c~B, u B , uB~v;B~, tO find a staircase k-path in N" joining a to c. Without loss of generality, assume c e B,~ Then e lies on a segment [b, q] where b ecI B and q is on the south edge of B,. Recall that there is some point p~ of N south of B having smallest y coordinate, and q lies on a horizontal segment at Ps. As in earlier arguments, if a is directly north, south, east, or west of ct B, then there is a staircase 2-path in N" from a to c.

If a is southwest (or southeast) of B~, use Lemma 1 to find a k-path 2 in N" from a to b. (See Figure 2.) Let v be the first point of 2 such that [v, c] is horizontal, and observe that v is not on the last segment of 2 unless v = c. Since N" is or thogonal ly ConveX, [v, c] c_ N". Path 2 f rom a to v, followed by [v, c], yields a p a t h of length at mos~ k in N " from a to c.

}D I t I I .~C iBs [ I I

Af i t

- - - ~ ' - lpsq t

t

Figure Z

Vol. 63, 1994 Staircase k-kernels for orthogonaI polygons 185

I f a is northwest (or northeast) of B~, we make use of a k-path # in N from a to p~. (See Figure 3.) Let u be the first point of # in B, u B e u Bw. Clearly u is not on the last segment of/~. If u is on one of the first k - 2 segments of #, then # from a to u, followed by a staircase 2-path from u to c in B~ u B, u B~ u B~, gives an appropriate a - c path. A parallel argument holds if u is on the (k - 1) *t segment of # provided this segment is horizontal. If u is relatively interior to the (k - 1) ~t segment of # and this segment is vertical, then it contains a point u' such that [u', c] is horizontal. Path p from a to u', followed by segment [u', c], will produce an appropriate a - c path in N'. Hence N" is staircase k-convex.

~

i ....... .b

I

_ ~ L _ i

ps~ Figure 3.

It follows that if N is a staircase k-convex polygon in S, then N lies in a staircase k-convex polygon which is a union of sets cl B for B in N, Since there are finitely many members of N', some maximal staircase k-convex polygon M must exist with N ~ M. Certainly each M will be a union of sets cl B for B in N, and hence there are finitely many sets M. This finishes the proof of Theorem 1.

Theorem 2. Let S + 0 be a simply connected orthogonal polygon in IR 2, and let k be a fixed positive integer. The staircase k-kernel of S is the intersection of all maximal staircase k-convex polygons in S.

P r o o f. If k = 1, observe that every maximal staircase 1-convex polygon in S is a segment. The intersection of alt such sets is a segment (in case S itself is a segment), a point (in case S is a union of two perpendicular segments), or the null set, and the theorem holds.

For the remainder of the argument, assume k > 2. Let ~ r be the set of alt maximal staircase k-convex polygons in S. We will show that Ker k S = c~ {M : M in ddk}. It is easy to show that the second set lies in the first: Let xec~ { M : M in JCk} and let seS. By Theorem 1, s belongs to some set M s in -Mk, and since x ~ Ms, x sees s via a staircase k-path in S. Hence x e Ker k S.

For the reverse inclusion, select point x in Kerk S to prove that x belongs to every M in ~ k . It suffices to show that if there is a staircase k-convex polygon N in S with x dF N, then N is not maximal. We will extend N to a staircase k-convex set N o in S which contains x: For each point n in N, there is in S at least one associated staircase path )4 (x, n) from x to n having the minimal number of segments. Let 2~ ~ denote the union of all such paths 2 (x, n), for n ~ N. Finally, let ~ represent the union of all horizontal and

186 M. BREEN ARCH. MATH~

vertical segments whose endpoints lie in N w Lf, and define N O = N ~ 5r w ~. We will show that N o has the required properties.

To show that N o ~ S, it suffices to examine a horizontal or vemcat segment La, b] whose endpoints are in N w s F o r point a, select an associated pa th 2(x, n~) in ~ which contains a and joins point n~ of N to x. Similarly, for point b select pa th 2 (x, nb). Using the facts that N O is or thogonal ly convex and S is simply connected, it is easy to see that [a, b] ~ S. Thus No ~ S.

To prove that N o is or thogonal ly convex, let a, b ~ No with [a, b] either horizontal or vertical. Wi thout loss of generality, assume that a e [ai, az] and b e [bl, b2], where a l , a2, ba, b z e N w s Each of these four points is jo ined to x via an associated staircase path ); (x, ai), 2 (x, b~) in No, i = 1, 2. (See Figure 4.) I t is clear that each point of [a, b] must lie either on a horizontal segment or on a vertical segment determined by the 2-paths. Hence [a, b] _= N o, and N O is or thogonal ly convex.

Figure 4.

Finally, we must show that the set N o is k-staircase convex. Select points p, q in N o to find a suitable p - q path. Since No = N ~) f w cg, point p is either on some 2 in ~ Qr on some horizontal or vertical segment [c, d] where c and d belong to paths in ~ . The following prel iminary result will be helpful.

Lemma 2. For p ~ No, p belongs either to a staircase k-path in No from x to N or to one of the first k segments o f a staircase (k + 1)-path in No from x to N. Of course, the path may be chosen to be minimal in length with the required properties.

P r o o f. If p belongs to some 2 in 5 ~ the argument is finished. Hence without loss of generality assume that p # s and that p lies on the horizontal segment [c1, c2], where ci belongs to minimal length staircase path 2i = 2 (x, n~) in Y from x to ni, i = i, 2. Assume that c 1 is west of c2, with x south of the line of c 1 and c2, and let 2 be a staircase k-path i n N f r o m n 1 t o n z.

In case x is east (that is, either southeast or northeast) of cl but west of e 2, proceed as follows. Suppose p is west of x. (See Figure 5.) Fol low 21 to a point directly south of p, then go nor th through p to 6. Since p ~ 21, the resulting pa th uses at most k - 1 segments of 21, and its total length is at most (k - 1) + 1 = k:

Otherwise, assume that x is west of both c 1 and c2. Fo r convenience of notation, assume that ci is on the j~h segment of 2~, i = 1, 2. (See Figure 6.) There are two cases to consider.

C a s e 1. I f j2 < J l , follow 2 2 to a point directly south of p, then go north through p to a point z ~ 5 w 21. If z ~ , we have an appropr ia te path with at most (J2 - 1) + I =

Vol. 63, 1994 Staircase k-kernels for orthogonal polygons 187

nl L cl P

x

Figure 5.

I c

i n2

J2---<-k segments. If z e21, follow 21 to n 1. The corresponding path has at most (J2 - 1) + 1 + (k - J l ) =J2 + k -.11 < k segments.

C a s e 2. If j l < J2, follow 21 to a point directly west of p, then go east through p to c 2, and follow 22 from c 2 to n 2. The resulting path has j l + I + (k - J 2 + 1) segments. Since j l + 1 <J2, the number of segments cannot exceed J2 + (k - J 2 + 1) = k + I. Furthermore, p is not on the (k + I) st segment of the path.

Clearly each of the selected paths is in N o, so the lemma is satisfied.

~ n

Z I I Inl

"~l i c 2 C1 -- I - - - P l - I

j 42

Figure 6.

Now we are ready ot verify that No is staircase k-convex. By the lemma above, point p in No belongs either to a staircase k-path or to a staircase (k + 1)-path in N o from x to N. In the latter case, p is not on the (k + 1) st segment of the path. Without loss of generality, assume that points p and q lie on paths 2p and 20, respectively, where 2p and 2q satisfy these requirements. Also, assume that 2p and 2q are chosen so that each one has minimal length. For convenience, order the segments on each path, with the segment at x called the first segment. Let 2p join x to np ~ N, 2q join x to nq E N, and let c5 be a minimal length staircase path in N joining np to nq, with its segments ordered from np to nq.

Assume that point p is northwest of point q. There are several cases to consider, depending on the position of point x.

C a s e 1. Assume that point x is south of both p and q and west of both p and q. Then there is a staircase 2-path in N O from p to q. (See Figure 7.)

A parallel argument holds if x is north and east of each point p and q.

C a s e 2. If point x is south and east of each point p and q, start at p and follow 2p toward x until reaching a point z which is either directly north of q or directly west of q. (See Figure 8.)

I 8 8 M. BREEN ARCH. MATH.

X

I

_ _ j q I

Figure 7.

1 //q

np

n 7 p

Ap

q

Figure 8.

--X

If z is on the first segment of 2p, then q is also on this segment, and we obtain a p - q subpath of 2p having at most k segments. If z is not on the first segment of 2p, then 2p from p to z, followed by [z, q], has at most (k - 1) + 1 = k segments and lies in N 0,

The proof is similar if x is north and west of each point p and q.

C a s e 3. Suppose that x is west of each point p and q but north of q, south of p. (See Figure 9.)

]

6

i'/q

Figure 9.

Observe that this forces p to lie on a staircase 2-path in the simply connected reg!on determined by 2p, 2q, and 3. By the way we have used minimal paths tO define No'

Vol. 63, 1994 Staircase k-kernels for orthogonal polygons 189

it is not hard to see that point p must lie on a staircase 2-path in N O from x to some point of N. Since 2 v is minimal, it must be such a staircase 2-path.

F o r the moment , assume p, q 0/6. G o directly nor th or directly west from q to point z of 6, and observe that z is not on the last segment of 6. If z is north of p, then there is a staircase 2-path in N o from q to p, and the argument if finished. Otherwise, follow 6 to point z' directly east or directly south ofp. Since p ~ 6, point z' is on one of the segments of 6 numbered 2 through (at most) k - 1. The corresponding q - p pa th uses at most k - 2 segments of 6, together with [q, z] and [z', p], for a total of at most k segments in all.

If p or q is in 6, very minor modifications yield a suitable path. Clearly there are three other cases analogous to this one, each with a paral lel argument.

C a s e 4. Suppose that x is northwest of q and southeast to p. As in Case 3, assume that p, q 4 6. Go directly south or directly east from p to point z ~ g, then follow 6 to point z' directly nor th or directly west of q. Since p is southeast of np, q is northwest of nq, and neither p nor q is on 6, we will avoid the first and last segments of 6. Hence we use at most k - 2 segments of 6 as we travel from z to z'. Count ing [p, z] and [z', q], we use at most (k - 2) + 2 -- k segments in all.

Again minor modifications yield a suitable path if p or q is on 6. We conclude that the set No is staircase k-convex and, of course, can be extended to

a maximal staircase k-convex polygon in S by Theorem 1. Hence if x E Ker k S, then x lies in every maximal staircase k-convex polygon in S. Thus Ker k S ~ ~ {M : M in ~k} , and the two sets are equal. This completes the proof of Theorem 2.

In conclusion, it is interesting to observe that the set Ker k S need not be connected. Consider the following example.

Example 1. Let S be the union of the three square regions A, B, C in Figure 10. The maximal staircase 2-convex subsets of S are

A w [a, , b,] u [a 2, b2] ,

B to [a2, b2] u [b2, C2] ,

and Ker2 S = {bl, b2}.

B w [at, bl] k) [a2, b2] ,

B to [bl, cl] w [b2, c2] ,

B to [a 1, bd w [bx, C l ] ,

and C to [bl, cd w [b;, c2] ;

a 1

Cl

Figure 10.

190 M. BREEN ARCH. MATH.

References

[1] M. BREEN, Staircase kernels in orthogonal polygons. Arch. Math. 59, 588 594 ;1992). [2] L. DANZER, B. GRONBAUM and V. KLEE, HeIly's theorem and its relatives. Proc. Sympos. Pure

Math. 7, 101 180. Providence, R. t. 1962. [3] M.A. Kr~ASNOSEL'SKn, Sur un crit4re pour qu'un domaine solt 6toi16. Mat. Sb. 19. 309-310

(1946). [4] S. R. LAY, Convex Sets and Their Applications. New York 1982. [5] R. MOTWANI, A. RAGHUNATHAN and H. SARAN, Covering Orthogonai Polygons with Star

Polygons: The Perfect Graph Approach. J. Comp. System Sci. 40. 19-48 (1990). [6] J. O'Rour~KE, Art Gallery Theorems and Algorithms. New York 1987. [7] A. G. SPARKS, Characterizations of the generalized convex kernel. Proc. Amer. Math. Soc. 27,

563-565 (1971). [8] E A. TORANZOS, Radial functions of convex and star-shaped bodies. Amer. Math. Monthly 74,

278-280 (1967). [9] E A. VALENTINE, Convex Sets. New York 1964.

Eingegangen am 10. 9. 1993

Anschrift der Autorin:

Marilyn Breen Department of Mathematics University of Oklahoma Norman, Oklahoma 73019-0315 U.S.A.