M A R I L Y N B R E E N

P O I N T S O F L O C A L N O N C O N V E X I T Y A N D SETS

W H I C H ARE A L M O S T S T A R S H A P E D

ABSTRACT. Let S # ~b be a bounded connected set in R 2, and assume that every 3 or fewer lnc points of S are clearly visible from a common point of S. Then for some point p in S, the set A ==- {s:s in S and [p,s] ~ S} is nowhere dense in S. Furthermore, when S is open, then S in starshaped.

AMS (MOS) subjec~ classification (1980). Primary 52.A30, 52.A35, secondary 52.A10.

Key words and phrases. Krasnosel' skii-type theorems, starshaped sets, convex kernel, points of local nonconvexity.

1. I N T R O D U C T I O N

We begin by generalizing some familiar definitions. Let S _~ R 2. Point s in the closure of S, cl S, is said to be a point of local convexity of S if and only if there is some neighborhood N of s such that N c~ S is convex. If no such neighborhood N exists, then sEcl S is said to be a point of local nonconvexity (lnc point) of S. For points x and y in S, we say x sees y via S if and only if the corresponding segment Ix, y] lies in S. For xEcl S and yES, x isclearly visible from y via S if and only if there is some neighborhood N of x such that y sees each point of N c~ S via S. As usual, set S is called starshaped if and only if there is some point p in S such that p sees each point of S via S, and the set of all such points p is called the (convex) kernel of S, denoted ker S.

A well-known theorem of Krasnosel'skii [4] states that if S is a nonempty compact set in R d, then S is starshaped if and only if every d + 1 points of S see a common point via S. Furthermore, the concept of clearly visible (introduced by Falconer in [3]) has been used to obtain the following Krasnosel'skii-type theorem involving the points of local nonconvexity of the set [2]" For S compact, connected, and nonempty in R 2, S is starshaped if and only if every 3 lnc points of S are clearly visible from a common point of S. In general, this kind of result concerning compact sets does not have direct analogues for nonclosed sets. (See [1].) Hence, the theorem obtained here is somewhat surprising. While the condition on lnc points stated above does not imply that a bounded connected set S be starshaped, it does imply that S be almost starshaped in the following sense: There is some point p in S such that the set A =- {s :s~S and [p, s] g; S} is nowhere dense in S. Moreover, when in addition set S is open, then S is starshaped.

The following terminology will be used throughout the paper. Conv S, cl S, int S, bdry S, and ker S will denote the convex hull, closure, interior, boundary, and kernel, respectively, for set S. Similarly, lnc S will be the set of points of local nonconvexity of set S. For x # y, R(x, y) will represent the

Geometriae Dedicata 13 (1982) 201 213. 0046-5755/82/0132-0201501.95. Copyright ~) 1982 by D. Reidel Publishing Co., Dordrecht, Holland, and Boston, U.S.A.

202 MARILYN BREEN

ray emanating from x through y, L(x, y) will be the line determined by x and y, and L1 and L 2 will denote distinct open halfplanes determined by line L. The reader is referred to Valentine [5] for a discussion of these concepts.

2. THE RESULTS

We will be concerned with the proof of the following theorem.

T H E O R E M d Let S ~ (9 be a bounded, connected set in R 2, and assume that every 3 or fewer lnc points of S are clearly visible from a common point of S. Then for some point p in S, the set A - {s : s in S and [p, s] ~ S} is nowhere dense in S. Moreover, A c_ bdry S.

The proof will be established by a sequence of preliminary lemmas and theorems, each using the hypothesis of Theorem d above. The first two theorems will consider the cases in which int cl S ~ S 4; qS, while Theorem 3 will be concerned with the case for int cl S ,~ S = ~b.

LEMMA 1. The set cl S is starshaped. Proof. We begin by showing that lnc cl S _~ lnc S. Let q~lnc S, and let

N be an arbitrary neighborhood of q. Select points x and y in Nc~cl S with Ix, y] ~ cl S. There are disjoint neighborhoods M~. and My for x and y, respectively, each contained in N, with no point of Mx c~ cl S seeing any point of My ~ cl S via cl S. Moreover, we may select sequences {x,} and {y,} in S converging to x and y, respectively, with {x,} ___ mx and {y,} _~ My. Then for each n, [x, , y,] ~ cl S so [x, , y,] ~ S. We conclude that qelnc S, and lnc cl S _ lnc S, the desired result.

Since every 3 lnc points of S are clearly visible from a common point of S and since lnc cl S _c lnc S, standard convergence arguments reveal that every 3 lnc points of cl S are clearly visible via cl S from a common point of cl S. Then since cl S is compact and connected, by [2, Theorem 1], cl S is starshaped, and the lemma is proved.

LEMMA 2. I f xe in t cl S ,,~ S, then xelnc S. Proof. Suppose on the contrary that for some convex neighborhood N

of x, N c~ S is convex. Without loss of generality, assume that N _ int cl S. We may select points Yl, Y2, Ya in N _ cl S such that x~int cony{y1, Y2, Y3}. For each i, 1 ~< i ~< 3, let {Yin} be a sequence in N c~ S converging to Yi- Then for m sufficiently large, x sconv {YI,,, Y2m, Y3,,} -~ conv(N n S). Since N ~ S is convex, we have x e N ~ S, a contradiction. Our supposition is false, and xe lnc S.

LEMMA 3. I f x~ in t c lS , , -S , then x lies in a nondegenerate segment in int cl S ~ S.

Proof. Let xe in t c l S ~ S. By Lemma 2, xe lnc S, so by our hypothesis

P O I N T S OF L O C A L N O N C O N V E X I T Y 203

for set S, there exist some point p of S and corresponding neighborhood N of x such that p sees via S each point of N c~ S. Without loss of generality, assume that N is convex and N __ int cl S. It is clear that (R(p, x) ~ [p, x)) c~ N is disjoint from S, and this segment satisfies the lemma.

LEMMA 4. In case int cl S ,,~ S has 2 or more noncollinear segments, then these segments are collinear with some unique point p in (ker cl S) c~ S. Moreover, all lnc points of S are clearly visible from p via S.

Proof. Assume that int cl S ~ S contains noncollinear segments T and T', with t e T and t ' eT ' . Since t, t 'e lnc S, they are clearly visible via S from some point p in S, and certainly p is unique. Moreover, by our hypothesis for set S, every lnc point of S must be clearly visible from p via S. It remains to show that peker cl S: If cl S is convex, the result is immediate, so assume cl S has lnc points. For qe lncc l S, qelnc S, so q is clearly visible from p via S. Using standard convergence arguments, q is clearly visible from p via cl S. Thus every lnc point of cl S is clearly visible from p via cl S, and by the proof of [2, Theorem 1], p ~ ker cl S.

T H E O R E M 1. I f int cl S ~ S has at least 2 noncollinear segments, then for some p in S, the set A = { s : s e S and [p, s] ~ S} is nowhere dense in S. Moreover, A ~ bdry S.

Proof. Let p be the unique point of S satisfying Lemma 4, to prove that the corresponding set A is nowhere dense in S.

As a preliminary step, we show that for ze int cl S n S , [p, z] c S: Notice that since peker cl S, if p ~ z, then (p, z] __ int cl S. Suppose that there is some point u e [p, z] ~ S to reach a contradiction. Then p =p u, u eint cl S ~ S, and by Lemma 3, u must lie in a nondegenerate segment T__ int cl S ~ S, Certainly T _c L(p, z). Assume that T is maximal, having endpoints t and t'. Then for an appropriate labeling, p ~< t ~< t' ~< z. It is easy to see that t' is necessarily an lnc point of S. However for N any neighborhood of t', by the maximality of T, N n S c~ [z, t'] 4; qS, so p cannot see N ~ S via S. This contradicts Lemma 4, our supposition must be false, and [p, z] c S.

To establish the lemma, we must show that S _c cl(S ,-~ cl A). Let x eS. In case xecl( in t cl S), then there is a sequence in int cl S converging to x, and standard arguments yield a sequence {s,} in int cl S ~ S converging to x. If M, is a neighborhood of s, in int cl S, then by the proof above, p sees each point of M, n S via S. Hence M, is disjoint from A, {s,} _~ S ~ cl A, and x e cl (S ~ cl A).

In case x e S ,-~ cl (int cl S), we will show that x e S ~ cl A. Choose a neighbor- hood U o f x disjoint from int cl S. For y e U n S , we assert that yogA. Recall that int cl S 4; ~b. Since peker cl S, pecl( int cl S), and p ~ y. Moreover, again using the fact that peker cl S, it is easy to see that y sees via S no point of S ~ L(p, y).

If ye lnc S, then by Lemma 4, [p, y] ~ S, and y ¢ A , the desired result. If

204 MARILYN BREEN

y¢lnc S, then by comments above, points of S near y lie on L(p, y). Let K be a segment of maximal length at y in S ~ cl(int ct S), and let k and k' be the endpoints of K. Note that since S is connected, k =p k'. Furthermore, since p e cl tint cl S)~ we may assume that p ~< k ~< y ~< k'. If p = k, then [p, y] S and y ¢ A, again the desired result. Hence we may assume that p < k ~< y ~< k'.

We assert that k slnc S: If k ¢lnc S, then since (k, k') __q S ~ cl(int cl S), points of S near k necessarily lie on L(p, y). Thus k ¢ cl(int cl S). Moreover, since K is maximal, there must be an interval on (k, p) at k disjoint from S. However, then it is easy to see that ( k , p ) ~ clS, a contradiction since pe ker cl S and kecl S. We conclude that kelnc S and the assertion is established. Then k is clearly visible from p via S, and since (k, k') ~ S, again [p, y] _c S and y ¢ A. We have proved that for y in U c~ S, y ¢ A. Therefore x has a neigh- borhood disjoint from A, and x e S ,,~ cl A.

Combining this with our previous result, we see that for every x in S, xecl(S ~ cl A). Hence S _~ cl(S ~ cl A), and A is indeed nowhere dense in S. It is clear from the proof that A ~ bdry S, and Theorem 1 is established.

This finishes the proof of Theorem ~ ' in case int cl S ,-~ S contains 2 or more noncollinear segments. However, there are other cases to consider. The following result will be useful.

LEMMA 5. Let T be a segment in int cl S ~ S, T of maximal lenoth, with endpoints t and t'. Then for one o f t and t', say t, if t is clearly visible from p via S, then p is in the component of L ~ T whose closure contains t. Moreover, all such points p lie in a (possibly degenerate) interval V in S havin 9 endpoint t. And [p, t) c_ S for each such p ~ t.

Proof. It is easy to show that t and t' are lnc points of S, and any point which sees a neighborhood of t or t' via S must lie on L = L(t, t'), Suppose that neither t nor t' has the required property concerning components of L ~ T. Then there exist points p and p' on line L, p ~< t < t' <~ p', such that t is clearly visible from p' via S and likewise t' is clearly visible from p via S. If t' = p', then t ' eS but p cannot see t' via S, impossible. Thus t' =p p'. Similarly t ¢ p, so p < t < t' < p'. Since t' is clearly visible from p via S and (t, t') ~ int cl S, standard arguments reveal that (p, t] _ int cl S. Since T is maximal in int cl S ~ S, there must be a sequence {t,} in (p, t] c~ S con- verging to t. However, p' can see no point of {t,} via S, impossible since p' sees all points of S in some neighborhood of t. Our supposition must be false, and at least one of t or t' has the required property.

Assume that point t has the required property. If t is clearly visible from p via S, show that p lies in an interval (possibly degenerate) V in S having endpoint t. If t~S, the result is immediate, so suppose t¢S. Let t be clearly visible from distinct points p and p' in S. Then by comments above, p, p 'e R(t', t) ~ [t', t). Standard arguments reveal that (p, t] u(p' , t] ~ int cl S. Since Tis maximal in int cl S ~ S, there is a sequence {t,} on (p, t) converging

P O I N T S OF L O C A L N O N C O N V E X I T Y 205

to t, so for n sufficiently large, [p , t , ]w[p ' , t , ]~_S . Thus [p , t )w[p ' , t )~_S, and [p, p'] ~ S. We conclude that all such points p lie in an interval V in S having end point t, and the lemma is established.

LEMMA 6. Let T be a segment in int cl S ~ S havin9 maximal length, and let V be the interval described in Lemma 5. Then ker cl S ~ V 4: q~.

Proof. Since every 3 lnc points of S are clearly visible via S from a common point of S, every 2 lnc points of cl S are clearly visible via cl S from a common point of cl V. By adapting the proof of [2, Theorem 1], ker cl S c~ cl V =p qS. Select point p in this intersection. Ifp ~ V, the lemma is established. Otherwise, p is necessarily an endpoint of the interval V. For every lnc point q of S, q is clearly visible via S from some point rq of V, so q is clearly visible via cl S from rq. Since p e ker cl S, q is also clearly visible via cl S from p, and we may select a neighborhood Nq of q such that both p and rq see cl S c~ N~ via cl S. Then by standard arguments, every point of [p, rq] sees each point of cl S c~ Nq via cl S.

For each q in lnc S, use the argument above to select a corresponding neighborhood Nq of q and interval [p, rq] ~ cl V such that every point of [p, rq] sees each point of Nq n cl S via cl S. It is easy to show that the set lnc S is compact. Hence we may reduce to a finite subcollection of these neighbor- hoods, say N1 . . . . , Nk, which covers lnc S. The corresponding finite inter- section c~ { [p, ri] :1 ~< i ~< k} is a nondegenerate interval at p. Moreover, for v in n { (p, ri] : 1 <~ i <<. k} ~ 4, v sees via cl S each point of (N1 u . . . u Nk) cl S. Therefore, every lnc point of S (and every lnc point of cl S) will be clearly visible via cl S from v. Again adapting the proof of [2, Theorem 1], v~ker cl S. Hence ker cl S c~ V ~ q5 and the proof of Lemma 6 is complete.

THEOREM 2. I f int cl S ~ S ~ ~ has all its segments on line L, then there is a point p in ker cl S ~ S c~ L such that A = {s :se S and [p, s] ~ S} is nowhere dense in S. Moreover, A ~_ bdry S.

Proof. Let T be a segment in int cl S ~ S having maximal length, and let V be the corresponding interval described in Lemma 5. Using Lemma 6, we may select a point p in ker cl S c~ V _~ ker cl S ~ S n L. We will show that p satisfies the theorem.

As in Lemma 5, let t and t' denote the endpoints of T, with t an endpoint of interval V. We begin by establishing the following preliminary result: For z in int cl S c~ S, p =p z, then [p, z] __c S. Since zeint cl S and p~ker cl S, certainly (p, z] ~ int cl S. We assume that [p, z] ~ S to obtain a contradiction. Then [p, z] must contain points of int cl S ~ S, and since all segments of int cl S ~ S are collinear, T ~_ L(p, z). We will examine the positions of z, t, t', and p on L(p, z). Using the facts that [p, t) _ V __ S and T _~ --~ S, it is easy to show that te[p, t'). Moreover, since we are assuming that [p, z] g; S, z~[p, t]. There are two cases to consider, depending on the position of z.

206 M A R I L Y N B R E E N

Case 1. p <<, t < z, then since z~S, we must have p ~< t < t' ~< z. But since (p, z] ~_ int cl S and T is maximal in int cl S ~ S, either t ' = z eS or there is a sequence in (t', z) n S converging to t'. Either way, since some point of V clearly sees bo th lnc points t and t' via S, we conclude that It, t'] % S, a contradict ion. Therefore Case 1 cannot occur.

Case 2. If z < p ~< t < t', then since (p, z) contains points not in S, there is a maximal segment Win int cl S ~ S which lies in (p, z). Let w and w' denote the endpoints of W. Since p, zeS, we may assume that z ~< w' < w ~< p. As in Case 1, either w ' = z eS or there is a sequence in (w', z)c~ S converging to w'. Since t and w' are clearly visible via S from some point of V, [w, w'] _ S, im- possible. Hence Case 2 cannot occur either.

We have a contradict ion, our assumption that [p, z] ~; S must be false, and [p, z] _ S, establishing our prel iminary result.

To show that point p satisfies the theorem, we must prove that S c_ el(S cl A). Let x e S. If x ecl (int cl S), then using our prel iminary result above and an adaptat ion of an argument in Theorem 1, we obtain a sequence in int cl S n S _ S ~ cl A converging to x, and x ~ cl(S ~ cl A).

If x ~ S ~ cl(int cl S), we will show that x ~ S ~ cl A. As in Theorem 1, let U be a ne ighborhood o fx disjoint f rom int cl S and let yE U n S to prove that y ¢ A. Certainly p 4= y and y sees via S no point of S ,,~ L(p, y).

In case y e l n c S, then y sees via S some point v of V, and v~L(p, y). If V~_L(p,y), then since p~Vc_S , it is easy to show that [p,y]~_S, and y~A, the desired result. If V ~ L(p, y), then ve V n L(p, y) = {p}, [p, y] ___ S, and again y¢A.

Hence, we may restrict our at tent ion to the case in which y~lnc S. As in Theorem 1, let K be a segment of maximal length at y in S ,,~ cl(int cl S), k and k' the endpoints of K, with p ~ k ~< y ~< k'. If p = k, the result is immediate, so assume that p < k ~< y ~< k'. By an argument in Theorem 1, k~lnc S. There- fore, by previous comments, k and t are clearly visible via S from a point u of V. And since (k, k') ~_ S ~ cl(int cl S), u~ L (p, y). If V ~ L(p, y), then [p, y] __ S and y ¢ A, the desired result. If V ~ L(p, y), then u = p, [p, y] _~ S, and again y¢A. Thus x has a ne ighborhood disjoint from A, and x ~ S ,,~ cl A.

We see that for every x in S, x ~ cl(S ,,~ cl A), and A is nowhere dense in S. Fur thermore , a close look at the argument reveals that A ~ bdry S, and the proof of Theorem 2 is complete.

It remains to establish Theorem d in case int cl S ~ S -- qS. The following technical lemmas will be useful.

L E M M A 7. Let p~ker cl S and let R and R' be distinct collinear rays emanat- ing from point p, with R' c~ cl S = {p}. For q in cl S, if q is not clearly visible via cl S from any point of R ~ {p}, then at least one of the followin 9 must occur: Either q is clearly visible via cl S only from points on some line L which contains

P O I N T S OF L O C A L N O N C O N V E X I T Y 207

both p and q, or q is clearly visible via cl S only from points in one of the closed halfplanes determined by the line of R

Proof. If q is clearly visible via cl S only from points on a line L which contains both p and q, then the argument is finished, so assume that this does not occur. We use a contrapositive argument. Suppose that q is clearly visible via cl S from points r and ro which lie in opposite open halfplanes determined by the line of R, to show that q is clearly visible via cl S from points ofR ,,~ {p}.

We begin with the situation in which qq~R. Then for one o f t and ro, say for r, points q and r are in opposite open halfplanes determined by the line of R. Select a closed convex neighborhood N o fq such that r~N and r sees via cl S each point of N n cl S. There are two cases to consider.

Case 1. If pC(q, r), without loss of generality, assume that N has been cho- sen so that p~conv(ruN). Then [r,q] necessarily meets R,,~{p} and conv( ruN)nR is an interval Ix, y], with p < x < y. Select st(p, x). Using the facts that p 6 ker cl S and r sees via cl S each point of N n cl S, a simple geometric argument shows that each point of [p, s] sees via cl S each point of N n cl S. This completes our contrapositive argument, and the lemma is satisfied.

Case 2. If pc(q, r), then since R ' ~ cl S = {p}, all points of N n cl S neces- sarily lie in the closed halfplane cl L 1 determined by line L(p, q) and contain- ing R. By our opening remarks, we may select point r'~L(p, q) such that q is clearly visible via cl S from r'. Certainly r'ELI. Choose a closed convex neighborhood N' of q, N' ___ N, such that r'f~u{R(p, x):xsN'} and such that r' sees via c lS each point of N'nclS . The intersection c o n v ( r u N ' ) n (R ~ {p}) c~ cl S is nonempty. For s in this intersection, it is not too hard to show that each point of [p, s] sees via cl S each point of N' n cl S. Again our contrapositive argument establishes the lemma.

For the situation in which qeR, use the fact that q is clearly visible via cl S from points r and ro which lie on opposite sides of the line ofR. Ifq ~ p, minor changes in the argument above yield the result. If q = p, then by our opening remarks, we may assume that point p is clearly visible from some point r'o~L(p,r). Point r~ necessarily lies in the open halfplane determined by L(p, r) and containing R, and point r~, together with an appropriate choice of r or ro, may be used to produce the required nondegenerate interval at p on R. This completes the proof of Lemma 7.

The following definitions from [2] will be useful in Lemma 8.

DEFINITIONS. For q~lncclS, define Aq = {x :q is clearly visible from x via clS}, and let Cq= n { H : H a closed halfplane with A ~ _ H and q~ bdry H}.

208 MARILYN BREEN

LEMMA 8. Let p6kerclSc~bdryclS. Then there exist rays Ro and R1 emanating from p such that the following statements are true: Point p is clearly visible via cl S only from points in the closed convex wedge cl B bounded by Ro and R1, and this wedge cl B is minimal. Moreover, if Ro 4; R1, then no point of S lies in the opposite open convex wedge B' bounded by the corresponding opposite rays R'o and R'~.

Proof. Assume that p is clearly ,~isible via cl S from some point other than p itself, for otherwise the result is trivial. In case pq~lnc cl S, select a neighbor- hood N of p such that N c~ cl S is convex, and let cl B be the tangent cone of Nc~cl S at p. (That is, cl B is the closure of the set ~ {R(p, s):seN c~ cl S, s 4: p}.) Choose Ro and Ra to be the corresponding boundary rays for cl B. Since p E ker cl S, it is easy to see that cl S ~ cl B, and the lemma is satisfied.

In case pelnc cl S, then by results in [2], set Cp is defined. By our opening assumption, Cp =p {p}. Moreover, using the fact that p~ker cl S ~ l n c cl S, it is not hard to show that Cp is not a line. Let Ro and R ~ denote the boundary rays for Cp. It is clear that the wedge cl B is exactly Cp, so wedge cl B is minimal. IfRo 4; R~, let R; and R'I denote rays emanating from p and oppo- site to Ro and R1, respectively, with B' the corresponding open convex wedge opposite to cl B.

It remains to show that B' c~ cl S = q~. Assume on the contrary that there is some point x in B'• cl S, to reach a contradiction. Then Ix, p] __ cl S. Furthermore, if La and L2 denote distinct open halfplanes determined by line L(x, p), we assert that p cannot be clearly visible via cl S from points in each of L1 and L2: For the moment, suppose that R0 and R~ are not colli- near. If p were clearly visible via cl S from yeL~ c~ Cp and from z E L 2 ~ Cp,

then for an appropriate x'e(x,p) and y'~(y,x'), conv{z,y',x'} would lie in cl S and would contain a neighborhood of p, impossible since pelnc cl S. If R0 and R~ are collinear, since Cp is not a line, minor changes in the argu- ment above again show that p cannot lie in lnc cl S, a contradiction. Thus our assertion is established, and p is clearly visible via cl S only from points in one closed halfplane determined by L(x, p). Hence the open ray R(x, p) Ix, p] either is disjoint from Cp or lies in bdry Cp. However, each situation is impossible, since R(x, p)~ Ix, p] lies between Ro and R~, the boundary rays for Cp. We have a contradiction, our assumption is false, and we conclude that B' c~ cl S -- 4). This completes the proof of Lemma 8.

LEMMA 9. Assume that int cl S ~ S = qS. I fker cl S contains a segment, then ker cl Sc~ S 4; ~b.

Proof. We assume that cl S is not convex, for otherwise the result is trivial. Let V = [p, p'] be a segment in ker cl S. If V ~ int cl S 4; ~b, then V c~ S :p ~b and the argument is finished. Hence we may assume that V __ bdry cl S. Since V _~ ker cl S, S must lie in one of the closed halfplanes cl L1 determined by the line L of V. To complete the proof, we consider two cases.

POINTS OF LOCAL N O N C O N V E X I T Y 209

Case 1. If every lnc point of S is clearly visible via S from some point in the open halfplane L~, then we use an argument similar to one employed in Lemma 6: For each lnc point q of S, select a point rq in L1 and a neighborhood N o of q such that every point of the triangular region conv (Vwrq) sees via cl S each point of Nq (3 cl S. Using the compactness of the set of lnc points, we obtain a nondegenerate triangle T at V from which every lnc point of cl S is clearly visible via cl S. Then by the proof of [2, Theorem 1], T _~ ker cl S. Since int T _~ int cl S ~_ S, any interior point of Tsatisfies the lemma.

Case 2. If some lnc point q of S is not clearly visible via S from any point in L1, then qo is clearly visible via S only from points on line L. Using the fact that every 3 lnc points of S are clearly visible via S from a common point of S, it is not hard to show that for at least one of the rays R(p, p') and R(p', p), say the former, every Inc point of S is clearly visible via S from a point on this ray. If p s S , then peker cl S ~ S, so we may assume that p~S. Hence every lnc point of S is clearly visible via S from a point of the open ray R(p, p') ~ {p}. As in the proof of Lemma 6, for each lnc point q of S, we may select a corresponding point rq on (R(p, p') ,,~ {p})c~ S and a neighborhood Nq ofq such that every point of [p, rq] sees every point of Nq c~ cl S via cl S. Re- duce to a finite subcollection of neighborhoods NI . . . . , Nk which cover lnc S. Then for some particular ri, 1 <~ i <~ k, say for r l , [p, rl] = ~ {[p, ri~: 1 <~ i <~ k). By the proof of [2, Theorem 11 , [p, r l ] c _ k e r c l S , so r lekerc lSc~S, and rl satisfies the lemma. This completes the proof.

LEMMA 10. I fcl S = cl (int cl S) and p e ker cl S c~ bdry cl S ~ S, then ker cl S contains a seoment.

Proof. Select rays R0 and R1 emanating from point p and satisfying Lemma 8, with cl B and B' the wedges at p described in that lemma. In case Ro = R~, then since peker cl S and int cl S 4 qS, it is not hard to see that p must be an lnc point for cl S. Thus pelnc S. Recall that p is clearly visible via cl S only from points in cl B, so p is clearly visible via cl S only from points of Ro. Therefore, p is clearly visible via S only from points of Ro. By our hypothesis for set S, this implies that every lnc point of S must be clearly visible from a point of Ro "~ {p}. We use the argument in Lemma 6 to obtain a nondegenerate segment at p on R 0 in ker cl S, and the lemma is satisfied.

Throughout the remainder of the argument we assume that Ro 4: R1. Let R2 denote the ray from p contained in wedge cl B and bisecting the angle determined by Ro and R~. Then the opposite ray R~ lies in B 'w{p} , so R~ c~ cl S = {p}. Now if every lnc point of cl S is clearly visible via cl S from points in R2 ~ {p}, then an argument similar to the one in Lemma 6 may be used to obtain a nondegenerate segment at p in R2 ~ ker cl S, completing the proof. Otherwise, for some lnc point q2 of cl S, q: is not clearly visible via cl S from any point in R2 ~ {p}. Thus by Lemma 7, either q2 is clearly

210 MARILYN BREEN

visible via cl S only from points on some line through p and q2 or qz is clearly visible via cl S only from points in one of the closed halfplanes cl L21 deter- mined by the line L2 of R2. If the former occurs, then a variation of the argu- ments in Lemmas 6 and 9 will yield our result, Hence we assume that the latter occurs. Since q2 is not clearly visible via cl S from any point of L~ ,,~ {p}, q2 is clearly visible via S only from points in L21. Recall that by the proof of Lemma 8, either cl S __ cl B or p is an lnc point for cl S. Therefore, using our hypothesis for set S, it is not hard to prove that every lnc point of cl S is clearly visible via cl S from a point other than p in the convex wedge cl B2 - cl Bc~cl L21. Define R3 to be that member of {Ro, R1} which lies in the boundary ofcl B2. Then cl B 2 is bounded by rays R 2 and R 3.

Inductively, for some n/> 1, assume that rays Ro ,RI . . . . . R2n, R2,+I, wedges cl B, cl B2 . . . . . cl B2., and lnc points q2 . . . . . q2. have been defined. Let R2,+2 denote the ray from p contained in cl B2. and bisecting the angle determined by R2, and Rzn+l. If R~,+ 2 is the corresponding opposite ray, clearly R~,+2 c~cl S = {p}. If every lnc point of cl S is clearly visible via cl S from a point of R2,+2 ~ {p}, an earlier argument completes the proof. Otherwise, there is some lnc point q2,+2 of cl S such that either q2.+2 is clearly visible via cl S only from points on a line through p and q2,+2 or q2, + 2 is clearly visible via cl S only from points in a closed halfplane cl Lz, + 2.1 determined by the line L2. + 2 of R2, + 2. If the former occurs, again the argu- ment is finished, so we assume that the latter occurs. Let cl Be,+ 2 = cl B2, cl Lzn+2,1, and define R2n+3 to be that member of {R2., R2n+l} which lies in the boundary of cl Bzn+2. Then cl B2n+2 is bounded by rays Rzn+2 and Rzn+3. Moreover, using previous comments and the hypothesis for set S, it is not hard to show that every lnc point of cl S is clearly visible via cl S from a point other than p in cl Bzn+2.

In our inductive construction, either the procedure terminates at some step (and the argument is finished) or we obtain an infinite sequence of rays, wedges, and lnc points. We may assume that the latter occurs, Then for every n, ray R, is defined, and clearly the sequence {R.} converges to some ray R emanating from p, with R = cl B c~ (n {cl Bzl :i >/ 1} ). We will prove that ker cl S contains a segment at p on ray R.

We begin by disposing of the case in which R is one of the rays in {R, : n ~> 0}. If R = Ro, then for every n, R2,+l = Ro, and the sequence of distinct rays {R2,} converges to R o. If {q2,} is the corresponding set of lnc points, let t be a limit point for {q2,}. Clearly t~lnc S. If pelnc S~ then it is easy to see that points p and t together are clearly visible via S only from points in Ro "-~ {p}. By our hypothesis for set S, every lnc point of S must be clearly visible from points in Ro ~ {p}, and the argument of Lemma 6 may be used to obtain a segment in ker cl ScaR o, finishing the proof. If p¢lnc S, then cl S ___ cl B, so t itself is clearly visible via S only from points on Ro ~ {p}. Again every lnc point of S is clearly visible from points on Ro "~ {p}, and the

P O I N T S OF L O C A L N O N C O N V E X I T Y 211

proof is finished. A parallel proof holds in case R = R1. Therefor e, throughout the remainder of the argument we will assume that R :p Ro and R ¢ R 1.

If Re{R,:n>~O} and R:/=Ro,R1, then R = R 2 k for some particular k ~> 1. Moreover, R2n+~ = R for 2n >~2k + 2, and the sequence of distinct rays {R2n} converges to R. Letting t be a limit point for {q2,}, t and q2k together are clearly visible via S only from points in Rzk "~ {p}, and our previous arguments may be used to finish the proof.

In case R~{Rn:n>~O}, then we may select disjoint infinite sequences (~R"} {R)} and~ 2 in {R,:n >~ 2} such that {R)} and {R;} lie in opposite open

halfplanes determined by the line of R. Let {q)} and {q)'} denote the sets of lnc points associated with {R)} and {R}'}, respectively, and let q' and q" be limit points for {q)} and {q}'}. Then it is not hard to see that the lnc points q' and q" are clearly visible via S only from points on R ~ {p}. Again our previous argument yields a segment in ker cl S c~ R, and the proof of Lemma 10 is complete.

LEMMA 11. Assume that i n t c l S ~ S = q 5 and c lS=cl ( in tc lS) . Then ker cl Sc~S :p qS.

Proof. Select point p in ker cl S ¢ ~b. If peS, the argument is finished, so assume that p~S. Then p~int cl S, so pebdry cl S ,,, S. By applying Lemmas 9 and 10, we conclude that ker cl S c~ S ~ qS.

At last we are ready to establish Theorem ~4 in case int cl S ~ S = q~.

THEOREM 3. I f in t cl S ~ S = ~b, then there is a point p in S such that A - {s :s in S and [p, s] ~ S} is nowhere dense in S. Moreover, A ~_ bdry S.

Proof. We consider three cases.

Case 1. If c l S = cl(intcl S), then using Lemma 11, select any point p in ker cl S c~ S ~ ~b. For s in S, there is a sequence {Sn} ~-- int cl S c~ S ~ {p} converging to s. Standard arguments reveal that (p, s,] _~ int cl S. Moreover, since int cl S ~ S = ~b, it is easy to show that (p, s,] ___ int cl S ,~ cl A. Thus {s~} c S ~ cl A, s~cl(S ~ cl A), and A is nowhere dense in S.

Case 2. In case int cl S ¢ ~b and cl S 4: cl(int cl S), select points x~S ,,~ cl(int clS) and reker cl S_c cl(int cl S). Arguments similar to those in Theorem 1 may be used to show that x lies in a nondegenerate segment K of maximal length in S ,-~ cl (int cl S), and K ~_ L(r, x). Let k and k' denote the endpoints of K, with r ~ k ~< x ~< k'. Again adapting an argument from Theorem 1, kelnc S. Thus k is clearly visible via S from a point of S, and since (k, k') is disjoint from cl(int cl S), this point must lie on L(r, x). Choose a segment W in S containing (k, k') and having maximal length. Then k is clearly visible via S only from points in IV, Hence every 2 lnc points of S are clearly visible via S from some point of W. By adapting the proof of

212 M A R I L Y N BREEN

[2, Lemma 1] and using an argument from Lemma 6, ker cl S c~ W =~ ~5. For point p in this intersection, we will show that the corresponding set a = {s :s in S and [p, s] ~ S} is nowhere dense in S.

Let seS to prove that secl(S ,-~ cl A). In case secl(int cl S), the argument in Case 1 above may be used to obtain the result. Hence we restrict our atten- tion to the case in which s e S ,,- cl(int cl S). By the argument above, s lies in some maximal segment in S ,,~ cl(int cl S) having endpoints z and z', with p ~ z ~ s ~< z', and z e lnc S. Then z and k are clearly visible from some point w of W, and this point w must lie on L(p, s). Moreover, since w sees via S points near z on (z, z'), [w, s] ~ S. Now if W c_ L(p, s), since p, we W ___ S, we have [p, s] _ S. Similarly, if W ~ L(p, s), then W n L(p, s) is the singleton point p = w, and again [p, s] _c S. We conclude-that [p, s] _c S for every s in S ~ cl(int cl S). Using this, it is easy to show that such an s lies in S ~ cl A, and we have the desired result. We conclude that S _~ cl(S ,-~ cl A), and A is nowhere dense in S.

Case 3. If int cl S = qS, we will show that S is starshaped. In case all points of S are collinear, then the connected set S must be a segment, and the proof is established. Hence we assume that this does not occur. Select point p in ker cl S :p ~b. Since int cl S = 4), it is easy to see that p must be an lnc point of S, so p is clearly visible from some point r in S. Moreover, an easy geometric argument reveals that p = r. Hence by our hypothesis for set S, every lnc point of S is clearly visible from p via S, and p e S.

Select point x in S to show that [p, x] _c S. If xe lnc S, this is immediate, so assume that x¢lnc S. Since int cl S = q~ and peker cl S, all points of S near x necessarily lie on L(p, x). As in Case 2 above, choose a nondegenerate segment K at x of maximal length in S, and let k and k' denote the endpoints of K, with k ~< x ~< k' and p < x on L(p, x). If k ~< p, then [p, x] ___ S and the argument is finished. Otherwise, p < k ~< x ~< k'. If kelnc S, then k is clearly visible from p via S, so p sees points of (k, k') via S, and [p, k') ~ S, contradict- ing the maximality of K. Thus k ¢lnc S. Since int cl S = qS, points of S near k are on L(p, x). Furthermore, [p, x] ~ cl S, so there must be a sequence {k,} in (p ,k )nS converging to k. Since k¢lnc S, for n sufficiently large, k, sees points of (k, k') via S, and again (k,k') cannot be maximal. We conclude that the situation p < k ~< x ~< k' cannot occur, so k 4 p and [p, x] __ S, the desired result. We have proved that S is starshaped at p, so A = q~, finishing the argument in Case 3.

A close look at the proof above reveals that A _~ bdry S in each case. This completes Theorem 3 and establishes the proof of Theorem d .

COROLLARY. Let S ~ q6 be a bounded, open, connected Set in R 2. Then S is starshaped if and only if every 3 or fewer lnc points of S are clearly visible

from a common point of S.

P O I N T S OF L O C A L N O N C O N V E X I T Y 213

/ / /

/ /

/ /

/ /

/ /

/ /

/ P

Figure 1.

In conclusion, it is interesting to notice that set S in Theorem d need not be starshaped, as the following example reveals.

E X A M P L E 1. Let S be the set in Figure 1, with dot ted lines representing segments not in S. Then every lnc point of S is clearly visible from point p via S, yet S is not sharshaped.

R E F E R E N C E S

1. Breen, Marilyn, 'A Krasnosel'skii-type Theorem for Nonclosed Sets in the Plane'. J. Geom. 18 (1982), 28-42.

2. Breen, Marilyn, 'A Krasnosel'skii-type Theorem for Points of Local Nonconvexity'. Proc. A. M. S. 85 (1982), 261-266.

3. Falconer, K. J., 'The Dimension of the Kernel of a Compact Starshaped Set'. Bull. London Math. Soc. 9 (1977), 313-316.

4. Krasnosel'skii, M. A., 'Sur un crit6re pour qu'un domain soit dtoil6'. Math. Sb. (61) 19 (1946), 309:-310.

5. Valentine, F. A., Convex Sets. McGraw-Hill, New York, 1964.

(Received: October 23, 1981)

Author's address:

Marilyn Breen, Depar tment of Mathematics, University of Oklahoma, Norman,

O k l a h o m a 73019 U.S.A.