Low-cost Computational Astrophotography

Joseph YenElectrical Engineering

Stanford Universityjoyen98@stanford.edu

Peter BryanElectrical Engineering

Stanford Universityjpbryan6@stanford.edu

Abstract

When taking photographs of the night sky, long ex-posure times help photographers observe many stars,but the rotation of the earth results in long star streakarcs about the celestial pole. An algorithm for re-moving these star streaks and substituting accuratelyplaced point-like stars is designed and implemented.Images containing star streaks are transformed intoones with un-streaked starry skies while retainingcolor and brightness information. The procedure isshown to work consistently on a 35-image dataset.

1. Introduction

Astrophotography is a widespread pastime in whichDSLR cameras are used to take images of objectsin space. Certain factors make this a very expen-sive hobby. One of these is the rotation of the earth.Very long exposure times are required to capture pho-tos of faint celestial bodies, and the earths rotationcauses these objects to move in the sky during the ex-posure. This causes the formation of ”star streaks,”bright lines in the photograph tracing out stars’ trajec-tory across the sky. These streaks can be aestheticallypleasing and many photographers deliberately createstreaked images. However, in producing an accurateimage of the sky, they are detrimental. To compen-sate for this motion, astrophotographers use movingcamera and telescope mounts to keep objects steady inthe frame. These mounts can cost thousands of dol-lars, making astrophotography prohibitively expensivefor most people. We hope to use computational tech-niques to remove the need for these rotating mountsand instead post-process these long-exposure images.

2. Related Work

Currently, most astrophotographers compensate forthe rotation of the earth by physically moving the cam-era along with the earth. However, some attemptshave been made to remove this requirement. To accu-rately localize stars in streaked images, one researcherhas re-mapped the star streak to polar coordinates, sothat all stars will have the same point-spread function.This technique was designed for locating specific starsfor the purpose of identifying specific stars, but willbe applicable for our purposes when removing starstreaks [4]. Other factors besides streaking can causequality degradation in star images, and attempts havebeen made to correct this. These have used such tech-niques as Richardson-Lucy deblurring [2] and maxi-mally sparse optimization [1].

3. Data Collection

Initially, we intended to produce our own imagesof starry skies by taking photos of the night sky atStanford. However, due to unfortunately cloudy andrainy weather, we were unable to take acceptable pho-tos that could be post-processed. Instead, we collecteda data set of 35 photos of star streaks gathered fromthe internet, via a Google Image search. To ensurethat our procedure would be able to handle the images,we selected images in which the celestial pole waswithin the photograph. The images also had variedparameters such as foreground/background elements,amount of star streaks, illumination, and noise. Anywatermarks and borders left in the photographs werecropped out to avoid the possibility that they could besegmented as stars.

1

Figure 1. Image processing pipeline

4. Image Segmentation

In order to effectively convert the star streaks intopoints, image segmentation is needed to separate outthe streaks, as detailed in Figure 1. The star streakscould be of varying brightness and color, even along asingle streak. In addition, even though the majority ofthe sky is relatively dark at night, there could still besome color and brightness gradients in different partsof the sky as well. Finally, there will often be objectsin the foreground that will need to be separated fromthe remaining star streaks in the background.

4.1. Thresholding

We perform three types of thresholding to segmentthe star streaks from the rest of the image. Eachmethod is performed independently before combiningto minimize the chance for unwanted segmentations tobe compounded forward, as shown in Figure 2.

First, we use global thresholding by looking at thegrayscale versions of the original images. If the pixelvalue on the black and white image is greater than aglobal threshold, we consider it part of a star streak inthe color image. This will allow us to catch the bright-est star streaks, while retaining the color information.However, this could cause large patches of images tobe considered as stars and removed from the image iftheir grayscale values are too high. To address this, weperform a second iteration through the image and refillthe pixels with their original values if the proportionof information in a given window that was removedexceeded a certain percentage, as shown in Figure 3.

We then use Otsu’s method as a clustering-basedimage thresholding technique to effectively separatethe background of the image from the foregroundwhile segmenting a different set of star streaks [3].Using a binary version of the original image, Otsu’s

Figure 2. Different thresholding methods

method calculates an optimal threshold such that theintra-class variance is at its maximum while the inter-class variance is at its minimum. The stars are thensegmented based on this calculated threshold and re-moved from the image. Again, to minimize the amountof dark patches, we iterate through the image again andreplace the removed pixels with their original values iftoo much information was removed in a given region.

Finally, we perform thresholding by looking at theHSV values converted from the RGB image. Convert-ing the image to HSV representation allows us to lookat the hue, saturation, and value separately unlike inthe RGB space. This will help us to catch some ofthe more faded streaks in the image. After convert-ing the the HSV space, we look at the value array andonly segment out the pixels that have a higher value incomparison to a modified mean. Similar to the previ-ous thresholding methods, we post-process the result-ing image to remove the dark spots.

4.2. Width Estimation

To take into account the individual star streakscould be varying in brightness and color along its path,we also include the neighboring pixels that are within acertain gradient from the pixels that were removed bythe thresholding methods. This will allow us to cap-ture a more representative portion of the star streakwhile leaving less behind, as even though the bright-ness tends to be highest in the center of the streak,dimmer portions along the width of the star are alsopart of the streak. This method also helps account forvarying streak widths, in comparison to just using adefined region size of neighbors, as shown in Figure 4.

2

Figure 3. Patch removal: before vs. after

Figure 4. Width estimation

4.3. Combining Methods

After performing the three independent threshold-ing methods and accurately estimating the star widths,we combine the resulting star streaks in order to max-imize the number of streaks we remove from the im-age segmentation. As an extra check, we look at theamount of information that had been taken out of theoriginal image before combining the results. As cer-tain methods would work better than others on imageswith varying parameters such as illumination and qual-ity, we only combine the ones that removed a balancedamount of pixel values so that we do not lose too muchinformation in comparison with the original image, asshown in Figure 2.

We also experimented with other segmentationtechniques such as edge detection, but the results wereeven worse than those of global thresholding. Thismay be because the star streaks are so densely packedin the data images. Other problems included manyartifacts along the foreground/background boundaries,especially along the horizon. As such, we balancedmultiple thresholding methods instead, which gavesignificantly better results.

5. Star Streak Modification

5.1. Polar PSF Calculation

After identifying the star streaks, the next step is totransform the streaks into individual stars, as detailedin Figure 1. Because the star streaks are caused bythe rotation of the earth, they form a circular pattern

Figure 5. Polar transform vs compressed clusters

around the celestial north pole, as shown in Figure 5.Because of this, there is not a uniform ”point spreadfunction” applied to the stars in Cartesian coordinatesthat could be used to perform deconvolution. We trans-form the star streak image into polar coordinates withthe origin at the celestial pole to have each individualstreak appear as an approximately horizontal line of auniform length [1].

A few possibilities exist to identify the star’s axis ofrotation. An algorithm is used to automatically locatethe coordinates of the axis, even if the axis lay outsidethe image [1]. We chose to instead test our procedureexclusively on images in which the celestial pole laywithin the image, and to have the user manually in-dicate the location of the axis by clicking on it whenprompted.

5.2. Star Clustering

After transforming the streaks into polar coordi-nates, we were able to convert them into points. Ini-tially, deconvolution based on a uniform point-spreadfunction was attempted. However, the star streaks inthe images tested did not form uniform circles, and sodid not form perfectly uniform lines in polar coordi-nates. Deconvolution, then, was not an effective strat-egy. Instead, a recursive clustering algorithm was im-plemented. In this algorithm, a bright pixel was iden-tified, and its neighbors were added to a cluster if theywere not surrounded entirely by dark pixels. This pro-cess was repeated until the complete cluster had beenformed. The cluster (i.e. an individual streak) was thencompressed into a single pixel, by averaging the val-ues of all pixels in the cluster, and setting the values ofeach cluster in the pixel to zero, except for the centerpixel, which was set to this average value. This pro-cedure was then repeated until each cluster had beenidentified, as shown in Figure 5.

3

Figure 6. Star replacement

5.3. Star Replacement

After blurring the image to remove the dark patchesand transforming the star streaks into individual pix-els, these images must be combined. The star pixelsare still in polar coordinates, so the clustered and com-pressed image is converted back to Cartesian coordi-nates, with the origin again at the celestial pole. Af-ter this transformation, the stars are correctly placed inspace. However, they consist of single pixels, whichwill not appear aesthetically pleasing to a user. Theimage containing only the star pixels is convolved witha 3-pixel wide Gaussian kernel in order to cause thestars to appear broader. The standard deviation of theGaussian was made to depend on the size of the im-age, so as to avoid creating huge stars in small imageof the sky. At this point, any stars that overlap witheach other are removed. This is done because any pix-els thought to be stars that are found so close to eachother are almost certainly artifacts, originating frombackground details that were not removed during thesegmentation process. After this widening and filter-ing of the star image, the background image and thestar image are simply added to each other.

6. Image Post-processing

6.1. Hole Filling

After segmenting the bright star streaks from theimage, many empty holes remain in pixel locationswhere the star streaks originally occupied. In orderto remove these dark spots, we fill these holes withinterpolated values from the surrounding pixels usingboth a linear and a two-dimensional averaging func-tion. First, we iterate through the pixels in the imagewhere the stars have been removed. For each horizon-

Figure 7. Hole filling: before vs. after

tal line of connecting pixels that have empty values, wereplace them with the average value of the edge pixelsthat bound them. Then, we run a two-dimensional av-eraging function in the remaining darkened regions ofthe interpolated image in the night sky, as shown inFigure 7.

6.2. Bilateral Filtering

After filling the holes, parts of the night sky stillseem noisy in the images, with sometimes sharp colorchanges between neighboring pixels. To diminishsome this noise, we use a bilateral filter based on aGaussian distribution to smooth the image. This is ap-plied to the images with high enough resolution, wherethe modified window size will not impact the newlyadded point stars. One of the benefits of bilateral fil-tering is that it allows us to preserve edges while fil-tering, in comparison to a simple median filter. Thisis useful because we want to try to preserve the edgesin the foreground while smoothing out the night sky inthe background. The point stars are also smoother andbetter blended into the background night sky.

7. Results and Discussion

Qualitatively, the images produced by the algorithmare visually pleasing, as shown in Figure 8. Stars arelocalized at the center of what was a star streak inthe original image and retain their color. The back-ground sky is filled in and blurred so the spaces wherestar streaks were removed do not appear harsh to theviewer, and this process works well even in imageswith color gradients. The edges are also preserved wellqualitatively when bilateral filtering is performed, butthe degree is difficult to measure quantitatively due tothe lack of ground truth images. Some problems doremain, however. For instance, the segmentation is notperfect so some star streaks, especially fainter ones,are not removed. Some extremely bright sections of

4

Figure 8. Sample processed images

the foreground may also be interpreted as stars, result-ing in false ”stars” in the foreground. The filling-inprocess is also not ideal, so the background can appearsomewhat blotchy. All stars are also the same shapeand size, which is not true for the real sky.

Quantitatively, in order to evaluate how well the starstreaks were removed, we measured the variance ofa window of the original color image and comparedit to the variance of the same patch after hole filling.The lower the variance, the more star streaks were re-moved. As shown in Figure 9, we see that for a 31x31pixel window surrounding the celestial axis, the aver-age variance decreases from 1.09e-2 to 5.62e-3, whichmeans that a significant amount of star streaks wereremoved. In order to evaluate the denoising done bythe bilateral filter, we again measure the variance, butthis time of a window of the image before inserting thestars, both before and after the bilateral filter was ap-plied. This approximates the noise level of the image.As shown in Figure 9, the variance decreases after thebilateral filter is applied across all images. Taking theaverage, the variance decreases from 5.62e-3 to 1.81e-3. In addition, we find the timing for our algorithmto be quite efficient and mainly dependent on the im-age resolution. On average, the full image processingalgorithm ran in 20 seconds for medium-sized images(around 500k pixels) and 80 seconds for large, high-quality images (around 3 million pixels).

8. Conclusion and Future Work

We were able to create an algorithm that removesmost star streaks from an image, and replaces themwith appropriately-colored point-stars. Some streaksremain, causing the sky to appear much uniform thanin the unedited image, but by denoising using bilat-eral filtering, we are able to create more aestheticallypleasing images that accurately represent the place-ment of stars in the sky. To further improve the pro-

Figure 9. Variance

cess, we could investigate other image segmentationprocedures, including machine learning algorithms,that could allow us to more effectively distinguish starstreaks from the rest of the image. The stars are alsocurrently the same shape, which is not representativeof actual stars, so we can consider better visualizationmethods such as incorporating the width estimation.We could also develop better strategies for filling holesleft in the image by the removal of star streaks. Whilewe were unable to obtain our own images during thecourse of this project, we would also want to test thisalgorithm on images obtained directly from a camera,without any post-processing.

Contributions

Joseph: data collection, thresholding, width estima-tion, other segmentation, polar PSF, hole filling, de-noising, results and discussionPeter: data collection, polar PSF, clustering, star re-placement, results and discussion

References[1] B. D. Jeffs and M. Gunsay. Restoration of blurred

star field images by maximally sparse optimization.IEEE Transactions on Image Processing, 2(2):202–211, April 1993.

[2] L. W. H. W. Y. H. Laili Su, Xiaopeng Shao.Richardson-lucy deblurring for the star scene under athinning motion path. 9501, 2015.

[3] N. Otsu. A threshold selection method from gray-levelhistograms. IEEE Transactions on Systems, Man, andCybernetics, 9(1):62–66, Jan 1979.

[4] B. Sease and B. Flewelling. Polar and spherical imagetransformations for star localization and rso discrimi-nation. 01 2015.

5