Monitoring a Fast Flux botnet using recursive and passive DNS: A case study

Dhia Mahjoub Umbrella Security Labs, OpenDNS

San Francisco, USA dhia@opendns.com

Abstract— Fast flux, an evasion technique that has been around for years, continues to be widely used by cybercriminals today. In this case study, we describe a real-time monitoring and detection system that leverages recursive and passive DNS to track the Kelihos fast flux botnet. We track how the botnet grows its population of infected hosts, and detect, in real-time, the newest Kelihos fast flux domains that are being hosted by the botnet. Our analysis will present results on various components and attributes of the infrastructure leveraged by the Kelihos fast flux botnet. These include: domain TLD distribution, botnet geo-distribution, botnet daily cycles, distribution of operating systems used by the botnet machines, daily-discovered fast flux domains, domain and IP lifetime distribution, as well as specific examples of usage that highlight malicious campaigns.

Keywords—fast flux; botnet; Kelihos; real-time; passive DNS

I. INTRODUCTION Fast flux came about a few years ago as an evasion technique used by cybercriminals to circumvent the takedown of their malicious domains or blacklisting of hosting IPs. Fast flux (IP fluxing) is a DNS-based technique where a domain resolves, with a low TTL, to a large number of IPs that are geographically diverse or resolves to a single IP with a very low TTL (typically zero). Fast flux is known as “single flux” where only the domain shows a fluxing behavior. Double flux occurs when both a domain and its name server(s) have a fluxing behavior. Fast flux is an effective technique and is often used by domains serving diverse criminal campaigns such as malware downloader domains (via Blackhole and Redkit exploit kits), trojan CnCs, and spam and phishing domains. This technique has received a lot of attention from academic research [1-4], industry [5,7-12], Internet regulating bodies [6] and security research blogs [12-21].

The Kelihos fast flux botnet is an info-stealer (e.g. passwords and virtual currency) and a spamming botnet with a peer-to-peer structure and fallback fast flux CnC domains. It was taken down by the combined initiatives of Microsoft and Kaspersky Labs in 2011, and again by Kaspersky Labs and several security firms in 2012. However, security researchers have been witnessing its recent reemergence [9-12]. There is a daily stream of new fast flux domains that are created and hosted by the Kelihos botnet and used to drop the Kelihos malware on new victim machines. The Kelihos fast flux botnet is used in combination with Blackhole and Red kit exploit kits to direct user traffic to malware downloader domains [21]. In this case study, we describe how we monitor, in real-time, the Kelihos fast flux botnet using recursive and passive DNS. Starting with a seed of Kelihos domains, we track the growing population of hosting IPs and detect new fast flux domains hosted by the botnet the instant they appear in our DNS traffic. This allows us to track and measure the various components and attributes of the infrastructure used by the Kelihos fast flux botnet. These include: domain TLD distribution, botnet geo-distribution, botnet daily cycles, distribution of operating systems used by the botnet machines, daily-discovered fast flux domains, domain and IP lifetime distribution, as well as specific examples of usage that highlight malicious campaigns.

II. REAL TIME MONITORING SYSTEM A. Description

The monitoring system consists of two phases. In Phase 1, we take a set of fast flux domains

confirmed to be hosted by the Kelihos botnet, and begin constantly resolving them via recursive DNS. We add any new IP to the set of IPs of the botnet (obtained by resolving the initial set). We also probe these IPs to keep count of those that are live. This gives us an approximation on the current size of the botnet (put NAT and DHCP churn effects aside). In Phase 2, using our in-house DNS database (which is fed in real-time with authoritaive DNS traffic coming from all data centers), we poll the passive DNS by doing inverse lookups against it to detect any new domains that resolve to the botnet’s IPs. This ensures that we catch these new domains as as soon as their domain, IP association is seen in our traffic. The two phases of the monitoring system run non-stop and in parallel. B. Initialization

To start, we seed the system with a set of domains that fit the Kelihos profile. The Kelihos profile consists of domains registered very recently (a few days before or on the same day they are detected) under various gTLDs and ccTLDs, and which generally resolve to a single IP with a TTL=0. There are exceptions to this profile (i.e. cocala.asia which resolved to 6 IPs on Aug 23rd with a TTL=600, and other cases of domains resolving with a TTL=300). These IPs whose number is increasing rapidly, are hosts of the Kelihos botnet and typically represent infected individual machines. Furthermore, a Kelihos fast flux domain is recognizable by the fact that it hosts malware payloads with specific names.

In practice, we tested building this “Kelihos profile” set with two different methods. The first one is to select seed domains from the output of our Fast Flux classifier [14]. The second method is to extract from our blacklist of domains “Kelihos profile” domains. The blacklist of domains is very diverse so it requires pre-processing to extract the domains we need. For this, we resolve all domains to IPs (using recursive DNS or the passive DNS database), keep only domains with a TTL of zero, and eliminate all known sinkholes. We also resolve domains to nameservers (NS), and build the graph associating domains to IPs and NSs. Looking at the connected components in the graph, we find that the largest connected component corresponds to the bulk of Kelihos domains that we use to seed the monitoring system. Since the fast flux domains are hosted on

the botnet’s large IP pool, this component has a large number of domains and IPs. If a component is composed of a large number of domains but just one or a few IPs, it typically represents a shared hosting IP, which is discarded because it doesn’t match the Kelihos profile. C. Post discovery check As the system starts, it will constantly discover new IPs in the botnet and detect new domains hosted on the botnet as soon as they hit our DNS authoritative traffic. To eliminate any false positives, we apply a post discovery check that only keeps the domains that fit the Kelihos profile and feed them back into Phase 1 to eventually grow the monitored IP pool. The post discovery check also eliminates any domains that are already sinkholed, and following all domains that combine the unwanted features: high TTL, not hosted on the known botnet IP pool, sharing IP with other unrelated domains or not hosting known malware payloads, and old registration. This filter is necessary as we observed that new malicious fast flux domains having other profiles (e.g. a TTL of 150, and serving other types of malicious intent) are occasionally hosted on some of the IPs that are part of the Kelihos “TTL=0” botnet. These other fast flux domains should not be confused with the Kelihos fast flux domains. The pseudo-code below describes the aformentionned monitoring system process.

TABLE I. MONITORING/DETECTION SYSTEM PSEUDO-CODE

while true 1. select a seed of domains with a confirmed Kelihos profile 2. continuously milk domains for IPs 3. continuously “inverse lookup” IPs in passive DNS, for new domains that start resolving to these IPs 4. check detected domains for known profile (e.g. TTL, registration, existence of payload, etc) 5. add new domains to the initial seed

D. Examples We were interested to determine whether this

real-time monitoring and detection system could be used to track other fast flux, malicious campaigns. For example, we monitored the fast flux domains used for daily spam campaigns. Anywhere from one-hundred to several thousands of these domains are registered daily and observed in our traffic. These domains typically have a TTL of 1440, similar naming conventions (fat loss, work from

home, news, etc.). However, we discovered that these domains are not hosted on a botnet, rather on a small pool of IPs located on either abused or bulletproof hosting providers. As an illustrative example, we take a sample of fast flux spam domains fitting the “TTL=1440” profile and we use our system to monitor them and discover new domains fitting the same profile. Table II shows the top most popular spam hosting ASNs of these “TTL=1440 spam profile” domains.

TABLE II. SPAM HOSTING ASNS

ASN Name Country

54444 Avesta Networks LLC US

22923 Yesup Ecommerce Solutions Inc. CA

4134 Chinanet CN

31430 ISP Firma TEL RU

54718 SYNAPTICA-NA CA

52284 Panamaserver.com PA

42926 Radore Veri Merkezi Hizmetleri A.S. TR

24940 Hetzner Online AG RZ DE

4808 CNCGROUP IP network China169 Beijing Province Network

CN

47869 Netrouting Data Facilities NE

44050 Petersburg Internet Network LLC RU

16019 Vodafone Czech Republic a.s. CZ

III. RESULTS We implemented the monitoring and detection system using Python and shell. The indexed DNS database is HBase based. In the following section, we describe the different results of this case study. A. TLD distribution In this study, we took a sample of 712 Kelihos fast flux domains discovered over a period of 6 months. The set consists of 541 2LDs and 171 3LDs. Figure 1 shows the distribution of top level domains (TLD) that Kelihos domains have been using. The ccTLD .ru was the most abused up until early August 2013. At which point, about a hundred live domains using .ru were taken down [18]. Following that takedown, a variety of TLDs have been utilized by the Kelihos gang. The TLDs .com, .nl and .net come next in popularity of use by Kelihos.

Fig. 1. TLD distribution of the Kelihos domains.

B. Botnet geographical distribution In Figure 2, we show the geographical

distribution of a sample of 10,000+ unique botnet IPs. We observe that infected hosts are highly concentrated in Eastern Europe (Ukraine, Kazakhstan, Belarus, Russia), as well as Japan and Taiwan.

Fig. 2. Geographical distribution of the botnet hosts.

Figure 3 shows the country breakdown of the botnet’s IP population.

Fig. 3. Kelihos botnet country distribution.

Using another sample of botnet IPs, Table III highlights the top 15 botnet IPs hosting countries.

TABLE III. BOTNET HOST POPULATION BY COUNTRY (TOP 15)

% Population. Country

50.47 Ukraine

6.62 Kazakhstan

6.33 Russia

6.27 Belarus

5.82 Japan

5.31 Taiwan

1.66 Turkey

1.58 India

1.25 USA

1.04 Chile

0.97 Armenia

0.95 Romania

0.77 Lithuania

0.77 South Korea

0.76 Hong Kong

C. Botnet daily cycle

Figure 4 shows the growth of the number of unique IPs and unique live IPs over a period of 3 days and a half (from 8:47am UTC on July 11th to 10:06pm UTC on July 14th). The number of unique IPs (plotted in blue) is constantly growing as more and more hosts are infected and recruited into the botnet. The number itself however does not indicate the real size of the botnet because of NAT and IP

churn effects. The number of live IPs (plotted in red) gives a better idea of the growth of the botnet.

As Figures 2 and 3 showed, the bulk of the botnet machines is concentrated in Ukraine and the neighboring Eastern Europeans countries. This becomes clear as we see the daily cycle of the botnet where its live population drops on evening hours and rises again in the morning following the Eastern European time zone (UTC+2). This is another indication that most infected hosts belong to corporate or residential users in Ukraine and neighboring countries who follow a daily working hours schedule. On Figure 4, we see that the botnet starts with 1,316 live IPs at the beginning of this experiment, reaching a minimum of 854 IPs on the night of the 1st day (1am UTC or 3am in Ukraine), and a maximum of 3307 in the late afternoon of the 3rd day (5:53pm UTC).

Fig. 4. Evolution of botnet IP population and daily cycle.

D. Most infected ASNs Using a sample of 10,967 unique IPs of the

botnet, we show the top 15 infected ASNs in Table IV. Table IV highlights once more that Ukrainian hosts are the most infected and that infected IPs belong to ASNs serving both corporate and residential users.

TABLE IV. BOTNET HOST POPULATION BY ASN (TOP 15)

% ASN Name Country

7.85 15895 "Kyivstar" PJSC Ukraine

5.41 9198 JSC Kazakhtelecom Kazakhstan

3.89 3462 Data Communication Business Group Taiwan

3.55 25229 Kyivski Telekomunikatsiyni Merezhi LLC

Ukraine

3.16 6849 JSC UKRTELECOM Ukraine

2.91 6697

Republican Unitary Telecommunication Enterprise Beltelecom

Belarus

2.64 31343 Intertelecom Ltd Ukraine

2.31 34661 TOV TRK "Briz" Ukraine

2.28 13188 TOV "Bank-Inform" Ukraine

1.90 21219 PRIVATE JOINT STOCK COMPANY "DATAGROUP"

Ukraine

1.85 45025 Online Technologies LTD Ukraine

1.11 12530 Golden Telecom Ukraine

1.05 52091 FOP Trubnikov Valeriy Muhaylovich Ukraine

1.00 15377 ISP "Fregat" Ltd. Ukraine

0.96 25106 Mobile TeleSystems JLLC Belarus

E. Operating Systems distribution

We took a small sample of live Kelihos IPs and probed them for an OS fingerprint. We noticed that 85% of the hosts are running a variant of Windows XP or Vista and one third of those hosts are running Microsoft Windows XP PocketPC/CE. PocketPCs, frequently used in a corporate setting, are another indication that the botnet might be targeting corporations. These types of victims are more prone to offer valuable information for the cybercriminals to harvest. Other observed operating systems were embedded versions of Linux, SCO, and FreeBSD typical in DSL and Cable modems. F. Daily new detected domains

Figure 5 shows the number of daily new detected Kelihos domains for a period of 1 month. The blue bar represents the number of discovered domains (2LDs only) and the red bar represents the total of discovered FQDNs (counting both 2LDs and 3LDs) for a given day. For example, giqhab.info represents a domain (2LD) and gvhvma.giqhab.info is a subdomain (3LD) under that 2LD. We observed that a new domain or subdomain were active everyday.

Cybercriminals maximize the utilization of fast flux through both 2LDs and 3LDs. On Aug 22nd, for example, the system detected 16 FQDNs, comprised of 13 domains and 3 subdomains.

Fig. 5. Daily counts of newly detected Kelihos domains over a month.

G. Domains’ lifetime distribution In Figure 6, we show the lifetime distribution of the sample of 712 Kelihos fast flux domains. 280+ domains were active for less than a day, and the majority of domains had a lifetime of less than 20 days. About 4 domains lasted for 50 days. These domains served as name servers for a large number of Kelihos domains and stayed active for a long period before they were taken down [18].

Fig. 6. Kelihos domains’ lifetime distribution.

Typically, a newly registered Kelihos domain either stays dormant for a few days or is immediately used in a malicious campaign after being registered [12,13]. Consequently, we usually observe sudden spikes towards Kelihos domains in our DNS traffic. Some Kelihos domains were also observed to be registered and stay dormat for close to 2 weeks before being used.

Fig. 7. Kelihos fast flux domains with very similar DNS traffic patterns.

Figure 7 shows how several Kelihos fast flux domains have nearly identical traffic patterns: izytexuf.ru, niqtasoz.ru, ahfamzyk.ru, tosahrux.ru, and bomuxvis.ru. Our monitoring system detected these domains as soon as they triggered DNS traffic, which is sometimes a couple days before they are used in malicious campaigns, or before actual

malware payload urls are reported by the security community (e.g. on Urlquery and Virustotal). H. IPs’ lifetime counts We took a sample of 27,200+ IPs from the Kelihos botnet and we used our historical DNS database to get the lifetime of every IP by calculating the duration between the first time and the last time any Kelihos domain has resolved to that IP. Figure 8 shows the lifetime distribution of the IPs sampled. Close to 1% of the IPs served in the botnet for 2 months or more, with a few IPs hosting fast flux domains for up to 3 months. 19,416 IPs were used for less than a day, and 2,624 IPs served between 24 and 48 hours.

Fig. 8. Kelihos botnet IPs’ lifetime distribution.

I. Kelihos domains’ usage Kelihos fast flux domains have been used for at least the following three purposes: as redirectors for Blackhole and Red kit exploit kits via iframe injection into compromised sites; as malware dropping domains mainly for the Kelihos trojan through spam campaigns (e.g. fake Walmart, USPS notifications, or the tragic Boston bombing news) and as trojan CnC domains [7,15,21]. In Figure 9, we show a list of domains compromised with iframe injections where Kelihos domains are used as the landing Exploit url.

Fig. 9. Kelihos fast flux domains used in Iframe injections.

A variation of the iframe injection known as the Cookie Bomb attack [19,20] is also shown in Figure 10 as using Kelihos domains in the landing Exploit url [15].

Fig. 10. Code snippet of the Cookie Bomb attack.

We took a sample of 200 Kelihos domains, and looked at how they are used by different types of malware. Typically, these domains are either used to drop the malware on a soon-to-be infected machine, or used as CnC for malware already installed on an infected host. Therefore, for a given domain, we looked at malware detected that either was downloaded from the domain, or communicated with the domain. This follows the taxonomy employed by Virustotal’s domain information search. Table V shows the percentages of utilization of the sample fast flux domains. 58.5 % of the

domains were used either as a dropper or CnC and 30% were used as both a dropper and a CnC. Common malware labels displayed by Virustotal that are served by the fast flux domains are Trojan Zbot, Trojan ransomware, Trojan Kazy and Trojan Kelihos.

TABLE V. FAST FLUX DOMAINS USAGE BY MALWARE

Domain usage %

Only download, no communication 10.5

No download, only communication 18

Both download and communication 30

No download, no communication 41.5

J. Malware executables distribution The Kelihos botnet serves malware payloads with known names (e.g. as angrim2.exe, rasta02.exe). In Figure 11, we show the distribution of payload names extracted from the botnet domains in the period July-September.

Fig. 11. Kelihos malware payload names’ distribution.

IV. CONCLUSION In this case study, we described a monitoring system that tracks, in real-time, the growth of the Kelihos botnet, and detects new malicious fast flux domains served by the botnet as soon as they appear in OpenDNS’ DNS traffic. We discussed several results on the fast flux domains’ TLD distribution, botnet geographical distribution, botnet’s daily cycle, the operating system of the infected machines, the daily-discovered domains, domain and IP lifetime distribution and examples of usage of Kelihos domains for malicious campaigns. We relied in this study on DNS and network perspectives to study the botnet and its domains.

Future work will involve a description of the P2P structure of the botnet by adding malware and host-based analysis. We have also extended the use of the real-time monitoring system from fast flux Kelihos domain detection to a more generalized framework that monitors other profiles of fast flux and detects a wide variety of other malicious domains such as Exploit kit domains, trojan dropping domains and CnCs, and ransomware domains. This generalized system has shown promising results as a means for early detection of DNS-based threats [16,17]. It is important to note, that this system is as fast as the latency of the process of parsing our authoritative

traffic, clean-up, deduplication, and insertion into our indexed DNS database. Other future work will involve using streaming technologies such as Storm, Kafka, ZeroMQ to acquire the authoritative DNS traffic flow as soon as DNS responses come through from our resolvers. This is currently being implemented within our research team, and will supplement the passive DNS-based detection system described in this paper.

V. ACKNOWLEDGEMENTS We would like to thank the reviewers for their contructive comments and several researcher friends for their valuable technical feedback and support.

REFERENCES [1] T. Holz, C. Gorecki, K. Rieck, F. C. Freiling, “Measuring and Detecting

Fast-Flux Service Networks”, NDSS Symposium, 2008. [2] J. Nazario, T. Holz, “As the Net Churns: Fast-Flux Botnet

Observations”, 3rd International Conference on Malicious and Unwanted Software, pp. 24-31, 2008.

[3] M. Konte, N. Feamster, J. Jung, “Dynamics of Online Scam Hosting Infrastructure”, Proceedings of the 10th International Conference on Passive and Active Network Measurement, PAM '09, pp. 219–228, 2009.

[4] R. Perdisci, I. Corona, and G. Giacinto, “Early Detection of Malicious Flux Networks via Large-Scale Passive DNS Traffic Analysis”, IEEE Trans. on Dependable and Secure Computing, vol. 9(5), pp. 714-726, 2012.

[5] The Honeynet Project, 2008, http://www.honeynet.org/papers/ff/ [6] “SAC 025: SSAC Advisory on Fast Flux Hosting and DNS”, 2008,

http://www.icann.org/committees/security/sac025.pdf [7] “Fast Flux Botnet/Black Hole nameserver”, 2013,

http://www.spamhaus.org/sbl/query/SBL175713 [8] “ATLAS Summary Report - Real-time global report of fast flux

activity”, http://atlas.arbor.net/summary/fastflux

[9] T. Werner, “Peer to peer poisoning attack against the Kelihos.C botnet”, 2013, http://www.crowdstrike.com/blog/peer-peer-poisoning-attack-against-kelihosc-botnet/index.html

[10] A. Adamov, “Update on Kelihos Botnet (August 2013)”, Lavasoft, http://bit.ly/16bLkwW

[11] A. Escobar, “Kelihos Worm Emerges, Takes Advantage of Boston Marathon Blast”, 2013, http://blog.trendmicro.com/trendlabs-security-intelligence/kelihos-worm-emerges-takes-advantage-of-boston-marathon-blast/

[12] D. Mahjoub, “Kelihos is back (with a vengence) is its third incarnation”, 2013, http://labs.umbrella.com/2013/02/13/kelihos-in-its-third-incarnation

[13] D. Mahjoub, “Details on Exploit kits as told by the Umbrella Security Graph”, 2013, http://labs.umbrella.com/2013/05/17/a-quick-look-at-domains-used-for-exploit-kits

[14] D. Mahjoub, “Discovering new malicious domains using DNS and big data, Case study: Fast Flux domains”, BSides NOLA 2013, http://bit.ly/14RZ02V

[15] D. Mahjoub, “Tracking versatile Kelihos domains”, 2013, http://labs.umbrella.com/2013/07/30/tracking-versatile-kelihos-domains/

[16] D. Mahjoub, “Real Time Monitoring of Kelihos Fast Flux botnet: a case study for APWG eCrime 2013”, paragraph 3, http://bit.ly/19wPmnn

[17] D. Mahjoub, “Fast detection of new malicious domains using DNS”, BSides Raleigh 2013, http://slidesha.re/1ad0kQO

[18] “The result on 48hours+ in battle with Kelihos < request for FURTHER block/dismantle cooperation & support. #Tango is going down..”, 2013, http://malwaremustdie.blogspot.com/2013/08/the-quick-report-on-48hours-in-battle.html

[19] “Proof of concept of CookieBomb code injection attack”, 2013, http://malwaremustdie.blogspot.jp/2013/07/proof-of-concept-of-cookiebomb-attack.html

[20] “What is behind #CookieBomb attack? (by @malm0u53)”, 2013, http://malwaremustdie.blogspot.com/2013/07/what-is-behind-cookiebomb-attack-by.html

[21] “What happened if Red Kit Exploit Kit team up with BlackHole EK? = Tripple payload + infection of Khelios!”, 2012, http://malwaremustdie.blogspot.com/2012/12/what-happened-if-red-kit-team-up-with.html