CODE BLUE 2014 : [Keynote] IDA and digital security by Ilfak Guilfanov

IDA and digital security Hex-Rays Ilfak Guilfanov

2 (c) 2014 Ilfak Guilfanov

Presentation Outline

 About me  Digital security is more important than ever  Evolution of development tools  Story of IDA  What we could do to improve the situation  Your feedback


About myself

 Started a programmer career more than 27 years ago  The author of IDA Pro and Decompiler  Founder and CEO of Hex-Rays  I have passion about programming and beautiful code

– I'm not a reverse engineer but a software developer  I believe that efficient and robust software makes our lives

better


Digital security

 We store more and more personal data digitally: – Medical records – Bank account info – Communications (emails, sms, etc) – Photos and videos – etc...  More and more (all?) decisions are taken based on digital

data  Virtually all our devices are connected to each other, which

makes them powerful and vulnerable at the same time


Security: current state

 Far from perfect  Most our software has vulnerability holes due to:

– Design flaws (security was not built in from the start) – Buggy implementations – Poor or no testing – Changed environments  We will hear more stories about thousands of stolen credit

cards, disclosed sensitive information, etc


Everything is lost? Nope

 I think that the situation is improving over time – The software improves – Most new systems get designed with security in mind – Society better handles digital security related issues


Development methods are constantly improving

 Compilers become better (the very first compilers would not issue any warnings!)  We have more compilers (clang, for example)  Programming languages evolve (initially c++ had no

templates)  Better programming paradigms and design patterns  Better software revision systems  Agile and other development approaches


Software analysis and testing

 There are many source code analyzers on the net (25 years ago there was just “lint”)  Many testing frameworks appeared too  Debuggers, fuzzers, valgrind, etc


How difficult was it to develop software in the past?

 Overall, much harder than today. One proof of this difficulty is the software size (measured as the number of source lines of code, not as the size of the binary image)  Example, MS Windows (data from wikipedia)

Year Operating System SLOC (mil) 1993 Windows NT 3.1 4-5 1994 Windows NT 3.5 7-8 1996 Windows NT 4.0 11-12 2000 Windows 2000 More than 29 2001 Windows XP 45 2003 Windows Server

2003 50


Principal reasons of difficulty

 Missing or inefficient development tools: editors, compilers, debuggers, code analyzers, testers, etc.  Slower computers  No memory protection or program isolation provided by the

operating system  Immature programming paradigms and methodologies  Lack of program verification and testing tools  Let us take a tour


Editor 'ed'

 A text editor with virtually no visual feedback  Yet considered a powerful tool:

– It has regular expressions – Programmable  Even modern Linux distributions still include it


Compilers

 No or little warnings  Could produce buggy code  Poor code optimization  However, as a software developer I still consider compiler

writers as semi-gods :)  Turbo C compilers from Borland were a breath of fresh air


Debuggers

 My favorite was Turbo Debugger. Compared to other debuggers it was a feast to the eyes  Powerful, robust, can display the source code  Supported hardware and conditional breakpoints


SoftICE: system level debugger

 Can debug the entire operating system  Very powerful  Its popularity was a disadvantage in some cases, anti-

SoftICE tricks were employed by some software

(from wikipedia)


Disassemblers

 Debuggers were not enough in some cases – Programs get bigger – Algorithms became more complex – Anti-debugger tricks were used more often – More detailed analysis was required, especially for viruses – Compiler bugs required to check non-executable (object) files  Disassemblers would:

– Analyze the program in depth – Show cross references – Assign meaningful names to functions and data – etc...


The most popular disassembler: Sourcer

 Sourcer from V Communications was a great tool  It was like magic for a newcomer, in fact

– It would tell apart code from data – It would assign meaningful names and comments


Sourcer: batch mode disassembler

 The biggest shortcoming: it was a batch mode program  Could handle programs of

limited size  Would occasionally misidentify

code or data  Slow for big programs


Interactivity is the answer

 There was a disassembler called Dis*Doc from RJ Swantek  I haven't used it myself so can not tell you much  But I liked the idea very much:

– No need to wait for the results – The user can browse the listing and annotate it – The user can guide the disassembler by marking locations as code or data – WYSIWIG (what you see is what you get) was a la mode at that time (remember 'ed'?)  This was the reason why I decided to create IDA


Initial design and implementation

 I tried a few approaches and rewritten the code at least 4 times before I hit the right thing  The result was either too heavy and slow, either too

lightweight and limited  Remember about 640KB memory and slow processors!  I needed a robust and fast database


Database choice

 Requirements: – Fast – Capable of storing variable sized objects – Robust  I tried the available databases like Paradox from Borland but

quickly abandoned the idea, they were way too slow  Fortunately my friend Pavel Rousnak implemented a B-tree

engine  We are still using his database in IDA, upgraded and

improved over many times but still the same code


IDA 0.1, the first public version

 It took me 6 months to implement version 0.1  The basic functionality was present but the user interface

was ugly  It supported only x86 instructions  Yet it was interactive and working!


IDA v2.09: nice text interface

 IDA v2.09 was released in 1994  TurboVision library from Borland fixed the user interface  It was robust, supported 3 processor families (x86, i51, and

z80), 8 input file types, had a built-in C like language  Since it was already over 500KB, it was a heavily overlayed

program. I was saving every byte


IDA v3.5b: with symbol files

 It was released in 1996  It had symbol files (IDS), could run on OS/2, MS DOS

extender, had loadable modules, etc  There were many other releases I won't mention in order not

to bore you


IDA Roadmap

 My initial plans for IDA were really ambitious. They included: – AI (artificial intelligence) with a LISP like language – Building a binary program optimizer on top of IDA – Using IDA for binary translation – Building some kind of knowledge database about common program snippets and their meanings – IDA would point out suspicious or problematic parts of the code (vulnerability scanner?) – Etc,etc,etc  However, with ever growing users of IDA I was simply

overwhelmed by the user requests and bug fixes  Even today it is like this


Datarescue and IDA

 I was lucky that Mr. Pierre Vandevenne got interested in IDA. His contribution can not be overestimated  Datarescue converted my

hobby project into a commercial program in 1996  The first GUI version of IDA

was built there, in 1999  We made a long and very

interesting way together  BTW, Pierre found the lady we

use for IDA logo


PC Magazine: Technical Excellence Award

 In 2001 IDA Pro was nominated as a finalist of the Annual Awards for Technical Excellence  We went to Las Vegas to participate in the

award ceremony  We lost the competition... to Microsoft's

Visual Studio .NET  It was still fun :)


IDA and pirates

 Unfortunately we were plagued by piracy  There were more pirates than legitimate users  Pirates were eating our time and resources  A typical conversation would start with a compliment from a

stranger; he would ask for a “little help” in the second message  It was even boring, so predictable  I do not understand when clever people pirate software and

then shamelessly ask for help. Probably they aren't that clever after all


IDA piracy map 2006

 Just a map of places where a pirated version of IDA was used (circa 2006)

(from www.datarescue.com)


Decompiler: a plugin on top of IDA

 Was greatly inspired by Cristina Cifuentes' PhD thesis on decompilation  After reading the thesis it was clear how to build a

decompiler  But the devil is in the details... many subproblems were still

not solved. For them: – Come up with an idea how to solve it – Implement it – Test it – Throw away and start over if it did not work  “Wash, rinse, repeat” – for years... (I liked it!)  The first attempts were made in 1998 or even earlier  The first public version appeared in 2007


Decompiler details

 Decompilation is a complex problem, insolvable in general  Very time consuming to

develop  Seemingly minor design

mistakes haunt and hinder development  One has to cut corners in

order to come up with a working decompiler  Question: which corners to

cut?


Hex-Rays

 Unfortunately Pierre decided to quit in 2007  I had to continue with the decompiler alone  Hex-Rays quickly became a strong and passionate team

– We do care about our code – We want to publish as bug free software as we can – We care about our users


Why IDA, after all

 I created IDA because there was no interactive and robust disassembler at that time; on the other hand there was a strong need in such a tool  I kept maintaining IDA all these years because

– IDA helps to solve some problems we face, like viruses – IDA improves our digital security – IDA users are very nice people in general (legit ones)  Like any tool IDA can be used for lowly deeds. Examples:

– Cracking software – Stealing code and algorithms


IDA as a seeing aid

 I usually compare it to a microscope  Basically useless to general

public but indispensable to professionals  Requires skills to use it

efficiently


Who uses IDA?

 This is a frequent question  I can only mention some users categories:

– Anti-virus companies – Security oriented organizations – Governments and military – Hobbyists – Shady persons of all kinds – Pirates (the dogs bark but the caravan goes on)  Overall it is a motley crew


How IDA improves digital security

 Our legit users are white hats (or at least they pretend to be so :)  IDA itself is not in the spotlight and stays in shadow but many

of our users are famous security researchers  We are glad they we can help them with their tasks  We want IDA to be safe for them (and for all our users)


How we improve security of IDA

 We run tests  We compile our code with various compilers on different

platforms  We use code reviews  We use lint, valgrind, and other verification tools  User reports are handled by the developers (there are no

first/second help lines). This ensures that developers really suffer from their bugs :)


Testing IDA

 We continuously work on improving our coding style  We keep adding more test cases. Every new reported bug

ideally creates a new test case  We keep adding more testing methods

– We have an extensive test suite for our analysis engine – We recently added tests for the user interface – We have a constantly growing set of decompiler tests – Our decompilation test suite is about 500GB (only output files)  Virtually every day we add a couple of new test cases  There are dedicated computers for running tests  We have a bug bounty program for critical bugs  We know that we are still not testing IDA enough


Bug bounty

 The idea is simple: if you find a critical bug in IDA, we will pay you a bounty  Many other companies do that; we think that it is really a

good idea  It is difficult to come up with a reliable exploit for IDA:

– IDA kernel is personalized for each user – We use ASLR – IDA randomizes the heap at the start – We use stack canaries – Our stack is not executable  Nevertheless we offer bounties for memory corruption bugs

even if there is no POC code


Things to improve in the nearest future

 We have to add a fuzzer to out test methods  A good static code analyzer is in the plans (in the past we

used one quite famous one but were disappointed)  More tests for the debuggers (since we support remote

debugging for all platforms, there are 24 possible combinations of local and remote computers)  More tests for IDA Python


Why is security hard? Because we are blind

 Making a watertight liquid recipient is hard for a blind person. He has to palpate it entirely to ensure there are no holes  If he misses a tiniest hole, the liquid will leak out  The same for us with security: we are essentially blind to

security holes and can see only the most obvious ones  This means that humans are hopelessly bad when it comes

to security, at least today  We need help, we need tools so we can see the light


Computers to rescue us

 Since us humans can not cope with the task, our only hope are computers  Computers can

– Test our software – Monitor its use – Prove its correctness – Serve as a seeing aid – Eventually computers would develop software


Testing

 Unfortunately it is impossible to test all cases  Not an excuse to abandon testing altogether  Testing must be continuous  Test as many different aspects as possible  Think as an attacker, try to foresee the possible scenarios  Keep adding tests for all newly discovered bugs  Write a test case before fixing the bug


Monitoring systems

 The digital world changes over time  New threats and attack vectors are discovered  We must monitor our systems  Many solutions exist: Tripwire, Nessus, OSSEC, …  Simple custom scripts have the advantage of being

unknown to the attackers


Proving the software correctness

 Software verification tools  Need support from the programming languages

– C++ is not the best language for verification – If not, at least good coding practices must be adopted – Unfortunately this comes with a price to pay (MISRA et other guidelines)  Code generation tools  Eventually these tools will become mainstream


Thank you!

Thank you for your attention! Questions?

Presentations & Public Speaking

CODE BLUE 2014 : [Keynote] IDA and digital security by Ilfak Guilfanov