Rechkov. Lomonosov Report

Introduction Assembler as a native language Anomalies detection

Detecting abnormal executable files usingbinary code mining

Rechkov Anton

TU Berlin Germany & TTI SFU Russia

21th March 2012

Rechkov Anton Lomonosov Scholarship Report 21th March 2012 1 / 31


Malware evolution

CipheredEncrypted malware code of viruses

OligomorphicGeneration of a decryptor by randomly selecting each piece of the decryptorfrom several predefined alternatives.

PolymorphicGeneration of a sample by encypting malware body and modifying decryptoreach replication

MetamorphicReprograming all virus body by some obfuscation engine.



Modern detection technique

Signature analysisSearching a determine pattern in code.

EmulationUnpacking and analysis through the emulation of malware code and continuesignature analysis.

Behavioral analysisAnalysis of functions graph flow.



Code modification

ObfuscationTransformation of executable program code which preserves functionality, butcomplicates the analysis and understanding algorithms.

DeobfuscationResolving irrelevant code by

Algebraic models

Formal grammars



Code modification

ObfuscationTransformation of executable program code which preserves functionality, butcomplicates the analysis and understanding algorithms.

DeobfuscationResolving irrelevant code by

Algebraic models

Formal grammars



Outline

1 Assembler as a native languageBinary code miningNative language processingStochastic models

2 Anomalies detection



Binary code mining

Table of Contents


2 Anomalies detectionPreparationCode generator lexemesAnomalies detection by neural networksAnomalies detection by probability model



Binary code mining

Structure of compiler

Code generator engine:Machine code generator,Optimizers:

interproceduraloptimization (IPO),profile-guidedoptimization (PGO),high-level optimizations

Mutation code generator /obfuscator.

Common compiler scheme



Binary code mining

Common Code generator features

high-level optimizations

Unique intermediate language

Preoptimizing in intermediate representation

Code generation

Code templates from Intermediate to Target

Number of used instruction types

Machine dependent optimizer

Instructions cost



Binary code mining





Code generation




Instructions cost



Binary code mining





Code generation




Instructions cost



Binary code mining

Approving theory

Experiment

Determine instruction sequences

Compile source code with compilers

Compare distributions

Compilers

⇒ MSVC

⇒ LLVM

⇒ GCC

⇒ Intel C++ Compiler



Binary code mining

Approving theory

Experiment

Determine instruction sequences

Compile source code with compilers

Compare distributions

Compilers

⇒ MSVC

⇒ LLVM

⇒ GCC

⇒ Intel C++ Compiler



Binary code mining

XTEA distribution test

Frequency of words in binary.

(a) LLVM (b) MSVC

(c) Intel C++ (d) GCC



Binary code mining

Optimize binary’s mean distribution



Native language processing

Table of Contents





Native language processing

Text Mining

Language detection

Author detection

Text Classification

Document clustering



Stochastic models

Table of Contents





Stochastic models

Neural networks

Advantages

+ effectively with small number of training vectors

+ assessment of all samples proximity

Disadvantages

- predetermining model

manual words definitionmanual excessive elements analysisreeducation limitations



Stochastic models

Probability model

Advantages

+ self-sufficient word definition

+ education only by positive vectors

+ education unification(flexible reeducation)

Disadvantages

- big sample set for education

- errors while distribution determination

- computational complexity



Outline

1 Assembler as a native language




Preparation

Table of Contents





Preparation

Collect statistics samples

Python

Detection list of max repeated sequences

Disassembling

Searching strings

MatlabStochastic models



Preparation


Python


Disassembling

Searching strings




Preparation


Python


Disassembling

Searching strings




Code generator lexemes

Table of Contents






From disassembling to lexemes

Lexem3 to 6 instruction length sequences

ignore unknown bytes

maximum repeated sequences




Lexemes analysis

Suffix tree:Economy memory,String searching faster then O(N2),Fast assessment of maximumrepeats in strings

Suffix Tree example



Anomalies detection by neural networks

Table of Contents






Radial basis networks

no need to choose the number ofhidden layerslack of the pathology convergencefast convergence through acombination of learning algorithms.

Neural net architecture




Detection compilers

Compiler detection testing



Anomalies detection by probability model

Table of Contents






Multivariate Gamma

Using a set of bi- and 3-variateGamma:

Suggest GammadistributionSample proximityFast education

Empirical and theoretical PDFof element

−0.02 0 0.02 0.04 0.06 0.08 0.1 0.120

5

10

15

20

25

30

35

40

X

PD

F

Gamma PDF

Empirical PDF




Probability model testing

Error graphs of compiler probabilities based on coefficient ofminimal value P i

p = P imin ∗ 10coef

0 1 2 3 4 5 6 7 8 9 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

coeff for min value

err

or

false positive GCC O0

false negative Clang

false negative Intel

false negative GCC O2

false negative MS

0 2 4 6 8 10 12 14 16 18 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

coeff for min value

err

or

false positive MS

false negative LLVM




Probability model testing

Problem of existing zero elements

0 1 2 3 4 5 6 7 8 9 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

coeff for min value

err

or





false negative MS

0 1 2 3 4 5 6 7 8 9 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

coeff for min value

err

or





false negative MS




Conclusion

Proposed connection between native language andassemblerDeveloped algorithms of lexical assembler languageanalyzesDeveloped experimental stochastic models:

Based on neural networksBased on probability model

Realized lexical assembler language analysis.Approximate false positive errors of compiler detection:

27%10-15%




Questions?


Education

Rechkov. Lomonosov Report