Design and develop a fuzzer framework to security test a

Miskolci Egyetem

Gépészmérnöki és Informatikai Kar

Általános Informatikai Tanszék

Design and develop a fuzzer framework to security test a Cloud based application

Balázs Attila

CVXTCT

1164 Budapest, Bányász utca 11.


I

Table of Contents 1. Preface ...................................................................................................................... 1

1.1 The Company .................................................................................................................. 1

1.3 Explanation of topic ......................................................................................................... 2

1.4 Task description .............................................................................................................. 3

2. The importance of the testing in the development process ........................................ 5

2.1 Software quality assurance and its impact on security .................................................... 5

2.2 The Agile software development ..................................................................................... 5 2.2.1 Scrum methodology ...................................................................................................................... 6 2.2.2 Definition of “Done” ..................................................................................................................... 7

3. Cloud based web-applications .................................................................................... 8

3.1 Well architected framework ............................................................................................ 8

3.2 Abstraction of a distributed system ................................................................................. 9 3.2.1 Communication between components ......................................................................................... 9

3.3 Definition of Cloud ........................................................................................................ 10

4. Testing in practice, introduction of test methods ...................................................... 11

4.1 Testing the functionality ................................................................................................ 11 4.1.1 Unit test ........................................................................................................................................ 11 4.1.2 Component test ........................................................................................................................... 12 4.1.3 Smoke test .................................................................................................................................... 12 4.1.4 Functional test ............................................................................................................................. 12 4.1.5 Mutation based testing ............................................................................................................... 13 4.1.6 System stress test ......................................................................................................................... 13 4.1.7 Code coverage ............................................................................................................................. 14

4.2 Nonfunctional testing ..................................................................................................... 14 4.2.1 Security tests ................................................................................................................................ 14 4.2.2 Static code analysis ..................................................................................................................... 15

5. Fuzz testing ............................................................................................................. 16

5.1 fuzzer types, methods, features ...................................................................................... 16

6. Research a fuzzer tool .............................................................................................. 19

6.1 Considerations for selecting a fuzzer ............................................................................. 19 6.2.1 Kitty / Katnip fuzzer tool ............................................................................................................ 20 6.2.2 Sulley fuzzer tool .......................................................................................................................... 20 6.2.3 Peach fuzzer tool ......................................................................................................................... 21 6.2.4 Burp as a possible fuzzer ............................................................................................................ 22 6.2.5 American Fuzzy Loop ................................................................................................................. 22 6.2.6 Protocol learning, simulation and stateful fuzzer ..................................................................... 22

6.3 The chosen fuzzer .......................................................................................................... 23

7. Technologies involved .............................................................................................. 25

7.1 Resources in use at Amazon Web Services .................................................................... 25 7.1.1 Amazon Elastic Compute Cloud (EC2) – virtual machines .................................................... 25


II

7.1.2 Networking in nutshell at AWS ................................................................................................. 26 7.1.3 Serverless technology .................................................................................................................. 26 7.1.4 Databases, RDS, DynamoDB ........................................................................................................ 27 7.1.5 Simple Storage Service ................................................................................................................. 27 7.1.6 How to control AWS resources .................................................................................................... 28

7.2 Development environment ............................................................................................. 28 7.2.1 The programming language ......................................................................................................... 28 7.2.3 Project dependencies ................................................................................................................... 29 7.2.4 Integrated Development Environment ....................................................................................... 29

7.3. Further technologies involved ........................................................................................ 30 7.3.1 Docker container .......................................................................................................................... 30 7.3.2 Redis in-memory database .......................................................................................................... 31 7.3.3 Google Protocol Buffer ................................................................................................................ 32

8. Software specification ............................................................................................. 34 8.1 Functional requirements ................................................................................................................. 34 8.2 Nonfunctional requirements .......................................................................................................... 35

9. Software design ....................................................................................................... 36

9.1 Software architecture ..................................................................................................... 36 9.1.1 Kitty core ...................................................................................................................................... 38 9.1.2 Katnip component ........................................................................................................................ 39 9.1.3 Framework core and the config parser ....................................................................................... 40 9.1.4 Protobuf parser ............................................................................................................................ 41 9.1.5 Protobuf message validator ......................................................................................................... 45 9.1.6 Protobuf target ............................................................................................................................. 47 9.1.7 Protobuf controller ...................................................................................................................... 47 9.1.8 Protobuf runner ........................................................................................................................... 48 9.1.9 HTTP target ................................................................................................................................... 48 9.1.10 HTTP controller .......................................................................................................................... 49 9.1.11 HTTP runner ............................................................................................................................... 49 9.1.12 DNS target .................................................................................................................................. 50 9.1.13 DNS controller ............................................................................................................................ 53 9.1.14 DNS runner ................................................................................................................................. 53 9.1.15 Redis component ....................................................................................................................... 53 9.1.16 Reporter component .................................................................................................................. 54 9.1.17 Utilities component ................................................................................................................... 54 9.1.18 Runner ........................................................................................................................................ 55

10. Installation and integration into CI system ............................................................. 56

10.1 The executor environment ............................................................................................ 56

10.3 Install the application ................................................................................................... 56

10.4 Integration into Continuous Integration System ............................................................ 57

10.5 Results after the first runs ............................................................................................. 59

11. User Guide to operate ............................................................................................ 60

11.1 Configuration ................................................................................................................ 60

12. Further development possibilities .......................................................................... 63

13. The future: increase app resilience with symbolic execution. .................................. 64

13. Summary ............................................................................................................... 65


III

14. Összegzés ............................................................................................................... 66

15. Bibliography .......................................................................................................... 67

16. Printed appendix .................................................................................................... 69

16.1 Appendix 1. .................................................................................................................. 69

16.1 Appendix 2. .................................................................................................................. 70

16.2 Appendix 3. .................................................................................................................. 71

16.4 Appendix 4. .................................................................................................................. 72


1

1. Preface Public facing internet services could be much more vulnerable and could face more

suspicious activity rather than private ones. Because of the properties of public service, we

should consider and keep in mind specific factors. One of those factors is the enormous

amount of load which needs to be served in a strict time window. It occurs in a smooth, easy

to handle way sometimes peak growing as fast as it is a virtue. That is, the nonfunctional

requirements are getting more emphasis on functional requirements. This means tighter

restrictions. This approach needs a higher level of quality insurance.

My employer is the Sophos Ltd., which is a worldwide IT security company. I work for them

at the Budapest engineering site as a software engineer. My basic tasks are related to DevOps

methodology.

It was evident that I process a theme which helps the quality and trough of our product as

well as my skills too. It aids my team to manage higher quality.

1.1 The Company Sophos was founded by Jan Hriska and Peter Lammer. The main profile is the cybersecurity,

which embodied in several software products and devices such as antivirus software,

firewalls, hardware-based infrastructure protection like intrusion detection platforms, routers

and so on. Complete hardware and software-based security system against cyber criminals

designed for small and mid-size companies across the Planet. Sophos Central is a centralized

solution for companies to manage their entities across their company network. Abingdon is

the flagship site, the basis of the Company, which also has got Sophos Labs.

It has several other sites around the Globe (Vancouver, Ahmedabad, Bangalore, Sidney,

etc.). The Budapest engineering site has a Sophos Labs where the threat researchers are

seeking after threats by reversing compiled code using special tools to develop the AV

software effectiveness with new researches.


2

1.3 Explanation of topic To achieve the highest possible standard of the functionality defined in the software-

specification and to assure the operation safety, the testing is a mandatory and fundamental

process. Those tests results might reveal high- or low-level software faults arise from wrong

design patterns or in from the point of low-level view failures might arise from wrong

program code, which possibly could be a syntax or semantic fault which was not covered

during the process of development. A programmatical error, which is not covered in time,

could lead to runtime exception, which could cause harmful program and system state or

behavior and might open backdoors for criminals or other threats. This could lead to

endangered software and system security. So, testing can help systems avoid becoming a

victim.

During the thesis, I have been introducing several testing methods. Most of them are high-

level views or abstractions. One of them will be the Fuzz testing, which I will be autopsy in

deeper. In nutshell, some words about the Fuzz testing are indispensable in an introduction.

So, the essence of this type of testing method is to flood the system with dummy data (or

mutation of valid request data) it did not prepare, in a way that the hidden failures get into

the field of view.

A possible question could be “Do we need and why we need another fuzzer software because

solutions are all over the internet which is already developed by open source society.”. The

answer is the unique solution we use, for instance Protobuf over HTTP(s) which so far no

one has implemented (according to my research) in a fuzzer yet, or the DNS protocol we use

interpreted differently, or a fuzzer framework which is easy to configure no matter how

complex the protocol is. So, teams can apply it easily. fuzzer solutions on the market mostly

inflexible to use them in an agile environment or harsh to configure and manage them.

Another important aspect is the Continuous Integration and Continuous Delivery,

Continuous Testing methods and their tools used by teams over the Company. The easy to

set up and secure integration is a must-have attribution of any tool. Of course, a fuzzer

framework must have these attributes too. Teams must be able to use it with low cost and be

able to integrate into CI/CD processes.


3

Accordingly, I decided to search an opensource fuzzer project, which is suitable to build on

a frame. The tilt factor is the architecture of the fuzzer which allows me to implement the

framework and the protocols we want to fuzz. The final goal is a well-equipped fuzzer

framework integrated into our automation processes.

1.4 Task description The objective is to design and implement a fuzzer framework to security test a Cloud-based

application. To understand the task, it is necessary to give some abstraction about the system

I want to test. Imagine an endpoint with antivirus software installed on it. In the past, usually,

it grabs the current threat definition database from a server. Nowadays, users expecting fast

answers and fast resolutions, and of course, they make more and more load on the network

as time goes on. If a client running on an endpoint needs information about a suspicious

object (Potential suspicious or threat object could be a website, an email or it’s attachment,

an application, domain and so on.) it will start a lookup in the direction of a cloud service to

ask whether the suspect is a known threat or not. The server in the Cloud have the ability to

query the most maintained database. The service in the Cloud is a part of a distributed system

and has a lot of interface in conglomeration with several Cloud components. I want to

security test that distributed system with the fuzzer framework.

I have the main task, which is describe the above items, I divide it into smaller ones to handle

them quickly and will be introducing them in this document from step by step. Those steps

are the next ones as they appear in logical sort order. The first task is to set up the

requirements against the fuzzer. Some of those requirements are that the fuzzer implements

the Object-Oriented Programming; it is robust, modular or component-based. It has got a

well-designed and easy to understand easy to automate template mechanism to create the

mutations of the data. It is template-based, and its template mechanism is not stiff. It has

already implemented network protocols and or can extend them with new ones, or there is a

possibility to override and or mix them.

The second task is to research the opensource market to grade the reachable fuzzer solutions

according to the requirements that have been set up. After I make the sort, I will be describing

all of them and make a decision, which I will be making a detailed description to prove my

decision.


4

In the implementation task I will be designing and implementing a fuzzer framework based

on the chosen fuzzer, which will have a configuration file to parse the system set up

according to what the user wants to Fuzz, for instance, DNS, Protobuf, HTTP based systems.

So, there will be three main components of the framework. I will be making a high-level and

low-level design and implement the method according to the designs. The integration tasks

are about creating documentation, which is about how to handle, maintain, debug, and

integrate the framework. To incorporate the structure into the CI system will be another task

of the post work. The latest job is about to describe further developing possibilities.


5

2. The importance of the testing in the development process The second chapter is about the software testing which importance is unquestionable. Still,

it is worth examining some aspects such as tools, methods which support the impact of

testing on security and quality.

2.1 Software quality assurance and its impact on security The two terms described in the title go hand in hand, and they have an impact on each other.

Obviously, under a certain quality threshold, there is no security and no functionality. Fact

that the quality has an impact on the safety and functionality.

The software quality assurance can interpret as the internal and external features of the

software. The external quality is defined based on the functional software specification and

which software performs in real-life scenario and how useful it is for users or services which

they use. For instance, the software could be called over its API; in this aspect, the

functionality means that the software responses and behaves like that as the API

documentation describes it. The internal quality is the code quality. No matter how proper

the functionality by the specification is. If we do so, it with a messy codebase, that means,

in one hand, that the code is hard to maintain, and on the other hand, it is hiding

vulnerabilities, failures, security flaws.

The quality assurance aims to evaluate the code through testing and compel the standards to

achieve software security.

2.2 The Agile software development The agile method is one of the most powerful tools for teams to handle everyday challenges

in the developing process. The toolset provided by the agile approach is the basis of the

quality workflow.

Agile meaning: „Agility is the ability to both create and respond to change in order to profit

in a turbulent business environment.”


6

In practice, it provides a framework for developer teams to assure the synchrony with their

actual tasks to each other. In small atomic pieces, they work and do iterations from sprint to

sprint from day to day to make designs, solve problems, implement solutions. [1]

2.2.1 Scrum methodology If the agile methodology is the framework, then the Scrum holds the toolset to manage kinds

of stuff. Definition of Scrum: Scrum is a type of toolset which allows teams to assign

complex tasks to its members to solve them with creativity and in high quality and value.

The roles are the Product Owner, who is in charge of the value of the product.

Development team, members who implement the logic. Scrum Master's part is to manage

the Scrum methodology.

The most significant part of the Scrum methodology is the Project, the next level is

the Sprint. Usually, a sprint is a 2-4 week long period. During the Sprint, tasks are escalated

to developers. A developer can hold one to more tasks per Sprint, depending on the task's

complexity. Sprints started with Sprint meetings during its planning. The responsibilities are

escalated to developers. The type of the task is depending on the state of the Project, it can

be a design task, or an implementation one, or a bug fix and so on.

Each task has a technical complexity, which is usually translated into Story Points; these

points determine the degree of difficulty of the task currently under in definition. Teams

Scrum policy mostly decides the Story Pointing rule, which means if a job has voted as a

high value of Story Point, it could be better to break the task smaller apart. Voting, teams

vote each task Story Point during the estimation of the task, which could be the part of a

Sprint meeting. Estimated duties are included in the Scrum supporter system.

During the Sprint teams holds a daily ritual called Stand Up. Stand Up about to report to the

Scrum Master and to each other about the things they done previously, and they further plan

to the upcoming day. If someone has a blocking factor, the Scrum master try to resolve and

involve necessary resources to manage the problem. To achieve easily the aims with the

Scrum there are several software tools available for teams which are really effective. These

software tools can interact and communicate with each other. For instance, the ticketing

system can open up a fresh new branch in the code repository handler system to support the

developer to manage the administrative stuff that comes with the start of the task. The CI

system also has the ability to put messages on a form of commit for the repository system to


7

indicate whether the code developed on a specific branch is built successfully or not. The CI

and repository system can insert messages about the building process into the ticket handler

system. Known statuses that we developed for us during a task are the “in progress,” “in

review,” “in test,” “completed”. In the state of “in review,” members judge each other work

to verify its legitimacy for company policy or professional consideration.

It’s evident that the process ensures the ability to handle the Project and provide the ability

to interevent any time if it is necessary. The product is made on small iterations and can

adapt new solutions if the needs are changing over time. Team members getting involved in

every aspect of the Project from designing to implementing no matter what component they

will work during the creation. This last approach has been also developed by us.

The Project closing in general followed by a Retrospective, where the team members can

discuss they comment about the past project, those comments can be negative or positive

ones regarding the process, how they feel things could be done better. [2][3]

2.2.2 Definition of “Done” A simple, useful list about how to start a task and how to reach success on it. It could depend

on how the company restructures it through its policy, but mainly it holds the following

elements in our case:

• Task understood;

• Feature implemented;

• Unit tests are passing;

• Build/code is evaluated by the CI system;

• Code is evaluated by tests in the CI system;

• Well documented;

• Code approved by at least two team members, merge into master branch;

• Ticket closed;


8

3. Cloud based web-applications Modern software architectures, designs, and implementations shape the market through the

demand they generate. Server providers try to pick up the pace to adjust their supply to fit it

to the new demand. The Continuous Delivery and Integration and the Agile software

development as well as the economics and society’s multiplicators, the demand for the

novelty generates the evolution from the view of demand and supply on the market of the

server providers. This is a powerful breeding ground for technology to evolve. The field of

on promise systems slowly narrows. If a company wants to participate in the competition, it

will do better to migrate the infrastructure into a highly available, scalable cloud basis. The

best technology to achieve this aim is Cloud technology.

3.1 Well architected framework The AWS (Amazon Web Services) allows everyone to access their documentation about how

to design a well-architected framework if a reliable Cloud infrastructure is a need.

It defines the fundamentals of a Cloud-based infrastructure. One of the main pillars is the

infrastructure as code which is a principle about how sysadmins and developers should

handle the infrastructure. They recommend that every piece of the infrastructure must be

written in code, and every simple modification on it must be through code. The infrastructure

code has to be version controlled. To achieve this principle, there is a powerful tool called

CloudFormation, which is a type of language developed by AWS to provide the ability for

developers to write down the infrastructure as code. Another recommendation is to document

the code to make it more understandable for others. Worth noting here too the small extent

of steps to take when planning to make any modification on the infrastructure. Rollback is

easier. [4]

Why do these recommendations are so important when we talk about a well architected

framework? Because these recommendations are providing the basis and a design pattern to

achieve a trustworthy Cloud infrastructure that is maintainable in any cases.


9

3.2 Abstraction of a distributed system In IBM wording, the distributed system is a bunch of independent computers deployed with

independent software-components on them, which seems for the outsiders as one system.

Machines and components are interacting with each other on the network they operate. [5]

Distributed system’s greatest advantage is scalability, redundancy, reliability. Cloud systems

mostly scale in the vertical direction. Vertical scalability is an approach to involve resources

if load rise. For instance, a system stands about ten virtual machines which serve the clients,

horizontal scalability means that if peak arises up, the system utilizes more instances to

maintain the service and bear the load. Before the Cloud technology, engineers used to

design vertical scalability, which means using greater virtual machines or bare metals instead

of small ones to maintain the service under higher load. [6]

Redundancy is an option provided by Cloud services, of course not mandatory but highly

recommended to use in the design. In practice, AWS provides “MultiAZ” which is a multiple

availability zones inside a region. Mostly every region has a minimum of 3 “AZ”, but there

are zones with much more “AZ” equipped. The architect’s responsibility is whether to

implement them or not. Another feature is the recoverability and reliability. AWS has got a

term called “shared responsibility” which means that they are in charge of the underlying

infrastructure, mostly the border is the hypervisor if thinking in virtual machine terminology,

from the networking view this border is the Virtual Private Cloud internet gateway. [7]

It is a general problem for developers to solve and provide communication between services

and maintain data persistency across the Cloud. Service discovery protocol is to manage the

communication and interaction between components. It could be a Load Balancer, or/and a

bunch of DNS record zone to be able to route the appropriate traffic from producer to

consumer. Databases and transaction systems, in-memory databases are technologies whit

developers able to maintain data consistency. The messaging system is in charge of manage

to handle the message traffic between services acting like an Inter-Process Communication

on the network.

3.2.1 Communication between components An interesting fact that the first service developed by Amazon was the SQS. This highlights

that the most demanded Cloud service is the communication between services. The Simple


10

Queue Service is a message queuing service that enable developers to send, store and receive

messages across distributed systems and let them to focus on the workload. [8]

Components can send SQS messages which act as a pipe. On one side, the producer puts the

message on it, and on the other side, the consumer removes it. In steps:

1. X component puts its message into the message provider, at this time a counter starts

which measure the elapsed time;

2. Y component removes the message from the queue and reads its content. Another

counter starts to maintain the elapsed time from the reading;

3. Y component process the message then removes it permanently from the queue;

a. There is a Dead Letter Queue implemented on every SQS to store the

messages which is defined in the cloud configuration. The default value of

the amount of message retention is 14 days.

Developers can define the processing order logic according to the order of the incoming

messages, which could be for instance, First in First Out model. The whole SQS has got an

API and SDK toolset to develop feature-rich applications.

3.3 Definition of Cloud On-premise systems are mostly owned by companies to serve their services all over the

world. If a company decided to step into a new market on the Globe or the demand starts to

grow for their service, they should upgrade and scale they datacenters or should build a new

one, which has got a huge impact on their costs. While sysadmins can keep in touch

physically with their infrastructure and hold the responsibility to operates it, then the Cloud

is provided by a third party who is responsible for all factors to assure the operation. The

Cloud is much more flexible than on-premise systems.


11

4. Testing in practice, introduction of test methods Heading to the implementation of fuzzer Framework, it is necessary to create an overview

of how testing looks like in practice and what type of methods available on the paleta of

testing.

4.1 Testing the functionality Functionality is why software birth, its aim, set of objectives, features. The function is the

co-ordination of interactor components, which are having objects communicating with each

other. How to behave an object is regulated by methods and functions. No matter which

software layer or component is the target of the inspection, those pieces must serve and

assure the functionality described in the specification. By the specification, the software

must output the expected response to the given input.

The smallest unit whereat a test can proceed is a function or method. The term came here,

unit testing. The unit test is explicit. As the process moves away from the lower levels to the

higher, the testing transforms into functional testing and be more implicit. [9]

Possible stages of preparing and running a test:

1. Declare functions, expectations;

2. Prepare input data set based on specification;

3. Define expected output according to specification;

4. Define test case(s), rather more case than less;

5. Run test case(s);

6. The compartment of the result and the expected result;

4.1.1 Unit test A unit test is about testing the building elements of a component which are classes, methods,

functions, interfaces.

Every attribution of a function is known by its API documentation or the document string,

generally known its parameter signature, their type, the return value, and its type. In practice,


12

a unit testing framework supports to manage the tests. Mostly a function handles and runs

the unit test of a target function called test case.:

def test_proto_path_config(self): proto_path = self.config.get_generic_proto_path() self.assertEqual(proto_path, '/framework/data/proto/')

In the Python unit test framework, an assertEqual method will compare the functions return

value with a given expected value if equal occurs, then the test passed. Of course, there is a

whole assertXY library for each unit test framework. It can test numeric, alphanumeric types,

booleans.

4.1.2 Component test

The component or module test is about to test an independent component itself. To make a

component test, it is necessary to know its interfaces and its use through the API

documentation. [10]

4.1.3 Smoke test During the CI/CD process, the smoke test provides the ability to immediately indicate if the

software functionality deviated because of the wrong code. It covers not the whole

functionality but the main ones and runs after the code compile in the Continuous Integration

system pipeline. If a code fails, the smoke test the CI pipeline must fail too, because of the

error that occurred in the code. [9]

4.1.4 Functional test The functional tests might run under the same circumstances as the production environment.

It must equal the production infrastructure on each level and aspect, which means if the

production environment is running in the Cloud, it is recommended to run those tests with

the same deployment. The intermediate solution could be a local environment equipped with

fake services which simulate the Cloud environment. Sometimes it takes more effort to

simulate locally.


13

The phase of functional testing the product must feature complete state. Or has features that

are currently under development and fast finished. By the API documentation, can a

functional test to design.

4.1.5 Mutation based testing With a specified testing tool, Quality Engineers can proceed a mutation-based test. For

instance, it mutates the operators and operands inside a decision branch. Its advantage is the

ability to evaluate the test cases written before because it will reveal conditions that were not

covered properly through the test case.

Assume that the code which is under in test has a method that passing every test case, and

the coverage report showing around 90% and everyone is happy with that. The next level of

Quality Assurance could be the question: Do we design the test case properly? The. Mutation

based tests will support to unmask the underlying logical failures.

function highTemp(temp) { if (temp > 1000) { return true; //too hot! } else { return false; //it’s a very nice… } }test(“test highTemp”) { expect(highTemp(500)).False(); expect(highTemp(1000)).False(); // this is the appropriate case } If the mutator starts to mutating the operators, there can be a state when it changes the > to

>=, but with the first case where the function got value 500 will not reveal the problem

because the case will always pass, in the second case the value is 1000 which can lead the

decision into problem if the mutator switch the operators to >= because the case will fail if

1000 >= 1000 with expectation of False result it will cause True. The conclusion is the

perfect parameter for this test case is the value on the level of 1000. [11]

4.1.6 System stress test This test is to measure the resistance of the system against a high load. It can reveal design

problems regarding component communication for instance locks, mutual exclusions,

database connectivity problems, worker process problems during high stress. Infrastructure


14

resistance information could extract too by stress test because revealing network-level

problems, file system issues. While the system is at the edge of the tolerance, latency

measurements, response time collection is also an objective. The reaction of Cloud services

such as Autoscaling according to CPU utilization or database connection or any composition

of them could be a basis of measurement.

4.1.7 Code coverage Code coverage is a tool to measure how perfectly is the coding unit tested. During the tests,

a third-party tool called mostly coverage maintains the process and measure which part of

the code was driven via the unit tests. After the test is succeed a coverage report will be

available to evaluate the degree of coverage. It depends on the Company’s policy where the

threshold line is defined that they could say the coverage is good enough. Coverage about

90% is a good one in Enterprise.

4.2 Nonfunctional testing Beyond the functional testing after convinced about the functionality, the glance is turning

to the system security. Security could be reached among others with the code quality, and a

nonstandard code can hold noxious calls, variables, memory management. Coding

conventions without caution of use could result unmaintained memory addresses. These and

many other faults could lead to code quality, which is presumably open for threats.

4.2.1 Security tests Penetration tests are trying to crack the system in no matter what way and does not care

about the weapon. It can proceed with system infrastructure and software as well. To

penetrate a system, the knowledge of network protocols is a must-have. Specialists and their

tools are known the methods of how to test a system. If a software system contains third-

party modules, it is indispensable to test them too. The objective from the operator’s side is

to maintain the running versions, upgrade them if new releases are released and patch them

in time.


15

4.2.2 Static code analysis Static code analysis is an approach of testing when the test evaluates the code with a toolset

that is made to recognize common errors, vulnerabilities, cognitive complexity, code

maintainability, nonstandard code usage, and another aspect. Some of these terms may need

an introduction, the first is the cognitive complexity, which is a way of measuring code

understandability, which occurs when the developer uses too many decision branches,

embedded scopes, iterations, which reduce the ability to understanding the code. [12] The

code smell means that how good the code is measured by a linter which is seeking for

nonstandard code usage and so on. One of these linters is Pylint, which will be in use during

the implementation by me. Another well-known is the Flake8 which is also a Python based

static code analyzer.

Most of these tools can be run locally by an IDE to get rid of failures as soon as possible,

but for enterprise projects, there are also solutions to automate them to doing analysis on a

huge amount of code lines. The maintainability means that how subtly divided the

component is. In practice, reusable code, small and clearly formulated methods, well divided

logic and structure can lead to a well-structured narrative which makes the code easy to

consume for humans.


16

5. Fuzz testing Fuzz testing is a nonfunctional test type mostly used to testing the system security, one of

the most powerful strategies for identifying security issues.

The fuzz (fəz) verb meaning is “make or become blurred or indistinct”. During a fuzz test,

the testing tool creates mutations of a known valid or a given input value and sends them in

the direction of the system through a specific channel. When fuzzing the system, the main

aim is to observe its response and behavior to explore hidden security failures or abnormal

program operation or system performance ruin.

The history of Fuzz testing started in the ‘50s, but the really first widespread method was

developed by Apple in 1988, which was called the Monkey test. The method based on the

theory if a monkey randomly starts hitting a computer’s keyboard after “t” time, there could

be a combination hits that possibly contains Shakespeare’s all works. [13]

The first Unix adaptation of the testing tool got the name Fuzz. It was able to create mutated

output into a file, then send that input for a specific target, which was mostly a CLI software.

For instance, if we grab the well-known “ls” CLI tool which has the default usage:

usage: ls [-@ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1%] [file ...]

With a fuzzer it is possible to fuzz its flags like “ls -%/+!+%”/=../../../ foo.bar” and so on as

it does not hit a state which causes an error in the software mechanism because of the lack

of exception handling or anything else.

As time passed it was used to test Graphical User Interfaces (GUI). In 2012 Google

introduced the Clusterfuzzer, which is a tool to testing Cloud based infrastructures. Earlier,

the AFL (American Fuzzy Loop) uses a generic algorithm. So, the Fuzz testing method is

based on that every software hides its failures, which are waiting just to explore them.

5.1 fuzzer types, methods, features

fuzzer types in terms of function and features:

• General or mutation-based: It creates its mutations from nothing by a given

specification or mutates a basic dataset given by the tester or the combination of both

approaches.

• Protocol based: beyond the ability to randomize the data, it is able to use protocols

such as TCP, HTTP.


17

• The most important feature is how the fuzzer acts whit the system, which depends on

whether it knows and controls the inside structure and functionality. If we know its

inner structure, then call it whitebox fuzzing. If no information about the system, then

it called blackbox fuzzing. The tool I will implement to is a blackbox fuzzer, but we

will see it has the ability to be a whitebox one.

By the whitebox approach, the tool has information about the systems inside structure. It

knows its inside functionality it can influence the system, for instance, observing services,

gather their log data, restarting daemons if necessary. Because of more information it has it

is able to send sophisticated mutations for the system.

By the blackbox approach, the fuzzer takes no care about the structure of the system, the

only aim is to send as many mutations as it can and take observations. In order to fuzz an

application is it has to know the application programming interface.

Assume a system with a REST API over HTTP, which described by the API documentation

as it follows: “somedomain.com/foo/bar/addItem” with HTTP POST method clients can

send data in JSON format “{“item_id” : 1234}”, the item_id variable is an integer type

in the system according to the documentation but we have no information about whether it

is an unsigned 16 bit integer or a signed 32 bit integer and of course we have no information

about what happens if the client send for instance “-2343535345343” or

“2.2342424243234”. The question is, did the developers implemented a mechanism to

protect these variables? Indeed, basically, client has no idea what type of programming

language is being used on the server side. Fuzzer has to have the ability to bring to light

whether there is a mechanism implemented in the software to protect the variable or not. On

one hand, it could be but the other hand there could be a query which ruins the application,

which will initiate system exit in better case, the worst scenario could be an illegal SQL

query or a terminal access, of course not in this simple example but it could happen.

Steps of the Fuzz test:

• Identify and define the target system

• Define the communication protocol

• Define inputs according to the API documentation and system specification

• Generate data


18

• Run tests (Fuzz Data).

• Transfer cases to the target

• To observe the behavior of the system via its response or logs.

• Report problems if it occurs.


19

6. Research a fuzzer tool The open source society maintains several tools to run such tests on diverse programming

languages through standard protocols.

6.1 Considerations for selecting a fuzzer Before getting into the research, expectations against the fuzzer shall be defined.

Expectations based on the specification of the framework I will implement, which are from

the framework point of view, is the fuzzer must be platform-independent, provide a client –

server model, implements network protocols, implements an object-oriented approach

which allows building a framework around.

According to my research, basically, there are two main types of fuzzer out there according

to their use, the ones are out of the box solutions equipped with network protocols to test

network applications, or to fuzz desktop applications the second group is more plastic,

having the possibility of further development. Evident, that the first group is out of scope for

this project.

I have also identified features that are not yet important but would be in the future, for

instance, the ability to maintain a session between client and server. [14]

The requirements of the fuzzer summarized in the following table.

Modularity Each part of the fuzzer stands on its own, which means parts

are reusable. Extensibility If wanted to extend the fuzzer and would implement

something new, then it could be done without the touch of the core elements.

Rich data modeling It supports rich sets of datatypes at low and high level. Stateful Take care of the message order and able to maintain a session

while simulating the client or server role. Client–server model To have the ability to simulate server or client to fuzz client

or server application. Platform independent or cross-platform

Supports a language, which is platform-independent such as Python and the attribution of working in Linux environment.

Logging system Well architected logging module extended with a reporting tool.

Usability, maintainability

Open source fuzzer basis in Python language.

Table 1 fuzzer requirements


20

A further expectation is to be well documented and maintained actively. Because of the

Python 2.7 deprecation at the start of the year 2020, the Python 2.7 implementation is a strict

exclusion.

6.2.1 Kitty / Katnip fuzzer tool Kitty/Katnip is an open-source modular fuzzer, which was developed by Cisco. The core

module is the Kitty component, and the Katnip holds the modules implemented by society

such as file fuzzer, TCP fuzzer, UDP fuzzer. The aim of the develop of Kitty was to provide

a core component which is able to receive new type of targets and protocols.

According to its developers it is not an out of the box fuzzer. On its own, it cannot run fuzz

tests as long as the user does not set it up or implements its own solution. Kitty provides only

the mutation mechanism to creating data to send in the direction of the target, a template

system with a bunch of fields, monitoring system, and controller system to control the target

during a whitebox fuzzing. The components and classes are well divided; each functionality

border is easy to recognize and easy to build up a case. Great advantage is the Katnip, which

is another layer on top of Kitty to provide higher level solutions. [14]

Modularity Yes Extensibility Yes Rich data modeling Yes Stateful Yes Client-server model Yes Cross-platform Yes Logging system Yes Python3 Yes

Table 2 Features of Kitty fuzzer

6.2.2 Sulley fuzzer tool Sulley is a fuzzer engine and a framework in one package it is extendible, and protocol-based

also written in Python. Unfortunately, it is not maintained for 6 years ago by the readme at

the repository page. The document recommends using the BooFuzz tool, which is actively

maintained. Why I mention Sulley despite it is not maintained anymore is the reason that it

was the one of the first fuzzer, which succeeds. The BooFuzz also mention it and inform the

visitor about that the BooFuzz wanted to “fuzz everything” as the ancestor did. [15]

It seems that they actively develop it, but checking the structure, I was not convinced that it

is obvious and comfortable to extend it with such a framework and toolset I have planned.


21

In practice, for instance, the target component charges for too many objectives which they

should not be in the same class. Another functions like the logging component I found it too

complex for a fuzzing and another example is the Connection class, which constructor has

more than 20 arguments, which I found too many to handle it easily.

Modularity Yes Extensibility No Rich data modeling No Stateful No Client-server model Yes Cross-platform Yes Logging system Yes Python 3 Yes

Table 3 Features of Sulley fuzzer

6.2.3 Peach fuzzer tool Maintained by Mozilla Security, and there is no link to the repository or any download link.

Any try leads to a contact form and to marketing. That means the mainline of Peach fuzzer

is a commercial product. But it has a fork named MozPeach with the goal to continue to

deliver Peach as open-source project for the community. They only just have to work on

Python 3 support and replacing deprecated python code. According to this, it is not a

welcome solution to my framework. Even if it already supporting Python 3 because of its

structural simplicity, which not fits to the objective. Getting into the code, it is too stiff to

cooperate with another components it is a good light-weight solution in its own. It uses XML

to describe the data model and have the term “Pit” to define how the fuzzer obey with the

target. [16]

Modularity No Extensibility No Rich data modeling No Stateful Yes Client-server model Yes Cross-platform No Logging system No Python 3 -

Table 4 Features of Peach fuzzer

It does not make sense to inspect deeper because of deprecated Python code.


22

6.2.4 Burp as a possible fuzzer Basically, the BurpSuite is a special Linux distribution installed whit a bunch of security

tools to test systems protection. Researchers and security professionals use it with a

predilection to test their systems or simulate a threat or during the process of research

standing in the middle of the threat and its command server to tap the traffic and grab the

communication that they do between each other. This method called Man in The Middle.

Burp has a subsystem that can act as an HTTP server or client, can terminate TLS, and able

to send any type of HTTP method. It can mutate those requests or responses too. I just

wanted to mention because it is a renowned tool but except for HTTP it is not a good starting

point. [17]

6.2.5 American Fuzzy Loop Maintained by Google is a free software tool to assist the detection of software bugs. It has

significantly another approach as the current objective needs. The AFL mostly is to test

binary. To achieve this, it is necessary to compile the binary with a special tool. There is no

way to implement this approach on a network based fuzzer framework but worth to mention

it because of its importance on binary testing. [18]

6.2.6 Protocol learning, simulation and stateful fuzzer The PULSAR was so far the embarrassingly special solution with its sophisticated network

protocol learning capability. The tool is capable of modeling a protocol through machine

learning techniques such as clustering and hidden Markov models. Really modern and

innovative solution. The program requires the user to provide a Wireshark .pcap file to

analyze. Those types of files are from Wireshark program after running a network session,

which is taped. After the protocol has been analyzed by the tool, it wanted to try to figure

out its rules. Unfortunately, it has a lot of dependencies and really hard to deploy and

configure, it not to mention the resources that machine learning algorithms needs. The

further problem is that it uses the “R” framework for statistical computing instead of

implements the Machine Learning algorithms via a Python framework such as TensorFlow.

Another problem is how the model can carry out the analysis of encrypted traffic because

the Wireshark .pcap file has not provide information about TLS communication except the

handshake, but that is come after it will be hiding because of the specialty of TLS. It is clear


23

that the TLS handshake is a part of the protocol but there could be special headers and service

messages after the handshake which are mainly encrypted.

The presence of deprecated Python2.7 code is shortcut every question described above.

Modularity No Extensibility No, but learning mode Rich data modeling No, but learning mode Stateful Yes Client-server model Yes Cross-platform Yes, but hard to deploy and configure Logging system Yes Python3 No

Table 51 Features of Protocol learning fuzzer

6.3 The chosen fuzzer After reviewing of each fuzzers codebase and architecture the best solution to implement a

fuzzer skeleton is the Kitty/Katnip because of the structure and relationship of modules, the

robustness, and the approach of how they interact and how easy to instrument a basic fuzzing

with them. This is the only fuzzer from the above which has modules that can overwrite and

could connect with the framework I will design. During the initial tryouts it turned out that

its parts are designed to be able to maintain a sustainable and long-term project with potential

of further feature developments. Its structure allows to act on several level. It provides the

core fuzzer module, a sophisticated Graph Model which orchestrates the Template on the top

of a sophisticated templating module, which can also be influenced through a framework.

Figure 1 Relation Between Modules, Source: Kitty documentation

The templating model stands from fields that could be low-level binary values or high-level

types. It is possible to create targets or derived from existing ones depending on the protocol

we want to fuzz. The controller module gives the opportunity to maintain the target server

or services as we would start to run whitebox fuzzing. It has a web interface, which has a

graphical surface to follow the process but easily can reach its methods to automate the

testing. The monitor can observe the victim’s behavior through its network traffic or memory


24

consumption. The Kitty was refactored in 2019 to get rid of deprecated Python2.7 code and

keep the project alive. [19]


25

7. Technologies involved During the development, I use a diverse technological environment that can be divided into

two parts. The first one is related to the target services on which the tests are performed. This

includes Cloud technology and network protocols. The second group is related to concrete

development, such as programming language, integrated development environment. And

there are overlapping technologies as well because inevitable to implement network

protocols or some Cloud services to manage to perform the tests.

7.1 Resources in use at Amazon Web Services I will introduce Amazon Web Services (AWS) resources that are being used related to the

objective.

There are services used by my employer with protocols, which are the DNS, HTTP, and

Protobuf over HTTP(s). These are basically held by EC2 instances, Lambdas, and Elastic

Container Service in the Cloud. During the development, I have to simulate these services

to support the development, and for this reason I have to create my mock services. Some of

them run at localhost and others in the Cloud. It was important to simulate to have all

properly implemented protocols prior to integration. I will introduce such technologies that

accessories such as API Gateways, CloudFormation, DynamoDB and so on.

7.1.1 Amazon Elastic Compute Cloud (EC2) – virtual machines The Amazon Elastic Compute Cloud service is a virtual machine service to utilize computing

resources to solve objectives depending on what the task is, which could be a simple web

application server or a machine learning optimized instance among others. AWS offers a

wide variety of operating systems including Linux distributions. These EC2 machines are

the same ones as we were used to VMware. They have launch templates with definition

whom to utilize them. The autoscaling group with scaling policies can regulate the number

of instances in operation. Several types of Load Balancer are available to balance the load

between virtual machines depending on the need of the application and its protocol. In that

system that is being tested, there are bunch of machines with different roles regulated by the

autoscaling group. Tests include internal and external communication.


26

7.1.2 Networking in nutshell at AWS The basic building block of an AWS network is the Private Virtual Cloud (VPC) for which

the AWS is responsible, and this is true all the services provided by AWS. It is up to the

engineer and of course, the market they would serve to that into which region they deploy

the application. AWS provides several regions around the Globe; regions have Availability

Zones (AZ), mostly three AZ per region. It is recommended to take advantage to utilizing all

AZ to achieve high availability. The AWS also provides subnets per AZ, Load Balancers and

gateways are available too. As we have become accustomed to in on-premise systems, it is

possible to form network security with ingress and egress roles, access control lists via

security groups. API Gateways are for applications, which are implementing REST. It

supports containerized or serverless workload and EC2 as well. It handles all the incoming

requests and transfers it after authentication to the appropriate resource. Route53 provides

the domain naming system across the Cloud in cooperation with CloudFront, which is a

service on the edge of the Cloud and its responsibility is to transfer traffic to the appropriate

endpoint. [20]

7.1.3 Serverless technology It is a possible scenario that one of the development teams will perform a fuzz test on

Lambda based service, but the main reason is why I deployed one is to have a system with

HTTP(s) with REST API to try out the fuzzer.

The deployed Lambdas per REST function are provided by AWS via a preconfigured

CloudFormation template, which is a service-specific language to deploy and maintain

infrastructure as code written in JSON or YAML format. Worth to note that is available with

Figure 2 Lambdas per REST function, Source: Aws account


27

the CDK in Python, which is a more developer-friendly way to handle infrastructure as code.

But after all, the Lambda is a serverless technology where developers don’t need to maintain

infrastructure, the underlying system is provided by AWS. Our only responsibility is to write

the handlers. They utilize just the case when a trigger comes from a direction, which could

be a Cloud Watch event, a Simple Notification Service notification, or a Simple Queue

Service message. After the trigger triggers the Lambda function, it does its task then go back

to the idle state. There is no networking and no server with Operating System.

In this task, I use them with API Gateway which was also a part of the sample template

provided by AWS. That configuration allows me to run tryouts with the fuzzer framework.

It is allowed to perform any test on an account and AWS infrastructure what we own

according to AWS policy. Obvious was in no way flooded.

7.1.4 Databases, RDS, DynamoDB There is a way to set up and scale relational databases or non-relational databases in the

Cloud, depending on the data structure designed by architects. RDS is mostly server-based

and supports MySQL, Postgre, Oracle, MariaDB. DynamoDB is a key-value store and

supports the document database as well. Both of them support replication between zones.

The Lambdas that have been deployed put the data they process into Dynamo tables (Figure

3.). [21][22]

7.1.5 Simple Storage Service To maintain objects in the Cloud ,AWS developed the service called Simple Storage Service

(S3), and regardless to the other services, it is region independent because of every S3 bucket

name is basically a domain name. Buckets are the storage where applications can upload

objects. It maintains object retention policy, access control, and versioning of objects is an

Figure 3 Lambda Table items, Source: Aws account


28

advantage too. The maximum size of an object to upload in one session is 4 Terra Bytes.

[23]

The fuzzer framework system use this service to archive fuzz session data (Figure 4.).

7.1.6 How to control AWS resources The AWS resource control use cases need to be distinguished depending on the goal. A

resource can be creating or modify via the web console, and it is recommended to do massive

deployment with CloudFormation which can perform with CLI tool. There could be cases

when automating something than in a shell script more comfortable to use aws-cli tool. AWS

SDK and Boto library giving the ability to control resources in a programmatic way. In this

context, the term control could be meaning a deployment or a resource configuration or

interact with resources inside an application. During the work, I use aws-cli which has been

configured to the Frankfurt region (eu-central-1) with access key and secret key, to control

S3 bucket creation and object upload inside from Python script it is managed by Boto 3

library. [24]

7.2 Development environment This subchapter is about what type of developer tools has been involved during the

development. It describes the programming language and other related technologies such as

IDE, data-interchange formats, libraries, code analyzers, virtual environments.

7.2.1 The programming language To develop the framework, it was clear to use Python3.7 because it is an industrial standard

at my company, and it was a restriction in the specification to use Python. There is no extra

expectation about the speed, so it is not necessary to implement, for instance, in C++. The

Python community is big enough to find well described tutorials and necessary module

Figure 4 Objects listed in S3 bucket, Source: Aws account


29

documentation if needed. Of course, this was the language in what I already have small

experience and have the willingness to extend my knowledge.

7.2.3 Project dependencies Python dependencies can be managed by several tools. The most common is the PIP, which

I also use during the project. The dependency list is maintained by the requirements.txt. The

method to create one is a really basic step, just run the command:

“pip freeze > requirements.txt”

– then it creates the dependency list with versions.

7.2.4 Integrated Development Environment The environment I was set up to work is containing the following. The Operating System is

OSX 10.15.2 x86_64-apple-darwin19.0 kernel with shell Zsh 5.7.1. For code editing, I prefer

PyCharm. Python version is 3.7.1 wrapped into a virtual environment. For running unit tests,

I use TOX ,which is a virtual environment builder, and it is able to bootstrap as many Python

environments as configured to evaluate the code on several environments. To analyze code,

I installed Pylint and Flake8 into the virtualenv an as well into the PyCharm to get rid of

most of the code lint errors during the coding to avoid pushing wrong quality code into the

repository system. To double-check the code, I prefer SonarQube (Figure 5.) which I run in

a Docker container on loclahost and periodically performing scans on project level.

Figure 5 SonarQube code quality measurements about Fuzzer Framework


30

Python coverage tool is periodically checking how perfect is the code coverage by unit tests.

An excerpt of the current coverage can see at Appendix 1. For version controlling on

localhost there is a git client, which has remote endpoint at GitHub.

I prefer using git branching, so I have a branching model for this task which is not so

sophisticated but conductive. There is a master branch, which I maintain and do not allowed

to push code straight into the master branch, this is for avoiding accidental code push into

master. Each component has its own branch, and after the feature is completed then it is

available for merging back into master.

Project packages handled by pip and dependencies are under the project root in

requirements.txt file. Because of the Kitty concept, it is no need to touch the Kitty core code

however the Katnip module necessary to deploy in developer mode using the “-e” (editable)

flag while installing it with the PIP packaging tool. It allows me to edit and extend its

components locally.

7.3. Further technologies involved Some specific services I use to manage it with Docker as I mentioned earlier. Docker is a

containerization mechanism to run processes independently on the operating system. It was

designed to run microservices. It is a great advantage that it is able to run any service under

a time of an average git clone or any download consume. The image can be pulled from the

docker repository than deploy itself into the container.

7.3.1 Docker container In practice the Redis container deployment described by a docker-compose.yml:

version: '2' services: redis: image: 'bitnami/redis:5.0' environment: # ALLOW_EMPTY_PASSWORD is recommended only for development. - ALLOW_EMPTY_PASSWORD=yes # REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL ports: - '6379:6379' volumes:


31

- 'redis_data:/bitnami/redis/data' volumes: redis_data: driver: local%

Then the only thing is to do is to run the command:

docker-compose up -d

After this step, the deployment starts it downloads the Redis docker images from the

maintainer’s repository, then start the container.

7.3.2 Redis in-memory database

The Redis is an in-memory database and message broker that is designed to high-

performance data storing. In this project, I use it to combine the Python basic data structures

with it for experimental purposes. Its key/value approach is similar to Python’s dictionary

idea.

In this application the HASH type was implemented to put the Protobuf API module name

into memory. This information gathered from the pb2_api Protobuf descriptor Python

object.

HASH types are key/value pairs and allows the developer to assign collections to the HASH

name. To explain its simplicity let’s have a look at a HAHS creation when a meta_data

object, which has key/value pairs: meta_data = {

"time_of_analyze": "2020-03-08-02:51:47",

"module": "xxrrr_pb2.py",

"UUID": "5dac2afe-60df-11ea-8a13-a45e60d02ac5",

"protobuf_syntax": "proto3",

"protobuf_package": "test_proto_msg"

}

Being put into the Redis memory as a HASH type. Where the first parameter of the hmset is

the name of the HASH, the second parameter is the object to refer with the name. r.hmset('some-id', meta_data)

True

Can be called up at any time: r.hgetall('some-id')


32

{'time_of_analyze': '2020-03-08-02:51:47', 'module': 'xxrrr_pb2.py',

'UUID': '5dac2afe-60df-11ea-8a13-a45e60d02ac5', 'protobuf_syntax':

'proto3', 'protobuf_package': 'test_proto_msg'}

The main benefit of the in-memory server is that the memory access is significantly faster

than a disk performance, it is allowing the application in the future to expand performance.

7.3.3 Google Protocol Buffer

The Protocol Buffer is developed by Google to efficiently and securely transfer data between

services. Currently it has two live version the Protobuf version 2 and Protobuf version 3.

Efficiently means that it could be in some circumstances faster than the JSON or XML packed

data transfer during HTTP request because of a Protobuf compiled messages in binary form,

definitely it allocates less space. Securely means that the binary form of data sent on the

network could be harder to decode if someone tapes the communication despite it provide

some extra security if sending it on an unencrypted channel that can carry the risk that

someone catches the payload and crack it. Highly recommended to use encrypted

communication even if the message is a compiled binary, there are tools in the GitHub

universe to decipher Protobuf binary message. On the other hand, security means that

debugging such communication could be circumstantial or difficult compared to a

conventional payload like JSON or XML.

It uses messages to transform the payload into Protobuf binary. To create the messages, it

uses a compiled API to pair the appropriate field with the corresponding value, then it creates

a binary form of it. The protocol buffer raw file must be compiled by the Protobuf compiler

to produce an output that gives the ability to access the API.

Designing an interpretative which is able to understand any Protobuf message structure

without prior knowledge of API in runtime is such a big challenge. During the work the

Google factory functions supports my work to iterate through for each message and its

accessary. The most useful tool is the DESCRIPTOR it helps to construct a mechanism that

attached to a message object can provide information about. Another gorgeous feature of

Python is the getattr() method which allows to construct a method call from mix of objects

and strings.

In under average circumstances, the Protobuf API have been imported during the work while

the code birth and the developer knowing everything about including the message types,


33

nested messages, fields, field types, he can decide easily whether a message or field is

optional, required or repeated one. All of this information has been provided to manage the

development process without any interrupt. In my case, the API is appearing after the fuzzer

have been launched, in runtime, which means I have to deploy several techniques to grab the

information that is mandatory to have a fuzz session with Protobuf message. The effort that

I have to invest is greater but in long term there will be a generic logic which is able to handle

Protobuf messages.


34

8. Software specification The initial software specification was not so detailed it was mostly equal with the task

description, which can be found on page 7. , that is about to design and develop a fuzzer

framework which is able to fuzz HTTP(s), Protobuf, DNS requests. Mostly it was a sketch

about how the configuration file should look like and what kind of parameters should it

contain. Actually, it is a high-level requirement definition and gives me a starting point

because the configuration sketch defines the expected functions. There was not any other

restriction regarding the technology except the programming language.

8.1 Functional requirements The functional requirements are based on the received configuration sketch what

transforming into a requirement list it contains the followings:

1. Generic attributes:

a. The application is able to run a complete fuzz test (DNS, or HTTP, or Proto

Buf) based on the provided configuration parameters.

b. The application must be able to run in verbosity mode to assist to debug;

c. Must be able to handle the target hostname;

d. Must be able to handle protocols depending on the target;

e. A noncritical requirement is to handle the TLS connection. The main reason

for this is that during the tests it is not necessary to use encryption because,

in this case, the services communicating with each other on the intranet.

Nevertheless, if someone wants to test right in the cloud, it is a must-have to

use TLS;

f. The application must be able to archive test results as artifacts. In this point

there were no more restriction or specification;

2. User must be able to select thru the configuration to on which protocol want to run

tests;

3. Protobuf type specification:

a. User be able to place compiled pb2 files into the project directory, and the

application must be able to decompile them, and there must be a mechanism

which decides the structures and types of the Protobuf API;


35

b. Preparation for the further Protobuf compile, currently not necessary;

c. Support the logic with further information in the configuration file is allowed.

For instance, restrict which classes have to be tested from the API, or provide

extra information which are cannot gather by decompiler;

d. Values to fuzz provided by the fuzzer except Protobuf version 2 default

values;

4. DNS type specification:

a. The ability to provide the TLD;

b. A field to define domain labels which the application will fuzz;

c. A timeout about how many seconds the fuzzer should wait for the response;

d. DNS record types: A, NS, TXT;

5. HTTP type specification:

a. The application must able to send HTTP requests with the following methods:

GET, POST. For further prepare the ability for DELETE, HEAD;

b. A field for the query string or REST path;

c. Provide a field and mechanism through the templating system to be able to

fuzz not only the query string or rest path or the payload but the URL too;

d. Provide a field for the content type description and the payload;

Restrictions regarding the subject of the fuzzed target are that never fuzz the target host and

port. The application must be able to integrate into a CI system that must be able to control

it via Groovy language or shell script.

8.2 Nonfunctional requirements There were no extra nonfunctional requirements, but in anticipation, it is a must-have that

the application could run in a virtual machine environment with a normal resource

consumption to provide the stable operation and not to overload the node which serves its

functions.


36

9. Software design In this chapter based on the specification, and on the Kitty fuzzer I will introduce the

designing steps. I will describe each component, their relation, and their communication and

functions. The first step is to create a “big picture” about the application to see how the

components interact to proceed with the goal. After that, I will introduce each component on

its level.

9.1 Software architecture

To design this specific application, it is an important to find connection points in the Kitty

API where or with the framework can interact with. Those points are identifying the basic

architecture of the application. So, I have to adjust the components and their behavior to

work with the Kitty API. To achieve this, it is a must-have to implement object-oriented

programming and with its use compose the application.

Regarding the specification the main “big picture” to start the design procedure as is follows:

• Available (third_party) components:

o Kitty core component takes care of the fuzz mechanism;

o Katnip component as it is extended with:

§ Protobuf target;

§ DNS target;

§ HTTP target;

• Components needs to be implemented:

o A runner that instruments the components depending on the protocol type.

o The core component holds the base object, which is the ancestor of objects,

every other inherited from this;

o Models to describe or hold data structures:

§ Maintain configuration file;

§ Maintain and hold the Protobuf API files;

o Services to do several tasks:

§ Configuration parser class;

§ Protobuf API parser class;


37

§ Protobuf message validator class;

§ File search class;

§ Protobuf compiler class;

§ Redis client class to maintain in-memory database related methods in

one place;

§ The artifact archiver class;

o Runners who orchestrate the fuzz test on each protocol:

§ HTTP runner class;

§ Protobuf runner class;

§ DNS runner class;

o Controllers to control a specific target. Not implemented in this version:

§ HTTP controller class;

§ Protobuf controller class;

§ DNS runner controller class;

o Reporter subsystem:

§ Creates a report for the Jenkins CI;

§ Creates a lazy report to the standard output (stdout);

o Utility component:

§ Helper class and methods;

The above order almost defines as well the project structure on the file system level.

The cloud_configs directory holds mock servers which I cloned from several maintainers at

GitHub for server simulation. Some of them I extended with features, for instance, with

Protobuf message processing received via HTTP POST message.

In the following subchapters I will describe each component and its classes and the features

they serve. Before I am hitting into the negotiation for each component, here comes a

common step to describe the relationship thru a relation model.

The terms used by the application can be found at Table 1. To design the Protobuf parser

and the message_skeleton it is necessary to create rules and concepts to handle the logic and

assure that the components speak the same language. The implemented logic can work along

these lines.


38

Table 1 Terms

Name Description Protobuf API Refers to a compiled *_pb2.py file that implements the messages, and fields defined in

the raw .proto file. Message skeleton

It is a digestible and simplified form of a Protobuf API for the fuzzer produced by the message validator component.

pb2_api Refers to the object that holds information about the Protobuf API after the pb2_parser component decompiled it;

Info string The term represents a Protobuf message and field hierarchy: __marker__.package.FooClass.FooMessage.FooField

__marker__ Help to decide from the actual level whether it is an optional, repeated, or nested. Optional obviously not mandatory, repeated indicates messages that can occur many times with no limitation.

9.1.1 Kitty core The Kitty core component provides the basic mutation mechanism to create mutated data

from zero or based on the user provided default values through its low-level field mutator

then the high-level Graph Model. With the Graph Model, it can manage by the programmatic

way how the fuzzer iterate over the templates and sub-templates, which one should be the

first and which one follows which. Worth to note, the templating system and the Graph

Model have a lot of potentials. Depending by the structure of the protocol or the target

system it allows to form a template no matter how complex it is.

Because all the components already have been described in an earlier chapter (6.3) it is

enough to introduce how the application will use the Kitty as a module and what type of

components will be in use generally. Each use will be explained in each component’s

description in the following subchapters.

To install Kitty, it is a pretty easy process it needs only a command via PIP packaging

manager: pip install kitty

After Kitty is installed, it can call in any component with the Pythonic method, for instance: from kitty.model.low_level.field import String

It will import the String field type. Fields are the atomic building blocks of a template, and

they can represent strings, numbers, integers, unsigned integers, signed integers in several

lengths. The application will use several components from the Kitty core, but it will do it

mostly through the Katnip, which is basically a toolset written over the Kitty core to

implement several targets and fields by default. This will be one of the connection points to

combine the framework application and the fuzzer tool.


39

In high-level abstraction, according to Katnip API documentation to build a fuzzer it should

be the next components to have:

• Template which describes a message that the accepted by the target;

• Target class, which will the fuzzed message actually transfer to the target (in our case

targets are mostly hosts with an interface, and a protocol);

• A controller if the fuzzing is a whitebox type and it can access the fuzzed target

service;

• A GraphModell which will traverse each sub template if the template is that complex;

• A fuzzer instance, which transfer the control to the core component and does the rest

of the job such as producing the mutations described by the template. So, the core

component provides the basic mechanism to build up a complete fuzzer.

• It has got an interface which is a common browser GUI to visualize the actual steps;

• The latest method that needs to be invoked is in charge to start the fuzz session;

All right, this was a basic fuzzer construction, this is what the framework application what I

develop wants to use to achieve the DNS, HTTP and PROTOBUF fuzzing. In my approach,

the pattern is fast similar as described by the API documentation, but in a bit more

sophisticated way because it wants to implement several protocols, each protocol should

have a well separated class. Other utilities regarding to artifact production and the separated

logging of the framework also needs individual classes. It will have a configuration, possibly

a data generator in the future and so on. It is an existing aim to develop this application from

year to year and serve feature requests that I receive. So, it really needs a framework.

9.1.2 Katnip component The Katnip installation is a bit different from the way of how Kitty was installed because its

targeting component expands with new target types during the development, so I have to

have access to its components at project level. In such a case, one of the most practical ways

to develop an active component is to install it editable. Katnip already has a TCP target

module that performs TCP communication over sockets, and it has a UDP target that does

the same mechanism but on UDP channel. Technically while coding the UDP or TCP there

is no need of extra effort from the developer except that simple definition on the socket to

use user datagram protocol while peers are communicating. Both targets are derived from


40

the Kitty ServerTarget class, and they act like a blueprint to rule the communication protocol

definitely.

9.1.3 Framework core and the config parser The framework core stands from two file, one is the fuzz_object.py which implements the

basic fuzzerObject class. This is the base class, most of the others are derived from this. The

framework core follows the Kitty object design pattern and it is an absolute reference of it

except the Redis which is a parameter in the fuzzerObject’s constructor, thereby the

StrictRedis client object is available to serve in aa early step. The fuzzerObjet class is also

responsible for managing the separate logging from Kitty’s logger to avoid interflowing with

it. This achieves that the fuzz session runs separated logging regarding to the tests and the

framework related operations.

The config parser uses one independent class, which is the GenerateUUID that implements

a very simple method to generates a unique ID for each fuzz test session. This unique ID

appears in system-wide and its objective is to assure the identification of each session.

The ConfigParser class is derived from the FuzzObject, and it is in charge of making the

configuration parameters accessible to the appropriate components via its class methods. To

avoid the spreading of multiple configuration objects in the memory of course with the same

data, that is does not make sense, it uses a toolset from a common class which is basically a

decorator function that prevents the unnecessary object creation called singleton. The

singleton is an object which exists in only one instance no matter how many time a function

call it. The tool can be found under the folder “utils” called decorators.py. [25] As the code

snippet showing the singleton decorator on Figure 6.


41

is a function which is wrapped into another function with the support of functools library

that returns a decorator and invoke the wrapped function which is decorated and returning

with the wrapped function. Technically, it evaluates whether the wrapped function has a live

reference to a live object or not.

9.1.4 Protobuf parser Definitely, the ParsePb class is one of the most complex ones because of the logic it

represents to decompile and preprocess the given Protobuf API which will after the

procedure more suitable for further processing. The idea is realized through several steps. It

is an important thing to control the dataflow simply because the fuzzer, during the start of

the application, initially, it has no information regarding to the complexity of the given

Protobuf API that provided by the user, during the process it must be ensured about its

structure. No matter how difficult the complexity is by the Protobuf API, the logic must be

able to parse it. That’s why I found that processing in more steps the Protobuf API is a good

approach. The pb2_parser class aims to proceed with the first step.

Figure 6 Singleton decorator accroding to RealPython's tutorial


42

The raw format of the Protobuf is a file with the .proto extension. A raw Protobuf version 3

file on the Figure 7. showing its syntax. At first glance, the data language rules are clearly

visible.

By its structure, it implements messages. A message stands from fields, a field can be any

well-known type such as INTEGER, STRING, or special one the ENUM, which is a list of

possible enumerated values. It is possible to use nested messages from the actual Protobuf

file or import other packages through messages and calling them whenever it needed. A field

type can be a message type too, which was defined earlier in the raw Protobuf file.

The ParsePb class reads and processes the compiled Protobuf and its power is that it can

read and decompile those APIs without any intervention or outside help. An excerpt from a

compiled Protobuf version 2 file on Figure 7. Depending on the Protobuf version, there are

differences that need to be taken care of during the design. Protobuf version 2 supports

default values while this feature was deprecated in version 3. Extensions are also

unsupported by the newer version; they were let the developers to declare a range of fields

Figure 7 Raw Protobuf 3 file - source Google


43

that are maintained for third-party extensions. Basically, it is a placeholder for external

messages. The Oneof type also a difference that makes sense. It provides a mechanism when

a message having many optional fields, but only one allowed to reserve the field, then the

Oneof enforce the behavior and apply only one message for the field. Setting a member of a

Oneof will drop all other previously added members.

Switching to Protobuf version 3 things are getting simpler, nevertheless, the logic must be

able to handle both.

The ParsePb has also been derived from the FuzzObject, and it uses the CheckProtoEnv

class which is the part of the Utilities component and its objective is to check whether the

environment is Protobuf capable or not. Through the Google Protobuf library, it consumes

compiled Protobuf API files and produces JSON output for the MessageValidator class. It

loads the Protobuf API module in runtime because the logic provides a generic way to read

and understand the API content. It needs only the path where the compiled Protobuf files can

be found. It puts the produced API structure in the memory as an object, and as well as JSON

file under <project_root>/bin/pb2_api_json path. This structure will be the blueprint for

the MessageValidator class to inspect and reveal the dependencies and hierarchy inside the

Protobuf API file.

Figure 8 Excerpt of a Compiled Protobuf 2 file, source Google


44

To crawl the information from the API the logic have to do the followings:

- Import *_pb2.py in runtime;

- Instantiate the component with an instance variable;

- Starting a loop to check after every possible message or field type occurrence:

o foo_object.DESCRIPTOR.message_types_by_name.keys() – will

returns between one and ‘N’ an object list that will show me the top-level

messages in the API

o Then I have to inspect for each message whether they have any other sub

message, or nested message, or fields, if fields occurs, what types they are?

o The logic should instantiate them in runtime as:

§ foo_message = getattr(foo_object, ‘Message_Name’)

o Then construct the descriptor again by calling foo_message.DESCRIPTOR.nested_types_by_name.keys()

o Repeat until more descendants can be found.

- A further objective is to maintain their relationship and find them a structure that fits

and provides a practical way to deal with them. To construct and pack the blueprint

of the API for the fuzzer to generate the proper data into the appropriate place is also

a major objective.

The greatest challenge regarding to this component is that the Protobuf API file parsing is

not as clear as ,for instance, parsing a JSON or XML structure. It has currently a deficiency

Figure 9 JSON file from the Protobuf API


45

that requires to provide ENUM relation via configuration to connect ENUM fields to the

appropriate message or field. An excerpt of the JSON type Protobuf API description file is

on the Figure 9.

It is true that the Google provide a MessageToJson function to get JSON format data from a

Protobuf message. But it is work only if the message itself is available in any form. In our

case, initially the application has no idea regarding to the message content or its structure.

In under normal circumstances an application applies the Protobuf as a whitebox, well

known component which is clearly defined across the components, and mostly it does not

change without any preliminary indication.

To reach out to the objectives I used to manage the problems with various toolset, to autopsy

a components structure, its classes, or methods in runtime, I voted to the Inspect library.

After the application able to get the first level classes, which named to ancestors or

top_levels I continue to go backward with collecting the method names from the API to

extract more information. Studying the Google’s relevant codes it was clear that the

application needs a mechanism which can iterate over a Protobuf API and has the right

intelligence to decide whether the actually under investigation level is a class, or a

MessageDescriptor, or EnumDescriptor perhaps a field and if it is, what kind of data type

represent it.

The MessageDescriptor describes a message defined earlier in a raw .proto file. It can give

miscellaneous attributes about the message as well the ContainingType that tell us whether

the actual message has a nested one or not, or about the Oneof, field type, or further nested

messages inside in it. So, the underlaying mechanism of the Protobuf is the key to understand

it. [26]

9.1.5 Protobuf message validator The ValidateMessage class is derived from the RPutCollections class, which is a helper class

for the Redis operations, and it is derived from the FuzzObject class. It has two main

objectives; one is to create a skeleton for the fuzzer about the Protobuf messages and the

other one is to validate whether the Protobuf APi parsing was successful or not. It discovers

the relationships between each message depending on the level they can be found during this

procedure; it also instantiates the messages and their fields and loads them with dummy data.

At the end of the procedure there will be a skeleton string and a test Protobuf message. The


46

skeleton string gives the ability for the fuzzer to easily decide the order of the fields to fuzz

and its types. It is also an encouragement for the ProtobufTarget class to instantiate each

message and put the fuzzed data into the appropriate message field. The skeleton is basically

an ordered list with strings. Strings delimited with dots and it describes the Protobuf message

relation see on this excerpt:

testRequest=main_pb2.TestRequest

TestRequiredRequestData = testRequest.TestRequiredRequestData.add()

TestRequiredRequestData.a_req.TYPE_STRING

It’s clear that this is a more straightforward form of a Protobuf API compared to the one

created by the ParsePb class. At this point, if we observe each state of an initially compiled

Protobuf API related to this skeleton, I mean compare the Protobuf API to the JSON API

description and to the skeleton, it is evident that in this form the messages are reproducible

any time anywhere without conflict.

The objective was to design an intermediate object group that supports the pairing, sorting

data. Currently, Python data structures such as list, dictionary support to compute, and

maintain the hierarchy between each message, or class. The helper structures are: Group of

nested messages, group of optional messages, repeated, and enum’s are also grouped. Groups

are starting points for further subdivisions. Then the component maintains the ancestor,

which is mostly the top-level class it should carry on the preceding messages. During other

further steps it dived the structures onward per message type. For instance, there will be

nested_optional, nester_repeated, repeated_imported message, and there will be

repeated_nested, repeated_optional, and so on every combination mapped to assure the

hierarchy. During the skeleton creation, the component simultaneously validates its job by

creating Protobuf message, decode then encode it back. The message creation without

critical error confirms that the process was successful.

The component puts its logs into the framework log file and stdout as well it also gives us

the ability to run the session with verbosity mode to inspect errors if it occurs.

The component basically works form three data source, the first is the configuration file the

second is the Redis in-memory database where some of the Protobuf related attributes are

stored. It uses the JSON descriptor files and as well the Python object which describes the

Protobuf API. The ability to force the fuzzer to fuzz only one class at the top level era it is


47

possible because if someone wanted to blackbox fuzz an application with this protocol, it is

not necessary to take care of the response messages, so it is allowed to exclude classes from

the mechanism in certain conditions such as the response message.

9.1.6 Protobuf target The ProtobufTarget class provides functionalities that are essentials to proceed a Protobuf

fuzzing on a given target. First, it uses the message skeleton to create each Protobuf message

in runtime. It is possible because of Python’s getattr, setattr methods which are in use

application wide. It implements two generic function to handle method calling and attribute

setting. After the messages are instantiated, it pairs them with the fuzzed data, which are

provided by the ProtobufRunner class; at this point the application use the setattr Python

method to set the attribute of a function. Worth to note, it could be a real scenario that the

fuzzer core could send such a fuzzed data, which could rise a type error, so proper exception

handling is a must-have. After the message is available it encodes it to bytes.

To put the Protobuf message into the finish line the application constructs a HTTP POST

method message with the appropriate HTTP headers and put it on wire via socket.

To get the proper result, the session needs a response from the target, so the fuzzer initialized

with the state of expect response, which is basically an abstract method in the Katnip’s

ServerTarget class which also overwritten by the application because the sophisticated

failure reporting. So, the response receiving maintained by a function which reads the

response from the socket and transfer it to the transmit function where, according to the

expectations, functions are unpacking every HTTP response header seeking for HTTP 50x

and 40x response codes. If any of them occurs, a handler will create the appropriate report.

Those reports will be processed by the Reporter component.

It also implements supporter classes and methods to assist the unpack mechanism to decide

hierarchy described by the message_skeleton.

9.1.7 Protobuf controller The controller class for Protobuf basically done by the Kitty API and its representation is

almost equal as it describes to how to set up a controller. Currently it checks whether the


48

actual target is alive or not, but it has got no influence on it. It is only a preparation of

whitebox fuzzing in the future. It uses all the Kitty and Katnip components.

9.1.8 Protobuf runner The ProtobufRunner class is the point where the Kitty and the framework marriage has been

made and this statement is true for every upcoming runner. In one hand, information came

from the framework side and goes to the Kitty fuzzer templating system, goes to the Katnip

Protobuf target than the direction again to the framework application. The template

mechanism receives the skeleton and grabs the types that it starts to fuzz. It recognizes what

kind of data type should to fuzz. According to the structure of skeleton it creates a dot

delimited string for instance: 1234.foo.bar.bazz.123 and sends to the

ProtobufTarget which creates a Protobuf message from them. The mechanism runs as many

times as the template and the input data mutation number is. If there is a structure which

describes, for instance, 500 mutations, the mechanism will turn 500 times. During the

integration it is possible to fine tune the template mechanism and the GraphModel to achieve

the proper mutation combination.

The rest of the class is about the fuzz session handling according to the Kitty/Katnip API

documentation as it was described earlier. The ProtobufRunner class invoked by the main

Runner class according to the configuration definition.

9.1.9 HTTP target The HttpTarget Class inherited from the Kitty ServerTarget class, constructed according to

its API documentation, and is in charge to send HTTP requests in the direction of the given

target using sockets. It also maintains the responses and seeking for HTTP error codes in the

HTTP response header and reports them if an exception occurs. Exception in this

terminology means that any of the 50x or 40x response codes can be found in the HTTP

response header. Obviously that the reporting mechanism is able to report in several other

conditions, it is only up to the needs. Currently every target initiates a report if the payload

raises an exception during the send or receive procedure. The case mostly is type error or


49

wrong byte start wrapped into try/except/finnaly scope to log the type errors and avoiding

the termination.

9.1.10 HTTP controller The HttpControler class representation, and the existence of it is equal with the

ProtobufController class. It is a basis of a further whitebox fuzzing. Currently it checks the

given target HTTP server whether it is available or not.

9.1.11 HTTP runner

The HttpRunner class implements the HTTP GET and POST methods according to the HTTP

version 1.1. It is able to fuzz the HTTP server target via REST API or the conventional way

during a POST method with form data or using a query string during a GET method.

It has two main templates constructed in it, each of them defines on HTTP method. It decides

the method according to the configuration provided by the user. The main challenge in this

runner class is to define a proper HTTP header which is good and sophisticated enough for

a well-equipped webserver. The best service to measure its perfection is the AWS

CloudFront.

A well-formed HTTP GET header according to the HTTP standard can see on Figure10.

screenshot taken in Postman:

On each fuzzer template, the application has to implement these forms, taking care of every

detail, for instance, the two double new lines at the end of protocol header, or the first line

must exactly be in the same form and order and so on. To simulate the structure of any HTTP

method, the application uses the Kitty template and its fields. An excerpt of the template,

which builds the HTTP’s ‘GET / HTTP/1.1’ header field can see on the next code snippet

on Figure 11.:

Figure 10 HTTP GET header, Postman


50

Technically a field is a list of methods, and each method represents a segment of a line.

Depending of the type of the actual segment the application calls the function, which

provides the functionality to fuzz the data value in it. It is clear that it simulates the new line

at the end of the method definition.

It has the ability to fuzz the whole protocol instead of fuzzing only the given path or payload.

So, if the feature requested through the configuration, the fuzzer can fuzz the protocol related

fields too. This feature gives the ability to cover errors if the webserver binary has any hidden

software bug or there is a misconfiguration elsewhere or wrong server logic implemented if

someone use a self-made webserver. The templating mechanism has the ability to fine-tune

it during the integration because of its flexibility and easy adaptation.

Basically, it generates fuzzed data by the default values provided via configuration, these

values are the path and the payload.

9.1.12 DNS target The DnsTarget class is also based on a factory extension called UdpTarget which is

implemented by the fuzzer framework, and it can be found under the targets in the

framework. It is also derived from ServerTarget, which is a core Kitty class. The reason for

the existence of a DNS fuzzer is that the Cloud service has specific versions which are

providing information about threat objects via DNS query responses. These are mainly “A”,

“NS” and “TXT” records:

• “A” record responses IP address;

• “NS” responses the name server;

Figure 11 HTTP GET template header


51

• “TXT” responses text record;

The following Figure 12. shows the DNS records in AWS Route53:

The first difference regarding to a TCP based connection is that in this case the application

uses sockets with user datagram as it seems in the following code snippet on Figure 13.:

The class receives the fuzzed data from the DnsRunner via its templating mechanism, in this

case the fuzzer mutates only the domain labels which are came from the framework’s

configuration. To dominate the data stream, it is necessary to take care of exceptions because

the fuzzer core in some cases changing the default character coding from ‘UTF-8’ to ‘iso-

8859-1’ that immediately raises a UnicodeDecodeError with wrong byte position. It is

obvious that useful to handle these exceptions. To put under pressure the victim server and

observe how to behave for a query that might never exists in real-world scenario but worth

to know what happens across the borders.

Further challenge is to construct a DNS query according to the given record type. The record

type is also provided by the configuration file. In the response side the objective is to grab

the DNS response code to decide whether there was any problem or not. But let’s see a DNS

request header first on the following code snippet in Figure 14.:

Figure 13 Socket creation for UDP, Katnip original method

Figure 12 Route53 DNS records


52

And the query’s response message at Figure 15.:[27][28]

Figure 15 DNS Response message layers.

The DNS target contain the implementation of a DNS query that is following a low-level

approach. Knowing the fuzzer core behavior I considered that is important to not use factory

library (DnsLib) for composing DNS query messages instead I implement the query from

layer to layer and concatenate the whole question message.

It is important that the fuzzer must fuzz only the domain labels and never fuzz the TLD and

the domain name:

• Domain name: somedomain; TLD: com

o Never fuzz: somedomain.com

Any preceding domain labels are allowed to fuzz in any order. For instance, from the domain

foo.bar.baz.somedomain.com the first three part is allowed to fuzz.

Back on the road, I voted for the factory module DnsLib process the DNS response and use

it to create the necessary reports, logs; there is nothing serious at first sight. But the responses

mostly arrive in disrupted structure (wrong start byte) too because of non-valid characters.

In order to handle these scenarios, the component inherits the DNSRecord and DNSBuffer

Figure 14 DNS Header format, Source RFC 2929


53

classes from DnsLib module and take care to override the appropriate methods to adjust its

bearing to the new circumstances.

The response codes which the logic triggers are the: FORMERR, SERVFAIL, NXDOMAIN,

NOTIMP, REFUSED, YXDOMAIN, XRRSET, NOAUTH, NOZONE. Initially the logic holds

all the response codes in a list and compare the index number with the response code number.

If the response code is not 0, the framework creates a report from the complete query and

response. With this mechanism, the application being able to collect problems during a DNS

fuzz session. Of course, it creates a report if any other common error occurs and raises an

exception. It is also providing the ability to extend or fine-tune the template structure or to

extend the reporting points, which means it is an easy task to introduce new error or

exception types to report.

9.1.13 DNS controller The DnsController class as all the followings is only an open door to implement a whitebox

fuzzer with a server agent. Currently it is only evaluating whether the target is alive or not.

9.1.14 DNS runner The DnsRunner class implementation makes the least problem because the domain label

structure representation in the template is obvious. It derived from the FuzzObject and

implements its logging in combination with the Kitty object logger. DNS also has a counter

to decide to after how many times of target failures should the controller forgive the

connection initiation. As a result of the DNS protocol the application uses timeout settings

which determine a time window to wait for the response. The rest of the class must apply

the anatomy of a fuzz session according to Kitty API documentation and as it is implemented

in other runners.

9.1.15 Redis component The RPutCollections class is derived from the FuzzObject and used by the ValidateMessage

class to put Protobuf API component name into the in-memory database. It is basically an

experimental class to try out to how easy the work could be with an in-memory server and


54

of course provide a class for the further developments. The Redis Python client connection

made by the FuzzObject. The class constructor waits the Protobuf API object which is

provided by the ParsePb class. They meet each other in the Runner class.

9.1.16 Reporter component After the application has been integrated into the CI system, it is a valid expectation that it

could be generating reports to publish them under the CI job’s reporting surface. These are

artifacts which are stands from in our case the Kitty logs the framework logs, the Protobuf

API JSON files, reports created by the Kitty web interface if a predefined error has occurred.

This report files are JSON files.

The reporter component transforms the application reports into artifacts in cooperation with

the FrameworkUtils class, which has several methods to collect each output into one specific

and easily identifiable folder. The class called PutArchiveToS3 that move the artifacts into

an AWS S3 bucket, and it is carrying on whether the bucket is exists or not.

The Reporter class privilege is to read and parse the reports generated by the Kitty web-

interface and produces a stdout summary and an XML file. The XML has to be compatible

with the Jenkins unit test publisher plugin. The Reporter class implements the mechanism

via the xml.etrre.cElementTree Python library.

There are several reusable methods that the mechanism listed above using, for instance,

searching for files, using UUID.

During the integration, the JUnit compatible XML has needed structural changes as I

planned earlier.

9.1.17 Utilities component This component builds to maintain helper classes and methods in one place to support the

framework with general features they provide.

• CheckProtoEnv class is to test the local environment whether it is capable of running

Protobuf related stuffs such as read or compiling;

• Decorators is a powerful collection of functions to help debugging code;

• GenerateUuid is a singleton class which creates a UUID for each fuzz session to help

keep track of them;


55

• Utils as mentioned several times above give the ability to use functions system-wide

by components such as the message decoder, encoder and file system related

operations, it also implements helper functions to search specific value by key in a

dictionary, or specific key by value, or list all values or all keys; It might better to

name it in the future to FrameworkLib;

9.1.18 Runner The Runner class integrates the Kitty/Katnip components and the fuzzer Framework

application to instrument a fuzz session according to the given configuration state, which

can be a DNS fuzz, a HTTP fuzz or a Protobuf fuzz session. It derived from the FuzzObject

and uses most of the classes implicit or explicit. This is the point where the Protobuf API

parser components has been utilize each other. It defines the function called tear down,

which triggers the arhiver components and the Redis memory release after a fuzz session

function.


56

10. Installation and integration into CI system To properly install and set up the framework, it is necessary to define its environment. In

this special case, when we speak about a complete integration into a CI system, it needs some

design decisions. The first thing that we should care about is the environment that runs the

framework, the next one is how to install on it, and the last one is the CI system job definition.

And of course, a target system, which is the victim of the fuzz test. According to this order

I will introduce the high-level design of the integration of the fuzz framework application.

10.1 The executor environment The virtual machine environment which serve the application is a deployed Centos7 Linux

node inside the Company’s VmWare cluster, which is an independent node from the CI

system. The mandatory Operating System packages are the Python 3.7 , PIP package

manager and Python virtualenv package, which allow us to bootstrap the necessary Python

environment. The CI will expect that the machine is always up and running and

preconfigured with the application.

To install the prerequisites the administrator should do the following steps:

1. Deploy a CentOS 7 Linux, with networking in the same subnet as the CI system;

2. Deploy Python 3.7 and the PIP package manager and the virtulenv package with a

common packaging manager like yum:

a. yum install python37, pip;

b. python3 –version; - to check it was properly deployed;

c. python3 -m pip –version; - to check pip version;

3. Install virtualenv:

a. python3 -m pip install virtualenv;

After these steps, there are no more configuration left from the virtual machines side.

10.3 Install the application After the virtual machine has been installed, the first step is to check out the fuzz_framework

repository from GitHub at https://github.com/netlounge/fuzz_framework and stay on the

master branch. Now it’s time to activate the virtual environment inside the project directory

with the command: virtualenv -p python3 .venv; It is not required but recommended


57

because of easy to handle Python packages and dependencies inside virtual environment.

There is a step by step guide inside a README.md under the project directory to fulfill the

integration or standalone installation: fuzz_framework/README.md

Going beyond the installation after running a test session from cli on the target system to try

out whether the application is working correctly or not, it is ready to integrate into the CI

system.

10.4 Integration into Continuous Integration System The CI system, in our case is the Jenkins CI, which was written in Java language, and it is

basically an engine that can receive wide variety of plugins to manage automated jobs to

support the software development’s process. A job is a configuration set or a script, which

automates several process depending on the definition it has. It can be a build job to building

source code to binary then it can run unit tests, functional, non-functional tests on it, or it

can mix up more work step into one huge pipeline as it looks on the Figure 16.

As it seems clearly it builds the source files into binary, runs unit tests separated the C++

and the Python related ones. Some of the steps are running in parallel mode. It creates

documentation from the code docstring placed inside the code, creates deployable RPM

packages from the binary and publish them as artifacts then grab a virtual machine from the

VmWare cluster and deploy the code, runs further tests then it publishes the results again.

These chapter’s objective is to design such a pipeline for the fuzz tests. Because of the

complexity the pipeline implementation, I will negotiate it from designers’ perspective

which are approximately as follows:

Figure 16 A Jenkins Pipeline view, source Sophos


58

1. First of all, we assume that there is a working AWS Cloud elsewhere pre-deployed

and available to run these tests; If so, the one possible first step is to scale-out the

deployed stack and grab information via aws cli about the state to know when the

stack is updated successful, and the servers are serving.

2. The following step can be to configure the fuzzer to run a DNS fuzz test on the target

system; The configuration can be done with a tool, which is able to modify a JSON

file; The test can run several times if it is needed and if more initial default DNS

query is available from a file; Another script can trigger the fuzzer application to run

the test;

3. The second step is to collect the actual tests artifacts and evaluate the results;

4. During the following steps, the Job starts the second one protocol but with the

Protobuf related configuration;

5. The last step is to tear down the system, collect application logs from the Cloud

services, scale it down, publish the collected application logs and framework logs,

results produced by the fuzzer framework;

6. Cleaning up on the Virtual Machine to preserve its initial state for the next run;

Jenkins gives the ability for the developers to scripting the above steps with the Groovy

language which is developed by the Apache Foundation to control the Jenkins scripted

pipeline. The high-level design of integration and data flow can be found at Appendix 2.

(Fuzzer Framework high-level integration and data flow chart). It demonstrates the on-

premise infrastructure where the Jenkins CI operates in the separated corporate network. The

application and its network connections, source code stream and interactors depict its

relations and place in the infrastructure. The data flow indicates the protocol types in the

direction of the AWS VPC. I tried to describe the VPC as obfuscated as I can to hide the

systems details and its structure, but even so the high-level sketch, some basic components

are well recognizable to understand the data flow targets.

It might worth speaking about a second decision that could be another integration setup. Its

existence is based on that to maintain a full Cloud deployment could raise the team budget

from period to period into a high rate. Definitely, an almost perfect condition could be to use

shared deployments in the Cloud to share its resources across several tests. In practice, one

deployment, is cheaper than two or more it’s obvious, but in case of one deployment other


59

facts are deserving to consider. One of those considerations is that one deployment used by

huge test cases receives higher load. Instances are being upscaled in a wider time window

with surely high load, database tables, or RDS instances also scaling during peak and stress.

These things are generating really high costs. Beyond the costs, engineers who are

responsible for the DevOps roles must manage each case to schedule the Jobs properly to

avoid deadlocks on parallel tasks and provide proper mutual exclusion against each other.

Based on these considerations, it could be an absolutely accepted solution to running a virtual

machine on a conventional Linux box to simulate the Cloud environment with fake services.

These services are fakes as their name refers to them and provided by AWS to give the ability

for the engineers to develop Cloud applications at their localhost to saving them time and

costs. The idea is at Appendix 3 (Fuzz Framework On-Premise integration).

After the integration into CI Pipeline the ongoing process of the fuzzer while runs a DNS

fuzz test is on Figure 17.:

Figure 17 Ongoing DNS fuzz test, source SOPHOS Jenkins CI Pipeline.

10.5 Results after the first runs After the fuzzer has been integrated and the first tests are initiated it has found a minor but

interesting issue while ran HTTP GET test. The requested data have been passed inside the

server core logic without validation and that somehow forced the logger subsystem to create

an empty log record, with such kind of flaw a threat can obfuscate its presence in the system.

It uses 7 test suites with two network protocols, the DNS on A, NS, TXT, and HTTP GET

launch every Saturday.


60

11. User Guide to operate To configure and operate properly, there are some further steps that the user must do before

running the first fuzz session. These steps are true in the case if the framework is a part of

the CI system. The CI must know, of course programmatically way, that what configuration

fields should modify to get the expected operation. After the runs and even if an error occurs

during the session, the log analysis is a must-have.

11.1 Configuration The starting point to set up the fuzzer, is the configuration file, which is under the project

folder framework/data/config/framework_config.json it has four main object type. The

generic object is about the system-wide settings:

• verbosity: set 1 to run on verbose mode;

• target_host: the host, which is the victim of the fuzz session;

• target_port: protocol specific port but it can be different according to the

implementation of the target listening port;

• tls: set true if the TLS is a must-have while fuzzing HTTP(s) methods;

• web_port: is a Kitty specific port where the Kitty web interface is available;

• archive_to_s3: if true, the framework will archive the results into an S3 bucket;

The next block is about the protocol type, set the appropriate one to true and be careful to

set only one protocol to true;

Protobuf related fields are about to configure the Protobuf fuzzing over HTTP:

• proto_path: is the path where the raw Protobuf files can be found, currently not

implemented to compile them;

• pb2_path: indicates for the system the compiled Protobuf API place;

• modules: The main Protobuf API component name, which holds all the imported

Protobuf packages;

• classes: The top-level message class name according to the Protobuf API structure,

it could be more than one on the top level;

• enum2class: The only insufficiency from the Protobuf parser component is that it

cannot decide to which ENUM filed belongs to which class, this is the only strict

point where the system needs help from the user; The preceding two field which is


61

the module and the classes are only to let the users decide the circle of the module

and class to fuzz; These two approach also a basis for a further development to

sophisticate the fuzz session during a Protobuf fuzzing;

The DNS fields are the followings:

• tld: The top-level domain name and the domain to query;

• default_labels: These domain labels are available represented by a DNS record and

this is the basis of the fuzzed query;

• timeout: The time window size what the fuzzer should wait for a DNS response;

• A: to ask A record;

• NS : to ask name servers;

• TXT: to ask TXT record;

DNS able to ask only one record type in a fuzz session.

HTTP related fields are:

• GET to fuzz a HTTP GET method;

• POST_PUT is to fuzz a HTTP POST method;

• fuzz_protocol: set true to fuzz not only the HTTP payload and path but the HTTP

header fields too;

• content_type: to define the HTTP header content_type;

• payload: Define the json format of a REST payload or form key value pairs;

With the appropriate configuration file, there is an open way to start a fuzz session. During

the session, the framework creates output files tagged with the actual session’s ID. At the

end of the session it puts all together into a folder under the framework/bin folder named

also to the session’s ID. Under this folder, every report and output log can be found for

further inspection if necessary. To debug the system, run it in verbosity mode and read the

log files produced by Kitty and the framework. The framework logs every essential step on

each stage it starts and finishes, and logs exceptions where it could be helpful. Each log line

contains a timestamp the component name, the file name and the line number where the

appropriate function step can be found. Whit these logging combined with the Kitty logs and

with the session ID it is an easy process to inspect errors or failures.


62

A complete fuzz session output folder content after a Protobuf session on Figure 18.:

To deep analyze the results:

In one hand, the framework will generate a result.xml , which is available in Junit compatible

format for the CI system. Each report generated into standalone file during the fuzz session

if any predefined error occurs. If the report in the CI system or on the stdout shows errors,

which can be interesting, then the result outputs will help to investigate deeply.

A report file name format was designed to help easily combine the information aggregation

to follow up the error. Worth to note that information provided in base64 code inside the

report files, but the Python’s Base64 library can decode it with the method b64decode().

The project contains a docs folder where the framework’s generated API documentation can

be found to help understand and deep analyze the mechanisms used by the framework if

someone needs it. An excerpt of the API Documentation can be found under Appendix 4.

Figure 18 Session result data, source Fuzz framework


63

12. Further development possibilities There are a lot of potential further development possibilities in this project, including a more

complex templating mechanism to create accurate mutation for the targets. It could also be

an objective to extend HTTP methods or the DNS query types to follow up the targets

evolution or extend the field type recognition mechanism when fuzzing Protobuf messages

or support more sophisticated Protobuf message structure. Some improvement around the

Protobuf message validator to get rid of its forced support from the configuration file, which

means it should handle the Protobuf message hierarchy from the view of enumerated fields.

The proper exception handling cases during the fuzzed data transfer could also be the field

of future improvement. Most of these are solvable during the integration because many of

them can be done with low effort.

To put the application into a higher level, it could be a possible feature request in the future

to run the framework as a background process or behind a http server and accept REST API

calls to reconfigure it in runtime to set up easily for the next fuzz session in a more elegant

way. From the background process approach it could be also a feature to run it in multi-

threading mode. A mechanism to check whether a Protobuf API has any change and

recompile it if changes are occurs. Store the file hashes of the API files to compare each

other to each version to decide is there any change on them.

A standalone seeder component which allows to produce input data in more sophisticated

way.


64

13. The future: increase app resilience with symbolic execution. In the near future of the field of fuzz testing, groundbreaking innovations are born according

to the newest studies. These studies do not argue that the blackbox fuzzing methods are not

good enough, but they agree that the technology is ripe for it to run symbolic execution,

which is basically the whitebox fuzzing that I also mention several times as a future

possibility. Google and Microsoft are pioneers in that kind of technology, and they make

software products more resilient with that type of testing. One method is that the test cases

are predefined by the testers bypassing the time that a simple fuzz test can take during brute-

forcing the application instead they focus on automatically generated test cases with the help

they can take a walk through on specific paths to support more accurate reveal of software

failures. Others went further and provide existing models to symbolic execution. One of

these methods is the Compositional fuzzing, which influences the system in very low-level

by isolating functions and apply seeders on them with special fuzzer drivers. They also take

care of the complex crash report, which can replay the test cases to generate reports about

the vulnerabilities. They are replacing functions with exploit test-cases to drive the code into

pathways leading to vulnerabilities.

While no test methods can ever explore every software errors, but there is the ability to

evolve the tests in parallel with the software to have more reliable and inclusive test result

set to which convince the responsible coworkers that the application is quite resistant.

[29][30]


65

13. Summary The topic of my thesis is to security test a Cloud-based application with a fuzzer application.

During the task, I researched an adequate fuzzer tool on the open-source software market

than implemented a framework on the top of the fuzzer to proceed fuzz tests and of course

the network protocols also implemented into the tool. In the first chapter, I have introduced

the task and provided some information about my employer. With this approach, the

justification of the task’s equity was done. The next chapter is about the importance of testing

in software development. The chapter also introduced the Agile software development and

its methods. Starting from the fact that the framework’s objective is to fuzz a Cloud

application, I introduced the Cloud, mainly the Amazon Web Services, and I gave an

abstraction about the distributed systems. The subsequent chapter was about the software

testing started with the unit testing method and discussed all the major tools up to the fuzz

testing. The fuzz testing was inspected in more detail.

After the introduction of fuzz testing, I started to describe the design procedure with the

research process of the possible fuzzer tools whereat I have implemented a framework. I

made an aspect system, that provided the benchmark between each fuzzer tool to help

ranking them and making a decision. I compared them pro and contra according to the

attribution and I made the decision to choose the one which fits the most for the expectations.

After these steps, I presented the technologies which were involved explicit or implicit into

the development. I discussed the Cloud resources and the integrated development

environment and the programming language that I choose.

The last two chapters were about the integration I made a clue about how to integrate it into

a CI system and gave advice about what kind of steps shall be done to implement in a specific

use case. I touched the installation procedures and the prerequisites too. Finally, I wrote a

User Guide to guide the prospect user to cope with the framework during operate with it. I

did notes about the configuration and the trouble shooting possibilities.


66

14. Összegzés A szakdolgozatom témája egy felhő alapú alkalmazás biztonsági tesztelés egy fuzz

rendszerrel, amelynek keretében a tesztelést vezérlő keretrendszert magam fejlesztettem,

valamint a megvalósítandó protokollokat magam implementáltam egy választott fuzzer-be.

A dolgozat első fejezeteiben a feladat ismertetésén túlmenően szót ejtettem a vállalatról ahol

dolgozom, így alátámasztva a feladat létjogosultságát hiszen az alkalmazásra szükség van a

minőség és biztonság fokozása céljából. A következőkben ismertettem a szoftver tesztelés

jelentőségét illetve az Agilis szoftverfejlesztést és módszertanát. Abból kiindulva, hogy a

tesztelt alkalmazás felhő alapú, bemutattam általánosságban a felhő technológiát, azon belül

is az Amazon Web Service-et, illetve az elosztott rendszert. A következő fejezet a szoftver

tesztelésről szólt ahol is a unit teszteken keresztül eljutottam a funkcionális és nem

funkcionális tesztelésen át a Fuzz tesztelésig ez utóbbi teszt bemutatására egy teljes fejezet

áll rendelkezésre.

A Fuzz tesztelés ismertetése után elkezdtem a feladat tervezését ami a lehetséges fuzzer

eszközök felkutatásával indult, a fejezetben felállítottam egy szempont rendszert ami alapján

értékeltem és választottam lehetséges eszközt a szabad szoftverek piacán rendelkezésre álló

fuzzer-ek közül, az egyes fuzzereket pro és kontra követelmények szerint elemeztem és

bemutattam, majd a szempontokat kiértékelve kiválasztottam az általam fejlesztett

keretrendszer számára legalkalmasabbat. Ezt követően ismertettem a fejlesztésbe közvetve

vagy közvetlenül bevont technológiákat kapcsolódóan a felhőrendszerhez valamint a

fejlesztői környezethez.

A következő fejezetben specifikáltam a szoftver funkcióit majd ennek alapján elkezdtem a

tervezést amin keresztük az implementálandó feladat problémáiba nyújtok betekintést.

Komponensről komponensre haladva ismertettem azok funkcióját, kapcsolatát más

komponensekkel, helyüket a rendszerben. A leírt terv szoros részét képezi az

implementációnak.

Az utolsó fejezetekben kifejtettem az alkalmazás integrációját, támpontot és egy lehetséges

use case-t valamint egy CI szkript tervet nyújtottam az integráció megvalósításához.

Végül a felhasználót segítő dokumentációt készítettem és ismertettem a lehetséges jövőben

fejlesztéseket, melyek közül némelyek alapja már elő van készítve mások komolyabb

beavatkozást igényelnek.


67

15. Bibliography [1] J. Highsmiths, Agile Software Development Ecosystems. Addison-Welsey, 2002. [2] Michael James, “Scrum Methodology.” https://scrummethodology.com. [3] D. CLAIRE, “What is Scrum?” Atlassian. https://www.atlassian.com/agile/scrum. [4] AWS Well-Architected. Amazon Web Services, Inc., 2019. [5] “Distributed computing,” Distributed Computing, 2020.

https://www.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.doc/dfhtm/topics/dfhtm09.html.

[6] S. Jignesh, “Vertical Scaling and Horizontal Scaling in AWS,” 2018. https://dzone.com/articles/vertical-scaling-and-horizontal-scaling-in-aws.

[7] Architecting on AWS Student book, Module 8., 6.4. AWS, 2019. [8] “Amazon SQS.” https://aws.amazon.com/sqs/. [9] “Functional Testing Types – Examples, Tips & More,” QA Symphony.

https://www.qasymphony.com/blog/functional-testing-types/. [10] Ficsor Lajos, Kovács László, Kusper Gábor, and Krizsán Zoltán,

Szoftvertesztelés. www.iit.uni-miskolc.hu, 2001. [11] Yotam Kadishay, “Tests Coverage is Dead — Long Live Mutation Testing,” Apps

Flyer, 2018. https://medium.com/appsflyer/tests-coverage-is-dead-long-live-mutation-testing-7fd61020330e.

[12] G. Ann Campbell, Cognitive Complexity. SonarQube, 2018. [13] “Fuzzing.” Wikipedia, [Online]. Available: https://en.wikipedia.org/wiki/Fuzzing. [14] Cisco, “Kitty documentation.” https://github.com/cisco-sas/kitty. [15] “Sulley fuzzer documentation,” OpenRCE. https://github.com/OpenRCE/sulley. [16] Mozzilla Security, “Peach fuzzer documentation.”

https://github.com/MozillaSecurity/peach. [17] “Burp Suite Documentation.”

https://portswigger.net/burp/documentation/desktop/tools/intruder/using. [18] “America Fuzzy Loop.” https://github.com/google/AFL. [19] Cisco SAS team, “Kitty - Framework Structure.” [Online]. Available:

https://kitty.readthedocs.io/en/latest/base_structure.html. [20] “Amazon Virtual Private Cloud (VPC).” https://aws.amazon.com/vpc/. [21] “Amazon RDS.” https://aws.amazon.com/rds/. [22] “Amazon DynamoDB.” https://aws.amazon.com/dynamodb/. [23] “Amazon S3.” https://aws.amazon.com/s3/. [24] “Amazon CloudFormation.” https://aws.amazon.com/cloudformation/. [25] “Primer on Python decorators,” RealPython.com, 2019.

https://realpython.com/primer-on-python-decorators/. [26] Google, “Protocol Buffer developer documentation,” Google Protocol Buffer.

https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/reflection/message-descriptor.

[27] D. Eastlake, E. Brunner-Williams, and B. Manning, RFC 2929 - Domain Name System (DNS) IANA Considerations. The Internet Society, IANA, 2000.

[28] “Let’s hand write DNS messages.” 2017, [Online]. Available: https://routley.io/posts/hand-writing-dns-messages/.

[29] Mel Liaguno, “Developers: Fuzzing, Symbolic Execution with Regression Testing Offers App Resilience,” The News Stack, 2019. https://thenewstack.io/developers-fuzzing-symbolic-execution-with-regressive-testing-offers-app-resilience/.


68

[30] Saahil Ognawala, Fabian Kilger, and Alexander Pretschner, Compositional Fuzzing Aided by Targeted Symbolic Execution. Faculty of Informatics, Technical University of Munich, 2019.


69

16. Printed appendix 16.1 Appendix 1.

Code Coverage of the application generated during unit tests with Python coverage library

Coverage report of parse_pb.py component


70

16.1 Appendix 2.

Fuzzer Framework high-level integration and data flow chart


71

16.2 Appendix 3.

Fuzz Framework On-Premise integration

.


72

16.4 Appendix 4.

API Documentation preview generated by Sphinx.

Documents

Design and develop a fuzzer framework to security test a