56
CONFIDENTIAL. ALL RIGHTS RESERVED. Optio is a subsidiary of Allied Minds, an innovative U.S. science and technology development and commercialization company. Operating since 2006, Allied Minds forms, funds, manages and builds products and businesses based on innovative technologies developed at leading U.S. universities and federal research institutions. Allied Minds serves as a diversified holding company that supports its businesses and product development with capital, central Hamilton Turner Oct 2015 1 Fast-paced Introduction to Android Internals

Fast-paced Introduction to Android Internals

Embed Size (px)

Citation preview

Page 1: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Optio is a subsidiary of Allied Minds, an innovative U.S. science and technology development and commercialization company. Operating since 2006, Allied Minds forms, funds, manages and builds products and businesses based on innovative technologies developed at leading U.S. universities and federal research institutions. Allied Minds serves as a

diversified holding company that supports its businesses and product development with capital, central management and shared services. More information about the Boston-based company can be found at www.alliedminds.com.

Hamilton TurnerOct 2015

1

Fast-paced Introduction to Android Internals

Page 2: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

$ whoami

2

@hamiltont@hamiltont

• Director of Malware Research, OptioLabs– Machine Learning, Data Science, Penetration

Testing, and Malware Prevention• Doctorate in Computer Engineering

– Mobile & Cloud Computing, Distributed Software Engineering, Security & Privacy

• Heavy contributor to open source software• Android framework hacker for ~4 years

Page 3: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

$ who…are we

3

and now a brief message from our sponsors

OptioLabs– Mobile security company founded in 2011– Caters to Government, Enterprise, and Mobile Carriers/Manufacturers– Founders Dr. Charles Clancey, Dr. Jules White, Dr. Brian Dougherty– Offices in Baltimore & Nashville– Our OptioCore product increases Android security and enables legal compliance with

numerous regulations– Recent news:

• AMD and OptioLabs partner to increase enterprise security• Rugged Meets Secure: OptioCore Powers Sonim XP7

www.optiolabs.com

Page 4: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

$ who…are we

4

We are hiring!

Mobile Developer (Application & Framework)Server Developer (Backend & Frontend)DevOps

Send resumes to our VP of Engineering

www.optiolabs.com

Andy [email protected]

Page 5: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Slide Set EULA• Android is…..

– A capital-B Big project– A complex project– Changing rapidly– Different on each physical product

• This presentation is…..– not comprehensive– probably not fully correct– already outdated

• Go explore yourself!• Please let me know when you find something wrong with this slide deck

5

Page 6: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

What does it mean to be ‘Android’?• Android is a (loosely) protected term!

– AOSP is all the open source code– Stock ROM is what manufacturer ships– Vanilla is very close to AOSP– Android is ROM that passed Android Compatibility

• Android Compatibility– Different for each version of Android– 3 main parts

• Compatibility Test Suite (automated tests)• Compatibility Verifier (human tests)• Compatibility Definition Doc (manual)

• Android + Google Mobile Services (GMS)– Google Play! Maps! YouTube! Gmail!– All these are not open source and require a license agreement with Google if you want to sell a device that comes with these

https://source.android.com/compatibility/index.html

6

Page 7: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Is Android really Open Source?~90% of development is behind closed doors

– Only Google&partners see source code– Only Google&partners accept (or ignore/deny) patches

• Google is both the KeyMaster and the Architect– Very few public discussions occur on AOSP design principles– No public roadmap of features– However….

• Google accepts huge responsibility for standardization & long term stability of platform• Major contributions to upstream Linux kernel• Huge money influx for PR releases, OEM standardization efforts, etc

“Open source is different than a community-driven project. Android is light on the community-driven side and heavy on the open source. Everything we do ends up in the open source repository.” - Andy Rubin, Cofounder of Android Inc, SVP@Google until 2013

See https://source.android.com/source/code-lines.html#about-private-code-lines

7

Page 8: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Super basic legality of Android• Pre-note: I am not a lawyer. This slide is extremely simplified for explanation purposes and there is therefore incorrect by

design• Google is very careful to minimize “threats” to parties interested in modifying AOSP

– Actively avoids using components that require ‘derived works’ to be open sourced• E.g. heavy modders are very familiar with the busybox install process ;-)• Why Android doesn’t come with the well-tested, POSIX correct glibc?

– Actively avoids using components that *might, possibly* require licensing fees or agreements• Apache Harmony vs Oracle (sun) Java – see https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google,_Inc.

• When modifying Android, you should take a strong interest in the software licenses!– Smartphone IP is extremely valuable and heavily protected – see https://en.wikipedia.org/wiki/Smartphone_patent_wars

8

Component LicenseAndroid applications Up to developer

GSM App suiteAOSP External libraries Apache 2 BSD, misc

Android Kernel GPLv2Bootloader ???

Page 9: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android

• Often explained using the stack on the right

9

Page 10: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android

• Or these…

10http://blogs.freescale.com/mcus/2010/05/android-makes-the-move-to-power-architecture-technology/http://www.techdesignforums.com/edasource/images/68/esd1004_mentor1_large.jpg

Page 11: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android, Simplified

11http://blogs.freescale.com/mcus/2010/05/android-makes-the-move-to-power-architecture-technology/http://www.techdesignforums.com/edasource/images/68/esd1004_mentor1_large.jpg

Page 12: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android• The running software stack is critically important to understand

– However, the build system is quite obtuse– And documentation is sparse & scattered– One of your big challenges will be mapping running code to

• Source code• Android filesystem

• Always be on the lookout for the ‘bigger picture’– All 5 of these are representations of ‘Android’– Of the docs you can find, 90% will be on the software stack– 10% will be on the source code– The rest is up to you

Page 13: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Filesystem of Android device• Roughly derived from standard linux filesystem

– See https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard– Uses ramdisk, /sbin, /sys, /dev (sometimes)

• Android adds multiple root folders – /data, /system, /cache– These are basically an Android filesystem Hierarchy

• Build system enforces this consistent hierarchy– Many of the FHS folders are symlinks

• E.g. /etc /system/etc– Just enough of a linux filesystem to run a kernel and some helpful functions

• Heavy use of partitions– Reliable update mechanisms are easier with multiple bootable partitions– Updates can be made smaller

13

Page 14: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Filesystem Partitions• System

– Android OS– Preinstalled APKs– Shared libraries– Executables

• Userdata– User data, user apps– App-specific data folders– Backups– Dalvik caches– Crash information– Encrypted and/or persistent system data

• Cache– Simple writable, always trashable partition

• Recovery, Boot– Not first-class citizens while AOSP is running!

14

Page 15: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Filesystem Partitions

• Adding main symlinks– /etc, /tombstones,

/vendor• Note: Some phones

have distinct vendor.img partition

15

Page 16: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Filesystem Partitions• Interactions with Kernel• /dev is tmpfs

– Android uses udev (ueventd) to fill /dev

• /acct is used by cgroups– CPU, Memory, DiskIO accounting

and limiting• /proc and /sys are standard linux

mechanisms for userspace and kernel interaction

16

Page 17: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Useful tooling: Android shell• Android uses MirBSD Korn shell (since 4.0

ICS)• Utilities

– Gnu core utilities? GPL!– Busybox? GPL!

• Android toolbox? BSD– See `ls –l /system/bin`– Emulates ~75 standard tools (ls, rm, rmdir, top,

sleep)– See AOSP_ROOT/system/core/toolbox for

source

17

Page 18: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Physical Android Architecture

• Most Android phones have two processors– Mobile app processor– Mobile baseband processor– Communication using interrupts– Which one ‘leads’?– Frequently on same Soc

• Sensors/USB/Screen/modem– Sometimes on SoC, sometimes connected externally

18

Page 19: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Physical Android Hardware: Sensors• General problem: sensors normally require proprietary code

– To be business-friendly, AOSP tries very hard to avoid GPL-ing someone’s sensor code

– This unfortunately means that hardware access is *not* done using one standard mechanism

• Access mechanisms:– /dev nodes (display, lights)– Shared library objects (touch, audio)– Sockets– Android HAL

• Android specifies header files, manufacturer provides *.so files– D-Bus (until recently)

19

Page 20: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Acquiring Android Source Code• Google has comprehensive docs

– https://source.android.com/source/downloading.html• AOSP uses git for version control

– Git repositories become huge given too much time– AOSP has a lot of pieces (~450 different git repos)– Working with ~450 git repositories is a nightmare….

• Repo– Google tool to automate working across many git repos– Enable branch/fetch/status/etc in multiple git repositories with one command

$ repo init -u https://android.googlesource.com/platform/manifest -b android-4.0.1_r1

See https://source.android.com/source/build-numbers.html#source-code-tags-and-builds

20

Page 21: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Acquiring Android Source Code• In repo terms, each git repo is called a project• A project has

– A remote (e.g. url)– A path– A name

• Repo will clone your projects into ‘detached HEAD’– aka anonymous branch– You should use `repo start <project name>` to do a

git branch• Repo has a concept of project groups

– Note the ‘notdefault’ group

See https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.txt

21

<manifest> <remote name="aosp“ fetch="https://android. googlesource.com" />

<project path="packages/apps/Mms“ name="platform/packages/apps/Mms" /> … … <project path="prebuilts/eclipse-build-deps" name="platform/prebuilts/eclipse-build-deps" groups="notdefault,eclipse" /> </manifest>

$ cat .repo/manifest.xml

Page 22: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Uploading Android Source Code• Doing code reviews across ~450 git repos is a pain!• Enter ‘gerrit’

– Google developed a code review website that is specific to AOSP modifications

– Extremely structured process– Can be self-hosted– Integrated with ‘repo upload’

• Note: Your dev team does not have to use gerrit– You can use git directly to push code– Not quite as painless as ‘repo upload’, but you don’t

have to setup/maintain/use gerrit

22

Page 23: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

23

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

Page 24: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

24

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

You want to work with AOSP in any capacity

Page 25: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

25

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

You want to port AOSP to a really-small IoT kind of device

Page 26: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

26

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

You want to modify what it means to be ‘Dalvik’

Page 27: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

27

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

You want to add a new system service to Android

Page 28: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

28

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

You want to build a custom ROM to is AOSP with different images

Page 29: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

29

abi C++ support for ABIart Android ART runtime, compiler, dex2oat, etcbionic Android C standard librarybootable Android Recovery image (no bootloader)build Makefile-based AOSP build systemdalvik Dalvik VM source code, dex format spec, libdexdevelopers Source code examplesdevelopment Resources for setting up dev environments. Config files, small scripts, USB driversdevice Device-specific files and binariesdocs Doxyfile for source.android.comexternal External projects used by AOSPframeworks Android core componentshardware Android Hardware Abstraction Layer and hardware supportkernel Precompiled kernel images with Android modificationslibcore All java.* packages from Object.java down, as supplied by Apache Harmonylibnativehelper Android wrapper to make JNI easier to work withndk Native Development Kit source code & tools (for using C/C++ from APKs)out Build output directory. make clean ~= rm -rf outpackages Stock Android apps, Input methods, content providers, etcpdk Platform Development Kit for OEMs to test upcoming AOSP versionsprebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etcsdk Software Development Kit – eclipse, ADT, SdkManager, etcsystem Native components of Android that run on top of Linuxtools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-onsvendor OEM chipset binaries and device definitions

You want to use a different Bluetooth stack

Page 30: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Navigating Android Source Code

• godir is your friend!– It’s find and cd rolled into one command

30

Page 31: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Building Android Source Code

• Use make!

• I don’t know why that needed an entire slide…

31

Page 32: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Building Android Source Code1. Setup computer

See https://source.android.com/source/building.html2. Setup your shell

$ source build/envsetup.sh 3. Choose compile target

$ lunch aosp_arm-eng4. Compile – full compile takes hours

$ make5. Run emulator (or skip this and run on device instead)

$ emulator6. Flash device

$ adb reboot-bootloader $ fastboot –w flashall

32

Page 33: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Building Android Source Code1. Setup computer

See https://source.android.com/source/building.html2. Setup your shell

$ source build/envsetup.sh 3. Choose compile target

$ lunch aosp_arm-eng4. Compile – full compile takes hours

$ make5. Run emulator (or skip this and run on device instead)

$ emulator6. Flash device

$ adb reboot-bootloader $ fastboot –w flashall

33

What is this?

How does this work?

Page 34: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Build System – High Level• AOSP has ~40k lines of makefiles under /build and /device

– Important to grok the main points before diving in there!

• Comparisons to standard projects using make– Uses ‘Android.mk’, not ‘Makefile’– Heavy usage of ANDROID_* environment variables– All output (even intermediate) goes into ‘out’ folder

• rm –rf out/– Smallest buildable unit is a ‘module’

• Similar to standard make targets, but there are 1000s of these• While many ‘modules’ are named statically, even more are created dynamically!

– For example, every module “foo” automatically gets an accompanying module “foo-clean”

– Non-recursive build system• See http://aegis.sourceforge.net/auug97.pdf• See build/core/build-system.html

– You never call gcc yourself! Multiple ‘build templates’ exist• Android application, shared library, static library, standalone executable, java library, prebuilt• Build templates enforce the Android filesystem layout• Accept “parameters” in the form of make variables (will default parameters if these variables are not set)

34

Page 35: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Build System – High LevelTo build a module:

• Setup expected variables:LOCAL_PATH := $(call my-dir)LOCAL_MODULE_NAME := some_example

• Then include the proper build template:include $(BUILD_JAVA_LIBRARY)

The build template will read the variables you set, compile as expected (in this case, as expected for a java library), and place the intermediates and final output in the correct locations inside the out/ folder

35

Page 36: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Build System – High Level

• Build system includes lots of ‘dark magic’• Not recommended to just start writing an Android.mk

– Find one that does what you want and modify it– Or you’ll have forgotten a variable

• Look into build/core/ to find makefiles for each build template– All options listed at build/core/config.mk line ~75– Missing variables are often set to default values

36

Page 37: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

What is lunch?• AOSP has many compile options

– Set options with env variables or buildspec.mk

– envsetup.sh defines functions to help set options using env variables

– One helper is ‘lunch’– “I’m having JellyBeans for lunch”

• Just calling lunch presents a menu of the form:

$TARGET_PRODUCT-$TARGET_BUILD_VARIANT

37

Page 38: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Build Targets• Target product ~= device + configuration

– E.g. nexus 6 (aka shamu)– E.g. en keyboard/apps/tts– Device name is arbitrary

• Example products– aosp_arm: AOSP default locale on Emulator– full_fugu: All locales on Nexus Player– aosp_shamu: AOSP default on Nexus 6– See https://source.android.com/source/build-

numbers.html• To build your own, take a look at the different

AndroidProducts.mk and vendorsetup.sh scripts– See https://source.android.com/source/add-device.html

38

Page 39: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Build Variants & Type• Target_build_variant determines the default apps and settings

– user: Include stuff plan user would want– userdebug: Include user stuff, and a few items/settings to make

debugging easier– eng: Include userdebug stuff, user stuff, and item/settings an engineer

might use• Target_build_type

– You can set this to release or debug to change the settings of ConfigBuildFlags.DEBUG

– This is used to enable/disable some extra logging in core system services

39

Page 40: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Building Android Source Code1. Setup computer

See https://source.android.com/source/building.html2. Setup your shell

$ source build/envsetup.sh 3. Choose compile target

$ lunch aosp_arm-eng4. Compile – full compile takes hours

$ make5. Run emulator (or skip this and run on device instead)

$ emulator6. Flash device

$ adb reboot-bootloader $ fastboot –w flashall

40

Page 41: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Modifying a Single Module1. Modify code

$ gedit frameworks/base/core/java/com/android/internal/os/ZygoteInit.java2. Compile that module

Note: findmakefile function is helpful to discover the LOCAL_MODULE_NAME $ make framework

3. Rebuild the system.img file using special snod module $ make snod

4. Reflash $ adb reboot-bootloader $ fastboot –w flashall

41

Page 42: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Bootup (Roughly) Explained

42

Page 43: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android init.rc• Android uses it’s own language for the init files

– Easy to understand– I recommend you read init.rc

• Main primitives– service: names a command string execution. Distinct from other uses of word

‘service’• Start with ‘start <servicename>’• Can be grouped into ‘classes’• Start class with ‘class_start <classname>’

– action: on some trigger, do this• Predefined actions

– early-init, init, early-fs, fs, early-boot, boot• User-defined action: property setting changed

– trigger: something happened• ‘on <trigger>’ defines an action to react• ‘trigger <triggername>’ fires the trigger

• Location of init.rc files– In running system (e.g. in ramdisk) : /init.*rc– In build system: system/core/rootdir/*.rc and /device

43

Page 44: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Properties• OS-wide key-value properties

– Not the same as Java properties– Not environment variables– Follow a hierarchy of naming

• Use getprop and setprop tools– Note: There is actually a socket file, but its tricky to

access• Setprop can be used to trigger actions in init.rc!

– Start/stop native code– Set logging levels– Change properties that persist across boot

44

Page 45: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android Properties• Where do properties originally get set?!

– Filesystem (/default.prop, /system/build.prop)– Init.rc files– Java code – SystemProperties.set– C++ code – property_set

• Some properties are controlled by UID/GID permissions– E.g. ril.* properties can only be set by RADIO user

• Special cases– ro.* is read-only. These can only be set once (normally at boot)!– persist.* properties are written to the filesystem and persist across reboots– ctl.* properties are not actually stored into the properties map. They enable you to start or stop an init.rc ‘service’ using the

properties system• E.g. ‘ctl.start foobar’ would start service ‘foobar’

45

Page 46: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

What’s a zygote?• At a native level, binary is called app_process• Encapsulates starting the Dalvik VM• Eases the use of JNI• Loads all Java libraries into itself• Is capable of being forked• Drastically speeds up creation of new, ready-

to-go processes pre-loaded with the Dalvik VM and java libraries

46

Page 47: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android System Server• Note that only one bit of Android talks to

the Hardware & Kernel• This is system server

– Multiplexes access to single-user systems like the display

– Does most of the complex book-keeping– Accepts remote messages from apps and

acts on their behalf (e.g. send SMS)• Most ‘Android System Services’ run as a

thread inside the system_server process– Although a number run as standalone apps

with the persistent=true flag

47

Page 48: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Main Android Services• Some biggies:

– WifiService– ActivityManagerService– PowerService– PackageManagerService– LocationManagerService

• There are about ~90 more– Individually they are not too bad– Look in source code under frameworks/base/services– Don’t start with ActivityManager ;-)

• Developers access these using Context.getSystemService• How does this all happen?

48

Page 49: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android IPC Architecture

“In the Android platform, the binder is used for nearly everything thathappens across processes in the core platform.” - Dianne Hackborn, Android Framework Engineer @ Google - https://lkml.org/lkml/2009/6/25/3

I highly recommend: https://newcircle.com/s/post/1340/deep_dive_into_android_ipc_binder_framework_at_andevcon_iv

49

Page 50: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android IPC Basics

• Android app calls into android.jar• In many places, android.jar wants

to call some core framework service– Proxy software design pattern!

• To send data through the Binder driver, you have to write data into a string– Marshalling/unmarshalling

50

https://i.stack.imgur.com/EkzyV.png

Page 51: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android IPC Basics

• Android app calls into android.jar• In many places, android.jar wants

to call some core framework service– Proxy software design pattern!

• To send data through the Binder driver, you have to write data into a string– Marshalling/unmarshalling

51

https://i.stack.imgur.com/EkzyV.png

Page 52: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android IPC Basics

• AIDL files are used to generate the Java source code for Proxy– Exactly the same as you can

do in an Android application!• Very useful to read

generated Proxy & Stub files

52

https://i.stack.imgur.com/EkzyV.png

Page 53: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Android: Files to Read

• Some source code files are important enough to actually read…

• Zygote, ZygoteInit• SystemServer• Init.rc

57

Page 54: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

How is Android not Linux?• ueventd vs udev

– Check for devices showing up and populate /dev• Kernel changes!

– Android Out of Memory (OOM) killer – Wakelocks – Binder RPC / IPC– Android’s Anonymous Shared Memory (ashmem)– Android Alarm (on top oc Linux’s Real time clock RTC)– Android’s ring logger mechamisms– Custom init.rc language– Active work to upstream most of these changes

• https://events.linuxfoundation.org/images/stories/slides/abs2013_stultz.pdf• Android Upstreaming group - https://wiki.linaro.org/WorkingGroups/Kernel/AndroidUpstreaming

• C standard library– Bionic vs glibc. Seems this is mainly for licensing reasons

• User management (e.g. uid/gid mapping)– Linux uses UID/GID. Android uses these for apps, and manages users in the AOSP code

• Hardware approach– Linux mainly uses /dev for hardware exposure– Android uses mostly shared object files

60

Page 55: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

Resources• Embedded Android

– Really excellent for everything below the AOSP java code• New Android Book –

– Good for Android AOSP code – E.g. ‘How does the PowerManager code work’

• Exploring SDK Add-ons– Very interesting talk covering some of the interaction between Google and OEMs– https://www.youtube.com/watch?v=1TY5rqyAyp4

• Deep Dive into Android IPC/Binder Framework at AnDevCon IV– Binder is critical to Android, more understanding == better!– https://newcircle.com/s/post/1349/andevcon_iv_video_presentations_android_developers– https://

newcircle.com/s/post/1340/deep_dive_into_android_ipc_binder_framework_at_andevcon_iv

• Binder IPC talk– http://www.dre.vanderbilt.edu/~schmidt/cs282/PDFs/android-binder-ipc.pdf

61

Page 56: Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015

$ who…are we

62

We are hiring!

Mobile Developer (Application & Framework)Server Developer (Backend & Frontend)DevOps

Send resumes to our VP of Engineering

www.optiolabs.com

Andy [email protected]