gsoc and grub4ext4

GSoC & grub4ext4

Peng Tao

BUPT—Broadband Network Research Center

Peng Tao (BUPT) Introduction to GSoC and grub4ext4 1 / 37

Agenda

1 Google Summer of Code

2 Google Summer of Code 2008

3 Grub4ext4: enable ext4 file system as boot partition


About Me

Peng Tao

Broadband Network Research Center, BUPT

Graduate at Apr,2010(expected)

[email protected]

Copyright c© 2008.

No rights reserved except that of others.

Bergwolf






What is Google Summer of Code?

Google Summer of Code (GSoC) is a program that offers student

developers stipends to write code for various open source projects.

Google will be working with several open source, free software, and

technology-related groups to identify and fund several projects over a

three month period.

Kicked off in 2005. In 2005, 2006 and 2007, the program has brought

together over 1,500 students with over 130 open source projects to create

millions of lines of code.







three month period.










three month period.





Goals of Google Summer of Code

Google Summer of Code has several goals:

Get more open source code created and released for the benefit of all;

Inspire young developers to begin participating in open source

development;

Help open source projects identify and bring in new developers and

committers;

Provide students in Computer Science and related fields the opportunity

to do work related to their academic pursuits (think ”flip bits, not

burgers”);

Give students more exposure to real-world software development

scenarios (e.g., distributed development, software licensing questions,

mailing-list etiquette).






development;


committers;



burgers”);









development;


committers;



burgers”);









development;


committers;



burgers”);









development;


committers;



burgers”);









development;


committers;



burgers”);





A Simple workflow

1. Open source projects who’d like to participate in Google Summer of

Code in 2008 should choose an organization administrator(s) to represent

them;

2. Organization administrators will submit the project’s application for

participation online;

3. Google will notify the organization administrators of acceptance, and

an account for the organization will be created in the Google Summer of

Code web app;

4. Students submit project proposals online to work with particular

mentoring organizations;


A Simple workflow



them;





Code web app;




A Simple workflow



them;





Code web app;




A Simple workflow



them;





Code web app;




A Simple workflow



them;





Code web app;




5. Mentoring organizations rank student proposals and perform any

other due diligence on their potential mentees; student proposals are

matched with a mentor;

6. Google allocates a particular number of student slots to each

organization;

7. Students are notified of acceptance;

8. Students begin learning more about their mentoring organization and

its community before coding work starts;






organization;









organization;









organization;





9. Students begin coding work at the official start of the program,

provided they’ve interacted well with their community up until the

program start date;

10. Mentors and students provide mid-term progress evaluations;

11. Mentors provide a final evaluation of student progress at close of

program; students submit a final review of their mentor and the

program;

12. Student who passes final evaluation uploads completed code to a

Google Code hosted project. Program close up.




program start date;




program;






program start date;




program;






program start date;




program;








Google Summer of Code 2008



GSoC 2008 was officially announced on February 25 and concluded on

September 2, 2008.

177 open source organizations with more than 3000 ideas.

over 1000 students from all over the world participated. Eighty-three

percent of them received passing final evaluations from their mentors and

communities, the highest rate of success in all four years running the

program.




September 2, 2008.





program.




September 2, 2008.





program.


GSoC 2008 organizations


GSoC 2008 organizations


openSUSE in GSoC 2008

openSUSE provided 30 project ideas this year. Ten of them became a

GSoC project and all ten projects passed final evaluation!


openSUSE mentors and students


GSoC 2008 and Me

Preparation: Started from last winter, I began to do the RTFC things. I

collected documents and manuals about GRUB legacy, got a slight idea

of how to patch GRUB from GRUB’s side.

Application: There was about two weeks to write my proposal(Officially

only one week, but there was one week’s delay). Communication is very

important while writing proposal, especially with the mentor.

Working on it: Reading kernel and e2fsprogs code, asking questions on

IRC, writing patch, bla bla. A lot of things happened.

Submitting: I submitted my code and documents to Coly Li on August

14.

The end: With Coly Li’s help, my patch is now in openSUSE 11.1 beta1.


GSoC 2008 and Me










14.



GSoC 2008 and Me










14.



GSoC 2008 and Me










14.



GSoC 2008 and Me










14.



GSoC 2008 and Me










14.







Introduction

The ext4 file system is merged into Linux kernel as ext4dev.

However, an ext4-formated partition can not be used as a boot partition,

because GRUB Legacy(referred to as grub in the following text), which is

the most popular bootloader nowadays, doesn’t yet support mounting

and reading ext4 partitions.

As part of Google Summer of Code 2008 program, grub4ext4 aims at

enabling gurb to boot on ext4 partitions.


Introduction

The ext4 file system is merged into Linux kernel as ext4dev.

However, an ext4-formated partition can not be used as a boot partition,

because GRUB Legacy(referred to as grub in the following text), which is

the most popular bootloader nowadays, doesn’t yet support mounting

and reading ext4 partitions.

As part of Google Summer of Code 2008 program, grub4ext4 aims at

enabling gurb to boot on ext4 partitions.


Introduction(ctd.)

Project home:

http://bergwolf.googlepages.com/grub4ext4

Packages:

http://download.opensuse.org/repositories/home:/bergwolf:/grub4ext4/


Linux VFS concepts

Generally, Linux Virtual File System(VFS) accomplishes following tasks:

1 Manage kernel level file abstractions in one format for all file systems

2 receive system calls from user level(e.g.write, open, stat, link)

3 Interact with a specific file system based on mount point traversal

4 receive request from other parts of the kernel, mostly from memory

management


Linux VFS concepts






management


Linux VFS concepts






management


Linux VFS concepts






management


Linux VFS concepts(ctd.)

VFS data structures:

1 VFS handle for the file: inode

2 user instantiated file handle: file

3 the whole filesystem metadata: superblock

4 a name to inode translation: dentry



Inodes:

Inode object is the kernel representation of the low level file

Inode has operations:

const struct inode operations *i op;

const struct file operations *i fop;

Inodes contain an FS specific information



File:

Store the process’s information about the file

File information: use count, flags, mode, owner, uid, gid, file operations,

etc.

File operations are copied from the i fop field of the inode object to the

file object by the kernel

Two file operation tables are maintained per FS: one for directories and

the other for regular files



Superblock:

Handel metadata only

Responsible for retrieving and storing metadata from FS media or peers

Struct superblock hold things like:

– block device, blocksize, filesystem type, flags, list of (inodes, dirty

inodes, etc) – super operations – wait queue – pointer to the root dentry

of the FS – etc...



Dentry:

The kernel uses dentries to represent the file system structure

Dentry is a name to inode translation structure

Dentry has Pointers to store the parent-child relationship of the files

Cached aggressively by VFS

Eliminate lookups by FS & private caches, faster lookups for NFS


From ext3 to ext4

There are four incompatible ext4 features against ext3 defined in

fs/ext4/ext4.h:

1 EXT4 FEATURE INCOMPAT EXTENTS

2 EXT4 FEATURE INCOMPAT 64BIT

3 EXT4 FEATURE INCOMPAT MMP(not implemented yet)

4 EXT4 FEATURE INCOMPAT FLEX BG

To enable grub to boot on ext4 filesystem is to provide grub with support for

these incompatible features.


From ext3 to ext4


fs/ext4/ext4.h:








From ext3 to ext4


fs/ext4/ext4.h:








From ext3 to ext4


fs/ext4/ext4.h:








From ext3 to ext4


fs/ext4/ext4.h:








From ext3 to ext4


fs/ext4/ext4.h:








EXT4 FEATURE INCOMPAT EXTENTS

Ext2/3 uses Indirect mapping to store data on disk.


extents

ext4 introduces a new on-disk data storage structure: extents.

ext2/3 indirect block maps are incredibly inefficient for large files

one extra block read per 1024 blocks

very obvious when moving or deleting big CD/DVD image files

an extent is a single descriptor for a chunk of contiguous blocks

efficient to represent large contiguous files

fewer metadata IO, better CPU utilization


extents

ext4 introduces a new on-disk data storage structure: extents.

ext2/3 indirect block maps are incredibly inefficient for large files

one extra block read per 1024 blocks

very obvious when moving or deleting big CD/DVD image files

an extent is a single descriptor for a chunk of contiguous blocks

efficient to represent large contiguous files

fewer metadata IO, better CPU utilization


on-disk extent format

12 bytes ext4 extent structure

48 bit physical block number to address 1EB filesystem

15 bit extent to store at most 128MB in a single extent

32 bit logical block number to address 16TB filesystem

struct ext4 extent {le32 ee block; /* first logical block extent covers */

le16 ee len; /* number of blocks covered by extent */

le16 ee start hi; /* high 16 bits of physical block */

le32 ee start lo; /* low 32 bits of physical block */

};


extent mapping

Ext4 uses extent mapping, which is an advantage when storing large

continues files.




EXT4 FEATURE INCOMPAT 64BIT

Currently(kernel 2.26.1), only changes in group descriptor are marked

with this feature. Group descriptor is fixed sized(32 bytes) in ext2/3 but can

be flexible in ext4(at least 64 bytes).


size of group descriptor

size of group descriptor is stored in s desc size of ext4 superblock which

is not present in ext2/3 superblock.

In grub4ext4, test EXT4 FEATURE INCOMPAT 64BIT bit to decide

whether to read s desc size.

#define EXT2 DESC SIZE(s)

(EXT4 HAS INCOMPAT FEATURE(s,EXT4 FEATURE INCOMPAT 64BIT)?

s–>s desc size : EXT4 MIN DESC SIZE)


size of group descriptor

size of group descriptor is stored in s desc size of ext4 superblock which

is not present in ext2/3 superblock.

In grub4ext4, test EXT4 FEATURE INCOMPAT 64BIT bit to decide

whether to read s desc size.

#define EXT2 DESC SIZE(s)

(EXT4 HAS INCOMPAT FEATURE(s,EXT4 FEATURE INCOMPAT 64BIT)?

s–>s desc size : EXT4 MIN DESC SIZE)


EXT4 FEATURE INCOMPAT FLEX BG

flex bg feature allows bitmaps and inode tables for a block group to be

placed anywhere on the storage media.

greatly decrease fsck time

grub doesn’t interfere with bitmaps but uses group desc structure to

access the inode table, so this feature is naturally supported.


EXT4 FEATURE INCOMPAT FLEX BG

flex bg feature allows bitmaps and inode tables for a block group to be

placed anywhere on the storage media.

greatly decrease fsck time

grub doesn’t interfere with bitmaps but uses group desc structure to

access the inode table, so this feature is naturally supported.


Solved problem 1

How about ext3 files that are left unchanged on ext4 file system?

There might be nonextent files on ext4 filesystem.

i flag in each file’s inode should be checked before mapping its logical block

number to physical block number in grub.


Solved problem 1

How about ext3 files that are left unchanged on ext4 file system?

There might be nonextent files on ext4 filesystem.

i flag in each file’s inode should be checked before mapping its logical block

number to physical block number in grub.


Solved problem 2

What if physical block number of a file exceeds 32bit limitation on a

48bit ext4 filesystem?

Physical block number is 64bit in ext4.

In grub4ext4, high 16 bits is checked to avoid reading files whose physical

block number exceeds 32bit limitation.

If a request for such files is received, grub will return ERR FILELENGTH

with error message ”Filesystem compatibility error, cannot read whole file”.


Solved problem 2

What if physical block number of a file exceeds 32bit limitation on a

48bit ext4 filesystem?

Physical block number is 64bit in ext4.

In grub4ext4, high 16 bits is checked to avoid reading files whose physical

block number exceeds 32bit limitation.

If a request for such files is received, grub will return ERR FILELENGTH

with error message ”Filesystem compatibility error, cannot read whole file”.


Thanks

grub4ext4 is originally Coly Li’s idea. Without his insightful guiding,

grub4ext4 would not be what it is today.

And special thanks goes to Aneesh Kumar, who have helped me a lot in

understanding new ext4 features.

Also, without Google Summer of Code program, this project would not

have been grown so quickly. Google, you rock!


Thank you!

Q & A

Peng Tao

Broadband Network Research Center

State Key Laboratory of Network and Switching Technology

Beijing University of Posts and Telecommunications

Beijing, 100876, China

[email protected]


mailto:[email protected]