Mirrors Primary (US) Issues October 2001

November 2001 Get BSD Contact Us Search BSD FAQ New to BSD?

DN Print Magazine BSD News BSD Mall BSD Support Source Wars Join Us

T H I S M O N T H ’ S F E A T U R E S

Bootstrapping Vinum: A Foundation for Reliable Serversby Robert A. Van Valzah

Any machine that is going to provide reliable service needsto have either redundant components on-line or a pool ofoff-line spares that can be promptly swapped in.Commodity PC hardware makes it affordable for evensmall organizations to have some spare parts available thatcould be pressed into service following the failure ofproduction equipment. These instructions show how tobuild a pair of disk drives where either one is adequate tokeep your server running if the other fails. Life is better ifthey are both working, but your server will never die unlessboth disk drives die at once. Read More

Growing FreeBSD filesystemsby Marc Fonvieille

Until today it was impossible to expand the size of afilesystem. FreeBSD 4.4-RELEASE comes with a newutility: growfs(8). This program permits us to enlarge, incertains conditions, the size of an existing ufs filesystem.Read More

From the Editor

Editorialby Chris ColemanWe announced last monththat we were producingFreeBSD starting withrelease 4.5. We are gearingup to add a few moreservices, FreeBSD trainingbeing one of the first.

Get BSD Stuff

Search

All of Daemon NewsSearch

Configuring IPSec on OpenBSD 2.9by Robert V. Sigillito

We began working with OpenBSD v2.9 to implement aVPN solution for little cost in order to spare a handful ofusers a 35 mile drive to another facility. Why drive out ofyour way, when you can use a public network to pass yourdata in a fully encrypted private tunnel? Being new toOpenBSD, but not Unix, we made more than our share ofmistakes. This guide is an attempt to document how wesuccessfully configured IPsec on OpenBSD v2.9. ReadMore

Logging Syslog to a Databaseby Zbyszek Sobiecki

When there’s a problem on your system, or in yournetwork, the first thing you check are system logs. Youidentify which system you should check, then locate thelogfile. Sometimes you even have to check your syslogconfiguration, only to discover that what you are lookingfor is not even logged, due to a misconfiguration. You mayalso run ’less’, ’more’ and ’grep’, to start digging into it.It’s nice when you find the answer to your problem in thelast few lines of log, but what if you can’t? Read More

IPv6 Trendsby Girish B Hampali

Girish B. Hampali gives us a quick little article on IPv6. Itexplains a vew of the reasons why we need IPv6 and is avery good starting point for exploring IPv6. It also gives afairly complete list of RFCs relating to IPv6. Read More

R E G U L A R C O L U M N S

Answermanby Todd Whitesel

Daily Daemon News

BSD == Christian &Biblical Software ? Beastie Jack-O-Lantern(andTux) OpenBSD 3.0 CDPre-Orders Being Accepted DN Magazine Issue 5Available Re-drawn Dixie T-Shirt OSNews InterviewsxMach’s Joseph Mallett FreeBSD Handbook, 2ndEdition now available Secure Communicationswith OpenSSH Run your render farm onNetBSD Mac OS X - A System forthe Post-Windows World

BSD Support Forum

ppp_nat vs. natd MySQL reinstall question Error when using vi withineterm Ftp, Connection closed byremote host. Where’d my X fonts go?

Source Wars

Week 22

Last column’s mailbag has been crushed by a witheringbarrage of answers from our readers. This time Todd takeson questions like: "How do I save my screen arrangementfor the window program?", "I’m trying to get taylor uucpworking on Mac OS X, help!", and "Tracking down culpritof spurious dialing with ppp -demand." Read More

Daemon’s Advocateby Wes Peters

I have been closely following the development of a numberof email clients for several years now, anxiously awaitingone that would allow me to ditch Netscape Communicatorfor the limited and crash-prone rusty tool it is. I’ve beeninstalling, trying, and deleting email clients so frequentlyover the past year my home directory on Homer becamecluttered with various and sundry email configurations,carefully moved away and saved as I tried the next client.Read More

Daemon News Mall

FreeBSD 4.5 Next ReleaseSubscription - $24.95 FreeBSD 4.4 Now Shipping- $35 FreeBSD-in-a-Box NowShipping! NetBSD 1.5.2 NowShipping - $23.50 Pre-Order FreeBSDUnleashed - $34.50 FreeBSD for your PC -$22.95 Pre-Order BSD Statuette -$65.95 Need Reseller Pricing - Goto Cylogistics

Miscellaneous

CreditsThe hard-workingcrew TarballDownload a tar.gzversion of this issue PDFDownload a PDFversion of this issue

Copyright © 1998-2001 DæmonNews. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

New BSD Products and Services

Chris Coleman, <editor@daemonnews.org>

October has been quite a busy month for Daemon News. We have been working as hard as we canto add new BSD products and services. But, we aren’t stopping here. We are determined to supportBSD in every way we can!

Stuff you should already know

One of the biggest announcements made is that Daemon News will be producing a 4 CD set ofFreeBSD 4.5. These will be made using the official FreeBSD project ISO images, packaged in astandard jewel case.

At the same time, we announced that you could purchase these on a subscription basis. Due topopular demand, we also added "Next Release" CD subscriptions so that you could start thesubscription with FreeBSD 4.5 instead of the currently shipping 4.4 version.

The latest release of Darwin, version 1.4.1, was released by Apple and the CDs are currently at thereplicators. This is a 2-CD set in a jewel case that is produced by Daemon News. We have beenproducing the CD sets since Darwin 1.2.1.

While looking through our online mall, I noticed we carried lots of BSD CDs and Books, but hardlyany BSD clothing. So, we have made it one of our goals to offer more BSD apparel. To start thingsoff, we added a new NetBSD T-shirt. We had also run out of the original, very popular DixieT-Shirt, so we took this opportunity to redraw it, and I think you will really like it.

At the same time, we’ve also released another issue of the Daemon News Print Magazine--Issue 5.It is full of lots of original articles and new, really cool artwork. The cover alone lets you know theamount of time we put into the design. Susannah and Seth have outdone themselves again.

New Stuff

Daemon Crossing T-Shirt

For as long as we’ve been making t-shirts, people have asked for a black one.We hadn’t been able to think of a good design for it, until now. The sign on thecover of the Issue #5 looks great on black. After many people requested it, wegot the hint and made it available for order.

FreeBSD Training Classes

Daemon News is proud to announce that we will be offering FreeBSD trainingclasses. We have already scheduled the first set, and you can sign upimmediately. The first set will be offered in Santa Clara, California, in January 2002. We haveplans to host future classes in additional locations, and possibly offer training for other BSDs aswell. We are committed to supporting BSD as a whole, but are being practical about what we canaccomplish at present.

The first class is a one day introduction to FreeBSD and covers most of the basics required to installand use FreeBSD. You will receive a copy of FreeBSD: An Open Source OS for your PC byAnnelise Anderson.

The second class is a four day course covering FreeBSD System Administration. You will receive acopy of FreeBSD Unleashed by SAMS publishing. There will be lots of hands-on experience andeach attendee will have access to a system. Space is limited to about 20 people, so order soonbefore space fills up. We do have early registration discounts.

We have lots more that we are working on that we can’t announce at this point, but rest assured thatwe are working as hard as possible to promote and support BSD. The support of the community iswhat makes this possible.

Thanks.

-Chris

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

Bootstrapping Vinum: A Foundation for Reliable Servers

Robert A. Van Valzah

In the most abstract sense, these instructions show how to build a pair of disk driveswhere either one is adequate to keep your server running if the other fails. Life is betterif they are both working, but your server will never die unless both disk drives die atonce. If you choose ATAPI drives and use a fairly generic kernel, you can be confidentthat either of these drives can be plugged into most any main board to produce aworking server in a pinch. The drives need not be identical. These techniques workequally well with SCSI drives as they do with ATAPI, but I will focus on ATAPI herebecause main boards with this interface are ubiquitous. After building the foundation ofa reliable server as shown here, you can expand to as many disk drives as necessary tobuild the failure-resilient server of your dreams.

1. IntroductionAny machine that is going to provide reliable service needs to have either redundant componentson-line or a pool of off-line spares that can be promptly swapped in. Commodity PC hardwaremakes it affordable for even small organizations to have some spare parts available that could bepressed into service following the failure of production equipment. In many organizations, a failedpower supply, NIC, memory, or main board could easily be swapped with a standby in a matter ofminutes and be ready to return to production work.

If a disk drive fails, however, it often has to be restored from a tape backup. This may take manyhours. With disk drive capacities rising faster than tape drive capacities, the time needed to restore afailed disk drive seems to increase as technology progresses.

Vinum is a volume manager for FreeBSD that provides a standard block I/O layer interface to thefile system code just as any hardware device driver would. It works by managing partitions of typevinum and allows you to subdivide and group the space in such partitions into logical devices calledvolumes that can be used in the same way as disk partitions. Volumes can be configured forresilience, performance, or both. Experienced system administrators will immediately recognize thebenefits of being able to configure each file system to match the way it is most often used.

In some ways, Vinum is similar to ccd(4), but it is far more flexible and robust in the face offailures. It is only slightly more difficult to set up than ccd(4). ccd(4) may meet your needs if youare only interested in concatenation.

1.1. Terminology

Discussion of storage management can get very tricky simply because of the terminology involved.As we will see below, the terms disk, slice, partition, subdisk, and volume each refer to differentthings that present the same interface to a kernel function like swapping. The potential forconfusion is compounded because the objects that these terms represent can be nested inside eachother.

I will refer to a physical disk drive as a spindle. A partition here means a BSD partition asmaintained by disklabel. It does not refer to slices or BIOS partitions as maintained by fdisk.

1.2. Vinum Objects

Vinum defines a hierarchy of four objects that it uses to manage storage (see Figure 1). Differentcombinations of these objects are used to achieve failure resilience, performance, and/or extracapacity. I will give a whirlwind tour of the objects here--see the Vinum web site for a morethorough description.

Figure 1. Vinum Objects and Architecture

The top object, a vinum volume, implements a virtual disk that provides a standard block I/O layerinterface to other parts of the kernel. The bottom object, a vinum drive, uses this same interface torequest I/O from physical devices below it.

In between these two (from top to bottom) we have objects called a vinum plex and a vinumsubdisk. As you can probably guess from the name, a vinum subdisk is a contiguous subset of thespace available on a vinum drive. It lets you subdivide a vinum drive in much the same way that adisk BSD partition lets you subdivide a BIOS slice.

A plex allows subdisks to be grouped together making the space of all subdisks available as a singleobject.

A plex can be organized with its constituent subdisks concatenated or striped. Both organizationsare useful for spreading I/O requests across spindles since plexes reside on distinct spindles. Astriped plex will switch spindles each time a multiple of the strip size is reached. A concatenatedplex will switch spindles only when the end of a subdisk is reached.

An important characteristic of a Vinum volume is that it can be made up of more than one plex. Inthis case, writes go to all plexes and a read may be satisfied by any plex. Configuring two or moreplexes on distinct spindles yields a volume that is resilient to failure.

Vinum maintains a configuration that defines instances of the above objects and the way they arerelated to each other. This configuration is automatically written to all spindles under Vinummanagement whenever it changes.

1.3. Vinum Volume/Plex Organization

Although Vinum can manage any number of spindles, I will only cover scenarios with two spindleshere for simplification. See Table 1 to see how two spindles organized with Vinum compare to twospindles without Vinum.

Table 1. Characteristics of Two Spindles Organized with Vinum

Organization Total Capacity FailureResilient

Peak ReadPerformance

Peak WritePerformance

ConcatenatedPlexes

Unchanged, butappears as a singledrive

No Unchanged Unchanged

Striped Plexes(RAID-0)

Unchanged, butappears as a singledrive

No 2x 2x

Mirrored Volumes(RAID-1)

1/2, appearing as asingle drive

Yes 2x Unchanged

Table 1 shows that striping yields the same capacity and lack of failure resilience as concatenation,but it has better peak read and write performance; hence we will not be using concatenation in anyof the examples here. Mirrored volumes provide the benefits of improved peak read performanceand failure resilience--but this comes at a loss in capacity.

Note: Both concatenation and striping bring their benefits over a single spindle at thecost of increased likelyhood of failure since more than one spindle is now involved.

When three or more spindles are present, Vinum also supports rotated, block-interleaved parity(also called RAID-5) that provides better capacity than mirroring (but not quite as good as striping),better read performance than both mirroring and striping, and good failure resilience. There is,however, a substantial decrease in write performance with RAID-5. Most of the benefits becomemore pronounced with five or more spindles.

The organizations described above may be combined to provide benefits that no single organizationcan match. For example, mirroring and striping can be combined to provide failure-resilience withvery fast read performance.

1.4. Vinum History

Vinum is a standard part of even a "minimum" FreeBSD distribution and it has been standard since3.0-RELEASE. The official pronunciation of the name is VEE-noom.

Vinum was inspired by the Veritas Volume Manager, but was not derived from it. The name is aplay on that history and the Latin adage In Vino Veritas (Vino is the accusative form of Vinum).Literally translated, "Truth lies in wine" hinting that drunkards have a hard time lying.

I have been using it in production on six different servers for over two years with no data loss. Likethe rest of FreeBSD, Vinum provides "rock-stable performance." (On a personal note, I have seenVinum panic when I misconfigured something, but I have never had any trouble in normaloperation.) Greg Lehey wrote Vinum for FreeBSD, but he is seeking help in porting it to NetBSDand OpenBSD.

WarningJust like the rest of FreeBSD, Vinum is undergoing continuous development.Several subtle but significant bugs have been fixed in recent releases. It is always bestto use the most recent code base that meets your stability requirements.

1.5. Vinum Deployment Strategy

Vinum, coupled with prudent partition management, lets you keep "warm-spare" spindles on-lineso that failures are transparent to users. Failed spindles can be replaced during regular maintenanceperiods or whenever it is convenient. When all spindles are working, the server benefits fromincreased performance and capacity.

Having redundant copies of your home directory does not help you if the spindle holding root,/usr, or swap fails on your server. Hence I focus here on building a simple foundation for afailure-resilient server covering the root, /usr, /home, and swap partitions.

Warning:Vinum mirroring does not remove the need for making backups! Mirroringcannot help you recover from site disasters or the dreaded rm -r -f / command.

1.6. Why Bootstrap Vinum?

It is possible to add Vinum to a server configuration after it is already in production use, but this ismuch harder than designing for it from the start. Ironically, Vinum is not supported by/stand/sysinstall and therefore you cannot install /usr right onto a Vinum volume.

Note: Vinum currently does not support the root file system (this feature is indevelopment).

It is a bit tricky to get started using Vinum, but these instructions take you though the process ofplanning for Vinum, installing FreeBSD without it, and then beginning to use it.

I have come to call this whole process "bootstrapping Vinum." That is, the process of gettingVinum initially installed and operating to the point where you have met your resilience orperformance goals. My purpose here is to document a Vinum bootstrapping method that I havefound works well for me.

1.7. Vinum Benefits

The server foundation scenario I have chosen here allows me to show you examples of configuringfor resilience on /usr and /home. Yet Vinum provides benefits other than resilience--namelyperformance, capacity, and manageability. It can significantly improve disk performance(especially under multi-user loads). Vinum can easily concatenate many smaller disks to producethe illusion of a single larger disk (but my server foundation scenario does not allow me to illustratethese benefits here).

For servers with many spindles, Vinum provides substantial benefits in volume management,particularly when coupled with hot-pluggable hardware. Data can be moved from spindle to spindlewhile the system is running without loss of production time. Again, details of this will not be givenhere, but once you get your feet wet with Vinum, other documentation will help you do things likethis. See "The Vinum Volume Manager" for a technical introduction to Vinum, vinum(8) for adescription of the vinum command, and vinum(4) for a description of the vinum device driver andthe way Vinum objects are named.

Note: Breaking up your disk space into smaller and smaller partitions has the benefit ofallowing you to "tune" for the most common type of access and tends to keep disk hogs"within their pens." However, it also causes some loss in total available disk space dueto fragmentation.

1.8. Server Operation in Degraded Mode

Some disk failures in this two-spindle scenario will result in Vinum automatically routing all diskI/O to the remaining good spindle. Others will require brief manual intervention on the console to

configure the server for degraded mode operation and a quick reboot. Other than actual hardwarerepairs, most recovery work can be done while the server is running in multi-user degraded mode sothere is as little production impact from failures as possible.

I give the instructions in Section 4 needed to configure the server for degraded mode operation inthose cases where Vinum cannot do it automatically. I also give the instructions needed to return tonormal operation once the failed hardware is repaired. You might call these instructions Vinumfailure recovery techniques.

I recommend practicing using these instructions by recovering from simulated failures. For eachfailure scenario, I also give tips below for simulating a failure even when your hardware is workingwell. Even a minimum Vinum system as described in Section 1.10 below can be a good place toexperiment with recovery techniques without impacting production equipment.

1.9. Hardware RAID vs. Vinum (Software RAID)

Manual intervention is sometimes required to configure a server for degraded mode because Vinumis implemented in software that runs after the FreeBSD kernel is loaded. One disadvantage of suchsoftware RAID solutions is that there is nothing that can be done to hide spindle failures from theBIOS or the FreeBSD boot sequence. Hence the manual reconfiguration of the server for degradedoperation mentioned above just informs the BIOS and boot sequence of failed spindles. HardwareRAID solutions generally have an advantage in that they require no such reconfiguration sincespindle failures are hidden from the BIOS and boot sequence.

Hardware RAID, however, may have some disadvantages that can be significant in some cases:

The hardware RAID controller itself may become a single point of failure for the system.

The data is usually kept in a proprietary format so that a disk drive cannot be simply pluggedinto another main board and booted.

You often cannot mix and match drives with different sizes and interfaces.

You are often limited to the number of drives supported by the hardware RAID controller(typically only four or eight).

In other words, Vinum may offer advantages in that there is no single point of failure, the drivescan boot on most any main board, and you are free to mix and match as many drives usingwhatever interface you choose.

Tip: Keep your kernel fairly generic (or at least keep /kernel.GENERIC around). Thiswill improve the chances that you can come back up on "foreign" hardware morequickly.

The pros and cons discussed above suggest that the root file system and swap partition are goodcandidates for hardware RAID if available. This is especially true for servers where it is difficult foradministrators to get console access (recall that this is sometimes required to configure a server fordegraded mode operation). A server with only software RAID is well suited to office and homeenvironments where an administrator can be close at hand.

Note: A common myth is that hardware RAID is always faster than software RAID.Since it runs on the host CPU, Vinum often has more CPU power and memoryavailable than a dedicated RAID controller would have. If performance is a primeconcern, it is best to benchmark your application running on your CPU with yourspindles using both hardware and software RAID systems before making a decision.

1.10. Hardware for Vinum

These instructions may be timely since commodity PC hardware can now easily host severalhundred megabytes of reasonably high-performance disk space at a low price. Many disk drivemanufactures now sell 7,200 RPM disk drives with quite low seek times and high transfer ratesthrough ATA-100 interfaces, all at very attractive prices. Four such drives, attached to a suitablemain board and configured with Vinum and prudent partitioning, yields a failure-resilient, highperformance disk server at a very reasonable cost.

However, you can indeed get started with Vinum very simply. A minimum system can be as simpleas an old CPU (even a 486 is fine) and a pair of drives that are 500 MB or more. They need not bethe same size or even use the same interface (i.e., it is fine to mix ATAPI and SCSI). So get busyand give this a try today! You will have the foundation of a failure-resilient server running in anhour or so!

2. Bootstrapping PhasesGreg Lehey suggested this bootstrapping method. It uses knowledge of how Vinum internallyallocates disk space to avoid copying data. Instead, Vinum objects are configured so that theyoccupy the same disk space where /stand/sysinstall built file systems. The file systems are thusembedded within Vinum objects without copying.

There are several distinct phases to the Vinum bootstrapping procedure. Each of these phases ispresented in a separate section below. The section starts with a general overview of the phase andits goals. It then gives example steps for the two-spindle scenario presented here and advice on howto adapt them for your server. (If you are reading for a general understanding of Vinumbootstrapping, the example sections for each phase can safely be skipped.) The remainder of thissection gives an overview of the entire bootstrapping process.

Phase 1 involves planning and preparation. We will balance requirements for the server againstavailable resources and make design tradeoffs. We will plan the transition from no Vinum toVinum on just one spindle, to Vinum on two spindles.

In phase 2, we will install a minimum FreeBSD system on a single spindle using partitions of type4.2BSD (regular UFS file systems).

Phase 3 will embed the non-root file systems from phase 2 in Vinum objects. Note that Vinum willbe up and running at this point, but it cannot yet provide any resilience since it only has one spindle

on which to store data.

Finally in phase 4, we configure Vinum on a second spindle and make a backup copy of the rootfile system. This will give us resilience on all file systems.

2.1. Bootstrapping Phase 1: Planning and Preparation

Our goal in this phase is to define the different partitions we will need and examine theirrequirements. We will also look at available disk drives and controllers and allocate partitions tothem. Finally, we will determine the size of each partition and its use during the bootstrappingprocess. After this planning is complete, we can optionally prepare to use some tools that will makebootstrapping Vinum easier.

Several key questions must be answered in this planning phase:

What file system and partitions will be needed?

How will they be used?

How will we name each spindle?

How will the partitions be ordered for each spindle?

How will partitions be assigned to the spindles?

How will partitions be configured? Resilience or performance?

What technique will be used to achieve resilience?

What spindles will be used?

How will they be configured on the available controllers?

How much space is required for each partition?

2.1.1. Phase 1 Example

In this example, I will assume a scenario where we are building a minimal foundation for afailure-resilient server. Hence we will need at least root, /usr, /home, and swap partitions. Theroot, /usr, and /home file systems all need resilience since the server will not be much goodwithout them. The swap partition needs performance first and generally does not need resiliencesince nothing it holds needs to be retained across a reboot.

2.1.1.1. Spindle Naming

The kernel would refer to the master spindle on the primary and secondary ATA controllers as/dev/ad0 and /dev/ad2 respectively. [1] But Vinum also needs to have a name for each spindlethat will stay the same name regardless of how it is attached to the CPU (i.e., if the drive moves, theVinum name moves with the drive).

Some recovery techniques documented below suggest moving a spindle from the secondary ATAcontroller to the primary ATA controller. (Indeed, the flexibility of making such moves is a keybenefit of Vinum especially if you are managing a large number of spindles.) After such adrive/controller swap, the kernel will see what used to be /dev/ad2 as /dev/ad0 but Vinum willstill call it by whatever name it had when it was attached to /dev/ad2 (i.e., when it was "created" orfirst made known to Vinum).

Since connections can change, it is best to give each spindle a unique, abstract name that gives nohint of how it is attached. Avoid names that suggest a manufacturer, model number, physicallocation, or membership in a sequence (e.g. avoid names like upper, lower, etc., alpha, beta, etc.,SCSI1, SCSI2, etc., or Seagate1, Seagate2 etc.). Such names are likely to lose their uniqueness orget out of sequence someday even if they seem like great names today.

Tip: Once you have picked names for your spindles, label them with a permanentmarker. If you have hot-swappable hardware, write the names on the sleds in which thespindles are mounted. This will significantly reduce the likelihood of error when youare moving spindles around later as part of failure recovery or routine systemmanagement procedures.

In the instructions that follow, Vinum will name the root spindle YouCrazy and the rootbackspindle UpWindow. I will only use /dev/ad0 when I want to refer to whichever of the two spindlesis currently attached as /dev/ad0.

2.1.1.2. Partition Ordering

Modern disk drives operate with fairly uniform areal density across the surface of the disk. Thatimplies that more data is available under the heads without seeking on the outer cylinders than onthe inner cylinders. We will allocate partitions most critical to system performance from these outercylinders as /stand/sysinstall generally does.

The root file system is traditionally the outermost, even though it generally is not as critical tosystem performance as others. (However, root can have a larger impact on performance if itcontains /tmp and /var as it does in this example.) The FreeBSD boot loaders assume that the rootfile system lives in the a partition. There is no requirement that the a partition start on the outermostcylinders, but this convention makes it easier to manage disk labels.

Swap performance is critical so it comes next on our way toward the center. I/O operations heretend to be large and contiguous. Having as much data under the heads as possible avoids seekingwhile swapping.

With all the smaller partitions out of the way, we finish up the disk with /home and /usr. Accesspatterns here tend not to be as intense as for other file systems (especially if there is an abundantsupply of RAM and read cache hit rates are high).

If the pair of spindles you have are large enough to allow for more than /home and /usr, it is fine toplan for additional file systems here.

2.1.1.3. Assigning Partitions to Spindles

We will want to assign partitions to these spindles so that either can fail without loss of data on filesystems configured for resilience.

Reliability on /usr and /home is best achieved using Vinum mirroring. Resilience will have tocome differently, however, for the root file system since Vinum is not a part of the FreeBSD bootsequence. Here we will have to settle for two identical partitions with a periodic copy from theprimary to the backup secondary.

The kernel already has support for interleaved swap across all available partitions so there is noneed for help from Vinum here. /stand/sysinstall will automatically configure /etc/fstab forall swap partitions given.

The Vinum bootstrapping method given below requires a pair of spindles that I will call the rootspindle and the rootback spindle.

Important: The rootback spindle must be the same size or larger than the root spindle.

These instructions first allocate all space on the root spindle and then allocate exactly that amountof space on a rootback spindle. (After Vinum is bootstrapped, there is nothing special about eitherof these spindles--they are interchangeable.) You can later use the remaining space on the rootbackspindle for other file systems.

If you have more than two spindles, the bootvinum Perl script and the procedure below will helpyou initialize them for use with Vinum. However you will have to figure out how to assignpartitions to them on your own.

2.1.1.4. Assigning Space to Partitions

For this example, I will use two spindles: one with 4,124,673 blocks (about 2 GB) on /dev/ad0 andone with 8,420,769 blocks (about 4 GB) on /dev/ad2.

It is best to configure your two spindles on separate controllers so that both can operate in paralleland so that you will have failure resilience in case a controller dies. Note that mirrored volumewrite performance will be halved in cases where both spindles share a controller that requires theyoperate serially (as is often the case with ATA controllers). One spindle will be the master on theprimary ATA controller and the other will be the master on the secondary ATA controller.

Recall that we will be allocating space on the smaller spindle first and the larger spindle second.

2.1.1.5. Assigning Partitions on the Root Spindle

We will allocate 200,000 blocks (about 93 MB) for a root file system on each spindle(/dev/ad0s1a and /dev/ad2s1a). We will initially allocate 200,265 blocks for a swap partition oneach spindle, giving a total of about 186 MB of swap space (/dev/ad0s1b and /dev/ad2s1b).

Note: We will lose 265 blocks from each swap partition as part of the bootstrappingprocess. This is the size of the space used by Vinum to store configuration information.The space will be taken from swap and given to a vinum partition but will beunavailable for Vinum subdisks.

Note: I have done the partition allocation in nice round numbers of blocks just toemphasize where the 265 blocks go. There is nothing wrong with allocating space inMB if that is more convenient for you.

This leaves 4,124,673 - 200,000 - 200,265 = 3,724,408 blocks (about 1,818 MB) on the root spindlefor Vinum partitions (/dev/ad0s1e and /dev/ad2s1f). From this, allocate the 265 blocks forVinum configuration information, 1,000,000 blocks (about 488 MB) for /home, and the remaining2,724,408 blocks (about 1,330 MB) for /usr. See Figure 2 below to see this graphically.

The left-hand side of Figure 2 below shows what spindle ad0 will look like at the end of phase 2.The right-hand side shows what it will look like at the end of phase 3.

Figure 2. Spindle ad0 Before and After Vinum

2.1.1.6. Assigning Partitions on the Rootback Spindle

The /rootback and swap partition sizes on the rootback spindle must match the root and swappartition sizes on the root spindle. That leaves 8,420,769 - 200,000 - 200,265 = 8,020,504 blocksfor the Vinum partition. Mirrors of /home and /usr receive the same allocation as on the rootspindle. That will leave an extra 2 GB or so that we can deal with later. See Figure 3 below to seethis graphically.

The left-hand side of Figure 3 below shows what spindle ad2 will look like at the beginning ofphase 4. The right-hand side shows what it will look like at the end.

Figure 3. Spindle ad2 Before and After Vinum

2.1.1.7. Preparation of Tools

The bootvinum Perl script given below in Appendix A will make the Vinum bootstrapping processmuch easier if you can run it on the machine being bootstrapped. It is over 200 lines and you wouldnot want to type it in. At this point, I recommend that you copy it to a floppy or arrange somealternative method of making it readily available so that it can be available later when needed. Forexample:

# fdformat -f 1440 /dev/fd0 # newfs_msdos -f 1440 /dev/fd0 # mount /dev/fd0 /mnt # cp /usr/share/examples/vinum/bootvinum /mnt

Someday, I’d like this script to live in /usr/share/examples/vinum. Till then, please use this linkto get a copy.

2.2. Bootstrapping Phase 2: Minimal OS Installation

Our goal in this phase is to complete the smallest possible FreeBSD installation in such a way thatwe can later install Vinum. We will use only partitions of type 4.2BSD (i.e., regular UFS filesystems) since that is the only type supported by /stand/sysinstall.

2.2.1. Phase 2 Example

1. Start up the FreeBSD installation process by running /stand/sysinstall from installationmedia as you normally would.

2. Fdisk partition all spindles as needed.

Important: Make sure to select BootMgr for all spindles.

3. Partition the root spindle with appropriate block allocations as described above in Section2.1.1.5. For this example on a 2 GB spindle, I will use 200,000 blocks for root, 200,265blocks for swap, 1,000,000 blocks for /home, and the rest of the spindle (2,724,408 blocks)for /usr. (/stand/sysinstall should automatically assign these to /dev/ad0s1a,/dev/ad0s1b, /dev/ad0s1e, and /dev/ad0s1f by default.)

Note: If you prefer soft updates as I do and you are using 4.4-RELEASE or better,this is a good time to enable them.

4. Partition the rootback spindle with the appropriate block allocations as described above inSection 2.1.1.6. For this example on a 4 GB spindle, I will use 200,000 blocks for /rootback,200,265 blocks for swap, and the rest of the spindle (8,020,504 blocks) for /NOFUTURE.(/stand/sysinstall should automatically assign these to /dev/ad2s1e, /dev/ad2s1b, and/dev/ad2s1f by default.)

Note: We do not really want to have a /NOFUTURE UFS file system (we want avinum partition instead), but that is the best choice we have for the space giventhe limitations of /stand/sysinstall. Mount point names beginning withNOFUTURE and rootback serve as sentinels to the bootstrapping script presented inAppendix A below.

5. Partition any other spindles with swap if desired and a single /NOFUTURExx file system.

6. Select a minimum system install for now even if you want to end up with more distributionsloaded later.

Tip: Do not worry about system configuration options at this point--get Vinumset up and get the partitions in the right places first.

7. Exit /stand/sysinstall and reboot. Do a quick test to verify that the minimum installationwas successful.

The left-hand side of Figure 2 above and the left-hand side of Figure 3 above show how the diskswill look at this point.

2.3. Bootstrapping Phase 3: Root Spindle Setup

Our goal in this phase is get Vinum set up and running on the root spindle. We will embed theexisting /usr and /home file systems in a Vinum partition. Note that the Vinum volumes createdwill not yet be failure-resilient since we have only one underlying Vinum drive to hold them. Theresulting system will automatically start Vinum as it boots to multi-user mode.

2.3.1. Phase 3 Example

1. Login as root.

2. We will need a directory in the root file system in which to keep a few files that will be usedin the Vinum bootstrapping process.

# mkdir /bootvinum # cd /bootvinum

3. Several files need to be prepared for use in bootstrapping. I have written a Perl script thatmakes all the required files for you. Copy this script to /bootvinum by floppy disk, tape,network, or any convenient means and then run it. (If you cannot get this script copied ontothe machine being bootstrapped, then see Appendix B below for a manual alternative.)

# cp /mnt/bootvinum . # ./bootvinum

Note: bootvinum produces no output when run successfully. If you get any errors,something may have gone wrong when you were creating partitions with/stand/sysinstall above.

Running bootvinum will:

Create /etc/fstab.vinum based on what it finds in your existing /etc/fstab

Create new disk labels for each spindle mentioned in /etc/fstab and keep copies ofthe current disk labels

Create files needed as input to vinum create for building Vinum objects on eachspindle

Create many alternates to /etc/fstab.vinum that might come in handy should aspindle fail

You may want to take a look at these files to learn more about the disk partitioning requiredfor Vinum or to learn more about the commands needed to create Vinum objects.

4. We now need to install new spindle partitioning for /dev/ad0. This requires that/dev/ad0s1b not be in use for swapping so we have to reboot in single-user mode.

a. First, reboot the system.

# reboot

b. Next, enter single-user mode.

Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel] in 8 seconds... Type ’?’ for a list of commands, ’help’ for more detailed help. ok boot -s

5. In single-user mode, install the new partitioning created above.

# cd /bootvinum # disklabel -R ad0s1 disklabel.ad0s1 # disklabel -R ad2s1 disklabel.ad2s1

Note: If you have additional spindles, repeate the above commands as appropriatefor them.

6. We are about to start Vinum for the first time. It is going to want to create several devicenodes under /dev/vinum so we will need to mount the root file system for read/write access.

# fsck -p / # mount /

7. Now it is time to create the Vinum objects that will embed the existing non-root file systemson the root spindle in a Vinum partition. This will load the Vinum kernel module and startVinum as a side effect.

# vinum create create.YouCrazy

You should see a list of Vinum objects created that looks like the following:

1 drives: D YouCrazy State: up Device /dev/ad0s1h Avail: 0/1818 MB (0%) 2 volumes: V home State: up Plexes: 1 Size: 488 MB V usr State: up Plexes: 1 Size: 1330 MB 2 plexes: P home.p0 C State: up Subdisks: 1 Size: 488 MB P usr.p0 C State: up Subdisks: 1 Size: 1330 MB 2 subdisks: S home.p0.s0 State: up PO: 0 B Size: 488 MB S usr.p0.s0 State: up PO: 0 B Size: 1330 MB

You should also see several kernel messages which state that the Vinum objects you havecreated are now up.

8. Our non-root file systems should now be embedded in a Vinum partition and hence availablethrough Vinum volumes. It is important to test that this embedding worked.

# fsck -n /dev/vinum/home # fsck -n /dev/vinum/usr

This should produce no errors. If it does produce errors do not fix them. Instead, go back andexamine the root spindle partition tables before and after Vinum to see if you can spot theerror. You can back out the partition table changes by using disklabel -R with thedisklabel.*.b4vinum files.

9. While we have the root file system mounted read/write, this is a good time to install/etc/fstab.

# mv /etc/fstab /etc/fstab.b4vinum # cp /etc/fstab.vinum /etc/fstab

10. We are now done with tasks requiring single-user mode, so it is safe to go multi-user fromhere on.

# ^D

11. Login as root.

12. Edit /etc/rc.conf and add this line:

start_vinum="YES"

2.4. Bootstrapping Phase 4: Rootback Spindle Setup

Our goal in this phase is to get redundant copies of all data from the root spindle to the rootbackspindle. We will first create the necessary Vinum objects on the rootback spindle. Then we will askVinum to copy the data from the root spindle to the rootback spindle. Finally, we use dump andrestore to copy the root file system.

2.4.1. Phase 4 Example

1. Now that Vinum is running on the root spindle, we can bring it up on the rootback spindle sothat our Vinum volumes can become failure-resilient.

# cd /bootvinum # vinum create create.UpWindow

You should see a list of Vinum objects created that looks like the following:

2 drives: D YouCrazy State: up Device /dev/ad0s1h Avail: 0/1818 MB (0%) D UpWindow State: up Device /dev/ad2s1h Avail: 2096/3915 MB (53%) 2 volumes: V home State: up Plexes: 2 Size: 488 MB

V usr State: up Plexes: 2 Size: 1330 MB 4 plexes: P home.p0 C State: up Subdisks: 1 Size: 488 MB P usr.p0 C State: up Subdisks: 1 Size: 1330 MB P home.p1 C State: faulty Subdisks: 1 Size: 488 MB P usr.p1 C State: faulty Subdisks: 1 Size: 1330 MB 4 subdisks: S home.p0.s0 State: up PO: 0 B Size: 488 MB S usr.p0.s0 State: up PO: 0 B Size: 1330 MB S home.p1.s0 State: stale PO: 0 B Size: 488 MB S usr.p1.s0 State: stale PO: 0 B Size: 1330 MB

You should also see several kernel messages which state that some of the Vinum objects youhave created are now up while others are faulty or stale.

2. Now we ask Vinum to copy each of the subdisks on drive YouCrazy to drive UpWindow. Thiswill change the state of the newly created Vinum subdisks from stale to up. It will alsochange the state of the newly created Vinum plexes from faulty to up.

First, we do the new subdisk we added to /home.

# vinum start -w home.p1.s0 reviving home.p1.s0 (time passes . . . ) home.p1.s0 is up by force home.p1 is up home.p1.s0 is up

Note: My 5,400 RPM EIDE spindles copied at about 3.5 MBytes/sec. Yourmileage may vary.

3. Next we do the new subdisk we added to /usr.

# vinum -w start usr.p1.s0 reviving usr.p1.s0 (time passes . . . ) usr.p1.s0 is up by force usr.p1 is up usr.p1.s0 is up

All Vinum objects should be in state up at this point. The output of vinum list should looklike the following:

2 drives: D YouCrazy State: up Device /dev/ad0s1h Avail: 0/1818 MB (0%) D UpWindow State: up Device /dev/ad2s1h Avail: 2096/3915 MB (53%) 2 volumes: V home State: up Plexes: 2 Size: 488 MB V usr State: up Plexes: 2 Size: 1330 MB 4 plexes: P home.p0 C State: up Subdisks: 1 Size: 488 MB P usr.p0 C State: up Subdisks: 1 Size: 1330 MB P home.p1 C State: up Subdisks: 1 Size: 488 MB P usr.p1 C State: up Subdisks: 1 Size: 1330 MB 4 subdisks:

S home.p0.s0 State: up PO: 0 B Size: 488 MB S usr.p0.s0 State: up PO: 0 B Size: 1330 MB S home.p1.s0 State: up PO: 0 B Size: 488 MB S usr.p1.s0 State: up PO: 0 B Size: 1330 MB

4. Copy the root file system so that you will have a backup.

# cd /rootback # dump 0f - / | restore rf - # rm restoresymtable # cd /

Note: You may see errors like this:

./tmp/rstdir1001216411: (inode 558) not found on tape cannot find directory inode 265 abort? [yn] n expected next file 492, got 491

They seem to cause no harm. I suspect they are a consequence of dumping the filesystem containing /tmp and/or the pipe connecting dump and restore.

5. Make a directory on which we can mount a damaged root file system during the recoveryprocess.

# mkdir /rootbad

6. Remove sentinel mount points that are now unused.

# rmdir /NOFUTURE*

7. Create empty Vinum drives on remaining spindles.

# vinum create create.ThruBank # ...

At this point, the reliable server foundation is complete. The right-hand side of Figure 2 above andthe right-hand side of Figure 3 above show how the disks will look.

You may want to do a quick reboot to multi-user and give it a quick test drive. This is also a goodpoint to complete installation of other distributions beyond the minimal install. Add packages,ports, and users as required. Configure /etc/rc.conf as required.

Tip: After you have completed your server configuration, remember to do one morecopy of root to /rootback as shown above before placing the server into production.

Tip: Make a schedule to refresh /rootback periodically.

Tip: It may be a good idea to mount /rootback read-only for normal operation of theserver. This does, however, complicate the periodic refresh a bit.

Tip: Do not forget to watch /var/log/messages carefully for errors. Vinum mayautomatically avoid failed hardware in a way that users do not notice. You must watchfor such failures and get them repaired before a second failure results in data loss. Youmay see Vinum noting damaged objects at server boot time.

3. Where to Go from Here?Now that you have established the foundation of a reliable server, there are several things youmight want to try next.

3.1. Make a Vinum Volume with Remaining Space

Following are the steps to create another Vinum volume with space remaining on the rootbackspindle.

Note: This volume will not be resilient to spindle failure since it has only one plex on asingle spindle.

1. Create a file with the following contents:

volume hope plex name hope.p0 org concat volume hope sd name hope.p0.s0 drive UpWindow plex hope.p0 len 0

Note: Specifying a length of 0 for the hope.p0.s0 subdisk asks Vinum to usewhatever space is left available on the underlying drive.

2. Feed these commands into vinum create.

# vinum create filename

3. Now we newfs the volume and mount it.

# newfs -v /dev/vinum/hope # mkdir /hope # mount /dev/vinum/hope /hope

4. Edit /etc/fstab if you want /hope mounted at boot time.

3.2. Try Out More Vinum Commands

You might already be familiar with vinum list to get a list of all Vinum objects. Try -v followingit to see more detail.

If you have more spindles and you want to bring them up as concatenated, mirrored, or stripedvolumes, then give vinum concat drivelist, vinum mirror drivelist, or vinum stripedrivelist a try.

See vinum(8) for sample configurations and important performance considerations before settlingon a final organization for your additional spindles.

The failure recovery instructions below will also give you some experience using more Vinumcommands.

4. Failure ScenariosThis section contains descriptions of various failure scenarios. For each scenario, there is asubsection on how to configure your server for degraded mode operation, how to recover from thefailure, how to exit degraded mode, and how to simulate the failure.

Tip: Make a hard copy of these instructions and leave them inside the CPU case, beingcareful not to interfere with ventilation.

4.1. Root file system on ad0 unusable, rest of drive ok

Note: We assume here that the boot blocks and disk label on /dev/ad0 are ok. If yourBIOS can boot from a drive other than C:, you may be able to get around this limitation.

4.1.1. Configure Server for Degraded Mode

1. Use BootMgr to load kernel from /dev/ad2s1a.

a. Hit F5 in BootMgr to select Drive 1.

b. Hit F1 to select FreeBSD.

2. After the kernel is loaded, hit any key but enter to interrupt the boot sequence. Boot intosingle-user mode and allow explicit entry of a root file system.

Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel] in 8 seconds... Type ’?’ for a list of commands, ’help’ for more detailed help. ok boot -as

3. Select /rootback as your root file system.

Manual root file system specification: <fstype>:<device> Mount <device> using filesystem <fstype> e.g. ufs:/dev/da0s1a ? List valid disk boot devices <empty line> Abort manual input mountroot> ufs:/dev/ad2s1a

4. Now that you are in single-user mode, change /etc/fstab to avoid the bad root file system.

Tip: If you used the bootvinum Perl script from Appendix A below, then thesecommands should configure your server for degraded mode.

# fsck -p / # mount / # cd /etc # mv fstab fstab.bak # cp fstab_ad0s1_root_bad fstab # cd / # mount -o ro / # vinum start # fsck -p # ^D

4.1.2. Recovery

1. Restore /dev/ad0s1a from backups or copy /rootback to it with these commands:

# umount /rootbad # newfs /dev/ad0s1a # tunefs -n enable /dev/ad0s1a # mount /rootbad # cd /rootbad # dump 0f - / | restore rf - # rm restoresymtable

4.1.3. Exiting Degraded Mode

1. Enter single-user mode.

# shutdown now

2. Put /etc/fstab back to normal and reboot.

# cd /rootbad/etc # rm fstab # mv fstab.bak fstab # reboot

3. Reboot and hit F1 to boot from /dev/ad0 when prompted by BootMgr.

4.1.4. Simulation

This kind of failure can be simulated by shutting down to single-user mode and then booting asshown above in Section 4.1.1.

4.2. Drive ad2 Fails

This section deals with the total failure of /dev/ad2.

4.2.1. Configure Server for Degraded Mode

1. After the kernel is loaded, hit any key but Enter to interrupt the boot sequence. Boot intosingle-user mode.

Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel] in 8 seconds... Type ’?’ for a list of commands, ’help’ for more detailed help. ok boot -s

2. Change /etc/fstab to avoid the bad drive. If you used the bootvinum Perl script fromAppendix A below, then these commands should configure your server for degraded mode.

# fsck -p / # mount / # cd /etc # mv fstab fstab.bak # cp fstab_only_have_ad0s1 fstab # cd / # mount -o ro / # vinum start # fsck -p # ^D

If you do not have modified versions of /etc/fstab that are ready for use, then you can useed to make one. Alternatively, you can fsck and mount /usr and then use your favoriteeditor.

4.2.2. Recovery

We assume here that your server is up and running multi-user in degraded mode on just /dev/ad0and that you have a new spindle now on /dev/ad2 ready to go.

You will need a new spindle with enough room to hold root and swap partitions plus a Vinumpartition large enough to hold /home and /usr.

1. Create a BIOS partition (slice) on the new spindle.

# /stand/sysinstall

a. Select Custom.

b. Select Partition.

c. Select ad2.

d. Create a FreeBSD (type 165) slice large enough to hold everything mentioned above.

e. Write changes.

f. Yes, you are absolutely sure.

g. Select BootMgr.

h. Quit Partitioning.

i. Exit /stand/sysinstall.

2. Create disk label partitioning based on current /dev/ad0 partitioning.

# disklabel ad0 > /tmp/ad0 # disklabel -e ad2

This will drop you into your favorite editor.

a. Copy the lines for the a and b partitions from /tmp/ad0 to the ad2 disklabel.

b. Add the size of the a and b partitions to find the proper offset for the h partition.

c. Subtract this offset from the size of the c partition to find the proper size for the hpartition.

d. Define an h partition with the size and offset calculated above.

e. Set the fstype column to vinum.

f. Save the file and quit your editor.

3. Tell Vinum about the new drive.

a. Ask Vinum to start an editor with a copy of the current configuration.

# vinum create

b. Uncomment the drive line referring to drive UpWindow and set device to /dev/ad2s1h.

c. Save the file and quit your editor.

4. Now that Vinum has two spindles again, revive the mirrors.

# vinum start -w usr.p1.s0 # vinum start -w home.p1.s0

5. Now we need to restore /rootback to a current copy of the root file system. These commandswill accomplish this.

# newfs /dev/ad2s1a # tunefs -n enable /dev/ad2s1a # mount /dev/ad2s1a /mnt # cd /mnt # dump 0f - / | restore rf - # rm restoresymtable

# cd / # umount /mnt

4.2.3. Exiting Degraded Mode

1. Enter single-user mode.

# shutdown now

2. Return /etc/fstab to its normal state and reboot.

# cd /etc # rm fstab # mv fstab.bak fstab # reboot

4.2.4. Simulation

You can simulate this kind of failure by unplugging /dev/ad2, write-protecting it, or by thisprocedure:

1. Shutdown to single-user mode.

2. Unmount all non-root file systems.

3. Clobber any existing Vinum configuration and partitioning on /dev/ad2.

# vinum stop # dd if=/dev/zero of=/dev/ad2s1h count=512 # dd if=/dev/zero of=/dev/ad2 count=512

4.3. Drive ad0 Fails

Some BIOSes can boot from drive 1 or drive 2 (often called C: or D:), while others can boot onlyfrom drive 1. If your BIOS can boot from either, the fastest road to recovery might be to bootdirectly from /dev/ad2 in single-user mode and install /etc/fsatb_only_have_ad2s1 as/etc/fstab. You would then have to adapt the /dev/ad2 failure recovery instructions fromSection 4.2.2 above.

If your BIOS can only boot from drive one, then you will have to unplug drive YouCrazy from thecontroller for /dev/ad2 and plug it into the controller for /dev/ad0. Then continue with theinstructions for /dev/ad2 failure recovery in Section 4.2.2 above.

A. bootvinum Perl Script

The bootvinum Perl script below reads /etc/fstab and current drive partitioning. It then writesseveral files in the current directory and several variants of /etc/fstab in /etc. These filessignificantly simplify the installation of Vinum and recovery from spindle failures.

#!/usr/bin/perl -w use strict; use FileHandle; my $config_tag1 = ’$Id: vinum,v 1.9 2001/11/02 03:08:35 gsutter Exp $’; # Copyright (C) 2001 Robert A. Van Valzah # # Bootstrap Vinum # # Read /etc/fstab and current partitioning for all spindles mentioned there. # Generate files needed to mirror all file systems on root spindle. # A new partition table for each spindle # Input for the vinum create command to create Vinum objects on each spindle # A copy of fstab mounting Vinum volumes instead of BSD partitions # Copies of fstab altered for server’s degraded modes of operation # See handbook for instructions on how to use the the files generated. # N.B. This bootstrapping method shrinks size of swap partition by the size # of Vinum’s on-disk configuration (265 sectors). It embeds existing file # systems on the root spindle in Vinum objects without having to copy them. # Thanks to Greg Lehey for suggesting this bootstrapping method. # Expectations: # The root spindle must contain at least root, swap, and /usr partitions # The rootback spindle must have matching /rootback and swap partitions # Other spindles should only have a /NOFUTURE* file system and maybe swap # File systems named /NOFUTURE* will be replaced with Vinum drives # Change configuration variables below to suit your taste my $vip = ’h’; # VInum Partition my @drv = (’YouCrazy’, ’UpWindow’, ’ThruBank’, # Vinum DRiVe names ’OutSnakes’, ’MeWild’, ’InMovie’, ’HomeJames’, ’DownPrices’, ’WhileBlind’); # No configuration variables beyond this point my %vols; # One entry per Vinum volume to be created my @spndl; # One entry per SPiNDLe my $rsp; # Root SPindle (as in /dev/$rsp) my $rbsp; # RootBack SPindle (as in /dev/$rbsp) my $cfgsiz = 265; # Size of Vinum on-disk configuration info in sectors my $nxtpas = 2; # Next fsck pass number for non-root file systems # Parse fstab, generating the version we’ll need for Vinum and noting # spindles in use. my $fsin = "/etc/fstab"; #my $fsin = "simu/fstab"; open(FSIN, "$fsin") || die("Couldn’t open $fsin: $!\n"); my $fsout = "/etc/fstab.vinum"; open(FSOUT, ">$fsout") || die("Couldn’t open $fsout for writing: $!\n"); while (<FSIN>) { my ($dev, $mnt, $fstyp, $opt, $dump, $pass) = split; next if $dev =~ /^#/; if ($mnt eq ’/’ || $mnt eq ’/rootback’ || $mnt =~ /^\/NOFUTURE/) { my $dn = substr($dev, 5, length($dev)-6); # Device Name without /dev/ push(@spndl, $dn) unless grep($_ eq $dn, @spndl); $rsp = $dn if $mnt eq ’/’; next if $mnt =~ /^\/NOFUTURE/; } # Move /rootback from partition e to a if ($mnt =~ /^\/rootback/) { $dev =~ s/e$/a/;

$pass = 1; $rbsp = substr($dev, 5, length($dev)-6); print FSOUT "$dev\t\t$mnt\t$fstyp\t$opt\t\t$dump\t$pass\n"; next; } # Move non-root file systems on smallest spindle into Vinum if (defined($rsp) && $dev =~ /^\/dev\/$rsp/ && $dev =~ /[d-h]$/) { $pass = $nxtpas++; print FSOUT "/dev/vinum$mnt\t\t$mnt\t\t$fstyp\t$opt\t\t$dump\t$pass\n"; $vols{$dev}->{mnt} = substr($mnt, 1); next; } print FSOUT $_; } close(FSOUT); die("Found more spindles than we have abstract names\n") if $#spndl > $#drv; die("Didn’t find a root partition!\n") if !defined($rsp); die("Didn’t find a /rootback partition!\n") if !defined($rbsp); # Table of server’s Degraded Modes # One row per mode with hash keys # fn FileName # xpr eXPRession needed to convert fstab lines for this mode # cm1 CoMment 1 describing this mode # cm2 CoMment 2 describing this mode # FH FileHandle (dynamically initialized below) my @DM = ( { cm1 => "When we only have $rsp, comment out lines using $rbsp", fn => "/etc/fstab_only_have_$rsp", xpr => "s:^/dev/$rbsp:#\$&:", }, { cm1 => "When we only have $rbsp, comment out lines using $rsp and", cm2 => "rootback becomes root", fn => "/etc/fstab_only_have_$rbsp", xpr => "s:^/dev/$rsp:#\$&: || s:/rootback:/\t:", }, { cm1 => "When only $rsp root is bad, /rootback becomes root and", cm2 => "root becomes /rootbad", fn => "/etc/fstab_${rsp}_root_bad", xpr => "s:\t/\t:\t/rootbad: || s:/rootback:/\t:", }, ); # Initialize output FileHandles and write comments foreach my $dm (@DM) { my $fh = new FileHandle; $fh->open(">$dm->{fn}") || die("Can’t write $dm->{fn}: $!\n"); print $fh "# $dm->{cm1}\n" if $dm->{cm1}; print $fh "# $dm->{cm2}\n" if $dm->{cm2}; $dm->{FH} = $fh; } # Parse the Vinum version of fstab written above and write versions needed # for server’s degraded modes. open(FSOUT, "$fsout") || die("Couldn’t open $fsout: $!\n"); while (<FSOUT>) { my $line = $_; foreach my $dm (@DM) { $_ = $line; eval $dm->{xpr}; print {$dm->{FH}} $_; } } # Parse partition table for each spindle and write versions needed for Vinum

my $rootsiz; # ROOT partition SIZe my $swapsiz; # SWAP partition SIZe my $rspminoff; # Root SPindle MINimum OFFset of non-root, non-swap, non-c parts my $rspsiz; # Root SPindle SIZe my $rbspsiz; # RootBack SPindle SIZe foreach my $i (0..$#spndl) { my $dlin = "disklabel $spndl[$i] |"; # my $dlin = "simu/disklabel.$spndl[$i]"; open(DLIN, "$dlin") || die("Couldn’t open $dlin: $!\n"); my $dlout = "disklabel.$spndl[$i]"; open(DLOUT, ">$dlout") || die("Couldn’t open $dlout for writing: $!\n"); my $dlb4 = "$dlout.b4vinum"; open(DLB4, ">$dlb4") || die("Couldn’t open $dlb4 for writing: $!\n"); my $minoff; # MINimum OFFset of non-root, non-swap, non-c partitions my $totsiz = 0; # TOTal SIZe of all non-root, non-swap, non-c partitions my $swapspndl = 0; # True if SWAP partition on this SPiNDLe while (<DLIN>) { print DLB4 $_; my ($part, $siz, $off, $fstyp, $fsiz, $bsiz, $bps) = split; if ($part && $part eq ’a:’ && $spndl[$i] eq $rsp) { $rootsiz = $siz; } if ($part && $part eq ’e:’ && $spndl[$i] eq $rbsp) { if ($rootsiz != $siz) { die("Rootback size ($siz) != root size ($rootsiz)\n"); } } if ($part && $part eq ’c:’) { $rspsiz = $siz if $spndl[$i] eq $rsp; $rbspsiz = $siz if $spndl[$i] eq $rbsp; } # Make swap partition $cfgsiz sectors smaller if ($part && $part eq ’b:’) { if ($spndl[$i] eq $rsp) { $swapsiz = $siz; } else { if ($swapsiz != $siz) { die("Swap partition sizes unequal across spindles\n"); } } printf DLOUT "%4s%9d%9d%10s\n", $part, $siz-$cfgsiz, $off, $fstyp; $swapspndl = 1; next; } # Move rootback spindle e partitions to a if ($part && $part eq ’e:’ && $spndl[$i] eq $rbsp) { printf DLOUT "%4s%9d%9d%10s%9d%6d%6d\n", ’a:’, $siz, $off, $fstyp, $fsiz, $bsiz, $bps; next; } # Delete non-root, non-swap, non-c partitions but note their minimum # offset and total size that’re needed below. if ($part && $part =~ /^[d-h]:$/) { $minoff = $off unless $minoff; $minoff = $off if $off < $minoff; $totsiz += $siz; if ($spndl[$i] eq $rsp) { # If doing spindle containing root my $dev = "/dev/$spndl[$i]" . substr($part, 0, 1); $vols{$dev}->{siz} = $siz; $vols{$dev}->{off} = $off; $rspminoff = $minoff;

} next; } print DLOUT $_; } if ($swapspndl) { # If there was a swap partition on this spindle # Make a Vinum partition the size of all non-root, non-swap, # non-c partitions + the size of Vinum’s on-disk configuration. # Set its offset so that the start of the first subdisk it contains # coincides with the first file system we’re embedding in Vinum. printf DLOUT "%4s%9d%9d%10s\n", "$vip:", $totsiz+$cfgsiz, $minoff-$cfgsiz, ’vinum’; } else { # No need to mess with size size and offset if there was no swap printf DLOUT "%4s%9d%9d%10s\n", "$vip:", $totsiz, $minoff, ’vinum’; } } die("Swap partition not found\n") unless $swapsiz; die("Swap partition not larger than $cfgsiz blocks\n") unless $swapsiz>$cfgsiz; die("Rootback spindle size not >= root spindle size\n") unless $rbspsiz>=$rspsiz; # Generate input to vinum create command needed for each spindle. foreach my $i (0..$#spndl) { my $cfn = "create.$drv[$i]"; # Create File Name open(CF, ">$cfn") || die("Can’t open $cfn for writing: $!\n"); print CF "drive $drv[$i] device /dev/$spndl[$i]$vip\n"; next unless $spndl[$i] eq $rsp || $spndl[$i] eq $rbsp; foreach my $dev (keys(%vols)) { my $mnt = $vols{$dev}->{mnt}; my $siz = $vols{$dev}->{siz}; my $off = $vols{$dev}->{off}-$rspminoff+$cfgsiz; print CF "volume $mnt\n" if $spndl[$i] eq $rsp; print CF <<EOF; plex name $mnt.p$i org concat volume $mnt sd name $mnt.p$i.s0 drive $drv[$i] plex $mnt.p$i len ${siz}s driveoffset ${off}s EOF } }

B. Manual Vinum BootstrappingThe bootvinum Perl script in Appendix A makes life easier, but it may be necessary to manuallyperform some or all of the steps that it automates. This appendix describes how you wouldmanually mimic the script.

1. Make a copy of /etc/fstab to be customized.

# cp /etc/fstab /etc/fstab.vinum

2. Edit /etc/fstab.vinum.

a. Change the device column of non-root partitions on the root spindle to/dev/vinum/mnt.

b. Change the pass column of non-root partitions on the root spindle to 2, 3, etc.

c. Delete any lines with mountpoint matching /NOFUTURE*.

d. Change the device column of /rootback from e to a.

e. Change the pass column of /rootback to 1.

3. Prepare disklabels for editing:

# cd /bootvinum # disklabel ad0s1 > disklabel.ad0s1 # cp disklabel.ad0s1 disklabel.ad0s1.b4vinum # disklabel ad2s1 > disklabel.ad2s1 # cp disklabel.ad2s1 disklabel.ad2s1.b4vinum

4. Edit /etc/disklabel.ad?s1.

a. On the root spindle:

i. Decrease the size of the b partition by 265 blocks.

ii. Note the size and offset of the a and b partitions.

iii. Note the smallest offset for partitions d-h.

iv. Note the size and offset for all non-root, non-swap partitions (/home wasprobably on e and /usr was probably on f).

v. Delete partitions d-h.

vi. Create a new h partition with offset 265 blocks less than the smallest offset forpartitions d-h noted above. Set its size to the size of the c partition less thesmallest offset for partitions d-h noted above + 265 blocks.

Note: Vinum can use any partition other than c. It is not strictlynecessary to use h for all your Vinum partitions, but it is good practiceto be consistent across all spindles.

vii. Set the fstype of this new partition to vinum.

b. On the rootback spindle:

i. Move the e partition to a.

ii. Verify that the size of the a and b partitions matches the root spindle.

iii. Note the smallest offset for partitions d-h.

iv. Delete partitions d-h.

v. Create a new h partition with offset 265 blocks less than the smallest offsetnoted above for partitions d-h. Set its size to the size of the c partition less thesmallest offset for partitions d-h noted above + 265 blocks.

vi. Set the fstype of this new partition to vinum.

5. Create a file named create.YouCrazy that contains:

drive YouCrazy device /dev/ad0s1h volume home plex name home.p0 org concat volume home sd name home.p0.s0 drive YouCrazy plex home.p0 len $hl driveoffset $ho volume usr plex name usr.p0 org concat volume usr sd name usr.p0.s0 drive YouCrazy plex usr.p0 len $ul driveoffset $uo

Where:

$hl is the length noted above for /home.

$ho is the offset noted above for /home less the smallest offset noted above + 265blocks.

$ul is the length noted above for /usr.

$uo is the offset noted above for /usr less the smallest offset noted above + 265 blocks.

6. Create a file named create.UpWindow containing:

drive UpWindow device /dev/ad2s1h plex name home.p1 org concat volume home sd name home.p1.s0 drive UpWindow plex home.p1 len $hl driveoffset $ho plex name usr.p1 org concat volume usr sd name usr.p1.s0 drive UpWindow plex usr.p1 len $ul driveoffset $uo

Where $hl, $ho, $ul, and $uo are set as above.

C. AcknowledgementsI would like to thank Greg Lehey for writing Vinum and for providing very helpful comments onearly drafts. Several others made helpful suggestions after reviewing later drafts includingDag-Erling Smørgrav, Michael Splendoria, Chern Lee, Stefan Aeschbacher, Fleming Froekjaer,Bernd Walter, Aleksey Baranov, and Doug Swarin.

Notes

[1] This assumes that you have not removed the line

options ATA_STATIC_ID

from your kernel configuration.

Author maintains all copyrights on this article.

Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

Growing FreeBSD’s filesystems with growfs(8)

Marc Fonvieille <fonvi@easynet.fr>

Introduction

During installation it is important to size filesystems so that we won’t be forced to resize them later.However, it often appears with time that one of the partitions is too small. Until today it wasimpossible to expand the size of a filesystem. FreeBSD 4.4-RELEASE comes with a new utility:growfs(8). This program permits us to enlarge, in certains conditions, the size of an existing ufsfilesystem.

Preparing the fight

Before detailing the operations through one example, there are many things to know:

Playing with filesystems is always a dangerous operation: do it only if it is mandatory and besure to backup your data. growfs(8) can only grow - not shrink - filesystems. You would not be able to shrink thefilesystem once it has been expanded. growfs(8) cannot enlarge a filesystem that does not have any free space around it. We will need to use growfs(8), disklabel(8), fdisk(8) and fsck(8). So print thesemanual pages and read them. Having a printed version under hand during the process isuseful. Remember, in single user mode you can’t read manual pages. To modify filesystems, we’ll have to calculate sizes and sector numbers, so if you’re bad atmental arithmetic use a calculator.

I did my experiments on a machine running 4.4-PRERELEASE with a 6.4G hard drive.

Here is the partition table of this hard drive:

# fdisk -s/dev/ad0: 784 cyl 255 hd 63 secPart Start Size Type Flags 1: 63 2088387 0x06 0x00 2: 2088450 2859570 0xa5 0x80 3: 4948020 7646940 0xa5 0x00

As we can see, first slice is a msdos filesystem (type 0x06) and the others are FreeBSD filesystem(type 0xa5). The 4.4-PRERELEASE is on the second one; on the last, there’s an old 3.5-STABLEsystem. I don’t need the latter, so I will remove it and use the free space to enlarge the second slice.

If I call fdisk(8) with no arguments, it prints the following:

******* Working on device /dev/ad0 *******parameters extracted from in-core disklabel are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512Warning: BIOS sector numbering starts with sector 1Information from DOS bootblock is:The data for partition 1 is:sysid 6,(Primary ’big’ DOS (> 32MB)) start 63, size 2088387 (1019 Meg), flag 0 beg: cyl 0/ head 1/ sector 1; end: cyl 129/ head 254/ sector 63The data for partition 2 is:sysid 165,(FreeBSD/NetBSD/386BSD) start 2088450, size 2859570 (1396 Meg), flag 80 (active) beg: cyl 130/ head 0/ sector 1; end: cyl 307/ head 254/ sector 63The data for partition 3 is:sysid 165,(FreeBSD/NetBSD/386BSD) start 4948020, size 7646940 (3733 Meg), flag 0 beg: cyl 308/ head 0/ sector 1; end: cyl 783/ head 254/ sector 63The data for partition 4 is:<UNUSED>

The first thing to do is remove that third slice. I booted on the 4.4-PRERELEASE system andhere’s how it went:

# fdisk -u******* Working on device /dev/ad0 *******parameters extracted from in-core disklabel are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl) parameters to be used for BIOS calculations are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl) Do you want to change our idea of what BIOS thinks ? [n] nMedia sector size is 512Warning: BIOS sector numbering starts with sector 1Information from DOS bootblock is:The data for partition 1 is:sysid 6,(Primary ’big’ DOS (> 32MB)) start 63, size 2088387 (1019 Meg), flag 0 beg: cyl 0/ head 1/ sector 1; end: cyl 129/ head 254/ sector 63Do you want to change it? [n] nThe data for partition 2 is:sysid 165,(FreeBSD/NetBSD/386BSD) start 2088450, size 2859570 (1396 Meg), flag 80 (active) beg: cyl 130/ head 0/ sector 1; end: cyl 307/ head 254/ sector 63Do you want to change it? [n] nThe data for partition 3 is:sysid 165,(FreeBSD/NetBSD/386BSD) start 4948020, size 7646940 (3733 Meg), flag 0 beg: cyl 308/ head 0/ sector 1; end: cyl 783/ head 254/ sector 63Do you want to change it? [n] ySupply a decimal value for "sysid (165=FreeBSD)" [165] 0Supply a decimal value for "start" [4948020] 0Supply a decimal value for "size" [7646940] 0

Explicitly specify beg/end address ? [n] ySupply a decimal value for "beginning cylinder" [0] 0Supply a decimal value for "beginning head" [0] 0Supply a decimal value for "beginning sector" [0] 0Supply a decimal value for "ending cylinder" [0] 0Supply a decimal value for "ending head" [0] 0Supply a decimal value for "ending sector" [0] 0<UNUSED>Are we happy with this entry? [n] yThe data for partition 4 is:<UNUSED>Do you want to change it? [n] nDo you want to change the active partition? [n] n

We haven’t changed the partition table yet. This is your last chance.parameters extracted from in-core disklabel are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl)

Information from DOS bootblock is:1: sysid 6,(Primary ’big’ DOS (> 32MB)) start 63, size 2088387 (1019 Meg), flag 0 beg: cyl 0/ head 1/ sector 1; end: cyl 129/ head 254/ sector 632: sysid 165,(FreeBSD/NetBSD/386BSD) start 2088450, size 2859570 (1396 Meg), flag 80 (active) beg: cyl 130/ head 0/ sector 1; end: cyl 307/ head 254/ sector 633: <UNUSED>4: <UNUSED>Should we write new partition table? [n] y#

Specifying a partition type of zero is equal to clear the partition and mark it as unused, but we mustspecify 0 values for all parameters concerning that partition (start, size...). Don’t forget that what iscalled "partition" in fdisk, is known as "slice" in FreeBSD.

Now fdisk -s prints:

# fdisk -s/dev/ad0: 784 cyl 255 hd 63 secPart Start Size Type Flags 1: 63 2088387 0x06 0x00 2: 2088450 2859570 0xa5 0x80

After that, I performed the command fsck to check the filesystem.

Expanding the /usr partition

Here is the current size of partitions:

# df -hFilesystem Size Used Avail Capacity Mounted on/dev/ad0s2a 39M 31M 4.4M 88% //dev/ad0s2f 1.1G 810M 208M 80% /usr/dev/ad0s2e 145M 308K 133M 0% /varprocfs 4.0K 4.0K 0B 100% /proc

The /usr is too small for me; I need something like 2.5G

Before using growfs(8), the slice must be labeled as a bigger size with disklabel(8) andfdisk(8).

Have a look at the current label of /dev/ad0s2:

# /dev/ad0s2c:type: ESDIdisk: ad0s2label: flags:bytes/sector: 512sectors/track: 63tracks/cylinder: 255sectors/cylinder: 16065cylinders: 178sectors/unit: 2859570rpm: 3600interleave: 1trackskew: 0cylinderskew: 0headswitch: 0 # millisecondstrack-to-track seek: 0 # millisecondsdrivedata: 0

8 partitions:# size offset fstype [fsize bsize bps/cpg] a: 81920 0 4.2BSD 1024 8192 16 # (Cyl. 0 - 5*) b: 131072 81920 swap # (Cyl. 5*- 13*) c: 2859570 0 unused 0 0 # (Cyl. 0 - 177) e: 307200 212992 4.2BSD 1024 8192 16 # (Cyl. 13*- 32*) f: 2339378 520192 4.2BSD 1024 8192 16 # (Cyl. 32*- 177*)

/usr is 2339378 sectors large. We have a 512-byte sector (look at bytes/sector parameter), and /usrin (human-readable) size is: (2339378/2)/1024 Megabytes, or 1142.27MB (same as given by the df-h command above, 1.1G).

I had said that I wanted a 2.5G large partition for /usr. Let’s do some calculations:

2.5G is 2.5*1024*1024Kwith 512-bytes sector it gives2.5*1024*1024*2= 5242880 sectors

We already use 2339378 sectors for /usr, we have to add5242880-2339378= 2903502 sectors to /usr and so to the whole slice.

The whole slice size will be: 2859570+2903502= 5763072 sectors

There are 16065 sectors/cylinder, so 5763072 sectors require about 359cylinders.

As summary, we will have to change these parameters in the label:

cylinders: 178 to 359 sectors/unit: 2859570 to 5763072 c: 2859570 to 5763072 f: 2339378 to 5242880

If I try to edit the label, disklabel(8) refuses these values; the slice is already full. I have to usefdisk(8) to enlarge the slice before labeling it.

The delicate work begins here, so to avoid problems I log in as single user then launch fdisk(8):

# fdisk -u******* Working on device /dev/ad0 *******parameters extracted from in-core disklabel are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl) parameters to be used for BIOS calculations are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl) Do you want to change our idea of what BIOS thinks ? [n] nMedia sector size is 512Warning: BIOS sector numbering starts with sector 1Information from DOS bootblock is:The data for partition 1 is:sysid 6,(Primary ’big’ DOS (> 32MB)) start 63, size 2088387 (1019 Meg), flag 0 beg: cyl 0/ head 1/ sector 1; end: cyl 129/ head 254/ sector 63Do you want to change it? [n] nThe data for partition 2 is:sysid 165,(FreeBSD/NetBSD/386BSD) start 2088450, size 2859570 (1396 Meg), flag 80 (active) beg: cyl 130/ head 0/ sector 1; end: cyl 307/ head 254/ sector 63Do you want to change it? [n] ySupply a decimal value for "sysid (165=FreeBSD)" [165] 165Supply a decimal value for "start" [2088450] 2088450Supply a decimal value for "size" [2859570] 5763072Explicitly specify beg/end address ? [n] ySupply a decimal value for "beginning cylinder" [130] 130Supply a decimal value for "beginning head" [0] 0Supply a decimal value for "beginning sector" [1] 1Supply a decimal value for "ending cylinder" [307] 489Supply a decimal value for "ending head" [254] 254Supply a decimal value for "ending sector" [63] 63Are we happy with this entry? [n] yThe data for partition 3 is:<UNUSED>Do you want to change it? [n] nThe data for partition 4 is:<UNUSED>Do you want to change it? [n] nDo you want to change the active partition? [n] n

We haven’t changed the partition table yet. This is your last chance.parameters extracted from in-core disklabel are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:cylinders=784 heads=255 sectors/track=63 (16065 blks/cyl)

Information from DOS bootblock is:1: sysid 6,(Primary ’big’ DOS (> 32MB)) start 63, size 2088387 (1019 Meg), flag 0 beg: cyl 0/ head 1/ sector 1; end: cyl 129/ head 254/ sector 632: sysid 165,(FreeBSD/NetBSD/386BSD) start 2088450, size 5763072 (2814 Meg), flag 80 (active) beg: cyl 130/ head 0/ sector 1; end: cyl 489/ head 254/ sector 633: <UNUSED>4: <UNUSED>Should we write new partition table? [n] y

#

As you can see, I used the value calculated earlier; the end cylinder is given by 130+359. After thatI decided to reboot:

Then always in single user:

# fdisk -s/dev/ad0: 784 cyl 255 hd 63 secPart Start Size Type Flags 1: 63 2088387 0x06 0x00 2: 2088450 5763072 0xa5 0x80

I edit the disklabel:

# disklabel -e -r /dev/ad0s2

as the following:

# /dev/ad0s2c:type: ESDIdisk: ad0s2label: flags:bytes/sector: 512sectors/track: 63tracks/cylinder: 255sectors/cylinder: 16065cylinders: 358sectors/unit: 5763072rpm: 3600interleave: 1trackskew: 0cylinderskew: 0headswitch: 0 # millisecondstrack-to-track seek: 0 # millisecondsdrivedata: 0 8 partitions:# size offset fstype [fsize bsize bps/cpg] a: 81920 0 4.2BSD 1024 8192 16 # (Cyl. 0 - 5*) b: 131072 81920 swap # (Cyl. 5*- 13*) c: 5763072 0 unused 0 0 # (Cyl. 0 - 177) e: 307200 212992 4.2BSD 1024 8192 16 # (Cyl. 13*- 32*) f: 5242880 520192 4.2BSD 1024 8192 16 # (Cyl. 32*- 177*)#

In my case, cylinders and sectors/unit were automatically changed to the new values; only c and fsizes have to be modified. My comments follow the hash marks (#).

Now we are ready to use growfs(8). We must umount the /usr partition before:

# umount /usr# growfs -s 52422880 /dev/ad0s2f

growfs(8) will ask you if you did a backup of your data; answer ’Yes’, and you will see theprocess begin. When it is over, we can check the new size of /usr:

# mount /usr# df -hFilesystem Size Used Avail Capacity Mounted on

/dev/ad0s2a 39M 31M 4.4M 88% //dev/ad0s2f 2.4G 810M 1.4G 35% /usr/dev/ad0s2e 145M 79M 55M 59% /varprocfs 4.0K 4.0K 0B 100% /proc

Now the /usr is 2.4G large, but I allocated 2.5G. The missing size is the percentage of space heldback from normal users, which is 8% of the whole filesystem by default. For more information seethe tunefs(8) manual page.

To check the new filesystem, I used fsck(8):

# fsck** /dev/ad0s2a (NO WRITE)** Last Mounted on /** Root file system** Phase 1 - Check Blocks and Sizes** Phase 2 - Check Pathnames** Phase 3 - Check Connectivity** Phase 4 - Check Reference Counts** Phase 5 - Check Cyl groups1145 files, 31975 used, 7672 free (136 frags, 942 blocks, 0.3% fragmentation)** /dev/ad0s2f (NO WRITE)** Last Mounted on /usr** Phase 1 - Check Blocks and Sizes** Phase 2 - Check Pathnames** Phase 3 - Check Connectivity** Phase 4 - Check Reference Counts** Phase 5 - Check Cyl groupsSUMMARY INFORMATION BADSALVAGE? no

BLK(S) MISSING IN BIT MAPSSALVAGE? no

123343 files, 829946 used, 1711476 free (7748 frags, 212966 blocks, 0.3% fragmentation)** /dev/ad0s2e (NO WRITE)** Last Mounted on /var** Phase 1 - Check Blocks and Sizes** Phase 2 - Check Pathnames** Phase 3 - Check Connectivity** Phase 4 - Check Reference Counts** Phase 5 - Check Cyl groups104 files, 80898 used, 67925 free (29 frags, 8487 blocks, 0.0% fragmentation)

We can ignore both SALVAGE messages. If we relaunch fsck(8), all is well.

Conclusion

For me, growfs(8) did the job well, but remember that modifying your filesystem isn’t a simpleoperation, so be careful and double-check everything.

I have not tried to enlarge a vinum filesystem, but the process must be the same.

Once again FreeBSD developers give us a superb tool. I hope a shrinkfs tool will be also developedin the future.

Marc Fonvieillefonvi@easynet.fr

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

A Quick Guide to Configuring IPsec on OpenBSD v2.9

by Robert Sigillito and Carol Thompson

We began working with OpenBSD v2.9 to implement a VPN solution for little cost in order tospare a handful of users a 35 mile drive to another facility. Why drive out of your way, when youcan use a public network to pass your data in a fully encrypted private tunnel? Being new toOpenBSD, but not Unix, we made more than our share of mistakes. This guide is an attempt todocument how we successfully configured IPsec on OpenBSD v2.9.

As we mentioned, this project was to be done for little or no cost. The PC’s we used for our VPNwere two Dell Dimensions (one 300mhz and the other 266mhz) that we pulled from the surpluspile. Each machine had a 3Com 3C509B NIC installed, which we supplemented with a second3C509B. And of course OpenBSD can either be installed via ftp (free) or from CDROM for thenominal cost of $30 US. We felt that the best way to go was to buy the CD. This avoids theproblems of ftp installs and supports a worthy project.

1. Install OpenBSD v2.9 per the instructions included with the CD. It’s a good idea to print outthe entire OpenBSD FAQ and have that with you before beginning installation. If you areinstalling this on a PC that has Plug-n-Play enabled in the BIOS disable it. OpenBSD doesn’tseem to work with Plug-n-Play. A problem we found, even after disabling P-n-P in the BIOS,was that the network cards (two 3Com 3C509B’s) would go to sleep after about 15-20minutes of inactivity. To get around this problem and to monitor our VPN machines weinstalled Big Brother on each machine. The Big Brother display monitor polls the two VPNmachines every five minutes which allows us to monitor their "health" and to keep theirinterfaces awake.

2. After installing the operating system you may want to make a number of minor changes.These changes include:

Edit the file /etc/group and on the first line, ‘wheel’ group, either remove "root" or addyour user ID. This will allow you to "su -" from your user ID. Edit the file /etc/ssh_config and remove the "#" from "ForwardAgent yes" and"ForwardX11 yes". This will allow you to tunnel X11 traffic through an SSH tunnel. Edit the /etc/sshd_config and change PermitRootLogin from "yes" to "no". This waysomeone will have to guess a user ID and password before they can even take a crack atroot. Edit the file /etc/sysctl.conf and remove the "#" from the lines"net.inet.ip.forwarding=1", "net.inet.esp.enable=1", and "net.inet.ah.enable=1". Youwant this so that when you restart the machine it will automatically come up routing andrunning ESP an AH. You do not have to run ESP and AH - you can run one or the otheror both together like we did. You can also activate each of these options from thecommand line by issuing a "sysctl" command. # sysctl -w net.inet.ip.forwarding=1

(This activates routing)

3. Copy the files isakmpd.conf and isakmpd.policy to /etc/isakmpd, then set the permissions ofthese files to read-write for the owner (root). You can do this by issuing the command"chmod 600 isakmpd.*". You can grab the sample files off of the CD. If you have enoughroom go ahead and copy the source files from the CD to /usr/src on your local drive, or getthem from the OpenBSD web site (www.openbsd.org).

4. We also need to edit the file /etc/rc.conf in order for isakmpd to start automatically uponreboot. Open /etc/rc.conf in your favorite editor and change the line "isakmpd_flags=NO" toread "isakmpd_flags=YES". You will also want to edit the lines in /etc/rc.conf pertaining toipfilter and ipnat (as needed) changing the default from "NO" to "YES".

5. Now we want to edit the files called "/etc/hostname.xl0 and /etc/hostname.xl1". It should benoted that the files on your machine may have different names, ie., "/etc/hostname.fx0" or/etc/hostname.de0", etc. The reason for this is that this reflects the different brands of NICs. Ifyour machine has two adapters (and a VPN router should have two adapters), the first onewill be named hostname.xl0 (for example) and the second hostname.xl1. We can modifythese files to include any routes we want them to have upon boot up. Remember that routesadded after boot up are stored in kernel memory, which is purged when the machine reboots.You can add routes to /etc/rc.local, but we prefer to add them to hostname.xl0 so that theywill be activated as soon as the network adapter is activated. For example, here’s the/etc/hostname.xl0 and /etc/hostname.xl1 from our machine "catbert" (please note that all IPaddresses used in this paper are purely fictitious):

/etc/hostname.xl0 ! /sbin/ifconfig xl0 152.7.15.253 netmask 255.255.255.0 ! /sbin/route add 152.7.68.0/24 152.7.15.254 ! /sbin/route add 152.7.72.0/24 152.7.15.254 ! /sbin/route add 152.7.66.0/24 152.7.15.254 ! /sbin/route add 152.7.3.0/24 152.7.15.254 ! /sbin/route add 152.7.1.0/24 152.7.15.254 ! /sbin/route add 152.7.7.0/24 152.7.15.254

/etc/hostname.xl1 ! /sbin/ifconfig xl1 152.7.20.253 netmask 255.255.255.0 ! /sbin/ifconfig 152.7.30.0/24 152.7.20.254 ! /sbin/ifconfig 192.168.0.0/24 152.7.20.254

These are the /etc/hostname.xl0 and /etc/hostname.xl1 from our machine "dogbert":

/etc/hostname.xl0 ! /sbin/ifconfig xl0 152.7.30.10 netmask 255.255.255.0 ! /sbin/route add 152.7.20.0/24 152.7.30.254 ! /sbin/route add 152.7.15.0/24 152.7.30.254

/etc/hostname.xl1 ! /sbin/ifconfig xl1 192.168.0.1 netmask 255.255.255.0

6. Your default gateway is stored in the file /etc/mygate, in case you want to change it.

7. You will need to insure that your /etc/isakmpd/isakmpd.conf & isakmpd.policy files on eachof your machines are correctly configured. Included below are samples from our machines"catbert" and "dogbert":

Catbert:

/etc/isakmpd/isakmpd.policy

Keynote-version: 2 Authorizer: "POLICY" Conditions: app_domain == "IPsec policy" && esp_present == "yes" && esp_enc_alg != "null" -> "true";

/etc/isakmpd/isakmpd.conf

# General options ########################################## [General] Policy-File= /etc/isakmpd/isakmpd.policy Retransmits= 5 Exchange-max-time= 120 Listen-on= 152.7.20.253

# # incoming phase 1 negotiations ########################################## [Phase 1] 152.7.30.10= ISAKMP-peer-dogbert

# # phase 2 connections ########################################## [Phase 2] Connections= IPsec-catbert-dogbert # # ISAKMP phase 1 peers ########################################## [ISAKMP-peer-dogbert] Phase= 1 Transport= udp Local_address= 152.7.20.253 Address= 152.7.30.10 Configuration= Default-main-mode Authentication= oursecretpassword

# # IPsec phase 2 connections ######################################### [IPsec-catbert-dogbert] Phase= 2 ISAKMP-peer= ISAKMP-peer-dogbert Configuration= Default-quick-mode Local-ID= Net-catbert Remote-ID= Net-dogbert

# # client ID sections ######################################## [Net-catbert] ID-type= IPV4_ADDR_SUBNET Network= 152.7.15.0 Netmask= 255.255.255.0

[Net-dogbert] ID-type= IPV4_ADDR_SUBNET Network= 192.168.0.0

Netmask= 255.255.255.0

# # main mode descriptions ####################################### [Default-main-mode] DOI= IPSEC EXCHANGE_TYPE= ID_PROT Transforms= 3DES-SHA

# # quick mode descriptions ####################################### [Default-quick-mode] DOI= IPSEC EXCHANGE_TYPE= QUICK_MODE Suites= QM-ESP-BLF-SHA-PFS-SUITE

Dogbert:

/etc/isakmpd/isakmpd.policy

Keynote-version: 2 Authorizer: "POLICY" Conditions: app_domain == "IPsec policy" && esp_present == "yes" && esp_enc_alg != "null" -> "true";

/etc/isakmpd/isakmpd.conf

# General options ########################################## [General] Policy-File= /etc/isakmpd/isakmpd.policy Retransmits= 5 Exchange-max-time= 120 Listen-on= 152.7.30.10

# # incoming phase 1 negotiations ########################################## [Phase 1] 152.7.20.253= ISAKMP-peer-catbert

# # phase 2 connections ########################################## [Phase 2] Connections= IPsec-catbert-dogbert

# # ISAKMP phase 1 peers ########################################## [ISAKMP-peer-catbert] Phase= 1 Transport= udp Local_address= 152.7.30.10 Address= 152.7.20.253 Configuration= Default-main-mode Authentication= oursecretpassword

# # IPsec phase 2 connections ######################################### [IPsec-catbert-dogbert]

Phase= 2 ISAKMP-peer= ISAKMP-peer-catbert Configuration= Default-quick-mode Local-ID= Net-dogbert Remote-ID= Net-catbert

# # client ID sections ######################################## [Net-catbert] ID-type= IPV4_ADDR_SUBNET Network= 152.7.15.0 Netmask= 255.255.255.0

8. After you have modified the isakmpd.conf and isakmpd.policy for your needs, now comes thetime to test. The OpenBSD FAQ (13.11) suggests that you test by running the isakmpddaemon in the foreground with debugging turned on. You will see lots of messages dumpedto the screen which can be helpful if you have to debug your configuration. To start thedaemon in this fashion, issue the following command: isakmpd -d -DA=99

9. Now comes time to test your VPN. The first tool to use is tcpdump. Tcpdump will allow youto capture the headers of packets on a network interface. To test, we suggest using ping. Starta ping between the two protected networks. In our example above, we have two protectednetworks: 152.7.15.0/24 and 192.168.0.0/24. To start a ping on catbert we issue the followingcommand:

# ping -I 152.7.15.253 192.168.0.1

In a separate session we need to start tcpdump. We are only interested in the traffic betweencatbert and dogbert so we issue the following command:

catbert# tcpdump -ni xl1 host dogbert tcpdump: listening on xl1 14:27:07.600423 esp 152.7.20.253 > 152.7.30.10 spi 0xB5804164 seq 34 len 116 14:27:07.602318 esp 152.7.30.10 > 152.7.20.253 spi 0xBAFB375B seq 34 len 116 14:27:08.600261 esp 152.7.20.253 > 152.7.30.10 spi 0xB5804164 seq 35 len 116 14:27:08.601728 esp 152.7.30.10 > 152.7.20.253 spi 0xBAFB375B seq 35 len 116 14:27:09.600233 esp 152.7.20.253 > 152.7.30.10 spi 0xB5804164 seq 36 len 116 14:27:09.602085 esp 152.7.30.10 > 152.7.20.253 spi 0xBAFB375B seq 36 len 116 14:27:10.600221 esp 152.7.20.253 > 152.7.30.10 spi 0xB5804164 seq 37 len 116 14:27:10.601939 esp 152.7.30.10 > 152.7.20.253 spi 0xBAFB375B seq 37 len 116 14:27:11.600221 esp 152.7.20.253 > 152.7.30.10 spi 0xB5804164 seq 38 len 116 14:27:11.602026 esp 152.7.30.10 > 152.7.20.253 spi 0xBAFB375B seq 38 len 116

You can plainly see the encapsulated pings. Congratulations, IPSEC is working.

10. We have found it helpful to mount a /kern filesystem and view the table of current SA/SPI’sincluding which have flows (outgoing SA’s) or not (incoming SA’s). There are also trafficcounters that you can use to see what traffic is going to where. To mount the /kern filesystemdo the following:

# mkdir /kern; mount -t kernfs /kern /kern

11. For more information, use netstat -nr -f encap to see your Security Associations.

Encap: Source Port Destination Port Proto SA(Address/SPI/Proto) 192.168.0/24 0 152.7.15/24 0 0 152.7.30.10/50/require/in

152.7.15/24 0 192.168.0/24 0 0 152.7.30.10/50/require/out

12. Now we have IPSEC configured and working. We’re almost done. OpenBSD includes astateful firewall called IP Filter (or IPF for short). We need to setup IP Filter to allow IPSEC,and a few other protocols, between the two VPN’s and block all other access.

13. The configuration file for IP Filter is /etc/ipf.rules. A tutorial on IP Filter is beyond the scopeof this document. For detailed, up-to-date information on IP Filter please see the official IPFhomepage (coombs.anu.edu.au/~avalon/ip-filter.html). The IP Filter HOWTO can be found at(www.obfuscation.org/ipf/).

14. That said, we can create a ruleset just to handle traffic between catbert and dogbert. With therules below we block in all outside traffic except IPSEC traffic between dogbert and catbert.On catbert we place the following rules in /etc/ipf.rules:

# xl0 is internal interface # xl1 is external interface # # Default deny all rule block in on xl1 all # # Passing in ISAKMP traffic from security gateways pass in on xl1 proto udp from 152.7.30.10 port = 500 to 152.7.20.253 port = 500 pass out on xl1 proto udp from 152.7.20.253 port = 500 to 152.7.30.10 port = 500 # # Passing in AH traffic from security gateways pass in proto ah from 152.7.30.10 to 152.7.20.253 pass out proto ah from 152.7.20.253 to 152.7.30.10 # pass in proto esp from 152.7.30.10 to 152.7.20.253 pass out proto esp from 152.7.20.253 to 152.7.30.10 # # Passing traffic from the designated subnets pass in on enc0 from 192.168.0/24 to 152.7.15/24 pass out on enc0 from 152.7.15/24 to 192.168.0/24 # # Passing SSH traffic for management pass in quick on xl0 proto tcp from 152.7.0/24 to 152.7.15.253 port = 22 keep state pass in quick on xl1 proto tcp from 152.7.30.10 to 152.7.20.253 port = 22 keep state

15. One thing that we recommend you do is install and configure Network Time Protocol (NTP).If the times on either end of your VPN drift too much, IPSEC may stop functioning.Fortunately, NTP is included as a package on CD1 (see OpenBSD FAQ 8.7 for moreinformation about packages). To retrieve the package off of CD 1 mount the CD using thefollowing command:

# mount -t cd9660 -r /dev/cd0a /mnt/cdrom

This assumes that you have already defined a mount point called /mnt/cdrom. The package isstored in /2.9/packages/i386/ntp-4.0.tgz. You might want to pull off the package ntp-doc.tgz,which contains full documentation for NTP.

16. To add the package, su to root and then type "pkg_add -v ntp-4.0.tgz". It’s that simple.

17. Configuring to use the NTP servers is simple. Create a file in /etc called "ntp.conf". Listedbelow is a sample ntp.conf entry:

# NTP configuration file # # broadcastclient server 192.5.41.41 server 128.175.1.3 server 128.46.129.95 driftfile /etc/ntp.drift

18. Create the file /etc/ntp.drift by entering "touch /etc/ntp.drift"

19. In the /etc directory there is a file called "localtime". This file is a symbolic link to/usr/share/zoneinfo/<yourtimezone>. If your timezone is set for Eastern StandardTime/Daylight savings time it will appear as /usr/share/zoneinfo/EST5EDT. You willprobably want to make sure that both ends of your VPN are set to exactly the same zone.

20. Finally, you will need to change the line in /etc/rc.conf (near the bottom of the file) that says"ntpd=NO" to "ntpd=YES". You will probably want to change the line in the same file thatsays "ntpdate_flags=NO" to "ntpdate_flags=YES".

21. The only other configuration change we made was we added SNMP (compiled from sourcecode) so that we could gather statistics from the network interfaces of the VPN’s.

Thanks

We are especially grateful to Tony Sarendal. Without Tony’s help we would still be struggling toconfigure ISAKMP.

Suggested References

OpenBSD, FAQ 13(http://www.openbsd.org/faq/faq13.html)

Patrick Ethier, "ISAKMPD and IPsec in the VPN environment"(http://www.secureops.com/vpn/ipsecvpn.html)

Alcatel, "Understanding the IPsec Protocol Suite"(http://www.cid.alcatel.com/doctypes/technewbridgenote/pdf/ipsec_nn.pdf)

Brendan Conoboy and Erik Fichtner, "IP Filter Based Firewalls HOWTO"(http://www.obfuscation.org/ipf/)

About the authors

Carol Thompson and Bob Sigillito work for Lockheed Martin in Gaithersburg, Maryland managingvarious LAN’s and WAN’s.

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

Logging Syslog to a Database

Zbyszek Sobiecki, <kodz@slonce.com>

0. Problem.

When there’s a problem on your system, or in your network, the first thing you check are systemlogs. You identify which system you should check, then locate the logfile. Sometimes you evenhave to check your syslog configuration, only to discover that what you are looking for is not evenlogged, due to a misconfiguration. You may also run ’less’, ’more’ and ’grep’, to start digging intoit. It’s nice when you find the answer to your problem in the last few lines of log, but what if youcan’t?

What if what you’re looking for is a bit more complex, and you have to analyze more and moredata, combined with log files from other hosts? What if you have no time to waste sitting anddigging through the many "useless" syslog messages? What if you have to find a backup, gzippedinto many files by newsyslog? This is a horrible waste of your valuable time.

1. Idea.

It would be a nice solution to have all network logs located in a central place, like a SQL database.This could solve many of the problems mentioned above: we have one place where we gather alllogs; we have a powerful way to find interesting messages (i.e., complex queries, as SQL supports),and it’s fast. Such a solution has many security aspects. For secure communication between theloghost and systems we can use SSL (e.g. using stunnel[1]), IPsec or any other similar method. Wecan also use something more advanced, like serial ports and a multi-port serial card on the loghost.Logging via serial port is totally independent from the network layer as it provides a transport layeritself -- we can have logs even when TCP/IP is broken, or the network is down. You should alsoconsider using VLANs on operating systems supporting it.

2. Solution.

It’s rather impossible to realize such an idea with standard (included in a system distribution) tools.However, there are many programs we can use. The two most extended and powerful syslogdaemons I know are syslog-ng[2] and msyslog[3]. Msyslog has built-in support for MySQL andPostgreSQL output (as modules). In syslog-ng you need to use some external application likesqlsyslogd[4]. Using external programs should be avoided where possible because ofineffectiveness. There is also a syslogd+mysql[5] package available, which is a patched version ofFreeBSD’s syslogd. It’s a good idea to place SQL procedures only on a central loghost and feed itwith raw data through TCP, UDP, or another protocol, using an encryption scheme, asforementioned.

3. Implementation.

In this chapter, I’ll describe how to configure everything to work the way we expect it to. I assumethat a smart administrator won’t have any trouble implementing solutions described in this article. Iwould like to point out that I’m just giving some ideas and hints, feel free to contact me if youwould like to discuss any of them.

Let’s say that we have an imaginary network with some servers - "sv1", "sv2" and "sv3" and onecentral loghost called... "loghost". sv1 is running IRIX, sv2 - FreeBSD, and sv3 is some otherdevice, not capable of doing anything more advanced than simply sending logs in udp packets.

loghost On our logging host, we should install an operating system that is able to do everything weexpect it to do, including:

run SQL database of a choice. support many encryption schemes - we can have some strange devices in our networkand our loghost should be as compatible with them as possible. support communication via some ’hardware’ channels like dedicated (serial) cables, etc.support effective packet filtering: it should be restrictive, in order to preventunauthorized access, DoS attacks and other nuisances. if you’re going to send logs over the network, a loghost should be placed in the middleof it, to have easy access to all logging hosts.

sv2 sv2 is running *BSD (or other modern opensource OS), so we have many ways to transfer logmessages The Right Way(TM); for example, we can use syslog-ng, which runs beautifuly onFreeBSD. However, default FreeBSD’s syslogd would be enough to use. We can chooseIPsec as our encryption scheme, so that our applications settings will be basic - when thesecurity layer is up, we simply set redirection of our logs to loghost ("*.* @loghost" or so).

sv1 sv1 is running IRIX. I haven’t seen any reasonable IPsec implementation for this platform,but it doesn’t mean that there aren’t any. However, we still can use a SSL tunnel to solve thiscase and set up the tcp connection to our msyslog tcp-plugin on loghost. We’ll have to runstunnel[1] in daemon (server) mode on our logging host and in client mode at sv1 (logs willbe simply written to stdin of stunnel, which will pass them into listening socket on loghost) asfollows:

------sv1--------------syslogd -pipe-> stunnel -ssl-tcp-> stunnel -loopback-tunnel-> msyslog(im_tcp) -> sql ----------------------loghost--------------------

sv3 Let say that sv3 is some mysterious device, not smart enough to transmit encrypted logsstraight to our loghost. Depending on what this device is able to do, we would choosedifferent ways of doing it. The solutions I can imagine at the moment are:

put another ethernet card into the device and connect it with loghost through a dedicatedlink, either a cross cable or a dedicated network. configure a VLAN on a switch with loghost and sv3 as its members.

connect them directly using a serial cable if there’s a serial port available. if anything else fails, put another computer near the device, feed logs into it securely(somehow) and send them via any secure channel.

Remember that it is very important to select the simplest solution, but with security in mind.

4. Management.

Log management is a very important thing. This includes log backups, rotation, and tools forbrowsing log messages. It’s not good for a database to continuously grow. We should dump it everynow and then, backup and leave the possibility to dig into them everytime. Default table structure inmsyslog for SQL database is simple and consists of: date/time of receiving, the host that logged themessage and the message line itself. I’ve added one more field - the self incrementing index, whichis useful when we need to locate something by message number, offset, or other time independentcriterion.

Generally it’s quite a good idea to write a backup script (you could use pg_dump for postgres orsomething similar for other databases) and run it periodically from cron as often as you need to fityour local backup policy. You should store log backups in some reasonable way, to have thepossibility of importing them back into the database for browsing, but this is not the subject of thisarticle. You can always have two instances of the database: primary, for gathering current logs inrealtime and searching through recent messages; and secondary, for feeding older records anddigging in them. This can look a bit complex, but with some simple perl/shell scripts the wholething gives us a fast and powerful log management tool.

5. Security.

While realizing the centralized loghost project, you must remember that logs contain very importantinformation. Not only for you as a system/network administrator, but also for an intruder trying tobreak in. There’s even a possibility to write automatic network activity and log gathering agents toperform distributed metastasis, using information from collected log messages. You should turn offany other service at loghost, leaving only these needed to receive log messages, filter ports, giveaccess to send logs only from specified hosts, setup MAC filtering on switches, etc. - but that’s adifferent story.

6. Notes.

I’m working on set of scripts and tools to automate log management, but they are not yet finished.You can always contact me for more information at kodz@slonce.com.

I would like to thank Kamil Andrusz and Maciej Kozak for support.

[1] - stunnel - SSL tunnel - http://www.stunnel.org/ [2] - syslog-ng - Syslog next generation - http://www.balabit.hu/en/products/syslog-ng/ [3] - msyslog - Modular syslog - http://www.corest.com/solutions/products.html [4] - sqlsyslogd - SQL syslog extension. - http://www.frasunek.com/ [5] - syslogd+mysql - Patched FreeBSD syslogd - http://keves.org/dev/files/syslogd+mysql.tgz

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

IPv6: Need and Trends

Girish B Hampali <shylabanu@technologist.com>

Abstract

This memo introduces the need of IPv6 and its trends in the current Internet. Suggestions andcomments are requested. Distribution of this memo is unlimited.

IPv6: Need and Trends

The network layer protocol, Internet Protocol (IP), was designed in September 1981 for use ininterconnected systems of packet-switched computer communication networks. The design goal ofthe network layer protocol was to provide the best effort way to transport datagrams from source todestination without regard to their network or any associated networks.

The purpose of IP is to move datagrams through an interconnected set of networks. In IP,datagrams are routed based on the Internet address. Internet addresses are defined and are of a fixedlength of four octets (32 bits). Internet addresses have multiple classes. Addresses are comprised ofnetwork identifiers and host identifiers. IP has been serving the Internet community for the last 20years.

The present trends and requirements of the Internet are entirely different and will be more differentin the days to come. The key problems with the present addressing scheme are: the difficulty ofconfiguration, the finite amount of address space, and its inability to cater to the needs of thepresent networking world. Various mechanisms were developed as solutions to alleviate theselimitations, including Dynamic Host Configuration Protocol (DHCP) and Network AddressTranslation (NAT). These are sufficient, but have their own limitations.

Hence, the network world started hunting for a new addressing mechanism. It was destined to find aspecific future direction for the replacement of the current version of IP, and the result was IPv6.Engineering is not a task -- it’s a phenomenon. You engineer something and require re-engineering;IPv6 is an effort in re-engineering IP.

IPv6 protocol recommendation includes a simplified header with a hierarchical address structurethat permits rigorous route aggregation. In addition, it is large enough to meet the needs of theInternet for the foreseeable future. The protocol also includes packet-level authentication andencryption along with plug and play auto-configuration. It also includes the ability to label trafficflows.

IPv6 is an evolutionary step, rather than revolutionary, from IPv4. Functions that are generally seenas working in IPv4 are kept in IPv6. Functions that don’t work or are infrequently used are either

removed or made optional. Few new features are added where the functionality was felt to benecessary. IPv6 is designed to enable high performance and scalability. Scalable networkingrequires careful utilization of human resources as well as network resources. Some of the importantfeatures of IPv6 are:

expanded addressing and routing capabilities, simplified header format, support for extension headers and options, support for authentication and privacy, auto-configuration and source routes, capability of providing ’quality of service’.

IPv6 addresses are 128 bits long. There are three types of IPv6 addresses: Unicast, Anycast, andMulticast. Unicast addresses identify a single interface. Anycast addresses identify a set ofinterfaces, such that a packet sent to an anycast address will be delivered to one member of the set.Multicast addresses identify a group of interfaces, such that a packet sent to a multicast address isdelivered to all of the interfaces in the group.

There are no broadcast addresses in IPv6; their function has been superseded by multicastaddresses. IPv6 addresses are 4 billion times the size of the IPv4 address space. This works out tobe 340,282,366,920,938,463,463,374,607,431,768,211,456.

The ’6bone’ is an international IPv6 test-bed network. It provides testing of IPv6 implementationsand standards, and testing of transition strategies. A diverse community of users, ISP’s anddeveloper organizations are involved. To date, ’6bone’ has tested numerous implementations ofIPv6 hosts and routers produced by data-com giants like Cisco Systems, 3Com, Bay Networks(which is now part of Nortel Networks), Digital Equipment Corp. (now owned by Compaq) andFujitsu. The University of New Hampshire’s InterOperability Laboratory is another testing venueof IPv6.

IPv6 is aimed as a simple and flexible transition from IPv4. Rapid adoption of IPv6 is not possible;at the same time, we cannot wait till IPv4 address space is exhausted. It is clear that the transitionmust to be completed before IPv4 routing and addressing breaks. The transition will be much easierif IPv4 addresses are still globally unique. The other two transition requirements are flexibility ofdeployment and the ability for IPv4 hosts to communicate with IPv6 hosts. There will be IPv6-onlyhosts, just as there will be IPv4-only hosts. The capability must exist for IPv6-only hosts tocommunicate with IPv4-only hosts globally while IPv4 addresses are globally unique. Manyupgraded hosts and routers will need to retain downward compatibility with IPv4 devices for anextended time period.

Further Reading

RFC 2373 IP Version 6 Addressing Architecture RFC 2374 An IPv6 Aggregatable Global Unicast Address Format RFC 2460 Internet Protocol, Version 6 (IPv6) Specification RFC 2461 Neighbor Discovery for IP Version 6 (IPv6) RFC 2462 IPv6 Stateless Address Auto-configuration RFC 2463 Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification RFC 1886 DNS Extensions to support IP version 6 RFC 1887 An Architecture for IPv6 Unicast Address Allocation RFC 1981 Path MTU Discovery for IP version 6 RFC 2023 IP Version 6 over PPP RFC 2080 RIPng for IPv6 RFC 2452 IP Version 6 Management Information Base for the Transmission Control Protocol

RFC 2454 IP Version 6 Management Information Base for the User Datagram Protocol RFC 2464 Transmission of IPv6 Packets over Ethernet Networks RFC 2465 Management Information Base for IP Version 6:Textual Conventions and General Group RFC 2466 Management Information Base for IP Version 6:ICMPv6 Group RFC 2467 Transmission of IPv6 Packets over FDDI Networks RFC 2470 Transmission of IPv6 Packets over Token Ring Networks RFC 2472 IP Version 6 over PPP RFC 2473 Generic Packet Tunneling in IPv6 Specification RFC 2507 IP Header Compression RFC 2526 Reserved IPv6 Sub-net Anycast Addresses RFC 2529 Transmission of IPv6 over IPv4 Domains without Explicit Tunnels RFC 2545 Use of BGP-4 Multi protocol Extensions for IPv6 Inter-Domain Routing RFC 2590 Transmission of IPv6 Packets over Frame Relay RFC 2675 IPv6 ’Jumbograms’ RFC 2710 Multicast Listener Discovery (MLD) for IPv6 RFC 2711 IPv6 Router Alert Option

Several web sites to navigate.

http://www.ipv6.org http://www.ipv6.com http://www.6bone.net http://www.ocean.ic.net/ftp/doc/nethist.html http://www.hitachi.co.jp/Prod/comp/network/pexv6-e.htm http://www.6tap.net/ipv6-exchanges.html

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

Hey! Mister Answer Man

by Todd Whitesel

The Mailbag defeated!

Last column’s mailbag has been crushed by a withering barrage of answers from our readers.

List of Topics

How do I save my screen arrangement for the window program? I’m trying to get taylor uucp working on Mac OS X, help! Tracking down culprit of spurious dialing with ppp -demand. Spurious ppp -demand dialing caused by named. Spurious ppp -demand dialing caused by sendmail. Mailbag: Questions I didn’t get to, or didn’t have a clue about.

Q:How do I save my screen arrangement for the window program?

Regarding question #1 of the September Mailbag:

I’m a long time user of NetBSD but a new user of NetBSD’s groovy window(1)program. I’ve figured out most of it but I don’t understand how to save the way I haveall of my windows arranged. Can you tell me?

Two readers write in:

A:

1. Take a look at the screen package. screen is not as fast as window, but it has more and betterfeatures, and saving your screen setup is trivial. You can leave most of your virtual consolesunconfigured, and use screen to call them up as you need them. The interface for screen isvery similar to minicom, if you have ever used that.

2. There is no automatic way to save your screen arrangement with window. You just writedown the screen coordinates of the windows as you want them, and then manually set up your.windowrc file. The syntax of the .windowrc file is pretty standard, but unfortunately the manpage lacks a good example. Still, a careful reading of the man page is enough to get youstarted.

Here is an example of what I use:

# start example --------# Test .windowrc file from Sascha Welter# Close all windows, in case any got openedclose all## now open one big window - approx. 3/4 of the screenmyrows = ( $nrow / 4 ) * 3 - 1mycols = $ncolwindow ( 1, 1, $myrows, $mycols )# the variables $nrow and $ncol get set by the window program # to the actual screen coordinates## Now another window, about the lower 1/4myrows = $myrows + 2mycols = 1window ( $myrows, $mycols, ( $nrow - $myrows ), ( $ncol - $mycols ))## assign fancy labels to our windowslabel ( 1, "Editing" )label ( 2, "Shell-Stuff" )## select window to work withselect ( 1 )# caution: vi (and maybe other screen oriented programs# don’t like a window that is too small!# end example --------

A quick summary of what it does: 1. Clean up (close all). 2. Set up any windows you like (window rows, columns, rows, columns). 3. Label the windows. 4. Select the one you want to have active when you start.

Note the spiffy little calculations to adjust for different screen sizes (or terminal windowsizes).

Q:I’m trying to get taylor uucp working on Mac OS X, help!

Regarding question #2 of the September Mailbag:

I’m trying to get taylor uucp working on Mac OS X. It builds, but I get the followingerror message when I try to queue jobs:

uuxqt - - (2001-05-31 18:03:01.41 16746) ERROR: opendir (LCK.XQT.0/X.): Is a directory

A reader writes in:

A:I’d suggest running it through strace, truss, struss, whatever system-call tracing utility is availableon Mac OS X.

Scour through the logs until you find exactly what is failing. It’s most likely something other thanthe actual opendir() call.

Q:Tracking down culprit of spurious dialing with ppp -demand.

Regarding question #3 of the September Mailbag:

Ever since I set up our home LAN, I have been running ppp on FreeBSD 4.3 in-demand mode. (I am using -nat if that matters, although I don’t see why it should.)I’m also using -demand to have it only take the phone line when it has something to do,but every once in a while the computer will spontaneously dial without my provocation.How can I track down the process that triggered the dialing?

Many readers write in:

A:

1. A way of tracking down the type of traffic, and thus narrowing down the processes that giverise to it, is to use tcpdump or ethereal.

I suspect that it’s either sendmail (which does many DNS lookups, even when sending mailto localhost), or netscape (which occasionally does lookups, for reasons known only toitself).

2. Try lsof -i to list the open network connections. 3. Try to catch the thing in the act with netstat or fuser. 4. The solution comes in 2 steps: determine what traffic is raising the link and then find some

suitable way to block it.

Start by logging all outgoing traffic on ppp0. A rule like the following could do the trick:

Pass out log on ppp0 from any to any

Next, reload ipfilter with

ipf -Fa -f /usr/local/etc/ipf.rules

(Those of us with firewalls already set up would just check to see that the log keyword isincluded in a rule associated with ppp0.)

Then, run ipmon and wait. You will eventually see something like:

Sep 17 15:02:35 gw-system ipmon80262: 15:02:30.894511 1x ppp0 @10:1 p10.2.3.4,2362 -> 214.1.1.253.666 PR tcp len 20 40 -A IN

So, we see our IP 10.2.3.4 is attempting to contact port 666 on a remote server. All we need todo now is add a rule like:

Block out quick on ppp0 proto tcp from any to 214.1.1.253 port = 666

The solution is more complicated if you only want to prevent traffic from raising the link,

instead of blocking it completely. Add commands to the ppp.linkup and ppp.linkdown scripts,to switch between two different firewall rule sets (one for when the link is up and one forwhen it is down). The "up" ruleset would not restrict normal traffic, and the "down" rulesetwould only allow traffic that is allowed to raise the link.

Add this to ppp.linkdown:

ipf -Fa -f /usr/local/etc/ipf.down

and this to ppp.linkup:

ipf -Fa -f /usr/local/etc/ipf.up

Traffic originating on the machine with the PPP link can be traced to its source by usingsockstat to see which PID is associated with the traffic.

Q:Spurious ppp -demand dialing caused by named.

A reader writes in:

A:I had a similar problem and found it was caused by name resolution. If you set up a local nameserver on the box, then it should not need to dial out to resolve names. This server would be keptup-to-date by a cron job which would dial out on a more reasonable schedule.

Q:Spurious ppp -demand dialing caused by sendmail.

A reader writes in:

A:On my network, this problem was caused by sendmail, which is enabled by default on manysystems. Typical configurations clear the mail queue every 15 minutes or so; perhaps this matchesthe frequency of your spurious dialing incidents.

If this is your problem, switch off sendmail and create a cron job to clear the queue once a day, orhowever often you don’t mind the phone being dialed.

# echo sendmail_enable=\"NO\" >> /etc/rc.conf# crontab -l > /tmp/crontab.root# echo 0 5 \* \* \* /usr/sbin/sendmail -q >>/tmp/crontab.root# crontab /tmp/crontab.root# rm /tmp/crontab.root

Another solution is to configure sendmail to only operate over the localhost interface, and allowonly local delivery of mail. If you do not need direct email service to the outside world, then this isa worthwhile option to consider.

Mailbag: Questions I didn’t get to, or didn’t have a clue about.

1. On FreeBSD 4.x, if I launch kdm from an rc script, the keyboard does not work; I cannot login unless I have a null password. The mouse, on the other hand, works fine. I have seen thison five different PCs, one of which was a laptop.

The first time I noticed it was when I installed KDE 2.0, but I seem to remember trying xdmand having the same trouble. It works fine if I log in as root once the system is up and execkdm from the command line, but I want something automatic.

2. Why, when rebuilding a FreeBSD kernel with some option commented out, does it recompilethe related source files anyway?

Example: I have all NFS kernel options commented out of my kernel, yet the kernel build stillcompiles /usr/src/sys/modules/nfs/../../nfs/nfs_serv.c anyway.

Do you have questions for the BSD Answer Man? Send them to bsd-answerman@toddpw.org.Any email sent to this address is assumed intended for publication and will become the propertyof Dæmon News. That’s all for this month, folks. Until next time, remember: there’s no shame in asking RTFMquestions any more, because these days, there is just too much FM to R.

About the Author

Todd Whitesel has been grokking computers for fun since his first grade school Apple II in 1980,and doing it for a living since 1992, when he escaped from Caltech with a B.S. degree. He helpspromote Japanese Animation in America by running Registration for Anime Expo, and helpspromote NetBSD by way of his NetBSD Architecture Farm.

[home| mail]

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. All Rights Reserved.

November 2001 Search Submit Article Contact Us Join Us Merchandise

Wes Peters, <wes@softweyr.com>

Disillusioned

Autumn is usually my favorite time of the year. The weatherin Salt Lake City is generally mild during the fall, with gentlebreezes, warm days, and cool nights. We spend weekendssailing on the Great Salt Lake, hiking and picnicking in themountains, and enjoying the sunsets and the stars at night.This fall has not been an enjoyable time for me, on manylevels.

This fall has been a time of failing financial markets, my own40th birthday and putting off my mid-life crisis,unprecedented terrorist attacks, international unrest, risingunemployment in the ranks of my friends and colleagues intechnology fields.

It’s not just those cataclysmic events, but the feeling that wewere somehow on the verge of something important, and nowit is all slipping away in the face of more immediate, but notnecessarily more important, concerns. Computers andnetworks seem pretty unimportant compared to jets flyinginto buildings and horrible diseases in envelopes. But what ifwe were on the verge of eliminating the need for "snail mail"or huge concentrations of people in one place?

I’ll try to focus on the BSD related parts of my ennui for theremainder of this column, and won’t bother you with tales ofhot weather lasting well into October, a 3-year drought thatshows no signs of letting up, or of having much of the civilizedworld crowded into my back yard this winter adding to theaccumulation of nonsense in my life.

Is there an email client in the house?

A couple of weeks ago I bought a new hard drive for my aginglaptop, Homer [1] . The 4GB drive that came with Homer, aSony VAIO, was filled to the brim with the software I useevery day, and with software I am experimenting with. Yes,those darned FreeBSD ports developers have made it way tooeasy for me to just plunk yet another email client on myfavorite computer.

The problem is, none of them suits my needs. I have beenclosely following the development of a number of email clientsfor several years now, anxiously awaiting one that would allowme to ditch Netscape Communicator for the limited andcrash-prone rusty tool it is. I’ve been installing, trying, anddeleting email clients so frequently over the past year, myhome directory on Homer became cluttered with various andsundry email configurations, carefully moved away and savedas I tried the next client.

What I really want is a really reliable, easy to use IMAP clientthat supports using folders on the server. I have a reasonablygood backup program in place on my mail server, and I havefast access to it from both office (typically 10 ms) and fromhome (switched 100Base-TX network).

I’m willing to settle for something that works correctly with alocal mailbox, has a decent editor or can call an external one,and has a reasonable degree of integration with otherapplications. Being able to send a URL to my running webbrowser, or to launch a picture viewer application like xv for agraphics file is essential. I do eventually want to get to theIMAP-based solution, but am willing to use fetchmail and alocal mailbox with a client that shows promise.

The situation is slowly improving. KDE2’s kmail application,for instance, almost works. It also drags along with it so muchadditional code, network object daemons, support for variousdongles and such, that it is unusable on a Pentium II 300 MHzlaptop. You youngsters can stop snickering now, a PII-300with 192MB SDRAM is a pretty considerable computer, andthe inability of an email client to run reasonably fast on it isabsolutely shameful. I realize kmail (and all of KDE) asdelivered by the default port still has a lot of debugging codeenabled, but good grief! When running IMAP, it seems towork well enough, but then some sort of I/O daemon dies andit falls into a paroxysm of trying to reconnect to the server.Sigh.

What are the Trolls up to?

On the other tentacle, we have the GNOME project and their"email client of the week" program. The first one of note,following in the tradition of Elm and Pine, was Balsa. Balsalooks lovely, and I’ve tried it a number of times over the pastyear, but each time I can’t quite get it to do what I want it to.Sometimes POP3 works well but IMAP can’t be configuredfrom the GUI. The next time I try it, IMAP works but canonly be configured by editing the undocumented configuration

file, and POP3 is broken. The next time, IMAP sort of works,POP3 works, but local mailbox support doesn’t, and I ofcourse have to upgrade to a newer version of the GNOMElibraries, requiring another 6 or 7 hours of downloading,compiling, and installing libraries before returning to buildingthe email application.

Just to be fair, I started a make in the ports directory for thelatest release of Balsa as I began to write this article. It’scurrently building an updated db3 library. Great. I justupdated everything last week, when I installed XFree86 4.1 onthis machine, but I guess several updates have come out sincethen. This 12GB disk is starting to feel mighty small all of asudden...

The next great advancement in Open Source,UNIX-compatible, GUI email clients is one that really puzzlesme. Ximian’s Evolution launched with quite a splash a coupleof months ago, being one of the few commercially supportedopen source programs available. That is, you can downloadthe source, compile and run it yourself as with any other GNUGPL software. Or, you can pay someone to put it on aCD-ROM for you, and have them provide a phone help deskyou can call if you have any problems with it. Wait a minute,hasn’t somebody else already tried that business model?

Even more puzzling than the anti-news surrounding thecommercial arrangements at Ximian is the overall design goalof Evolution: to duplicate the look and feel of MicrosoftOutlook. I find this puzzling because in my experience,Outlook is second only to Internet Explorer in the rage andhatred it raises among most of the open source folks I hangout with. I end up in this quandry over many of the GNOME

applications: if we hate the Microsoft applications so badly,why are we trying so hard to duplicate them? Is it just theMicrosoft-ness of them them we hate? The fact that they run[supposedly] on Windows? The outrageous prices?

Actually, this is my biggest ongoing complaint about the entireLinux movement. Why set out, in 1990, to build Yet AnotherUNIX? Why not latch onto something better, newer, morenetwork ready, more portable, more mobile, more something?This same thinking seems to dominate the thinking with bothKDE and GNOME projects: we can do just as good asMicrosoft. What Microsoft does isn’t nearly good enough forme, so just as good isn’t going to cut it. I want better,remarkably better.

I tried Evolution a couple of months ago. It was not ready forprime time then. Just to see how it’s going, I’ll build the latestport and give it a whirl if Balsa ever finishes building.However, this application is not likely to woo me. It does havean impressive array of features, including the ability tosynchronize address books with PalmOS PDAs via GNOMEPilot, but they seem to have still missed the boat on corestability.

The Balsa port is now updating gtk+, so I may never actuallyget to try Evolution. I suppose I’ll need gtk+ that to build themost recent Evolution anyhow. Oops, the Evolution portseems to be quite out of date. I guess I’ll skip that one for now.

Last, but not least, comes the newest entry in the GNOMEemail client race, Sylpheed. Sylpheed aims for quick responseand a graceful user interface, and pretty much achieves thesegoals. The version I have been testing has some

synchronization problems with IMAP servers, occasionallylosing the ability to download further messages from theserver. Exiting and restarting the program seems to clear upthe problem, so it, at least, isn’t saving bad data where it cancontinue to cause harm.

It finally built!

Sylpheed, that is. I fired it up and used it to respond to someurgent Dæmon News email, and it worked OK. It still gotstuck on an email from Gary Kline, insisting it couldn’t fetchit, until I exited the program and restarted it. Sigh. The Balsabuild, meantime, is dragging in an updated version ofGNOME Control Center.

This is one of the problems with the KDE and GNOMEapproach to building applications. I can see where it might beamusing to change a setting in a control panel somewhere andhave all your applications blink and display the new settings,but at what expense?

The UNIX world used to be one where you could count on thefunctionality being nailed down before the bells and whistlesshowed up. Now it seems that we get the gew-gaws first andthe functionality shows up gradually. I’m willing to foregoaddress book integration and theme support if it requiresdragging a relational database and an object broker into myemail application.

Is there a point in here somewhere?

Many of us use various version of BSD for our daily work, andwish the rest of the world would too. This evening, just beforeI started to write this column, my sister called and wanted

help with her computer. She was trying to print a simpleheading for a poster her daughter was making, but her ratherlimited old Windows 98 machine wouldn’t allow her to print asingle page of 72-point text from Microsoft Word, itcomplained of not having enough memory. I hate providingphone support for Microsoft products, and refuse to do it foranyone other than close family members.

Much as I’d like to, I realize that I cannot possibly move userslike my sister (or my wife, a Win2K and Word user) onto BSDmachines until we can offer them applications (and printersupport) that is the equivalent of Windows. We must becareful about rushing headlong into replacing Windows, lestwe fall into the trap presented by Ximian in Evolution: if wemerely strive to duplicate the Windows feature set, we willnever produce anything better than Windows. We will, asHenry Spencer warned of so many years ago, reinvent UNIXpoorly. Or in this case, reinvent Windows poorly--I shudder tothink how bad that might be.

In all this waiting I keep hoping to find an email client that issomehow delightfully better than the run of the mill. I’m notlooking for hand-chiseled gewgaws decorating the screen, justsomething where the designer spent some time thinking aboutthe tasks associated with reading, creating, and responding toemail, and implemented them in a clean, intuitive userinterface. I know there is an open source programmer or teamout there somewhere who have carefully considered the natureof email and produced just such a program, if I can just findit.

As I finish off this article, the Balsa build is downloading anupdated version of gnome-core while the Mahogany build is

dragging in Python 1.5, despite my already having Python2.1.1 installed. Both of them seem to require Ghostscriptwhich seems redundant since I have a PostScript printer atwork and home.

Heaven help me and my disk.

[1] No, not named after Homer Simpson! Doh!

Named for Homer the wandering writer. Remember him? The Odyssey, The Iliad, etc? return

Author maintains all copyrights on this article.Images and layout Copyright © 1998-2001 Dæmon News. AllRights Reserved.