Effective Linux Migration Processes

Copyright © 2005 Wind River Systems, Inc.

Effective Linux Migration ProcessesWhite Paper

Dan NoalSenior Director, Wind River Professional ServicesWind River Systems, Inc.

Tim FaheySenior Manager, Wind River Professional ServicesWind River Systems, Inc.

Migrating your device application software to a new operating system can be a thankless, risky prospect. But the risks can be mitigated and success assured if the proper plan is put in place. This white paper proposes a framework for just such a plan, and it notes specific areas of concern when the new operating system is Linux.

� EffEctivELinuxMigrationProcEssEs Copyright © 2005 Wind River Systems, Inc.

1 Why Linux for Device Software?Readers interested in the topic of accomplishing a Linux migration are likely to have made the decision to migrate a device, product line, or entire development staff to Linux. Recent industry data, such as The Embedded Software Strategic Market Intelligence Program by industry analyst Venture Development Corporation (VDC), indicates that difficulty or issues associated with migrating to Linux are not gating factors inhibiting Linux adoption.

The driving forces behind the growth in Linux adoption in the device software market typically consist of a combination of business and/or technical issues. Typical open-source benefits often cited as driving factors behind selecting Linux for device software projects include:

• No royalty fees• Access to source code • Software available from multiple commercial and noncommercial sources• High-quality reputation• A large pool of available development talent• Extensive communication or integration capabilities• Availability of support for desired processor

Of these benefits, perhaps the most frequently cited decision driver is the lack of royalty fees. Cost savings from a lack of royalty fees is the most prevalent contributor to projecting a lower total cost of ownership for Linux over other device operating systems.

2 Why a Commercial Distribution?One of the huge benefits of Linux is its ubiquity: Linux seems to be available anywhere from anybody. After selecting Linux, your next decision is whether to assemble the distribution yourself or buy a commercial distribution. Those doing serious due diligence on their Linux decision, especially when trying to project a total cost of ownership, are moving toward a commercial distribution.

Linux customers are asking themselves, “What business am I in? Am I in the operating systems business or the application software business? If I’m not in the operating systems business, why am I spending scarce R&D funds to source, assemble, test, and deploy an operating system, when I can get a preintegrated, pretested, open-source OS from a commercial vendor?”

They also ask, “What else can I do with those R&D funds? Can I create dynamic new features that solve my customers’ problems better and faster than my competition? Is that a better way to provide solutions for my customers?” Increasingly, the answers to these questions drive Linux users to commercial distributions.

Another factor driving a commercial distribution decision is time-to-market: Linux customers have realized that it takes time and resources to roll their own Linux distribution. If selecting a preassembled and validated distribution commercially mitigates time-to-market issues, this can help bring a product to market sooner.

Linux customers are

asking themselves,

“Am I in the operating

systems business or the

application software

business?”


For some, the availability of support services or consultants from their distribution provider bridges a gap between in-house skill sets and what’s needed to bring the product to market. Timely vendor support, as opposed to support from the open-source community, can decrease time-to-market and increase customer satisfaction. And these have all tipped the scales toward a commercial distribution decision for some clients.

Having access to the vendor’s partner ecosystem, especially if open-source support for the hardware you’ve chosen is not available, can mean the difference between success and failure. Again, the question comes down to this choice: Do you want to spend your resources linking an OS and hardware together, or writing application code?

Real business and technical factors are driving the open-source community to look to the commercial sector to help deliver on the promise of Linux. With the decision for your distribution’s source, you select the source for your development tools or development suite. Again, there are benefits to both an open-source and commercial approach.

When your tools come solely from the open-source community, you have the highest level of control, because you have the source code and can manipulate the tools in any way you want. Also, you can also choose from any number of other compatible tools to add functionality that you need for your project.

Open-source also means not having to sign any contracts for software or software support. For some users, not having to manage another vendor’s contract is relief from an unrewarding administrative task.

Open-source tools may have lower nominal costs, since there is no direct exchange of money (although you should note that, like going the roll-your-own route for your operating system, rolling your own tools does have opportunity costs that should be fully vetted before going down this road).

Figure 1. Why switch to a commercial Linux distribution?


Some have said that since the tools are free and widely available, the project can start more quickly. However, other developers have said that needing to integrate disparate tools has an impact on project start and completion times.

On the commercial side, several commercial vendors offer integrated development suites. Commercial products have commercial-grade quality assurance before they are shipped, although most open-source advocates will stand behind the QA delivered by the open-source community. More important than QA done on any given tool is QA done on the entire suite—and you’ll need to do that QA yourself if you take the open-source route.

Some of the commercial tools vendors offer development suites with specific industry development tool requirements preintegrated. With a commercial suite, the heavy lifting of tools integration has been done, leaving you to focus on application software. Some integration may still be required based on what the customer wants to add, like any third-party plug-ins. Commercial tools vendors may offer additional capabilities, like memory analyzers or on-chip debuggers, which may not be available in the free open-source community.

In addition, commercial solutions offer support and professional services to assist you when you come across obstacles in the development process.

3 Conversion Project IssuesYou’ve selected your target OS, you’ve selected your OS source, your development suite is idling out back—you’re ready to go. But before you jump into drive, there are a few things you should do to make your project more successful, as well as make your life easier later.

First, take the time to lay the foundation for measuring the success of your migration later. Ask yourself the question, Why are you undertaking this change? Have you set goals to define a successful outcome of your decision to migrate to Linux or to commercial-grade Linux, and have you established a way to measure the effectiveness of your decision? If you chose a commercial distribution to reduce the total cost of ownership (TCO) over time, how will you know if you do? Which factors will be measured, and which will not?

Look at it another way: How can you prove quantitatively that you made the right decision? What could the measurements be?

First, keep in mind the warnings about return on investment (ROI) or TCO, or whatever you’re going to use to measure success. At best, your mileage may vary. Common measurements of a reduced TCO include such factors as lack of royalty licenses, less expensive code maintenance, and reduced staffing costs. The more elusive measurement is strategic enablement—meaning that by selecting Linux, and by selecting a commercial distribution, you have enabled downstream product or investment options that created competitive differentiators or opened the doors to real benefits for your company, doors that would have been closed had you made a different decision. But if you don’t establish the baseline now, you’ll never know.

You should also thoroughly vet your assumptions—being wrong at this stage will impact your ability to achieve your desired TCO benchmark. Also, identify your risks and implement a rigorous risk mitigation plan. Risks have a nasty way materializing when ignored, and unaddressed risks will inevitably increase your TCO.

With a commercial

suite, the heavy lifting

of tools integration has

been done, leaving you

to focus on application

software.


Objectively assess your team’s capabilities compared to your project’s needs. From here, decide what help, if any, you’re going to need. You may need assistance with specific project skills like testing and validation, technology education, or overall project management. You may choose to outsource the entire project to an external service provider and allow your team to continue working on application software advancements.

Finally, you should ruthlessly scrutinize the project plan for the migration. As trite as that sounds, it appears that a lot of people don’t look at data. For participants in delayed embedded software projects, over 40% cite what they call unrealistic schedules as the predominant factor, again according to VDC.

And it gets a little worse when you look at how many projects these delays actually comprise. VDC reports that well over one third of embedded projects are completed behind schedule, and an additional 10% are cancelled outright. There were different factors cited, but in many cases, handling the migration right from the outset may have prevented the preventable from happening. So, how do you avoid becoming one of these statistics? Best practices in Linux migration.

Your team is probably excited about the challenge of a Linux migration. Similar to operating system migrations, conversions don’t always run smoothly or finish on time and on budget. While Linux was originally created as an enterprise operating system, and many programmers with Linux knowledge come from the enterprise industry, code not properly designed and tuned to the constraints of an embedded device can put a Linux migration project at risk. Follow the software development principles that led to your initial success with your task-based solution in your Linux migration.

The Big Rocks

Architecturally, most device operating system solutions are quite different from Linux. You can choose to simplify the migration by choosing the Linux software architecture closest to what the embedded operating system provides, but this may not be the appropriate architectural choice for other reasons:

• The Linux system calls are simply different from what your application code and middleware use today on your embedded-based platform. Your device operating system may not understand the concept of a process like Linux has deployed.

• There are likely to be differences between the I/O architecture of your current device operating system and Linux.

• Interprocess and/or interprocessor communication may look quite different. Scheduling a process and scheduling threads in Linux are likely to be very different than your application experiences today.

So, as you begin this migration process, it is essential to understand the key drivers by which you are going to measure success. The migration’s key performance metrics—which you will measure and define—are at the forefront. It’s absolutely critical to set these performance metrics up front and flow them down to the layers in your software architecture. Deferring performance tuning or deferring this flow down in requirements development to the optimization phases in your final system integration test phase can lead to unnecessary complications and difficulties in problem isolation.


Quality testing will be a major issue for this migration. Creating equivalent functionality and quality in your migrated device is considered table stakes. Functionality, performance, and robustness may need to be better with Linux than with the RTOS version of your device, since investments made in the migration drive a need to show more than just cost improvements.

Another major factor in migrating to a Linux device operating system is the impact on your software development environment. This is likely to represent a large change for your organization, and like any major change, it must be managed, architected, and communicated well. In fact, the impact on your applications is likely to be significant.

There may also be changes in how you handle software requirements and design, configuration management, integration testing, debugging, bug tracking, and other elements of your software development environment. A Linux environment is likely to open up great opportunities for improvement that carry risk to your programmers. A slew of different ways to develop Linux-based software has become available. Your engineers can now, perhaps for the first time, build their own tool chains, debuggers, test utilities, IDE frameworks, and test scripts. This is freedom—that’s what Linux offers. This freedom can lead to chaos, which can be a development manager’s enemy. But the savvy development manager balances liberating creativity with chaos.

In order to maximize the ease and positive impact of your Linux migration, be sure to understand the key technology metrics by which you will measure success of the migration performance. Define the drivers and set the metrics at the onset of the project, and track them through the project as you migrate the layers of your software architecture.

Planning

Understanding the technical issues of a Linux migration is essential, but not sufficient on its own. The project leader must also define the project phases, entry and exit criteria, and success metrics for each phase, and drive your team to meet project objectives.


Figure 2 shows the overall project methodology that Wind River Services employs on all projects, which we call our Device Software Optimization Methodology. It forms a common representation of a software project that follows each of these steps. Rigor and discipline are required, but it is equally important to document what you do, do what you document, and communicate the entire process well to your team members. Any software project follows these familiar elements. Next we’ll review each of these steps, as well as best practices specifically for a Linux migration project.

4 Linux Migration MethodologyStartup Phase

The startup phase defines the fundamentals of a project: end goal and output. During the startup phase, you define the Linux migration build and test strategies. You must also consider the impact of Linux and its changes to the most basic aspects of your software development environment.

For example, Linux will likely change how your organization will plan and execute activities such as design reviews and code reviews. It is likely that you will want to adopt some of the designing and coding practices from the Linux community and those who maintain the key open-source projects you will be leveraging.

Configuration management and software build issues are also likely to be affected heavily by Linux methods, compared to what you are doing today. Developing rock-solid configuration management and build practices and a dependable build-infrastructure will be critical to your project. One common mistake in this phase is building systems that are nearly impossible for

Figure 2. Migration Methodology

Linux will likely change

how your organization

will plan and execute

activities such as design

reviews and code

reviews.


individual engineers to build on the same Linux kernel and in the same Linux environment as other developers or the quality testers. Some inexperienced Linux developers have created build systems for the core kernel, kernel patches, middleware, and other open-source packages that are all distinct. They are all manual and incompatible, which can lead to an untenable and unproductive environment.

How you do testing is likely driven by Linux and the available testing resources provided by your Linux vendor and the open-source community. LTP (Linux Test Project), LM Bench, and other such projects are worth considering in this startup phase.

Requirements Phase

The issue of software requirements remains crucial—even if your migration project will not substantially change the functionality of your system—because you must flow down the requirements of your complete system to the fundamental architectural building blocks of your Linux-based system. This starts at the software foundation level (consisting of the boot loader, Linux BSP, root file system, and the kernel mode drivers your application will require), followed by the optional middleware level, and finally your application. In a migration, the requirements activity generally demands less time on functionality due to porting an existing system, and more time on attributes of your system like footprint, bootup time, and performance. This holds especially true if the migration doesn’t introduce new features to your system. In addition, quality requirements may mandate Linux architectural decisions, such as the partitioning of your application into multiple processes or multiple threads within processes.

Design Phase

The software design phase needs to document how the software will be built and designed, which requires some key decisions. For example, you must decide how your RTOS-based application will be converted into a Linux application, applying what you have already learned about the underlying architecture of the two operating systems, which may be quite different. The design issue potentially affects all of the layers: the software foundation, middleware, and the application. A highly successful strategy lets the middleware layer serve as the primary abstraction point for your application. This approach can isolate changes primarily to this layer, allowing much of your investment in your application code to be preserved.

At this stage, you also plan for and design an integration system task. You may be able to leverage the existing system’s test assets: benches, test equipment, test scripts, and full system testing. If not, these will need to be created. More fundamentally, test, unit test, and component test of each of the levels may look and behave differently with Linux than your previous device operating system.

Code and Unit Test Phase

In the coding stage, your software development environment finally needs to be stable and complete. At this point, your developers are actively using the configuration management environment and debug environment, as well as your build infrastructure and testing assets. The coding standards that you employ may have changed from your previous projects as a result of Linux, and they will be used heavily here.


Often overlooked until late in a project, testing approaches must be planned up front. Choosing Linux may help you to leverage your distribution vendor and/or the open-source community for improved test assets and testing technology. Automated testing may greatly improve test productivity and your time-to-market, but will require careful planning and analysis up front to reach these goals. System testing should be leveraged from your existing product infrastructure, whereas unit testing (such as software foundation testing) will likely look different from your existing system.

Test and Migration Phase

When testing migrations in your final system, you are likely to find that integration testing represents a large investment from your company in testing resources, tested equipment, and so on. To minimize testing costs, leverage the investments made for previous test suites and the test infrastructure built for the original system, and provide the correct system test bench for your Linux-based migrated project. A Linux-based system will impact your build systems, quality systems, release management, and all elements of your development environment. This necessitates careful planning for final test and integration. Final testing is always in the critical path, and it is a gateway to the release of your migrated system.

5 Migration Methodology AppliedWe can apply the framework of a migration project methodology to some of the technical issues that may arise during the conversion effort. This effort may be divided into three areas of activity within the methodology: the software foundation, the middleware, and the application. Of these, the software foundation may not require as much support. Your existing middleware and the third-party middleware that you use are the key focus of your effort. Your application code makes up the final area of consideration.

Software Foundation

The software foundation consists of a Linux bootloader and a Linux Board Support Package (BSP). The help tools for converting these elements have been covered very well in other literature, but it is worth noting that the boot time, footprint, self-test, and software upgrade requirements that exist for your current system must also be met in your migrated system. It’s one thing to have a functional bootloader, but entirely another to have a bootloader that meets your system’s specific requirements.

Many devices can leverage the Linux BSP that may be provided by your commercial Linux vendor, while others require a custom BSP. The custom BSP may start from a “close cousin,” meaning either a commercial BSP or one available in the open-source community. Most conversion projects cannot leverage the existing device BSP, even when using the same hardware in the migration project.

On the other hand, many can adapt existing device driver code, such as device net and memory maps. But in cases where existing driver code cannot be leveraged, the open-source community may prove a useful source for the drivers that are needed. The challenge in this phase of either converting existing drivers or sourcing a new driver from the open-source community or your Linux vendor, lies in this key question: “Does each driver meet my device’s requirements?”

It is easiest to attack

nascent performance

problems at the

bootloader, BSP, or

device driver layer in

isolation from your

middleware and file

applications.


You will also benefit from beginning performance tuning at this project phase. A performance tuning problem may start at the initial software foundation layer. It is easiest to attack nascent performance problems at the bootloader, BSP, or device driver layer in isolation from your middleware and file applications. Deferring all performance tuning to the final integration stages places undue pressure at the final project phase, making problem isolation much more difficult.

Middleware and Application Software

It may be easiest to approach coding middleware and application together. Many applications rely on a middleware layer that provides application services, then abstract the underlying operating system. In fact, this type of architecture can facilitate migration projects to accomplish the supporting layer for the application. This layer should be tested and isolated from the full application, again to assist in fault isolation, relying on the software foundation testing that should have been done already.

During this phase, it is important to determine if, assuming all or part of your middleware was provided by a third party, the vendor offers Linux support. Vendor-provided Linux support can significantly reduce a Linux migration challenge. If your middleware vendor does not offer a Linux-supported version, or if you authored your own middleware, you must determine what changes to the middleware architecture are required.

The middleware conversion strategy must be based on existing middleware or new middleware that has been added. When converting the middleware and application software, your options are to model, emulate, or virtualize your original device operating system. Middleware can either use native Linux constructs or a porting layer that you design. Keep in mind that any middleware changes may impact your application also.

Once you have selected the best way to accomplish the Linux migration and ported the application, full systems integration follows, where final performance metrics are taken. If you have done your job correctly and well in the software foundation and middleware layers, the chances of surprise in the full systems integration are much lower. Also, if the job has been done correctly to the lower levels, changes to the application code itself are likely to be well-isolated and easier to complete.

Integration Testing

There are several options for Linux test resources:

• Leverage your commercial Linux vendor for quality and testing infrastructure.• Use the Linux Test Project (LTP), an open-source community resource at http://ltp.

sourceforge.net.• Perform device driver testing and stress testing, which will benefit your project by resulting

in improved drivers and fault isolation later.• Use middleware testing, which builds on software foundation testing. You may be able

to leverage test suites from your commercial Linux vendor or your previous project. Developing confidence in your middleware apart from your application will require writing test applications, but this can aid fault isolation greatly later on.

Integration testing needs to be planned and executed at each level and at the final system level.

�0 EffEctivELinuxMigrationProcEssEs Copyright © 2005 Wind River Systems, Inc.

Software Foundation Testing

It pays dividends to have a solid software foundation with well-understood qualification and stress tests upon which to base higher-level tests. To get a solid foundation, you need clear requirements to flow down, such as boot time and device driver performance, with well-understood qualification stress tests. Device driver and stress testing isolated from your application code in middleware will benefit your project, resulting in an improved software foundation that helps you in the fault isolation layer.

Middleware Testing

Middleware testing builds on software foundation testing. You may be able to leverage a test suite from a third-party vendor or your previous project, developing confidence in this middleware layer apart from your software application. Alternatively, middleware testing may require additional work in writing a test application to exercise the middleware that you may not have done previously. However, as with software foundation testing, middleware testing can greatly aid fault isolation.

Application Testing

Application testing is the final system-testing phase, your last gating activity prior to product release. You may be able to leverage simulation environments before your product is introduced to its “go live” scenario, but at this stage, you will build upon the solid testing foundation that you have put in place. A well-defined system architecture, requirements flowed down to the lower levels, changes isolated from your application and the port itself, and a well-tested set of code will all help you complete your migration smoothly.

�� EffEctivELinuxMigrationProcEssEs Copyright © 2005 Wind River Systems, Inc.

6 Other ChallengesThere are additional migration issues, more generic than specific to Linux, but your project can fail if you ignore these issues, just as surely as if you chose not to test your device drivers.

Change Management

Moving to an open-source solution can cause profound changes in your development organization. Engineers who were responsible for integrating new technologies into your proprietary OS may now have the task of testing and validating Linux, instead of OS development. Support staff may now manage the responses from a commercial call-center, rather than coding fixes back into your old OS. Have you adequately conveyed their new roles within and outside the department? Is your staff truly on board with the changes? How will these employees’ success now be measured?

Processes

If this is your development team’s first foray into the world of open-source software, your development, distribution, and support processes can change as well. If you didn’t choose a commercial Linux distribution, how will you decide where to download packages and fixes? As a roll-your-own Linux consumer, you need to manage your operating system yourself, in effect putting yourself in the operating system business. Have your process changes been tested and validated? Has everyone who needs to know been informed?

Training

Do your engineers have Linux experience, and if not, how can you get them up to speed on Linux as quickly as possible? The provider of your commercial distribution likely has training services, and these can be a cost-effective way to jump-start your internal competence with Linux. Classroom, on-site, online, and CBT training are ways to gain Linux knowledge. Another way is to hire in consultants to help with your migration and shadow their production, so your internal staff learns as you go. This not only results in high-quality output from industry and technology experts, but also enables you to develop internal competence.

As a roll-your-own Linux

consumer, you need to

manage your operating

system yourself, in

effect putting yourself

in the operating system

business.


7 Criteria for Finding HelpReturn to the activities that started your project. Once your application is migrated, it is time to measure success. Did you get your product to market faster? Did it offer more or better features because you were able to focus more resources on it? Did you grow revenue, or achieve revenue growth more quickly, as a result? Did you reduce costs, without affecting productivity or functionality? Based on the criteria you chose and compared to the baseline you developed, take this opportunity to prove to everyone else how you delivered on what you promised. Demonstrate that you did, in fact, succeed.

To make sure you achieve that success, you might seek help for your Linux migration. How do you select the right service partner?

• Your service partner should have experience in device software migration: Have they done what you need done before?

• Your service partner should have solid project management expertise with a tested methodology—a project roadmap that will get you to end of the job on time, on budget, on spec, and with high quality.

• Linux experience also plays a role, but our research shows that companies like yours consider the first two criteria more important than just embedded Linux experience. But you may as well have it all if you can.

• Does your service partner have domain skills in your industry? This helps your partner to speak your language and understand the issues of greatest importance to you—but a service partner who offers solutions across a wide number of industries has the additional benefit of applying best practices from those industries to bring you the best possible solution.

• Does your service partner offer training to help your project get off to a quick, efficient start? A complete migration solution should include customer education to get your team up to speed rapidly, as well as a plan to keep them there over time.

Ultimately, your service partner should be able to bring people, process, and technology together to help you accomplish the goals you’ve set for your Linux migration.

8 ConclusionWhile a Linux conversion may seem daunting to anyone unfamiliar with the technology, a sound, tested migration process will lay the foundation for your success. Seasoned Linux expertise helps to avoid typical mistakes—if that expertise is not available in your organization, you will be well served to bring in resources who can augment your staff with the necessary experience to ensure a smooth migration. Selecting a commercial Linux distribution offers the dual benefits of saving time, since you do not assemble the distribution yourself, and gaining access to a ready pool of professional Linux migration experts.


About Wind River

Wind River Systems, Inc. 500 Wind River Way Alameda, CA 94501

Toll-Free: 800-545-9463 Tel.: 510-748-4100 Fax: 510-749-2010

www.windriver.com

Wind River, the Wind River logo, Tornado, and VxWorks are registered trademarks of Wind River Systems, Inc. Any third-party trademarks referenced are the property of their respective owners. For further information regarding Wind River trademarks, please see http://www.windriver.com/corporate/html/trademark.html.

.

Documents

Effective Linux Migration Processes