Embed Size (px)
Citation preview
Mainframe Application Reverse Engineering
Key to Unlocking Your Business Content
AUTHORS Srikrishnan Sundararajan, ETS Mainframe CoE HCL Technologies, Chennai Dr Usha Thakur, ATS Technical Research HCL Technologies, Chennai
Mainframe Application Reverse Engineering: Key to Unlocking Your Business Content
© 2010, HCL Technologies Ltd.
December, 2010
The authors would like to thank Abhijit Apte and Mahesh Hegde for their valuable inputs.
Mainframe Application Reverse Engineering 2
Mainframe Application Reverse Engineering 3
Contents Introduction .................................................................................................................................................. 4
Purpose ......................................................................................................................................................... 4
Significance of Mainframes ........................................................................................................................... 4
Limitations of Legacy Applications ............................................................................................................ 5
Moving Away from Legacy Applications ................................................................................................... 6
Relevance of Reverse Engineering ................................................................................................................ 7
Business Drivers of Reverse Engineering .................................................................................................. 8
Reverse Engineering Service from HCL ......................................................................................................... 8
Reverse Engineering ................................................................................................................................ 10
Forward Engineering ............................................................................................................................... 10
Implementation ...................................................................................................................................... 10
Benefits ....................................................................................................................................................... 11
Endnotes ..................................................................................................................................................... 12
Mainframe Application Reverse Engineering 4
Introduction More than ever before today’s end users are very intolerant of delays in obtaining relevant information, non‐interactive Websites, and non‐collaborative applications. Naturally, businesses need to incorporate changing user demands and preferences into their IT strategy. Whereas for modern systems that may not be a very tall order, for mainframe systems it is. Although there is a gradual trend to migrate to newer systems, the bulk of global business transactions continue to be processed by mainframes, especially in industrialized economies. For that reason mainframe systems continue to remain relevant to enterprises. Instead of throwing out the baby with the bath tub, mainframe customers are warming up to the idea of mainframe migration and modernization.
Purpose The audience of this paper is CIOs who are contemplating reducing operational cost of mainframes and supporting the business objectives of their enterprise via a modernization strategy that calls for reverse engineering. The paper examines the main drivers of reverse engineering by organizations that have decided to continue using mainframes and highlights key features of the reverse engineering service from HCL.
Significance of Mainframes Throughout the history of the mainframe, IBM has retained its dominance,1 though not without its share of controversy.2 It is estimated that IBM’s mainframe sales will reach $3.3 billion in 2010, which would represent about three‐quarters of the overall mainframe market of $4.5 billion.3
For sometime now, there has been a popular belief that mainframes are living on borrowed time and may not be around in the near future. Writing in 2003, John R. Phelps of Gartner pointed out that a number of developments over the past 15 years4 led to stories about the death of the IBM mainframe, but the reality has been to the contrary. Currently, a great deal of attention is being given to the declining revenue from mainframes and servers in general,5 and once again we are seeing stories about the end of mainframes in the very near future.
What is interesting, however, is that IBM has continued to modernize and invest in a range of mainframes, improve pricing, and reach out to new customers, especially in newly industrializing economies,6 which indicates that IBM still has confidence in continued business from mainframes. In fact, IBM is solidifying its position in niche accounts. For instance, the zEnterprise system (z196)‐‐latest mainframe model from IBM‐‐is targeted at the top 10% to 20% of its customers where the volume of
Mainframe Application Reverse Engineering 5
processing i.e., CPU usage (expressed in Millions of Instructions Per Second (MIPS)) is very high.7 MIPS is one of the most important criterion that IBM uses for calculating the cost of mainframe license.8
The mainframe story would be incomplete without its customers. Although some customers are on the look out to reduce dependency on mainframes by moving to systems powered by newer technologies, many still continue to use them for historical reasons. Mainframe customers are big companies in data‐intensive industries such as distribution, financial services, general business, and public sector that invested in mainframes several years ago. While reliability, security, and scalability continues to be top priorities for mainframe customers9 they consider the rising cost of owning and running mainframes to be undesirably high and are, therefore, willing to explore possibilities of mainframe modernization strategies such as reverse engineering.
Limitations of Legacy Applications1 Since the primary role of IT departments is to facilitate the smooth running of business operation, it is important for them to be in sync with the business objectives of their respective companies. Consequently, if for instance, the overall operations cost is higher than desired or productivity is sinking to unacceptable levels, then IT departments need to take stock of how they can contribute to bringing various business parameters to acceptable levels. This is precisely the question that business executives of mainframe customers are asking of their IT executives.
It is well known that legacy applications residing in age old mainframe systems are proving to be a liability for mainframe customers who need quick support from their IT systems in order to remain competitive. For businesses in the present day world, the most problematic aspects of the mainframe or legacy systems are as follows:
SILOS OF INFORMATION: The primary goal of legacy applications has been to meet the process flow requirements of individual departments. Users are usually required to login to multiple parallel applications to perform a single task. This is proving to be a problem today because several departments need to collaborate and reach decision quickly. Furthermore, the lack of integration among legacy applications across departments often leads to a loss of up‐selling and cross‐selling opportunities because of the complexity involved in implementing them.
CLOSED DATE STRUCTURE: Accessing information from legacy data structures such as COBOL files, and IMS DB and IDMS DB requires heavy involvement of IT. This leads to delays, which hinders timely availability of data for business decision making, analytics and data mining.
1 Legacy applications and mainframe applications as well as mainframe systems and legacy systems have been used interchangeably throughout the paper.
Mainframe Application Reverse Engineering 6
LACK OF INTEGRATION WITH THE EXTERNAL WORLD: Legacy systems do not adapt easily to new technologies such as service‐oriented architecture (SOA), Web services, enterprise architecture integration, virtualization, and cloud computing. The self‐contained nature of legacy applications makes it difficult to support newer business environments that cater to customer self service, real‐time enterprise support (e.g., straight through processing, event driven processing), business to business support (e.g., partner and supply chain integration), and much more.
DECREASED PRODUCTIVITY: The limitations of the character oriented user interfaces (CUIs) and limited real estate of 80x25 terminals makes applications complex. For instance, data entry runs into multiple pages, scrolling or popup are not allowed, field level help is limited, error message are displayed one at a time, to mention only a few. Since the on‐screen validation capabilities are limited, there is a possibility that data quality is compromised. Moreover, newer users are highly reluctant to learn CUIs and older generation users are retiring.
PROLONGED TURNAROUNDS FOR CHANGE REQUEST: This aspect of legacy application is very painful for business executives who want relevant stats and facts quickly in order to take timely business decisions. The lack of agility in legacy applications can be attributed to a number of characteristics including, though not limited to, monolithic architecture, rigid SDLC models, unintuitive IDEs, out dated documentation, short supply of knowledgeable SMEs, deterioration in the quality of code (because it has been maintained by multiple people), and hard coded business rules.
In light of the above, it is not surprising to see mainframe customers exploring avenues such as application migration and modernization for overcoming the constraints of legacy systems.
Moving Away from Legacy Applications Mainframe application migration and modernization are two popular approaches for enterprises to cope with constantly changing business requirements. Typically, application migration may include the migration of databases, languages, and software as well as hardware platforms. Mainframe modernization refers to integrating the legacy application with enterprise‐wide applications. This includes, among others, activities such as replacing the traditional green screens with GUI, integrating mainframe applications with the enterprise‐wide systems, and using a SOA approach to expose the underlying business or Web services in legacy applications. The timely and the successful completion of software projects are among the top most reasons for enterprises to modernize their legacy applications.10
A number of options are available for migrating from legacy applications to newer systems, each having its strengths and weaknesses. Which one is best suited to an enterprise depends upon a number of factors including, though not limited to, cost considerations, long‐term maintenance and durability, and system agility.
Mainframe Application Reverse Engineering 7
Some of the popular migration options are as follows:
Replace legacy applications with Commercial Off The Shelf (COTS) products: While quick to implement, this is a costly option. Moreover, the business logic specific to an organization is not available in COTS.
Re‐host: This option calls for moving an application ‘as is’ to a simulated environment such as that provided by Microfocus/Clerity and others. Although this option provides a tactical solution for retaining the legacy application code on a new platform other than mainframes, problems do crop up when the source applications are large in size or require immense processing power. In such situations the re‐hosting strategy usually starts to break down.
Automated code migrations: This option allows software engineers to take applications out of the legacy environment and develop identical applications on other platforms by using language translation tools. However, it is the least preferred option as it calls for high maintenance of the target system.
Rewrite: Considered as the most beneficial functionally, this option is very time consuming and the most costly one. As the name suggests, it calls for capturing the requirements of applications from business and IT SMEs, re‐writing the entire code in new languages, and then testing the new code, just like one would do when developing a new application. The eventual mapping and migration is not an easy task.
Re‐Engineer: This is the option that allows software engineers to gather functional specifications of legacy applications via semi automated tools and processes. Using a two fold strategy, engineers reverse engineer the legacy application and then use its results for forward engineering (the legacy application) to a new platform. This option gives the benefits of Rewrite, together with the advantage of reduced cost, time, and effort. Since the process of reverse engineering allows software engineers to deepen their understanding of the internal structure and functionality of a legacy application, they are in a good position to ensure that the migration of legacy applications is implemented in a holistic manner.
Relevance of Reverse Engineering The process of reverse engineering allows architects to study, analyze and document the internal workings of a software system. It also allows them to examine the application’s business functionality and technical characteristics such as security, interoperability, and other aspects with the aim of determining the make up of its original design. Following a detailed study of the underlying operational services such as transaction processing, mainframe batch processing, spooling, database and file management system, engineers prepare a detailed specifications document of the functionality provided by the current application. This document is the output of the reverse engineering process and can be used for application development, support and maintenance, effective capturing of requirements, and for architecting code reuse or change, or for system integration. The specifications document can also be used as inputs for forward engineering or as a reference document during the implementation phase.
Mainframe Application Reverse Engineering 8
The documentation of the existing system and extraction of the business logic out of the confines of the mainframe environment makes it available for ‘forward engineering’ to newer platforms (e.g., .NET and J2EE) and operating systems (e.g., Windows, UNIX, and Linux), which allow the building of flexible and agile software applications that can respond quickly to business changes. Reverse engineering reduces the dependency on legacy languages (e.g., Assembler, PL/I, COBOL, etc) and thereby on skills that are already scarce and hence difficult to find; by the same token it increases the productivity of engineers. Furthermore, the detailed application documentation makes it possible for engineers to not only integrate the legacy applications with enterprise‐wide systems but to also Web‐enable them.
Business Drivers of Reverse Engineering The goal of reverse engineering is to make software applications agile, highly responsive and scalable, and interoperable so that enterprises could meet the following business objectives:
Reduce the Total Cost of Ownership (TCO) of mainframe servers, especially that of the cost of running data centers
Reduce dependency upon scarcely available legacy skills Increase productivity Make the online applications available 24/7 to an increasing number of customers without any delay11
Improve end‐user experience e.g., comparing insurance quotes12 and the like transactions Increase customer retention13 Eliminate SLA violation
Reverse Engineering Service from HCL This service from HCL uses proven methodologies and frameworks in order to undertake a thorough assessment of mainframe applications, identify the target architecture, and to reverse engineer the mainframe system. Since reverse engineering calls for prior business knowledge and expertise of all the relevant tools and programming languages, HCL uses highly competent and qualified professionals for the task.
As illustrated in Figure 1, the HCL team first prepares a blueprint of the ‘as is’ state of the application under consideration. It does that by:
Studying the application portfolio Capturing details about the technology landscape Interviewing legacy application SMEs Identifying missing code as well as points of functional and business alignment Highlighting the degree to which an application meets business needs Listing the functionality that are not in use or have become obsolete Analyzing the functionality of the system Analyzing the technical characteristics of the applications such as security, performance, integrity and other technical aspects
Figure 1: Re‐Engineering Methodology
Figure 2: Reverse Engineering Methodology
Mainframe Application Reverse Engineering 9
As illustrated in Figures 1 and 2, following a study of the ‘as is’ state, the HCL team proposes a high‐level data migration strategy and plan of action in terms of effort estimates and timelines for the reverse engineering, forward engineering , data migration, implementation, and knowledge transfer.
Reverse Engineering The HCL team uses a number of standard and in‐house tools to analyze the internal structure of the mainframe application programs as well as that of the dataflow. This is done to weed out the dead and redundant code and to extract 70% or more of the business rules, using a Business Rules Extraction Engine (BREE). The latter helps to refine the extracted business rules.
Forward Engineering The detailed application documentation created during the reverse engineering phase proves as a basis for software engineers to develop the current and subsequent application functionality on a new platform such as .NET or J2EE. Forward engineering is best achieved using agile development methods.
Implementation During the implementation phase engineers prepare documents such as user manual, operating manual, training manual and other support documents. They impart training, prepare the production move, and implement the newly developed system in production. Following Implementation, customers continue to receive system support for a specific duration. The old application is decommissioned when customers accept the new system and support as stable.
As illustrated in Figure 3 prior to decommissioning the mainframe application, the HCL team identifies and archives the necessary workload (environment, z/OS, schedules) inventory (online & batch, shared resources), data, and interfaces (that need to be re‐directed).
Figure 3: Archival Phase ‐ Activities
Mainframe Application Reverse Engineering 10
Benefits The process of reverse engineering leads to a number of tangible IT benefits such as faster development cycle, having the ability to build agile and interactive applications, increasing productivity, and reducing reliance on legacy skills.
As illustrated in Figure 4, the benefits of reverse engineering are not limited to IT; they have a very
positive impact on business in the long term. Its impact on business is felt in terms of a reduction in the
overall cost of owning and maintaining mainframes, low migration cost, faster time to market, tapping
business opportunities that would otherwise have been lost, and a closer alignment of business
objectives and IT strategy.
Figure 4: Reverse Engineering Case Study
Mainframe Application Reverse Engineering 11
Mainframe Application Reverse Engineering 12
Endnotes 1 IBM’s share of the market for mainframe is believed to be around 90%. Other mainframe vendors include Computer Associates, BEA, Microsoft, Hewlett‐Packard, Fujitsu Siemens, Unisys, Sun, and Micro Focus.
2 The latest being antitrust complaints by TurboHercules, a French maker of open‐source software for mainframe computers, and T3 Technologies, an American distributor of Flex software that runs mainframes files. On the basis of those complaints, the European Commission has opened an investigation against IBM. For details, see Kevin J. O'Brien, "Europe to Investigate Antitrust Complaints over I.B.M. Mainframes" (July 26, 2010) http://www.nytimes.com/2010/07/27/business/27blue.html [October 2010]. See also "Antitrust: Commission Initiates Formal Investigations against IBM in Two Cases of Suspected Abuse of Dominant Market Position" (July 26, 2010; Ref.: IP/10/1006) http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1006&format=HTML&aged=0&language=EN&guiLanguage=en [October 2010]. See also "NEON Files Motion for Partial Summary Judgment in NEON v. IBM" (September 15, 2010) http://www.marketwatch.com/story/neon‐files‐motion‐for‐partial‐summary‐judgment‐in‐neon‐v‐ibm‐2010‐09‐15?reflink=MW_news_stmp [October 2010]. Similar concerns have been expressed in a comprehensive study by the Indian Council for Research and International Economic Relations. For details, see Rajat Kathuria, et al., "The Issues of Competition in Mainframe and Associated Services in India" (March 11, 2010) http://openmainframe.org/featured‐articles/the‐issues‐of‐competition‐in‐mainframe‐and‐associated‐servic.html [October 2010].
3 Kevin J. O'Brien, Ibid. According to Toni Sacconaghi of Bernstein Research, nearly 40% of IBM’s profits are mainframe‐related. Cited in “The Return of the mainframe Back in Fashion” (January 14, 2010) http://www.economist.com/node/15276714?story_id=15276714&source=hptextfeature [October 2010].
4 He is referring in particular to the movement toward client/server platforms and complementary metal‐oxide semiconductor (CMOS) technology combined with the availability, reliability, security, scalability, and manageability of mainframes, the growth of the Internet and e‐business, and the Y2K problem, all of which led many to believe that the mainframe was living on borrowed time. For details, see John R. Phelps, “The Future of Mainframes Looks Surprisingly Good,” (August/September 2003 issue of zJournal, now a part of mainframezone) http://www.mainframezone.com/it‐management/gartner‐cio‐update‐future‐of‐theibm‐mainframe‐the‐looks‐surprisingly‐good [October 2010].
5 According to IDC, the worldwide server revenue in 2009 was said to have declined 18.9% to $43.2 billion when compared to 2008, and worldwide unit shipments declined 18.6% to 6.6 million units over the same period. For details, see IDC, "Worldwide Server Market Rebounds Sharply in Fourth Quarter as Demand for Blades and x86 Systems Leads the Way" (24 Feb 2010) http://www.idc.com/getdoc.jsp?containerId=prUS22224510 [October 2010]. According to one estimate, IBM’s mainframe revenue took a plunge of 39% in the second quarter of 2009. See Steve Hamm, "IBM Defends Its Big Iron" (August 4, 2009) http://www.businessweek.com/technology/content/aug2009/tc2009084_001429.htm [October 2010].
6 There are several articles on this subjects. See, for instance, Zacks Investment Research, "IBM’s Mainframe Wins Customer" (June 30, 2010)
Mainframe Application Reverse Engineering 13
http://www.zacks.com/stock/news/36260/Zacks+Analyst+Blog+Highlights%3A+Union+Pacific%2C+IBM%2C+Hewlett+Packard%2C+Vornado+Realty+and+KKR+Financial [October 2010]. See also Robert L. Mitchell, ""Morphing the mainframe" (February 6, 2006) http://features.techworld.com/operating‐systems/2229/morphing‐the‐mainframe/ [October 2010].
7 The zEnterprise systems is said to start at about $1 million and according to Lou Miscioscia, an analyst with Collins Stewart, they are expected to generate a profit margin of about 70 percent, vs. a 46 percent margin for the company as a whole. Cited in By Katie Hoffman, "IBM Mainframes: Boring but Profitable" (July 22, 2010) http://www.businessweek.com/magazine/content/10_31/b4189041885991.htm [October 2010]. See also IBM, "IBM zEnterprise System," http://www‐03.ibm.com/systems/z/hardware/zenterprise/ and also "Success Stories," http://www‐03.ibm.com/systems/z/success/index.html [October 2010]. According to Martin Kennedy, the managing director of Citigroup's enterprise systems infrastructure, "we will be able to collapse multiple existing large systems into the new water‐cooled z196, which is a huge step in the company's attempt to shift more of its IT dollars away from internal operations and maintenance and toward customer‐facing efforts. For more details, see Bob Evans, "Global CIO: IBM's Blazing New Mainframe Wins Raves from Citigroup," (September 2, 2010) http://www.informationweek.com/news/global‐cio/interviews/showArticle.jhtml?articleID=227200199&pgno=2&queryText=&isPrev= [October 2010]. See also John R. Phelps and Mike Chuba, "IBM Adds Integrated Management Role for Mainframe," July 27, 2010; ID: G00205553) http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&resId=1411013&ref=QuickSearch&sthkw=mainframe [October 2010].
8 MIPS reflect the mainframe’s footprint, and over the past decade it has witnessed a steady rise with installed capacity reaching a high of 11.1 million in 2007. For details, see Timothy Prickett Morgan, "The IBM Mainframe Base: Alive and Kicking" (July 10, 2007) http://www.itjungle.com/big/big071007‐story01.html [October 2007].
9 For a quick overview on this aspect, see IBM, "The IBM Enterprise System Advantage ‐ Running the World's Most Sophisticated Business Transactions," http://www‐03.ibm.com/systems/migratetoibm/getthefacts/market.html [October 2010]. See also Anne Rawland Gabriel, "Debate over the Future of Mainframe Computing Rages On" (September 16, 2008) http://www.wallstreetandtech.com/it‐infrastructure/210601591?pgno=1 [October 2010].
10 According to a recent Standish Group report, the percentage of software projects that were on time and successfully completed was as low as 32%. Those that were challenged (i.e., later, over budget, and had insufficient features and functionalities) were as high as 44%. Failed projects (i.e., cancelled, incomplete, or never delivered) accounted for 24%. For further details see http://www1.standishgroup.com/newsroom/chaos_2009.php [November 2010].
11 For instance, in the future banks need to accommodate for the increasing number of customers accessing mobile banking application. According to a study by Berg Insight, "the number of active users of mobile banking and related financial services worldwide is forecasted to increase from 20 million in 2008 to 913 million in 2014." Berg Insight, "Mobile Banking and Payments," p. 3, http://www.berginsight.com/ReportPDF/ProductSheet/bi‐mbp‐ps.pdf [September 2010]. Anticipating such high volume, IBM has developed a 5.2GHz chip ‐ considered to be the highest speed rating to date ‐ for its fastest mainframe computers. See Brooke Crothers, "IBM Ships 5.2GHz
Mainframe Application Reverse Engineering 14
Chip, Its Fastest Yet" September 1, 2010) http://news.cnet.com/8301‐13924_3‐20015297‐64.html#ixzz10uOqdeX8 [September 2010].
12 Ryan Arsenault’s observation captures the views of many experts thus: “[t]he problem is not the mainframe platform, but the fact that many insurance companies are making use of legacy mainframe application code which was originally designed to provide quotations to real people.” What is needed is an improvement in the performance of hardware and software applications that use computer resources for running a transaction such as generating insurance quotes. For details, see Ryan Arsenault, "Mainframes Run into Performance Problems with Online Insurance Comparison Shopping," http://itknowledgeexchange.techtarget.com/mainframe‐blog/mainframes‐run‐into‐performance‐problems‐with‐online‐insurance‐comparison‐shopping/ [September 2010.
13 Improving performance of business applications is a key to delivering business value to customers and their end‐users. A recent survey by the Aberdeen Group found that companies that monitored and measured application performance at the point of consumption by end users had few end user complaints. For full details of the survey, see Jeffrey Hill, “End‐User Experience Monitoring and Management” (August 2010) http://v1.aberdeen.com/launch/report/benchmark/6778‐RA‐end‐user‐experience‐monitoring.asp [October 2010]. Writing about the excessive significance given to operational performance metrics in the post 1990s era, Michael Bitterman noted that "IT continues to focus on operations and...explain ‘how the clock works’ when users are asking, ‘what time is it?’ The availability of tools for monitoring hardware and network performance [has given] IT the ability to overwhelm users with statistics regarding IT performance. Users [become] baffled by the metrics that show that mainframe availability was 99.99% yet they couldn’t enter data or access systems.” For the complete report, see Michael Bitterman "IT Metrics for the Information Age" (July 25, 2004) http://www.performance‐measurement.net/news‐detail.asp?nID=198 [October 2010].
