Improving Team Communication for Efficient Software Development

Paul E. Martin, P.Eng.

Nortel 5050 40th St. N.E.

Calgary, Alberta T3K 4E6 Abstract-This paper will look at various communication scenarios and how they affect software development. A direct link between good team communication and efficient software development will be demonstrated.

Additionally, some strategies to improve software team communication will be suggested. In situations where strategic decisions have already been made and cannot be changed, additional tactics for first-line managers to improve communication will be looked at.

I. INTRODUCTION

The salad days of the late 1970’s and early 1980’s are over. Then, a single software developer could develop a “killer application” like Visicalc, WordStar or dBase all on his or her own. Contrast that with today, when modern applications are complex systems consisting of many different subsystems and components. These subsystems, depending on their complexity, often require teams of programmers, testers, architects and business primes to develop and maintain. In the end, large numbers of people are required to cooperate and communicate together for a successful project.

The paradox is that while software design has undergone a revolution, these larger groups of software professionals are operating in business structures that are at least as old as the industrial revolution [1]. All of us are familiar with the functional organization structure where companies are structured around a job function like “development” or “customer support.” Alternatively, the matrix system, which adds project managers to manage the functional managers (effectively giving them two bosses to report to), can be used. It improves upon a functional system by adapting more quickly to customer demands, but at the expense of more meetings and more managers.

Fig. 1. Shannon-Weaver information theory model

More meetings and more managers ultimately means more communication. A very useful model to show how communication of a single message happens is the Shannon-Weaver Information Theory model in Fig. 2 [2]. While originally designed to describe the electronic transmission of data, it is equally useful to model business communications. Please note that this model describes a one-way communication path. Since most business communication is two way, a second communication path—the feedback loop—pointing from the destination to the source of the message can be imagined.

This model will be used to explore some communication scenarios the author has experienced. First, the scenario where the customer is discovering bugs in the product is looked at. Then software streaming and how this affects team communication is investigated. After that, we consider the team communications required during design and integration phases. Finally, how different functional development groups communicate with each other and how this might be improved is considered.

II. BUGS IN THE FIELD: COMMUNICATING WITH THE END USER

While no software organization wishes to release major bugs into the field, it can and does happen. Pete Peterson, former COO of WordPerfect Corporation, captures the urgency of finding a serious bug in the first release of WordPerfect 5.0 in 1988.

“On Monday, May 9th, the calls started to trickle in, and by Thursday, the trickle of calls turned into a flood. On Friday, the thirteenth, there were so many people trying to call us that our busy signals brought down the entire AT&T 800 system in the Mountain West.”[3]

It is this sense of urgency, panic even, that drives the structured organization into a sort of frenzy, akin to disturbing an ants’ nest with a stick. Because of the breakdown in communication, nothing can be as wasteful of effort as diagnosing a serious bug at a customer site.

What causes this breakdown in communications? To understand this we have to look at the communications structure of the functional organization. If a serious problem is found in customer service, and the customer is “hot,” the message is escalated to higher and higher levels of

0-7803-9139-X/05/$20.00 ©2005 IEEE. 553

management within customer service. Often this escalation is done by the customer him or herself. Then the message is passed at a high level of management to the development organization and the message traverses the layers downward to the people actually solving the problem.

In the end, the people fixing the bug have a problem description that passed between many persons, each with a slightly different interpretation of a complex issue. The Shannon-Weaver model would define this as noise in the channel, and this noise is multiplied by having many serial channels. In fact, having so many people in between is reminiscent of the childhood game where a line of children whisper a message starting at one end and ending at the other. In this scenario, it can be surprising how garbled a message can become, especially if the message is a complex one (and software debugging inevitably requires complex, full duplex messaging).

In this scenario, the first duty of the manager is to shorten the feedback loop and improve the communications between the people doing the work and the end users. There are a number of ways to do this. One of the best ways is to setup a site visit, where a cross-functional “tiger team’ of bug fixers, log collectors and end users can sit face-to-face to communicate in a high bandwidth way with each other. If this is not possible, some modern technologies such as Instant Messaging TimbuktuTM and WebxTM can show the programmers the nature of the problem efficiently and in real time. As a last resort, email can be used, but this only works well with simple problems, and certainly does not work well with intermittent issues that can only be reproduced at a customer site.

To add to the scenario, the problem can be exacerbated when the upper levels of management become aware of the urgent issue. A flood of inquiries and demands from the top can overwhelm the people trying to diagnose the problem and actually slow the resolution of the issue. It is ironic that this extra attention can actually cause the organization’s muscles to work against each other by overwhelming the communication capacity of the people doing the work.

Therefore, the second duty of the manager is to shield the members of the ‘tiger team’ from communications demands from upper management. This is especially relevant to the matrix style of corporation, where first line managers may have multiple senior managers requiring updates. Email is one of the most effective tools to solving this. By expanding the “distribution list” as more and more senior managers get involved, the updates can be broadcast to all people who are interested.

To illustrate the full scenario, consider a real life example. In the fall of 2001, the author’s software development team was notified of a local site that wanted to return our software. While the customer explained the error scenario, no one could

reproduce it in the lab. A site visit was arranged, and when the development and support team arrived, we were drawn into a firestorm of meetings between the customer, the distributor, and our company’s representatives. During a lull in the action (after somewhat placating the parties involved) we began talking with the actual people who used the software system. From this conversation, we were able to replicate several scenarios that were causing the problems and gather the logs for these scenarios. When we discovered how the customer was actually using the system, one designer whispered with incredulity, “But they are not supposed to do that!”

In fact, the software had not been designed to cover the scenarios the customer was using. This was because the lower layers of software did not support it without lots of extra effort. Due to the time constraints on our group, and the opinions of some designers that customers would never exercise the program in such a manner, the application had to be used in certain restrictive ways to behave predictably. In the end, it was this epiphany the developer experienced talking point-to-point to the end user that made it possible to re-architect the system to cover all usage scenarios, even the unlikely ones. Of course, this took lots of extra effort, but the re-architecture went ahead because we could explain to upper management that real users would return their software without the work. In the end, we permanently fixed the code in the next release (six months later) and ended up with a much more flexible architecture from which to build future products.

III. CODE BALKANIZATION: DEVELOPING ON MULTIPLE STREAMS

Metcalfe’s Law proclaims, “the community value of a network grows as the square of its users increase” [4]. Based on this statement, it can be shown that if a single network was split it into N equal parts that did not communicate with each other, the community value of the total network would be diminished by N.

An actual illustration of this “code balkanization” principle can be found in the UNIX world in the early nineties.

“[McVoy’s 1993] research showed that UNIX developers were spending about a billion dollars a year on operating system development. Much of it was spent on doing the same thing over and over again.” [5]

Here the “networks” are the separate development efforts that each independent company (say, Digital, HP, IBM and Sun) had for their proprietary systems and the “community value” is the development “bang for the buck.” Intuitively, we can see how doing things four times over would result in four times more spending; perhaps had each company shared development on a single stream, development costs could have been reduced to 250 million dollars!

Let us consider a slightly different scenario. In this case, we have a team of developers working on a single stream. Then that stream is split into two equal parts. What happens?

0-7803-9139-X/05/$20.00 ©2005 IEEE. 554

Does the effectiveness of the development team decrease by half? The answer is that their effectiveness does decrease but by less than that amount. This is because the team continues to communicate within itself, so not all work is done twice over. Now a proportion of development team’s effort consists of porting bug fixes from one stream to another. In fact, by using advanced code management tools like ClearCaseTM, bug fix porting can be done more easily than in the past; however, it still requires extra effort. Additionally, human errors now creep in; occasionally some code ports will be missed. This puts extra stress on the software testing team, which now must test all streams produced by development to ensure that bug fixes are effectively propagated. (Note that in this case the software testing team’s effectiveness does decrease by half.)

What is not readily apparent from this example is how mind numbing and repetitive the extra stream management work is. This type of work is not appreciated by the best developers—who wish to create and contribute—and often leads to attrition in the group. Moreover, to get the details right, extra communications are now required both within the development and test teams and between many cross-functional managers. The danger is that with so much time spent on additional communication, there is no communications bandwidth to react to unexpected bugs or requirement changes in the product itself. The development team becomes like a platoon of soldiers so focused on not tripping on each other’s boots that they could march right into a minefield.

The author himself in the late 1990’s worked on a project that had several different software streams. The product worked on various different operating systems, and even included some driver level functions that were platform specific. As the product shipped to more and more countries, the number of independent streams to keep track of kept increasing… At one point, there were eight separate streams, all slightly different from one another, in the source control. With each additional software stream that was added, the effectiveness of the software team was reduced by about 10%. Therefore, with eight of these streams, more than half of the group’s effort went into porting and managing them. Something clearly had to be done.

When the idea was first floated for unifying the streams there was some concern. One of the most common worries was whether introducing a change into one stream would affect another stream in subtle and poorly understood ways. This fear of “bug bleed-through” is common and stems from two types of experiences, one with unstable code, where adding a slight change somewhere might cause a wild pointer to behave differently; and the other with fragile code, where adding something seemingly innocuous might cause a mismatch somewhere deep within the architecture. Both these failure patterns are much less of a concern these days because we have better memory debug tools and more robust memory protection in operating systems. Additionally, many modern

programming languages, like Java and C#, prevent unstable code by restricting designers from using memory pointers altogether.

In any case, our experience with unifying eight streams was a happy one. We were able to increase the throughput of the group by reducing porting effort in the medium and long run. One serendipitous result was that we discovered that many streams had not had all the bug fixes ported into them and some were missing substantial features! Over time, many of these quirks had been forgotten and plastered over; now we were able to fix them for all markets and platforms. Additionally, an unexpected advantage came with using conditional compiles and “runtime-ifs” to show code changes within a file. This allowed programmers to see the differences between the platforms at a glance, aiding their comprehension and allowed them to make fixes to all platforms simultaneously without having to edit multiple files.

IV. TEAM COMMUNICATIONS FOR GOOD SOFTWARE DESIGN

As good developers know, designing the architecture first is a crucial step in writing software for large projects. A good design—that includes understanding the requirements and scoping the architecture—breaks down the software into smaller parts. This makes the software both more understandable to the human mind (and therefore easier to work with) and more compartmentalized (so that the work can be split up between individuals and teams). Additionally, by using Object Oriented Design (OOD) techniques, improved code reuse and lower software maintenance effort are possibilities.

Conversely, if the design is done poorly, a complex project will neither “converge” nor meet customer requirements. In this case, a large amount of rework is generated when the original design decisions are reconsidered and changed.

What causes a poor design? The author was involved in one project where the design process went awry. In spite of great emphasis on this phase of the project, two groups of designers from different companies could not agree on the proper way to solve the problem. Both groups considered themselves subject matter experts and disagreed on practically every detail of the project definition. Ironing out these differences required large amounts of communications; however, there were a great number of restrictions hampering this. For example, designers could only communicate via telephone and only for one hour a day because of the seven-hour time zone difference. What did not help was that the companies both groups belonged to had different business objectives and cultures. This made it hard to reach consensus on many things. Eventually, the pressures of the market forced the design phase to close, and in spite of the large amount of time and attention that had gone into design, it closed prematurely. The results were predictable, with the software having a larger than average number of bugs and not

0-7803-9139-X/05/$20.00 ©2005 IEEE. 555

meeting some crucial customer requirements. In fact, it was surprising the software worked as well as it did, knowing the fundamental hurdles that had been overcome in later phases of the project.

In this example, poor communication during the design phase caused the project to run into difficulty. Had the designers met face-to-face early in the design phase to work together, perhaps many areas of contention could have been resolved. If the designers could have been co-located during this phase, Management by Walking Around (MBWA), perfected by David Hewlett of HP, would have been an option [6]. Of course, this was not possible for budgetary reasons, and instead, extensive use of such technology such as Instant Messaging (IM) did help to give better immediacy. However, it still lacked the power and richness of an impromptu face-to-face communication.

Perhaps most important though, is the ability to remove teams or individuals from the design phase if their contributions are hindering progress. In cases like these where there is limited bandwidth for communication between teams, it may be beneficial for one team to fully design the software and communicate the decisions made by publishing a specification to the other team. In fact, by making this one change in the example above, the speed and quality of designs has noticeably improved.

V. COMMUNICATIONS FOR SUCCESSFUL SOFTWARE INTEGRATION

Another high-communications project phase is the integration phase, where designers test the different software components and how they interoperate with each other. This is done with controlled “white box” testing [7]. Effectively, the integration phase ensures that the moving parts are working together by implementing a confidence-building internal milestone.

Of course, all the techniques mentioned for improving the design phase also help the integration phase: face-to-face integration sessions with MBWA, and IM and regular telephone conference calls in situations where this is not possible. However, another strategy can vastly improve communications during this phase. This strategy is what Cockburn calls using “software iterations” [7].

Fig. 2. Multiple software iterations smooth communication demands.

The software iteration is a complete design-integration cycle and typically happens with frequencies between one week and four months. The concept is to build smaller confidence-building milestones to test architectural concepts and designs on the way to a final integration. This reduces the volume of communications in the final integration, because many of the problems are solved earlier on as part of earlier software iterations. With this technique, the peak communications demand on team communications during both the design and integration phase is broken down into a series of smaller, more manageable peaks. If we ensure that the smaller more manageable peaks are less than the maximum communications bandwidth, we greatly reduce the chance of lost messages or “clipping” (see Fig. 2.). In the end, this should reduce rework and ensure the project proceeds more efficiently.

VI. ORGANIZING A TEAM FOR GOOD COMMUNICATIONS

If a project is successful and it continues to grow in scope, eventually, the team should be partitioned to ensure focus on the proper subcomponents. Ideally, the partitioning of the team should be done hand-in-hand with how the software architecture is compartmentalized. In situations when the code has not been written yet, there may be an opportunity to decide how to design the software to make best use of existing resources and communications channels.

One example of how a team can be organized comes from the world of open source. In 1998, Linux kernel king, Linus Torvalds, could no longer keep up with the volume of communications from the Linux open source project. He needed some lieutenants he trusted to buffer him from the flood of code requests and patches coming in from other contributors. The solution was to use a more efficient source code management system that would allow Linus to “surround himself with a ring of lieutenants who would report to him through a hierarchical tree structure.” By 1999, to keep up with the pace of development, Linus had eight trusted individuals, who in turn managed the input from 250 developers (most of them testers) who took patches and code from thousands of contributors worldwide [5].

The reason this model worked so well was that partitioning went hand in hand with the new software tools (the code management software). This new structure was designed specifically with Linus and his trusted lieutenants in mind. Specifically, it filters the overwhelming volume of communications and thereby allows Linus to spend no more that 50% of his time coordinating a massive worldwide software effort.

It is important to recognize that even when a team is partitioned successfully, that small inefficiencies are introduced in communications. The job of the manager is to ensure that these efficiencies are minimized, by ensuring that people who communicate well together are matched up and

0-7803-9139-X/05/$20.00 ©2005 IEEE. 556

that the software architecture is partitioned to take advantage of this. However, even in the most optimal case, delays and inefficiencies are introduced between communicating individuals. We can understand how this can get worse. If we have two people who do not communicate well with each other, or the team is partitioned so that the amount of communications required for maintaining the architecture overwhelms communication capacity, then we are well on the way to a communications breakdown and a “balkanized” team.

The manager should do everything he or she can to avoid a situation as shown in Fig. 3. In that example, the team has been “balkanized” or split along lines that cause subtle impairments in communication that will definitely affect the production of software. These impairments may not be visible at first, but as the software process gets more arcane (as a result of trying to “put process” around these communications bottlenecks) they will start showing up as delays. Of course, the real reasons for these delays will not be visible to upper management, who will only see the fact that the software process seems to be bogging down and may attribute it to “laziness” or “featherbedding” of the work estimates by the development team.

If team balkanization is allowed to exist for some time, something even more damaging may occur. In an attempt to increase the throughput of the team, additional resources may be thrown into the mix without a careful understanding of the paths of communication. The resulting additional process to keep these people marching in the same direction starts to result in a functional bureaucracy. Eventually, the communication divisions evident in a balkanized team may grow into functional silos as in Fig. 4!

Fig. 3. How a poorly partitioned team becomes “balkanized”

Fig. 4. Communication between functional silos.

Note that functional silos add many serial communication links (reminiscent of the bug-solving scenario presented earlier). Moreover, the upward links may be much weaker than the downward links, further reducing the communications bandwidth between the two silos.

Of course, in today’s economy we do not always have the luxury of avoiding silos and team balkanization. For business reasons, modern software development companies have to deal with many new barriers including: working across time-zones, where the opportunity to have impromptu telephone conferences is greatly restricted; using only email to communicate, which cannot give important emotional cues in the message; working across different cultures, including corporate cultures; working between different companies, which may introduce additional legal and contractual hurdles; being forced to work through a single contact who may refuse to cooperate; and many others. So how can the first line manager deal with this?

VII. BREAKING DOWN THE SILOS

Certainly, we can think of the obvious answers. Circumventing the single point of contact that refuses to communicate is an obvious solution, but may not always be possible. Similarly, moving everyone from the different time zones and cultures into the same building is most likely neither possible—nor even desirable in many cases. What can we do if constrained by these realities?

The answers lie in the Shannon-Weaver transmission model of Fig. 1. If we can “tune” the decoder and the encoder to each other, we might be able to send short simple messages that could represent large complicated ideas, a sort of “high-level language.” For example, let us assume the encoder and decoder can agree that the message “build release 10” also means doing all the proper checks and code reviews to ensure “release 10” is a quality product. This is a great improvement over sending detailed messages back and forth for each step of the process. Effectively, using a “high-level language” increases the bandwidth of the channel and improves the “noise resistance” of the message.

Perhaps the most powerful way to “tune” the decoder and encoder is by forming friendships and gaining trust with other developers and extended team members. This is particularly true in the software world and exemplified by the trusting partnerships between Jobs and Wozniak (in founding Apple Computer in the 1970’s) and Gates, Allen and Ballmer (in making Microsoft the largest software company in the world). The factor of trust, while an intangible, should not be underestimated as a way to improve communication. Individuals who trust each other know each other’s minds to some extent, and can often anticipate the content of requests. Therefore, the need for redundant messages, that ensure the receiver is carrying out the sender’s requests, is reduced. This is essentially what is meant by “tuning” the encoder/decoder!

0-7803-9139-X/05/$20.00 ©2005 IEEE. 557

Another way of improving the communication between silos is by eliminating the need for the silos themselves. For example, if a programmer needs to make a change in a communications protocol, he or she may need to modify several applications that use the protocol. If these applications are “owned” by this programmer, then it is easy as the programmer can do it directly without asking anyone’s permission. However, if another group owns the application, a request has to be made to that group and completing the work item requires additional communications. To further extend this example, if the application is owned by another functional silo, then some delicate negotiations have to go on, and there is no guarantee that it will be done in a timely fashion!

In software development, many companies have already standardized on a single code repository. This is an important step in eliminating silos between software development groups. The other important step is to allow developers to treat the software base like an “information commons,” similar to the structure adopted by successful open-source development projects. To do this, trust must be encouraged between key development groups. This can be done by allowing component primes to accept or reject code changes made in their area by other developers. In this way, primes can share key guidance with developers but still have these individuals do the actual work. This reduces turnaround time by reducing the need for extra communication yet retains safety by ensuring the component prime still checks the code changes.

An “information commons” can break down functional software development silos, but requires a new type of programmer. The new programmer must have knowledge that spans several components or functional layers. In this sense, the developer must be able to “play all the positions.” In fact, if we carry the sports analogy further, a comparison can be made to Johan Cryuff’s “Total Football” philosophy where:

“All players had extremely good ball skills and could play any position on the field. Every player was versatile and possessed a lot of tactical knowledge by seeing the game through different field positions. This system, lead by legend Dutch Johan Cryuff, helped Ajax and Holland to dominate world soccer in the 1970's.[8]”

For programmers willing to broaden their areas of expertise, the benefits include fewer meetings and more control over the feature they are implementing. This also helps managers by eliminating coordination meetings between software groups. Now, additional communications bandwidth within the software development team is freed up and can be used towards improving the software.

VIII. CONCLUSION

This paper presented some ideas for how a first-line manager in a large software development project might

achieve more efficiency though better team communications. “Point-to-point” communications between the development team and the end-user enables customer focus; multiple face-to-face design and integration ensures timely and accurate delivery; learning trust builds bridges across corporate bureaucracy; the “total football” programming concept can break down silos. By using these powerful techniques, software team efficiency can be improved.

Robert X. Cringely postulated once that under ideal conditions a handful of highly productive “genius” programmers who communicated well with each other could accomplish what a team of hundreds of average programmers who communicated normally could do [9]!

If this is true, then this paper is a clarion call for all first-line managers to resist things that impeded good communications, like team balkanization. If this is beyond our control, we can still reach out to like-minded individuals in other software groups and extended teams. By doing this, we can form alliances with them to create high bandwidth operational teams as in Fig. 5. High bandwidth team communication is a prerequisite for the ultimate result: delivering quality customer focused software efficiently.

Fig. 5. “Teaming” improves communications for more responsive software development.

REFERENCES [1] R. Semler, Maverick: The Success Behind the World’s Most Unusual

Workplace, Warner Books, Inc., 1993, pp. 109-118. [2] C. Shannon and W. Weaver, The Mathematical Theory of

Communication, University of Illinios Press, 1955, p. 9. [3] P. Peterson, AlmostPerfect: How a Bunch of Regular Guys Build

WordPerfect Corporation, Prima Publishing, 1994, p. 136. [4] [anon.], Metcalfe’s Law, Rpt. <www.whatis.com> [5] R. Young and W. Rohm, Under the Radar: How Read Hat Changed the

Software Business—and Took Microsoft by Surprise, 1999, pp. 117-122. [6] D. Packard, The HP Way: How Bill Hewlett and I Built Our Company,

Harper Business, 1995, pp. 155-156. [7] A. Cockburn, Surviving Object-Oriented Projects: A Manager’s Guide,

Addison-Wesley, 1998, p. 126. [8] [anon.], Dutch Total Football, Rpt. <http://www.expertfootball.com>,

Dec. 31, 2004. [9] R. Cringely, Accidental Empires: How the Boys of Silicon Valley Make

Their Millions, Battle Foreign Competitions, and Still Can’t Get a Date, Addison-Wesley, 1992, pp. 18-32, 73-92.

0-7803-9139-X/05/$20.00 ©2005 IEEE. 558