|
|
MIT Sloan Management Review - The Trouble With Enterprise Software
belongs to Industry  by nGenera on 2007-08-14 01:08 PM read 2308 times Source: http://evora.mit.edu/smr/issue/2007/fall/01/ Discovered by: Listener |
|
Has enterprise software become too complex to be effective?
TECHNOLOGY has always been about hope. Since the beginning of the industrial revolution, businesses have embraced new technologies enthusiastically, and their optimism has been rewarded with improved processes, lower costs and reduced workforces. As the pace of technological innovation has intensified over the past two decades, businesses have come to expect that the next new thing will inevitably bring them larger market opportunities and bigger profits. Software, a technology so invisible and obscure to most of us that it appears to work like magic, especially lends itself to this kind of open-ended hope.
Software promises evolutions, revolutions and even transformations in how companies do business. The triumphant vision many buy into is that enterprise software in large organizations is fully integrated and intelligently controls infinitely complex business processes while remaining flexible enough to adapt to changing business needs. This vision of software lies at the core of what Thomas Friedman in The World Is Flat calls “the Wal-Mart Symphony in multiple movements — with no finale. It just plays over and over 24/7/365.”1
Whole systems march in lock step, providing synchronized, fully coordinated supply chains, production lines and services, just like a world class orchestra. From online web orders through fulfillment, delivery, billing and customer service — the entire enterprise, organized end to end — that has been the promise. The age of smart machines would seem to be upon us.
Or is it? While a few companies like Wal-Mart Stores Inc. have achieved something close to that ideal, the way most large organizations actually process information belies that glorious vision and reveals a looking-glass world, where everything is in fact the opposite of what one might expect. Back office systems — including both software applications and the data they process — are a variegated patchwork of systems, containing 50 or more databases and hundreds of separate software programs installed over decades and interconnected by idiosyncratic, Byzantine and poorly documented customized processes. To manage this growing complexity, IT departments have grown substantially: As a percentage of total investment, IT rose from 2.6% to 3.5% between 1970 and 1980.2 By 1990 IT consumed 9%, and by 1999 a whopping 22% of total investment went to IT. Growth in IT spending has fallen off, but it is nonetheless surprising to hear that today’s IT departments spend 70% to 80% of their budgets just trying to keep existing systems running.
According to a multiyear study of over 400 companies by MIT researchers Jeanne Ross, Peter Weill and David Robertson,3 IT departments tend not to be innovative leaders within organizations, but rather conservative forces, viewed by business executives as cost sinks and liabilities. In many companies, it takes the IT department one to two years to implement a new strategic initiative — hardly the agility companies are striving for. The research shows the typical IT structure is so dense and extensive that it’s often a miracle that it works at all. The researchers observe: “Legacy systems cobbled together to respond to each new business initiative create rigidity and excessive costs. Every change becomes a risky, expensive venture.”
How did this happen? James Cordata, who has written extensively about the information economy, points out that as work became more complex and specialized over the 20th century, the use of data — numbers and facts — as fodder for more and more analysis and fact-based decision making intensified. And digital technology “was perfect for this kind of world.”4 Of course, digital technology not only supported that complexity but also played a large part in actually creating it, weaving a continuous web of unending data. “More computers are better than fewer” remains a key belief of American business, Cordata says. “There are no limits to how much is good.” Management became accustomed to the idea that buying more computers and more software would continue to cut costs and improve operations.
But there are limits, some of which are inherent in the nature of software itself. Software is code, lines and lines of code that runs sequentially. Building software programs entails accumulating more and more code. Much of the seemingly boundless complexity of enterprise software is founded on conditional branching (if-then statements) and a hierarchy of interacting objects, all of which manipulate information in a logical succession of small steps. Each step contains explicit instructions. To build software, programmers routinely break down processes into discrete steps, effectively systematizing and standardizing how work is done. An entire sequence of such instructions works more like a calculator than a “thinking machine.” Thus the so-called intelligence of digital technology arises not through magic, nor, in more contemporary terms, through some emergent or self-organizing principle, as some would believe. The result is not greater than the sum of the parts. Rather, it’s more akin to Adam Smith's division of labor and Frederick Taylor’s scientific management, a process dependent on relentless analysis and rationalization of the work to be done.
General software programming used in enterprise systems may contain intricate branching and handle a huge number of conditions, all of which allows it to control a certain amount of complexity. It does not, however, tolerate ambiguity, inconsistencies or illogical conclusions. To be sure, there are fuzzy logic programs, dynamic simulations, genetic algorithms and neural nets with subtler powers, but a vast amount of software working in today’s large organizations is not of these more advanced types. In fact, enterprise software systems are more likely to succeed at relatively straightforward tasks such as procurement and order processing. As the problems get more complex, so does the software that solves them. It is estimated that for every 25% increase in complexity in the tasks to be automated, the complexity of the software solution itself rises by 100%.5
Business users and management inevitably want changes in their automated processes as their needs and markets evolve. And they expect to be able to customize their software to fit their own needs. “Software is infinitely malleable,” says computer historian Martin Campbell-Kelly.6 This is in theory true; however, as enterprise software becomes increasingly comprehensive and complex, the costs and risks involved in changing it increase as well. No single person within an organization could possibly know how a change in one part of the software will affect its functioning elsewhere.
Software’s supposed flexibility and unending ability to manage complexity contributed to the discrepancies between the great expectations and mediocre reality that plagued the first round of implementations of enterprise resource planning systems. In the middle to late 1990s, U.S. corporations rushed to purchase and install such systems. These systems — Germany-based SAP Aktiengesellschaft's is the most common — promised to eliminate the complexity of multiple operating systems and applications by replacing them with a single set of interconnected modules to run the financial, manufacturing, human resources and other major functions of a typical multinational corporation. Theoretically, a single monolithic system would seamlessly connect various distinct and geographically separate locations through private networks. Companies understood that they could customize these systems as needed to suit their unique business processes.
That was the hope. But these massive programs, with millions of lines of code, thousands of installation options and countless interrelated pieces, introduced new levels of complexity, often without eliminating the older systems (known as “legacy” systems) they were designed to replace. In addition, concurrent technological and business changes made closed ERP systems organized around products less than a perfect solution: Just as companies were undertaking multiyear ERP implementations, the Internet was evolving into a major new force, changing the way companies transacted business with their customers, suppliers and partners. At the same time, businesses were realizing that organizing their information around customers and services — and using newly available customer relationship management systems — was critical to their success.
The concept of a single monolithic system failed for many companies. Different divisions or facilities often made independent purchases, and other systems were inherited through mergers and acquisitions. Thus, many companies ended up having several instances of the same ERP systems or a variety of different ERP systems altogether, further complicating their IT landscape. In the end, ERP systems became just another subset of the legacy systems they were supposed to replace.
ERP systems were expensive, too, costing companies more than they had ever paid for software when costs had been based on per-workstation usage. But that price tag was dwarfed by the installation charges, because companies had to hire brigades of outside consultants, often for a number of years, to actually get the software up and running. While the average installation cost $15 million, large organizations ended up spending hundreds of millions of dollars. Even such large expenditures did not guarantee success, however. In fact, 75% of ERP implementations were considered failures.7
Try as they might to measure the productivity gains of ERP implementations or IT in general, researchers have yet to arrive at any coherent or consistent conclusions. One problem is that there is little statistical evidence, especially about whether the benefits of ERP implementations outweigh the costs and risks. Researchers have even suggested that ERP implementations are so difficult that those companies that actually complete them with relative success gain a competitive advantage in the marketplace.8 It seems that ERPs, which had looked like the true path to revolutionary business process reengineering, introduced so many complex, difficult technical and business issues that just making it to the finish line with one’s shirt on was considered a win.
All that complexity and all those options created another conundrum. As Nicholas Carr famously pointed out in his book, Does IT Matter? Information Technology and the Corrosion of Competitive Advantage,9 simply implementing the plain-vanilla business processes that your competitors have does not provide any competitive advantage. On the other hand, as many companies learned the hard way, customizing the already complex ERP software created yet more complexity and even larger risks. Without intimate knowledge of how the integrated pieces of these modular software packages actually worked, customizing could lead to in-house bugs and glitches that were hard to foresee and expensive to fix. Perhaps even worse, customization made changing the software later — or upgrading to a newer version — far more difficult, and in some cases prohibitively expensive. Christopher Koch, executive editor of CIO, tells the story of one head of a corporate SAP installation group who bragged that he had his installation time down to a mere three months for various facilities around the world: “It didn’t matter that he was honing his skills on a 10-year-old version of the software because the costs of upgrading are so huge — tens, even hundreds of millions of dollars, or as much as it cost to install the stuff in the first place — that he keeps installing old versions of the software so that it will line up with the old software they already have.”10
Unexpected bugs present another type of difficulty that increases with complexity. Robert Pool, technology journalist and author of Beyond Engineering, explains it this way: “It’s possible to go through a program line by line and make sure that each individual instruction makes sense but it is not possible to guarantee that the program as a whole has no flaws.”11 The average professional coder makes 100 to 150 errors for every 1,000 lines of code, according to a Carnegie Mellon study conducted by Watts Humphrey.12 That means for every million lines of code there would be 500,000 mistakes. Software developers do extensive testing on the paths users seem likely to take and correct many of these errors. Nevertheless, they cannot test or even anticipate every possible usage path, so released software inevitably contains unknown defects. “Civilization depends on software. So although much software code is poorly written, you can’t just stop the world to fix it,” says Bjarne Stroustrup, the Danish-born computer scientist who designed the popular C++ programming language. On the other hand, Stroustrup does concede that “muddling along is expensive, dangerous, and depressing.”13
The data that software processes and generates is another constant and growing problem. Estimates of errors are astoundingly high. Single systems can have error rates of 50% or more from myriad sources — everything from mistyped data to stale information to data placed in the wrong fields within the database structure. But the really nasty, intractable data problems erupt when companies integrate multiple data sources, as was necessary for ERP implementations, so that they could have all their product, inventory and production records stored in one place. Because of differing formats, conventions, abbreviations and so on, such integrations can result in 100 or more records that actually point to a single product or customer. In the case of enterprise system implementations, data problems alone precipitate many of the failures perceived by business users and much of the added expense as well. Overwhelmed by the sheer difficulty and complexity of the new software itself, companies literally “forgot about the data,” as executive John Nicoli of Trillium Software in Billerica, Massachusetts, describes it, until the tail end of the project, thereby necessitating enormous reworking to properly clean up and integrate the data.14 And with corporate data stores doubling every three years, such data issues are only compounding.
Is enterprise software just too complex to deliver on its promises? After all, enterprise systems were supposed to streamline and simplify business processes. Instead, they have brought high risks, uncertainty and a deeply worrying level of complexity. Rather than agility they have produced rigidity and unexpected barriers to change, a veritable glut of information containing myriad hidden errors, and a cloud of questions regarding their overall benefits. Leaders in computer science are clearly worried. “Complexity is death,” says Chuck Thacker, one of 16 technical fellows at Microsoft Corp. “We are hanging on with our fingertips right now.”15
Business executives, however, simply want to continue to believe that technology will lower costs, improve processes and reduce the size of the workforce. They don’t want to understand IT issues. In part, this is because technology requires special skills and intellectual talents that are quite distinct from those needed to understand and manage business organizations, markets and strategy. But it is also because executives do not like to hear about the downside of technology. Observes Jim Shepherd, senior vice-president of Boston-based AMR Research Inc., “Senior managers often don’t particularly want to be told that there’s a high risk and that there’s a great deal of expenditure involved in minimizing it.”16 Yet the only sure thing about new technologies and the changes they introduce is their uncertainty. In summarizing decades of research into technological change, MIT Sloan School of Management’s Wanda Orlikowski and the National Science Foundation’s Suzanne Iacono conclude that changes involving technology are both “profoundly complex and uncertain.”17
For their part, CIOs and their managers rate aligning IT with business strategy as their number one priority. They struggle year after year to prove the value of IT to the business side of the organization. Yet the cost overruns, delays and outright failures of enterprise systems have if anything widened the digital divide between IT and the executive suite.
The proposed fix for these problems — the next new thing — is service-oriented architecture. Basically, SOA proposes to overcome the problems involved with updating and changing legacy systems by building modular cross-system business processes. These processes would connect the relevant pieces of functionality from various IT systems, thereby making it easier to change processes to adapt to new business goals. But technical realists point out that many difficult technical problems must be solved before SOA can become the backbone for a new strategic architecture, including robust protocols for accessing the applications, high quality integrated data stores and a sound methodology for managing the overall process. Researchers Ross, Weill and Robertson admit that most companies are in the early stages of a four-part transformation to SOA that may take many years — even decades — to realize.18 The estimates of how long this will take reflect a growing acknowledgment of just how deep and radical are the organizational changes these technological innovations mandate. It is a process of adoption and adaptation that by definition cannot occur overnight. Nor, conclude the researchers, can companies skip a step. Given that only 6% of companies have made it into the later stages, this model would suggest that companies are in for a long haul if they are to escape the tangle of technological complexity inherent in large organizations today, and it will be a journey fraught with cultural as well as technical problems.
The timeline itself for this kind of transformation may just be too long to be realistically sustainable and successful. The dynamic business environments of today, where whole industries and markets can undergo radical changes in a matter of a few years and the horizon for corporate strategies has shrunk from 10 years to three to five, makes it questionable whether companies can actually maintain a focused strategy long enough to align their core business processes with IT.
Technical problems raise additional questions about the feasibility of such an undertaking. The hallmark of service-oriented architecture — one might reasonably argue its entire raison d’être — is the fundamental modularity of its software business processes. A self-contained business process adopts parts of the functionality from multiple enterprise applications to automatically complete a set of tasks. For example, a single business process might begin with an order from a customer on the Internet in a web services system and send it to manufacturing in an ERP system. The same business process would set up delivery in a logistics system and then send all the relevant information to billing in an accounting system as well as a customer relationship management system. Companies would build (or purchase) business modules for their core processes. They would then be able to easily change these processes, snapping out and in functional pieces of code from enterprise systems in Lego-like fashion.
The Lego dream has been a persistent favorite among a generation or more of programmers who grew up with those construction toys. Unfortunately, however, software does not work as Legos do. For one thing, a unit of software code is not similar to other software code in terms of scale or functionality, as Legos are.19 On the contrary, code is widely various and heterogeneous. It contains different numbers and types of connections to other code, more like fractals, as Victoria University of Wellington researchers James Noble and Robert Biddle describe it, than Legos, with their uniform connections. Software engineering expert Robert Glass sees another problem with the Legos idea: The notion of reusable software works on a small scale. Programmers have successfully built and reused subroutines of standard functions. But as software grows more complex, reusability becomes a difficult or impossible task. “It is simply a problem too hard to be solved, a problem rooted in software’s diversity.”20
“Complexity is a deadly software killer,” says Yale computer scientist David Gelernter, and he argues that managing complexity is more of an art than a science, and a difficult one at that, especially given the monumental real-world systems today’s software attempts to automate.21 And to the extent that these service-oriented architectures use subsets of code from within ERP and other enterprise systems, they do not escape the mire of complexity built over the past 15 years or so. Rather, they carry it along with them, incorporating code from existing applications into a fancy new remix. SOAs become additional layers of code superimposed on the existing layers. That means it is possible that a process will fail at some point due to some fault in the layers below, and in order to understand and fix that problem, software engineers will need to deal with the layers of enterprise applications below the modular business processes.
Culturally, this long-term plan calls for closer and closer communication and collaboration between the IT and business sides of the organization. While much to be desired, this has proved difficult in the past, and with increasing complexity in software systems, it is unlikely to improve by itself in the future. Differing backgrounds and perspectives, goals, even vocabularies — all hamper efforts to improve communication across this internal digital divide. Biases intrude: A recent study by Forrester Research Inc. of Cambridge, Massachusetts, found that only 28% of CEOs thought their CIOs were proactive or creative in terms of business process improvement.22 Forrester’s advice to CIOs is to get more deeply involved in the business issues and educate executives on what IT is and what it actually does.
Sound advice, no doubt, but it may be time for business executives themselves to become more proactive. Executives could educate themselves more about technology. They could send promising younger executives to executive programs designed to teach business people how to better understand, communicate with and capitalize on their IT. And business schools, too, could do better at teaching the interdependence of business and IT. At present, however, corporations see in software’s seductive invisibility and seemingly open-ended flexibility a never-ending frontier of promise, where hope triumphs over reality and the search for the next new thing trumps addressing difficult existing problems. And hope, unfortunately, has never been a very effective strategy.
icon
Social networking applications. Sometimes viewed as an extension of the Web widget model, social networking applications are applications designed to run inside of popular social networking environments and usually have capabilities that tap into and make use of the 
While it's no longer quite so fashionable to label your Internet startup a 'Web 2.0' company these days, the popularity of the term remains extraordinarily high and is presently used today both far and wide in traditional media and social media. The Google Trends graph in the figure to the right tells the overall story; global search interest in Web 2.0 is more popular than 'social media' and 'social networking' 
It wasn't long ago that to be a credible participant in social media one only had to have a decent blog and keep it updated fairly regularly. The 
I've been spending a good amount of time the last several weeks getting ready for the 
Unless you've been hiding under a rock lately, you know that Facebook is presently the industry darling in social networking, having largely pushed MySpace off the industry's stage, as it seems to offer a more compelling model for social interaction to users overall. Just as importantly, Facebook also lets any other company that wants to join in party do so by building 3rd party 
Login to Reply
Log in to Tell a Friend
Log in to Copy
Report Abuse
RSS
Secure RSS