Continuous Architecture: Sustainable Architecture in an Agile and Cloud-Centric World

By Murat Erder and Pierre Pureur

Continuous Architecture provides a broad architectural perspective for continuous delivery, and describes a new architectural approach that supports and enables it. As the pace of innovation and software releases increases, IT departments are tasked to deliver value quickly and inexpensively to their business partners. With a focus on getting software into end-users hands faster, the ultimate goal of daily software updates is in sight to allow teams to ensure that they can release every change to the system simply and efficiently. This book presents an architectural approach to support modern application delivery methods and provide a broader architectural perspective, taking architectural concerns into account when deploying agile or continuous delivery approaches. The authors explain how to solve the challenges of implementing continuous delivery at the project and enterprise level, and the impact on IT processes including application testing, software deployment and software architecture.

• Covering the application of enterprise and software architecture concepts to the Agile and Continuous Delivery models
• Explains how to create an architecture that can evolve with applications
• Incorporates techniques including refactoring, architectural analysis, testing, and feedback-driven development
• Provides insight into incorporating modern software development when structuring teams and organizations

• Language: English
• Publisher: Elsevier Science
• Release date: Oct 21, 2015
• ISBN: 9780128032855

Murat Erder

Murat Erder is a Director at Deutsche Bank, and is currently working in Data Engineering group as part of the Chief Data Office. Prior to his current role, Murat was the Domain Architect for Reference Data. Before this role he led the Integration Services group at Deutsche Bank responsible for all managed services that dealt with integration; including core messaging, file transfer, ETL, Enterprise Service Bus, Workflow and Document Management. Murat also setup the Technology Standards group at Deutsche Bank and governed adoption of new technologies. Prior to joining Deutsche Bank, Murat worked for 13 years for management consulting organizations, including KPMG/BearingPoint and Deloitte providing architecture and technology strategy advice to leading Financial Services organizations. Murat started his career as a software developer in Silicon Valley and was employed as a Software Development Lifecycle and Object Oriented Analysis and Design specialist by Objectory and Rational. Murat has a MS degree in Electrical and Computer Engineering from the University of Massachusetts

Book preview

book.

Chapter 1

Introduction to Continuous Architecture

Abstract

Through our careers, we have noticed the increasing speed of delivery expected from information technology practitioners within enterprises. At the same time, the ease of use and 24/7 expectations of end users are driven by the overwhelming expansion of technology in our daily lives—we have progressed from PCs to tablets to smartphones to wearable technology. Today software delivery teams are expected to operate on an Internet time and scale. This has significantly increased the demand and resulted in the widening adoption of Agile and Continuous Delivery practices. As a result, the pendulum has swung away from traditional software architecture practices and in particular Enterprise Architecture. We do not believe that the pendulum will swing back to these traditional practices. However, there is still a need for an architectural approach that can encompass Continuous Delivery, providing it with a broader architectural perspective. This is the main topic addressed in this book.

Keywords

Continuous Architecture principles; Continuous Delivery; Agile development; requirements; Quality Function Deployment (QFD); Quality Attributes; Architecture Tradeoff Analysis Method utility tree; architecture and design decisions; loose coupling; microservices; Conway’s law; minimum viable product; Minimum Viable Architecture; legacy monolithic systems; continuous feedback

"We are called to be architects of the future, not its victims.

—R. Buckminster Fuller

Both of us call ourselves architects because we believe there is no better explanation of what we do every day at work. Through our careers covering software and hardware vendors, management consultancy firms, and large financial institutions, we have predominantly done work that can be labeled as software, solution, and enterprise architecture.

However, when we say we are architects, we always have a need to qualify it; we believe an explanation is required to separate ourselves from the stereotype of an IT architect that adds no value. Most readers are probably familiar with the common expression of: I am an architect, but I deliver/write code/engage with clients (fill in with your choice of an activity that is perceived as valuable).

How has this notion of the ivory tower architect become so predominant? We are confident that we are not alone in this mindset. We also believe that architects who exhibit the infamous qualities of abstract mad scientists, technology tinkerers, or presentation junkies are a minority of practitioners. A majority of architects work effectively as part of software delivery teams, most of the time probably not even calling themselves architects. In essence, all software has an architecture, and most software products have a small set of senior developers who create a consistent architecture regardless of whether it is documented or not.

Through our careers, we have also noticed the increasing speed of delivery expected from IT practitioners within enterprises. At the same time, the ease of use and 24/7 expectations of end users are driven by the overwhelming expansion of technology in our daily lives—we have progressed from PCs to tablets to smartphones to wearable technology. Today software delivery teams are expected to operate on an Internet time and scale. This has significantly increased the demand for and resulted in the widening adoption of Agile and Continuous Delivery practices.

As a result, the pendulum has swung away from traditional software architecture practices and in particular Enterprise Architecture. We do not believe that the pendulum will swing back to these traditional practices. However, there is still a need for an architectural approach that can encompass Continuous Delivery, providing it with a broader architectural perspective. This is the main topic addressed in this book. Before we define Continuous Architecture, let us first provide a definition of software architecture and then consider the historical perspective.

What Do We Mean by Architecture?

When we talk about architecture, we are concerned with software architecture. How do we define software architecture? Let us look at a few common definitions.

According to the Wikipedia entry maintained by IFIP WG 2.10¹ on software architecture:

Software architecture refers to the high level structures of a software system, the discipline of creating such structures, and the documentation of these structures. It is the set of structures needed to reason about the software system. Each structure comprises software elements, relations among them, and properties of both elements and relations. The architecture of a software system is a metaphor, analogous to the architecture of a building.

The International Standards Organization and Institute of Electrical and Electronics Engineers (IEEE) use the following definition*:

Architecture: the fundamental concepts or properties of a system in its environment embodied in its elements, their relationships, and in the principles of its design and evolution.

Specifically, we are dealing in this book with the IT concerns of software development within a commercial enterprise.

The four most common reasons for developing a software architecture are to:

1. Define the guiding principles and standards. Architecture is a vision of the future, and supporting tools to help you get there.

2. Develop architecture models. In this case, architecture is concerned with abstracting at an appropriate level to make business and technical decisions.

3. Build common services. The services could be systems or organizations. Architecture can be defined as focusing on defining interfaces.

4. Create a roadmap to an IT future state. Architecture deals with transition planning activities that lead to the successful implementation of an IT blueprint.

In accordance with these goals, building a large software system in an effective and efficient manner requires a blueprint—commonly referred to as a software architecture. Such an architecture includes descriptive models (defined in both business and technology terms) to aid decision makers in understanding how the entity operates today and how it wants to operate in the future and prescriptive models to help the project team successfully transition to its desired IT state.

Historical Perspective

Let’s look at a brief history of the evolution of the computer and software industry. Our objective here is not to provide a comprehensive history but highlight a few key developments to put Continuous Architecture in a historical perspective.

While looking at the history of computer technology, it is interesting to look at two aspects, the major technology epochs versus concepts evolving in software development and software architecture. We can articulate the major epochs in the computer industry as:

1. The age of the mainframe (1950s +): This started basically in the 1950s with the main computing paradigm being mainframe computers. These were expensive and powerful machines that enabled large corporations to start automating certain aspects of their businesses as well as supporting scientific development.

2. Getting personal (1970s +): This is when PCs (Personal Computers) as we know them today were introduced. It was a huge leap forward in the sense that computing power could be put in one machine that could sit on a desk. This enabled individuals to interact with computers in a much more personal manner. However, most individual users of such technology were still in large companies and research institutions.

3. Distributing the load (1980s +): As computers evolved in capacity and started becoming networked, the concept of client server architecture started evolving. This resulted in a view that not all software needed to run on a single computer but that the computing burden could be shared. At that time, client server architecture and how best to distribute responsibilities were major architectural topics. As the client server paradigm evolved, so did the concept of networked computers.

4. Era of connectivity (1990s +): The network is the computer term was coined in 1984 (the term is attributed to John Gage, Chief Researcher and fifth employee of Sun Microsystems), but it took another 10 years before the first commercial browser was introduced (The Mosaic browser was released in November 1993). The World Wide Web and Internet revolution has had tremendous impact on not only the software industry but also the entire world. We do not have sufficient space to detail its impact, but needless to say that everything has changed from the way we shop to the way we communicate and even to how our gadgets interact.

5. Back to the future (2000s +): With the ubiquitous Internet came the opportunity to unleash us from dependency on physical devices. The ability to store and retrieve information from the cloud creates even further opportunities, as well as challenges.

For example, we do not always physically own the books we buy on our e-readers.² The company that sells us the books is the owner of the content and has the right to remove it, which would not be an option for old physical books. We will leave the ethical discussion of the Internet and cloud computing to other authors and go back to our architecture storyline.

The cloud era has also enabled enterprises to only acquire the required software and compute capacity. We call this era back to the future because we are back to the model of leasing capacity, which was exactly the model in the mainframe days. The only difference is that we are implementing the model in a highly interconnected, standardized, and real-time world.

The computer and software world will continue to evolve, with the next phase currently coming from mobile devices and wearable computers. What has not changed during this approximately half a century journey is the fact that developers still write code. What is also interesting is that productivity among developers is still quite varied, and creativity is still associated with the developer role. Basically, we still are in a software as a craft stage and have not been able to fulfill the industrial revolution and automation that have been promised at each stage in the journey.

What has changed, though, is the speed at which we are expected to execute, as well as the scale in which we need to operate. Quarterly release cycles are no longer relevant, and commercial solutions have to cater to increasing demands from end users in terms of performance, usability, and scalability.

Figure 1.1 highlights some interesting data points in the five epochs we have listed in the brief history of the commercial computer and software industry.

Figure 1.1 Technology and architecture timeline.

Let us now look at how the approach to software development and architecture has evolved during this journey. As stated before, some things have not changed. Structured programming has been with us for most of the journey and highlights key elements any current architect and developer would recognize: creating logical structure based on modularity.

The 1980s saw some key architectural patterns starting to evolve. The Open System Interconnection (OSI) Model in the network domain was an excellent example of creating a layered architecture. Enterprise Architecture concepts can be considered to have been jump-started with the introduction of the Zachman framework, which created a technology taxonomy that is still valid today.

The early to mid-1990s saw an explosion of ideas, from use cases to UML and Extreme Programming (XP). This was when object-oriented programming was the rage, which resulted in a series of efforts to create methodologies and techniques for this new world. It was also at the beginning of the Internet explosion, although we were not quite there.

The impact of the Internet era and the demand for quicker delivery of software resulted in the Agile Manifesto being published. An interesting footnote is that this was published right after the dot-com bubble burst. Over time, we have seen a gradual acceptance of the Agile approach and its evolution to Continuous Delivery and DevOps concepts.

In parallel, the Enterprise Architecture concept has gone through several waves of popularity and disillusionment. The fact that The Open Group Architecture Forum (TOGAF) standard is still active is an attestation that there still is demand for addressing enterprise-wide architectural concerns.

However, we believe that the concept of software architecture and in particular Enterprise Architecture is becoming less relevant for developers in large enterprises. Even if Enterprise Architecture groups exist, their impact on the day-to-day development is limited.

The activities of the Enterprise Architecture groups fail what we call the junior developer litmus test. This litmus test acknowledges that a lot of architectural decisions are made by developers in action—where the rubber hits the road. The litmus test validates if a junior developer on a team makes his or her decisions based on guidance from architects.

We believe that the gap between the Agile delivery and architecture practices within a commercial enterprise is wider than before. This is where Continuous Architecture comes in; it is a set of principles and tools targeted at addressing this gap.

Another way of looking at Continuous Architecture is by using an analogy based on real-life architecture. If our objective is to build a cathedral, an Agile developer will start shoveling, but an enterprise architect will look at a 5-year plan. The goal of Continuous Architecture is to bridge this gap (Figure 1.2).

Figure 1.2 The Continuous Architecture context.

Current Challenges with Software Architecture

Now that we have a historical perspective, let us look at the state of software and Enterprise Architecture today. We can identify three main challenges with the current state of software architecture within most commercial organizations:

1. Focus on technology details rather than business context

2. Perception of architects as not delivering solutions or adding value

3. Inability of architecture practices to address the increase in speed of IT delivery

Software architectures are driven by business goals, yet gaining a deep understanding of the business and its needs is not done very often in architecture projects. Architecture teams tend to spend considerable amounts of time documenting the current state of existing systems and then creating elaborate architecture blueprints that try to fix the perceived shortcomings. Unfortunately, these blueprints are usually derived from current IT trends or fads; for example, most architectures designed these days include some amount of service definitions, cloud computing, and big data, regardless of their applicability to the problem at hand. The root cause of this problem may be that most IT architects are more comfortable solving technology problems than they are at solving business problems.

A second issue with architecture lies with the concept of Enterprise Architecture and Enterprise Architecture organizations. Broadly speaking, architects fall into two categories: solution architects and enterprise architects. Enterprise architects may be further categorized according to the discipline they are specialized in, such as application, information, security, or infrastructure. Whereas enterprise architects are normally located in a central group, solution architecture organizations may be either centralized or distributed.

Solution architects focus on providing project-level architectures, but enterprise architects attempt to create strategies, architecture plans, architecture standards, and architecture frameworks for the whole enterprise. In turn, solution architects are expected to abide by those strategies, plans, standards, and frameworks when working on a project. Because enterprise architects are not close to the day-to-day business processes, the artifacts they produce may not be considered very useful by the project teams and may even be seen as hindrances.

In addition, the implication of the solution versus enterprise architect terminology is that enterprise architects are not perceived as focused on providing solutions and therefore are perceived as being potentially part of the problem. Often, enterprise architects are portrayed as academic, theoretical, and out of touch with the day-to-day realities of delivering and maintaining IT systems. Of course, the same is sometimes said about some solution architects, too, or even about anyone who plays an architect role on a project.

A third issue with architecture is the accelerating speed of IT delivery. As the pace of the business cycles increases, the IT function is expected by their business partners to deliver systems more rapidly. A new class of systems has emerged: the systems of engagement, which provide capabilities that support the interactions between the outside world (prospects, customers, and other companies) and the enterprise. Because of advances in software engineering as well as the availability of new enabling technologies, the approach used for architecting modern systems that require agility is different from the approach used to architect older systems. This has created the need for a two-speed architecture approach, leveraging tools and techniques to architect systems of engagement that are fundamentally different from the tools and techniques used to architect the older systems of record.

The planning processes associated with Enterprise Architecture are best suited to the older systems of record, which do not need to change as often as the newer systems of engagement. Old school enterprise architects insist on using the same set of strategies, frameworks, and standards for every system, including the newer systems of engagement, and this attitude creates a further disconnect among the business, the software developers, and the architects.

Given these three issues, we have to accept that traditional architecture has failed. It is not surprising that Enterprise Architecture groups are being disbanded and that even the most practical of the solution architects are just tolerated in a world where agility and speed of delivery are paramount.

Architecture in an (increasingly) Agile World

The first Agile methodology, XP, was created by Kent Beck in March 1996 when he was working at Chrysler.³ The first book on Agile (Extreme Programming Explained) was published in October 1999,⁴ which was followed by several publications on various Agile methodologies inspired by XP.

Since then, adoption of these methodologies has been nothing short of explosive, and Agile methodologies have been maturing so quickly that Agile is rapidly becoming old news. Even the most conservative companies have started looking into Agile. Techniques such as pair programming, test-driven development, and Continuous Integration have become mainstream. How have software architecture practitioners responded to the Agile tidal wave? Poorly, in our opinion.

• The beginnings: architecture and extreme programming: In the beginning, Agile (i.e., XP) and architecture ignored each other. In the minds of XP practitioners, architects were part of the red tape that they believed was counterproductive and were trying to eliminate. If a role or an activity was not seen as directly correlated with the development of executable code, the XP response was to eliminate it, and architecture was considered part of this