Automated Integration Testing in a Microservices World.

Everyone's dabbling with microservices these days. It turns out that writing distributed applications are difficult. They offer advantages to be sure. However, there is no free lunch. One of the difficulties is automated integration testing. That is, testing a microservice with all the external resources it needs including any other services it calls, the databases it uses, any queues it uses. Just setting up and maintaining these resources can be a daunting task that often takes specialized labor.  All too often, integration testing is difficult enough that the task is sloughed off. Fortunately, Docker and it's companion product Docker Compose can make integration testing much easier.

Standardizing on Docker images for deployment artifacts makes integration testing easier. Most organizations writing microservices seem to adopt Docker as a deployment artifact anyway as it greatly speeds up interaction between application developers and operations. It facilitates integration testing as well as you can deploy services you consume (or mocks for the services you consume) temporarily and run integration tests against them. Additionally, consumers for your service can temporarily deploy your docker image to perform their own integration tests. However, as most services also need databases, message queues, and possibly other resources to function properly, that isn't the end of the story. I've previously written about how to dockerize (is that a word?) your own services and applications here.

Docker images already exist for most database software and message software. It's possible to leverage these docker deployments for your own integration testing. In other words, the community has done part of your setup work for you. For example, if my service needs a PostgreSQL database to function, I leverage the official Docker deployment for my integration tests. As it turns out, the Postgres docker deployment makes their image very easy to consumer for integration testing. All I need to do is mount the directory '/docker-entrypoint-initdb.d' and make sure that directory has any SQL files and/or shell scripts I need run to set the database up for use by my application.  The MySQL docker deployment does something similar. For messaging, similar docker distributions exist for RabbitMQ, Active MQ, and Kafka. Note that ActiveMQ and Kafka aren't yet "official" docker deployments.

Docker Compose makes it very easy to assemble multiple images into a consistent and easily deployable environment. Docker-compose configurations are YAML files. Detailed documentation can be found here. It is out of scope for this blog entry to do a complete overview of Docker Compose, but I'll point you to an open source example and discuss a couple of snippets from the example as an illustration.

The screen shot on the left contains a snippet of a docker-compose configuration. The full source is here. Note that each section under services describes a docker image that's to be deployed and possibly built. In this snippet, images vote, redis, worker, and db are to be deployed. Note that vote and worker will be built (e.g. turned into a Docker image) before they are deployed. For images already built, it's only necessary to list the image name.

Other common compose directives are as follows:

• volumes-- links a directory in the real world to a directory inside the container
• ports-- links a port in the real world to a port on the inside of the container. For example, vote links port 5000 on the outside to port 80 on the inside.
• command-- specifies the command within the Docker container that will be run at startup.
• environment-- (not illustrated here) allows you to set environment variables within the Docker container

Assemble and maintain a Docker compose configuration for your services. This is for your own use in integration tests and so that your consumers can easily know what resources you require in case they want to run integration tests of their own. It's also possible for them to use that compose configuration directly and include it when they set up for their own integration tests.

The Docker environment for your integration tests should be started and shut down as part of the execution of the test. This has many advantages over maintaining the environment separately in an "always on" state. When integration tests aren't needed, they son't consume resources regardless. Those integration tests, along with their environment, can be easily run by developers locally if they need to debug issues; debugging separate environments is always more problematic. Furthermore, integration tests can be easily and painlessly be hosted anywhere (e.g. on-premise, in the cloud) and are host agnostic.

An Integration Test Example

I would be remiss if I didn't pull these concepts together for an integration test example for you.  For my example, I'm leveraging an integration test generic health check written to make sure that a RabbitMQ environment is up and functioning. The source for the check is here, but we're more interested in its integration test today. 

This test utilizes the DockerProcessAPI toolset as I don't currently work in environments that require a docker-machine and the Docker Remote API (Linux or Windows 10 Pro/Enterprise). If your environment requires a docker-machine (e.g. it is a Mac or an earlier version of Windows), then I recommend the Spotify docker-client instead.

The integration test for the health check uses Docker to establish a RabbitMQ environment before the test and shut it down after the test. This part is written as a JUnit test using the @BeforeClass and @AfterClass annotations to bring the environment up once for the entire test and not for each test individually.

In this example, I first pull the latest RabbitMQ image (official distribution). I then map a port for RabbitMQ to use and start the container. I wait five seconds for the environment to initialize, then cause a logging for the current docker environment running.

My log of what Docker containers are running isn't technically required. It does help sometimes if there are port conflicts where the test is running or other problems with a failed test that need to be investigated. As this test runs in a scheduled manner, I don't always know execution context.

After the test completes, the @AfterClass method will shut down the RabbitMQ instance I started and once again cause a container listing just in case something needs to be investigated.

That's a very short example. Had the integration test environment been more complicated and I needed Docker Compose, that would have been relatively simple with the DockerProcessAPI as well. Here's an example of bringing up a Docker Compose environment given a compose configuration YAML:

Here's an example after the test of bringing that same environment back down:

In addition, there are additional convenience methods on the DockerProcessAPI that can log compose environments that are running for investigative purposes later.

Thanks for taking time to read this entry. Feel free to comment or contact me if you have questions.  

Resources

• Docker and Docker Compose documentation
• DockerProcessAPI toolset
• Spotify Docker-Client
• Integration Test Example from btm-DropwizardHealthChecks

Book Review: Reactive Services Architecture - Design Principles for Distributed Applications

A friend of mine gave me a copy of the new book "Reactive Services Architecture - Design Principles for Distributed Applications" by Jonas Bonér.  My friend was quite taken by the content and asked me what I thought of it. Having been working with microservices architecture and writing about them for a couple of years, I was intrigued and took up the gauntlet.

I was immediately struck by the word "Reactive" and wondered what was the difference between a "reactive" microservice and a non-reactive microservice. I started reading the book with this question nagging me along the way.

Book Summary

The book starts out by a traditional defining of the problem with monolithic applications; These are applications that have grown too large and complex to maintain and enhance with any reasonable ease and speed. It has a very well written introduction to the concept of microservices architecture and how it solves the problems presented by monolithic applications. The book describes basic principles of microservices architecture and also effectively compares microservices to traditional SOA in a manner that's very easy to read and understand. Principles described by this sections are:

• Microservices do one thing and do it well; they have a single functional (e.g. business) purpose.
• Microservices most often evolve from monoliths; not often used for completely new applications

The second chapter refines the single responsibility trait that microservices have and introduces us to several additional microservice principles. They are principles that are present in previous discussions of microservice architectures and aren't really new, but are explained very clearly and concisely.  These principles are:

1. Microservices act autonomously; they are context independent.
2. Microservices own their own state / data store.
3. Microservices should have location transparency; they use a service discovery mechanism so they can scale effectively and have clustering / resilience.
4. In a microservice world, what can be made asynchronous or non-blocking, should be made asynchronous (some term this "eventually consistent").
5. In a microservice world, planning for failure is a necessity.

I did find the discussion on isolation at the beginning chapter two extremely interesting. Isolation is presented as a coupling of a service to time (when execution occurs) and space (where the service is hosted). Removing this coupling (i.e. isolating services) is a prerequisite to adhere to principles 1, 3, 4, and 5 above. I had never explicitly identified this type of coupling previously in my writing and consider it insightful. 

The author does discuss reliance on messaging technologies as a part of the definition of a "reactive" microservice. I look at messaging as one of many techniques for making service calls asynchronous. I see messaging as a technique for supporting principle 4 and mitigating 5 and not really fundamental to the architecture. Usually, architecture definitions are principle based and not reliant on a specific implementation tactic. I agree with the author in that designers usually "assume" real-time needs instead of challenging them and making more portions of a system asynchronous. The author seems to assume the opposite. To be fair on pp 36-8 (chapter 3), the author does acknowledge that there are portions of a system that must be synchronous.

There are reasons for using ("persistent") messaging to make service calls asynchronous rather than spawning the work in a separate thread. Work in a spawned thread will be lost if a service instance dies where it won't be if persistent messaging is used. The author really doesn't discuss this point or address why "messaging" (persistent or not) is the only option he presents for making work asynchronous.

Chapter three discusses several cross-cutting concerns that frequently come up with people newly introduced to microservice architectures. They are:

• Service discovery as a means to support location transparency.
• API Gateways as a way to manage evolving contracts over time.
• Messaging as a way to support several of the principles outlined above.
• Security management 
• Minimizing data coupling and coordination costs

The author does point to event driven architecture and conflict-free replicated data types (CRDTs) as being natural complements to microservice architectures. Essentially, the event-log becomes the source of truth for a microservice and the underlying database is simply a convenient "cache" of that truth. These concepts are touched on, but not really explored in depth. To be fair, these are weighty topics and likely deserve books of their own; declaring in-depth discussion of them "out of scope" for this book is reasonable.

Reviewer Summary -- Rates 4 Stars out of 5

This book is a great summary for those looking for an overview of microservices architecture. All concepts are explained concisely, clearly, and in an easy to understand writing style. As Lightbend is handing out free copies (at the time of this writing), it's certainly cost-effective.

This book respects the readers time. Time is a scarce resource for me as I'm sure it is for many. At 47 pages, it can be easily be read in one or two sittings.

This book doesn't try to sell products. When I saw the company name "Lightbend" on the cover, I was a little nervous that there would be plugs for Lightbend products. There isn't. Akka is mentioned as an implementation option as well as Apache Camel and other products. Nothing, however, that's overt marketing.

I'm not convinced that a "reactive" microservice is any different than a normal microservice. While I agree that all service calls that can be non-blocking/asynchronous should be, that's not really new or different. It is a microservice design best practice to be sure, but not really a different architecture style. The book does contain mentions of reactive programming and the Reactive Manifesto. But not really enough to link the books content specifically to those constructs for me. In the end, it's this point that kept my rating out of the five-star category.

You might wonder why I published this review on my blog instead of on Amazon. As a book author myself in this genre, Amazon will not publish my reviews on other computer-related books. Thanks for reading this article.

All Monoliths Are Not Created Equal

All monoliths are not created equal. That is, not all monoliths became monoliths the same way nor do they have the same root causes. With this post, I'll identify different types of monoliths I've seen. This is useful as the tactics I use to break the monolith into smaller, more manageable, pieces is different depending on which category of monolith I'm dealing with. It's worth noting that some times elements of these different categories of monoliths are used in combination.

A monolith is an application that has grown too large to effectively manage. That is, it's expensive to enhance and fix. Change to monoliths often come with unintended consequences; you fix something and other things break. Monoliths effectively marry you to a technical stack, making it difficult to grow with new technologies as they evolve and mature. Due to the size and complexity of underlying monolithic code, it's often expensive and time-consuming to bring on new developers (there's so much to learn). The same labor characteristic limits the business in that outsourcing change often isn't an option.  

Feature-Bloated Web Application

This type of monolith is specific to web applications. It becomes larger and more complex as new features are added. Moreover, some of those new features weren't considered when the application was originally designed. Consequently, those new features are often inelegantly "tacked on" and don't really conform to the initial design of the application. Often, there's no time and budget to revamp the application design to add new features in ways that are easy to maintain and support. The consequence for this is often technical debt in the form of unwanted coupling.

New features add complexity to all parts of the application. New features impact not only the user interface but server-side backing code that supports that interface and the underlying database. Often that server-side backing code and database is tightly coupled with the user interface and "assumes" that the business processes currently implemented by the interface. Often, it is assumed that these business processes are static and rarely change. Unfortunately, that is often not the case. Developers react to the additional complexity in predictable ways.

Beware of undocumented developer assumptions. These assumptions are often made for developer convenience. Those assumptions make the new feature easier to code. I don't blame them; the extremely large feature set is often too difficult to keep in one person's head. Developers, as their assumptions produce initially more streamlined and easy to read code, believe that they are making a positive contribution.  Unfortunately, they are also inadvertently creating land-mines for other developers who aren't aware of those underlying assumptions and need to change that code for some other feature addition. As most of these assumptions go undocumented or even unidentified, other developers working on that section of code are often unaware of all the consequences of their changes.

Most monoliths don't measure feature usage. As a consequence, features, once implemented, never get removed. It would be logical to remove features that aren't used or aren't used often as the complexity for having that code remain will negatively impact your ability to add new features down the road. Sadly, getting budget for removing features is hard to do in most organizations as the benefit to doing so is hard to measure and too intangible.

 It's worth noting that often the design of the underlying database often becomes bloated along with the web application. Due to the large scope of business it supports, the underlying database often becomes an Achilles heel for the organization. Not only is database refactoring much more costly and difficult than refactoring code, it's not uncommon for that database to be used by multiple applications. The implication here is that change to the underlying database often has consequences for more than the monolithic application it was originally created for.

Data Store Strangle

Despite the length of time that databases (relational and non-relational) have existed, database design skills are woefully inadequate at most organizations. As a result, you often get a process-oriented database structure. That is, a database structure that is specific to processes currently implemented by the application that database supports. Usually, the mindset is that the database can be 'refactored' along with code as new features are implemented.

Database design impacts application code. At the risk of stating the obvious, applications have code dedicated to reading and writing to the database. Hopefully, database code is restricted to a data access layer within the application, but not all developers adopt the practice of separating data access concerns. However, the readability of that data access code is directly impacted by the quality of the design of the underlying database. If the database structure is complex to the point that its not understood, chances are the data access code that uses that database will be just as hard to understand. Furthermore, if the application uses a different domain mode than was implemented in the database, some type of data structure translation is present in data access code. The short story is that the database design used can greatly increase the size and complexity of data access code.

Refactoring databases is harder and more time-consuming than refactoring code. There are a couple of false assumptions buried in that line of thinking. First, it is assumed that refactoring a database is as easy as refactoring code. Not true. Any database refactoring often must take place while supporting existing code. Also, any database refactoring also must have some type of data conversion as a part of it.  Another way to look at it is that code only has state that's transient. That is, its state exists only at run time. Database state is persistent. Changes to that database must include converting the data that was stored into its new format.

Refactoring databases has more risk than refactoring code. If a code refactoring is discovered to have a defect, backing out that change is usually an option. Database refactoring changes usually don't have this safety net. Yes, you can restore a version of the database that was taken before the refactoring was implemented, but users will loose any data entered while that refactoring was in place. Unless you code and test a reverse conversion, implementing database refactoring changes commits you (pun definitely intended) past the point of no return. 

The Over-Engineered Tarball Component

Some developers can't resist over-engineering internal components they write. Often that happens because they are bored and looking for something more interesting to do. I can't fault them for that motivation. Unfortunately, when those mental musings get implemented in applications, additional complexity often results. This contributes to an application becoming a monolith.

I've seen a case where such a component was written to validate cash transfers from account to account. While laymen to this business process might look at such a transfer action as simple, those of you deeply embedded in bank know differently. The tax ramifications and considerations are incredible depending on how that transfer is recorded. In fact, we had over 100,000 permutations and combinations of transfers that needed to be verified and tested.  Anyway, the component in question utilized the same type of bit-switching algorithm that UNIX uses for file permissions, but with many more dimensions than just user, group, and all permissions.  Anyway, when that person left, nobody could understand the component nor could they effectively change the rules it applied.  Long story short: the application was held hostage by this embedded, but critical component.

Overly complex components have a way of creating complexity in the surrounding parts of the application. In my example, we had code to correct the validation verdict in places where it came up with the incorrect answer. This was needed as nobody knew how to change the overly-complex component handling validation.

The business process implemented by the component is often not understood. That is, replacing such a component is often more difficult as the business rules it implements are often not understood. It's hard to code a replacement without a specific target.

The Distributed Monolith
Separate services are supposed to increase developer velocity by reducing the size and complexity of code that needs to be changed. We all know that this doesn't always work out. We've all seen situations where feature additions require coordinated changes across multiple components. We've all seen that even though these services are theoretically decoupled and should be able to be changed/enhanced/deployed independently, that this doesn't seem to happen much of the time.  The reason is that these services are not truly decoupled. Let me explain.

Developers have a desire to consolidate code. We've incented this in developers for years by promoting DRY (Don't Repeat Yourself). Carrying this idea forward, developers consolidate repeated code whenever they find it. Invariably, business code gets consolidated into common libraries. It starts with implementing a service contract in code, making a library out of that, and sharing it across the publishing service and all consumers. It often progresses from there to include common business code that is listed as a dependency for multiple services. this common business code, which usually is used for manipulating common business data, tends to also be consolidated. Perhaps by publishing that logic in common libraries, and is used by several services.

Developers are often quite proud of the code consolidation. Unfortunately, whenever that common code changes, it forces a coordinated redeployment of all consumers along with the producer. In this world, there's no such thing as a non-breaking change. In essence, DRY, if used for business logic including service contracts, increases coupling between services. If that common code is purely technical and services are completely free to decide when they upgrade to new versions, then there's no coupling issue.

Distributed monoliths are caused by coupling between services that forces change across multiple services. In these cases, you're not getting the benefit of breaking the world up into smaller services. In fact, just the opposite. You get all the disadvantages of a highly coupled monolith along with the additional overhead of managing a larger number of services. In other words, you forgo all advantages of implementing microservices and pay more for the privilege.

Separate common business functionality into its own service instead. Deploy it once. call it from whatever services need it. Only maintain it once. 

Don't create common code that represents your service contracts. Consumers usually don't need all attributes in the request or response for a REST service call. Let's take an example of a REST call that returns customer information. For each customer, there might be 50 attributes. Of that, possibly a consumer only needs five or six. That consumer call should not break if new attributes are added or if an attribute that the consumer ignores is changed or withdrawn. If that contract is represented by common code, that common code will necessarily list all the attributes that the consumer ignores and such changes will be a breaking change. 

Services should always be able to choose when they upgrade included libraries.  Always - no exceptions. To get the benefits of breaking the monolith up into several services, you cannot permit any common code that could force a deployment of a service.

Trailer

In forthcoming blog entries, I'll suggest tactics for breaking each of these types of monoliths into more manageable applications. Thanks for reading.

AWS Lambda Reading List

I'm hearing a lot about AWS Lambda these days and "serverless" architectures. By "serverless", I mean the concept, not the product (here). Basically, AWS Lambda is computing power without management. You provide your code, AWS Lambda runs it. You have minimal setup and no responsibility for maintaining servers on which to run that code.

As with all new technology fads, there's a lot of buzz and a flurry of unorganized content. My intent with this entry is to keep a current list of relevant Lambda articles and categorize them. Hopefully, this will streamline your research if you're looking at using AWS Lambda at your organization.

I intend to update this list as new material comes to my attention. If you see articles that are worthy of mention, please add a comment providing a link. I'll take a look.

Getting Started

These articles provide overview material describing what AWS Lambda is.

AWS Lambda -- Run Code in the Cloud (AWS Blog)

• "It’s Amazon’s way of delivering a microservices framework far ahead of its competitors."

A Walk in the Cloud with AWS Lambda (Slideshare)

• Provides detailed overview and possible use cases.

What are the Business Benefits?

These articles highlight business benefits provided by AWS Lambda

It’s Game Over in the Cloud: Some Shocking Ideas About AWS Lambda

• Somewhat biased as it's an Amazon executive describing how they use Lambda internally.
• “No more glue code. No more servers. Just run your code.”

Flies in the Ointment

These articles highlight limitations and issues with AWS Lambda.

3 Reasons AWS Lambda Is Not Ready for Prime Time (Datawire Blog)

• "Lambda is a building block, not a tool"
• "Lambda is not well documented"
• "Lambda is terrible at error handling"
• Pay attention to the discussion below. There's some resistance to the error handling points.

Vendor Lock-in

Many fear becoming relient on AWS Lambda as AWS can raise prices with just a simple edit.  The following products propose solutions to mitigate that risk.

Project Kratos

• Want AWS Lambda without the Amazon lock-in? Meet Project Kratos

Project Serverless

Case Studies

This section details use of AWS Lambda in practice.

Author Interview: AWS Lambda and serverless architecture (Manning)

• Embedded case study for acloud.guru, a AWS education company.
• Also has material on the strategic reasons to consider AWS Lambda

AWS Case Study: VidRoll (VidRoll Blog)

AWS Case Study: Netflix & AWS Lambda (AWS site)

Implementation Issues
This section provides technical assistance for specific issues I''ve had implementing AWS Lambdas.

Running Python with Compiled Code on AWS Lambda (PerryGeo Blog)

• Getting Python Lambdas up and running is a completely aggravating experience.  This blog helped me quite a lot.
• In addition to Mathew's wise guidance, make sure you install/compile any binaries on an instance using the officially supported Lambda AMIs (listed here).  Note that you will likely need to install a compiler (e.g. yum install gcc)

Using Maven to produce Docker imarges

Docker is fast becoming the deployable artifact of choice. There are good reasons for this. It standardizes deployables across all technical stacks. This has the effect of streamlining work for a DevOps team working to manage large numbers of deployables. It allows developers to control the technical stack under which they deploy without and any tooling able to deploy Docker containers will just work. The reasons go on and on.

As a result, I've begun adding Docker artifacts to my standard builds for Java microservices and web applications. I begun using the Spotify Maven docker plugin.  Quite nicely done, although there were a few rough patches in getting everything to work correctly.  While the plugin itself is documented quite nicely, there's gaps in the documentation for the touch points between Docker and the Spotify Maven Docker plugin that created some headaches. This is especially true if you've a Windows development environment, which many of us have. I'm documenting this in my blog for myself as much as anybody else.

This entry uses my microservice example Moneta and its Spring Boot deployment build that can optionally produce a Docker artifact.  In fact the specific build I'm using as an example can be found here.  Pay attention to the Profiles section for id=Docker.

Using the Spotify Maven Plugin

This section guides you through adding the Spotify Maven plugin to your project.  It assumes that you've used Spring Boot, Dropwizard, or some other product to package your service into an executable jar.


Add the Spotify Maven Plugin to the build

Here's an example of the plugin include from my Moneta project.

<plugin>
 <groupId>com.spotify</groupId>
 <artifactId>docker-maven-plugin</artifactId>
 <version>0.3.258</version>
 <configuration>
  <imageName>moneta-springboot-${project.version}</imageName>
  <dockerDirectory>docker</dockerDirectory>
  <resources>
   <resource>
    <!-- <targetPath>/</targetPath> -->
    <directory>${project.build.directory}</directory>
    <include>moneta-springboot.jar</include>
   </resource>
  </resources>
 </configuration>
 <executions>
  <execution>
   <id>build-image</id>
   <phase>install</phase>
   <goals>
    <goal>build</goal>
   </goals>
  </execution>
 </executions>
</plugin>

The plugin does provide special tags so that you can effectively code the dockerfile in the POM.   I prefer to keep the dockerfile separate; more readable and easier to maintain. You do need to include all resources needed by your dockerfile.  As an example, I've incldued my Spring Boot application jar.  All resources get bundled and transmitted to the Docker host where the docker file is run. If you dockerfile has COPY statements, those resources need to be included as resources in the POM.

Create a Dockerfile

The Spotify plugin can accept your docker build information a couple ways. You can put tags in your maven build to specify your base image, exposed ports, and entry point. Or you can create a Dockerfile and specify the directory that contains it.  I prefer this method as developers already using Docker will be familiar with the Dockerfile.  Documentation for Dockerfile syntax can be found here.

Include COPY commands for any build artifacts you want to include in the Docker image. These artifacts must also be listed as a Resource in the docker plugin section of your build.  I'll talk more about that later.

Add Preparation Tasks

I have two preparation tasks.  The Dockerfile doesn't easily support Maven variables such as ${version}, hence I had to copy my Spring Boot jar artifact into a constant name without the version number.  For example, copying file moneta-springboot-${project.version}.jar to moneta-springboot.jar that's referenced in my dockerfile.

I also checked to make sure that the environment variables DOCKER_HOST and DOCKER_CERT_PATH were set. The advantage is that developers get a more meaningful error message than "connection refused" if they are not set.  Obtaining values for these variables isn't obvious if you're using a Windows environment; see section "Verifying Docker Environment" below for details on this.

Consider using Spring profiles to make production of your Docker artifact optional. Adding a docker deployable adds significantly. It might not be needed with all local development builds and integration tests. 

Verifying Docker Environment

You must set DOCKER_HOST and DOCKER_CERT_PATH environment variables for the Spotify Docker maven plugin to work.  Figuring out proper values for these variables wasn't obvious on a Windows environment. Well, it's easy once you've figured out where to look.

The Docker Quickstart Terminal is nice enough to give you the IP address of the default docker machine it creates, but not the port. To obtain the value specify in your DOCKER_HOST setting, use the command below.  My docker host setting from this output is tcp://192.168.99.100:2376

    docker-machine ls

You can obtain the DOCKER_CERT_PATH value to use by executing the command below in your Quickstart terminal.  The output is also shown below.
    echo $DOCKER_CERT_PATH

Considerations for adopting Canary releases

A canary release is a tactic for reducing deployment risk. The idea is to deploy a new release to one or two nodes in a service cluster, let them handle some portion of the work load, and see if any unexpected errors result. In effect, this is a kind of "testing" in production. The term comes from the practice of miners to bring canaries with them into the mines; if the canaries died, then the air wasn't safe to breathe and they should evacuate. Like all strategies, there are advantages and disadvantages to using this approach. This blog entry attempts to outline the more important considerations.

The reason this is attractive is that it mitigates risk without slowing down developer velocity as other forms of risk mitigation (e.g. additional testing before deployment) does. If you are adopting continuous delivery where there are far more deployments and changes are deployed at a greatly increased rate, risk mitigation techniques like this are attractive.

Canary deployments seek to mitigate risk as you'll have less impact should a defect accidentally be deployed. Fewer customers or end-users will be impacted by unintended defects making it to production as the portion of the load a canary deployment handles is a fraction of the total. For example, if the canary deployment only handles 2 percent of the load, that drastically reduces the impact of an unintended defect when compared to if the new release handled 100% of the load..

One problem is how to evaluate unintended defects. Logged exceptions, where the defect results in an exception, are easy to defect. Silent defects are defects that don't result in an exception (e.g. produce an incorrect answer or incorrect data). They are harder to catch and in fact might result in a derivative error in some other service that processes incorrect data. In fact, it might not be caught until an end user or customer complains. 

Canary deployments do not effectively mitigate the risk of silent defects. The reason is that they are often not caught until far after the defect occurs. Furthermore, they often are detected manually, which introduces a time lag in and of itself.

Canary deployment capabilities increase complexity associated with your automated deployment mechanism. There are a few strategies that can be used to manage canary deployments. The available strategies are a comp[lex enough subject that they should be addressed in a separate blog post. All of these strategies introduce additional complexity to automated deployments and back-outs.

Canary deployments require the ability to measure and compare metrics of the canary to metrics of other nodes in the cluster. To those operating at a larger scale, this metric comparison between the canary and baseline is automated. Those metrics differing substantially compared to baseline are automatically rolled back.

Some use canary deployments can be utilized instead of capacity testing. The idea is that load is transferred gradually from one release to another enabling administrators to spot resource consumption issues before 100% of the load is transferred over. With this idea, all deployments are canary deployments; it's just that the deployment happens gradually over a longer period of time.

Database changes can present a problem for canary deployments. If the database changes in a way that's not backward compatible. As an example, if tables and or columns are removed. This can be a large topic and should really be addressed in a separate blog post. 

Canary deployments are only possible in cases where there are no service contract changes. In an age where we're breaking up monolithic applications into larger number of smaller deployables, not every release will be a candidate for a canary deployment. Those deployments that introduce a contract breaking change will almost certainly require an all-or-nothing deployment of both the producing service and all consuming services.

Management Benefits from using Microservice Architectures

Much has been written on technical costs and benefits of using microservice architectures.  Also, we’re starting to see more management topics, such as prerequisites for implementing microservice architectures.  For an excellent summary of needed prerequisites to implementing microservices, see Fowler’s “Microservice Prerequisites”.  Among the prerequisites for succeeding with microservices is a dev-ops culture where environment setups and deployments are automated.  You also need skills in establishing, defining, and documenting web service operations and contracts; specifying the inputs/outputs of each service.  I believe that there is more to the list of what’s been written about prerequisites, but that will be the subject of another blog post.  Not enough has been written about the management and business benefits to utilizing microservice architectures.  I’d like to change that.

You might question why a description of management/business benefits to using microservices architectures would appear on a blog targeted at java architects.  The reason is that it often falls to the architect's to 'sell' the idea to management.  That makes the topic relevant to architects and worth including on this blog.

Resource Staffing Flexibility and Insulation

Microservice development teams are small and focused.   Developers *only* work on one service at a time.  That service is supposed to have a firm contract those developers are coding to preferably with containerized deliverables (i.e. Docker) that.  The point is that they’re mainly heads-down development.   They shouldn’t be burdened (or have to be burdened) with communication overhead beyond with other developers working on that service.

They are working as an independent team, completely asynchronously from all other service teams.  Combined with the fact that they have low communication overhead and defined deliverables, they can be anyplace in the world in any time zone.  It doesn’t matter whether they are in-sourced in some corporate office somewhere or off-shore.  

Given that the work of the team is confined to the focused requirements for the service, worries about code quality, etc. are at least contained to the small service source code base.  In other words, there’s a limit to how much code that team can mess up and create maintenance issues.  

    

This means that for development for microservices services can be more easily outsourced.

This also means that you’re better insulated from rogue developers.  We’ve all managed developers who produce unmaintainable code.   As rogue developers are confined to one small and focused code base, there’s a limit to how much damage they can do to that code base.   Furthermore, that code base is small, focused, and well-defined; it’s more replaceable if need be.  You’re not letting rogue developers loose in a 500K line monolithic application to cause havoc on a wider scale.

Increased Development Velocity

Assuming that contracts for needed services are fully defined along with the dependencies between those services, it’s easier to bring larger number of development teams on for ongoing efforts.  The concern about bringing on additional developers is that that typically means more communication overhead.  As a result of that increased overhead, developer productivity has diminishing returns.  You get less and less productivity out of each additional developer.  

While individual developer productivity does increase in microservice teams as they don’t have as much communication overhead, there is a limit to how much productivity increase you’ll get with any individual developer.  Your velocity increase really comes from the ease with which new teams can be added.  Developers never see or are impacted by the size of the overall effort as they are heads-down development in a small, focused, team.   Consequently, it’s easier for management to scale development to larger numbers of developers.  Note that in addition to having all service contracts fully defined, you also have to have identified any dependencies between contracts; some services might require other services be completed first before work on a particular service can finish.  Note that any development team can stub dependent services and start work before dependent services are finished.

What really happens here is that the constraint to scaling development shifts to management and enterprise architects responsible for identifying individual services and specifying the service contracts.  With larger efforts, will come more coordination points.  As some services call other services, those dependent services need to be built first.  The order in which work flows down to development teams needs to be coordinated.

Technical Stack Freedom

In a microservices world, services are containerized.  That is, the deployment artifact has an operating system and other software that it needs (e.g. Java JVM, PHP, Python, GoLang, etc.).  Each service can easily operate at different levels of these technologies.  The corollary is that new technologies can be more easily introduced or upgraded.   That makes it easier for you to capitalize on newer technologies much more quickly than you can with traditional monolithic web applications.

With traditional web applications, the code base is much larger.  Therefore, the impact of upgrades will be much larger and require more extensive regression testing.  Moreover, switching development languages is much more difficult with the larger code base.

This is a technical benefit, but I believe it to be a management benefit as well.  Managers are always looking for new ways to deliver software faster.  One of the avenues for increased developer productivity is technology advances with the available tool sets.  Your ability to utilize newly developed tool sets and bring about productivity increases the business wants is better with microservice architectures than with traditional web deployments.

A Word about Microservice Prerequisites.

This blog post doesn’t address the prerequisites you need to have in place before you can succeed with microservice architectures on a large scale.  That list of prerequisites is a long litany.  However, there are benefits available to offset the costs.

Writing Microservices in Java

I've been selected as a presenter at JavaOne in San Francisco (Oct 25-29) this year.  My session description is below.  I hope to see you there!

Using Docker to deploy Java Web Applications

Docker has fast become a favorite deployment format.  For those of you who haven't used Docker, Docker is "package once, run anywhere".  Docker does for software what shipping containers do for tangible goods.  It has standard connection mechanisms for disk and networking.  Additionally, it's easy to link Docker 'containers' together (eg link database Docker container with your application).

Advantages and Disadvantages

Not only is it a win for system administrators as it makes deployment management easier and more easily automated, but it's a win for Java developers and architects too.  As the applications environment is portable and is packaged with your application, environment-specific problems and defects are greatly reduced. What runs in production is more like what you test with.  In addition since your application runs with its own version of java and the operating system, you can upgrade your JDK and any native code you rely on with less fear of impacting other applications or even involving system administrators to make the native software change. 

Getting past the hype, there are costs and disadvantages to using Docker.  Running Docker and its associated tools is much easier on Linux; Windows has limited support.  This fact creates inconveniences in most corporate environments where most development occurs on Windows.  Yes, there is boot2docker, but that solution is far from complete or convenient.  Platform differences also present issues adding maven functionality to produce docker images; maven builds are supposed to be platform independent.  Additionally, Docker does have a learning curve.

Some may fear performance issues with an additional virtualization layer.  IBM did a good study showing the performance impact to be negligible.   All in all, the advantages of using Docker outweigh the costs.

A Java EE / Docker Example

As it turns out, producing a deployable docker image for Java Web application is easy.  I leverage a packaging product such as Dropwizard or Spring Boot to make such deployments even easier.  That said, Tomcat and many other containers have official Docker images available, so this step isn't strictly required.

As an example, I'll use a microservice I'm writing for a series of presentations called Moneta, named after the Greek goddess of memory.  Moneta provides a Restful web service interface to selected portions of a relational database.

Artifacts for the Moneta product include a Dropwizard deployment for which I've defined a Docker image, which I've open sourced (here).  I'll briefly go through the example as you might find it useful.

Sample Docker File

FROM java:7-jre

MAINTAINER Derek C. Ashmore

#  External volume definitions

RUN mkdir /jarlib

VOLUME /jarlib

RUN mkdir /config

VOLUME /config

RUN mkdir /logs

VOLUME /logs

ENV MONETA_URL https://github.com/Derek-Ashmore/moneta/releases/download/moneta-0.9.1-alpha/moneta-dropwizard-0.9.1-alpha.jar

RUN curl -SL "$MONETA_URL" -o moneta-dropwizard.jar

ENV CLASSPATH /config:/jarlib/*.jar:moneta-dropwizard.jar

EXPOSE 8080 8081

ENTRYPOINT ["java", "-classpath", "$CLASSPATH", "-jar", "moneta-dropwizard.jar", "server", "/config/moneta-dropwizard.yaml"]

Docker files are scripts that produce runnable Docker images.  A couple of items to note.  In a Docker file, you expose ports and disk volumes to the outside world.  I've exposed certain ports within the Docker container, but system administrators can map those ports differently when the container is run.  For example, admins might map 8080 to 9125.  In other words, the admins still need to coordinate ports; that is still managed outside Docker.

Additionally, I've created and exposed three file mount points.  Admins map those volumes to physical disk when they run the container.  I've elected to make the product configurable and extensible.  The configuration and even additional jars can be supplied when the container is run.  I've also exposed a file system for logs so they can be processed by a log management application, such as Splunk or Logsplash.

Docker Application Run Example

Bottom line, it's possible to run the same Docker container in multiple contexts.  I can run different instances of Moneta against different databases and can horizontally scale it ad infinitum.

These mappings become evident in the command used to run the Moneta image.

docker run -d -p 8080:8080 -p 8081:8081 \ 

-v /c/Users/TheAshmores/moneta/dropwizard/config:/config \ 

-v /c/Users/TheAshmores/moneta/dropwizard/logs:/logs \ 

-v /c/Users/TheAshmores/moneta/dropwizard/jarlib:/jarlib \ 

force66/moneta-dropwizard

There are other possibilities.  Docker recently announced a product called 'Compose' (here) that allows you to specify and configure multi-container applications.  This makes coordination of multiple containers much easier.  A compose discussion is best left for another post.