Sunday, April 24, 2016

AWS Lambda Reading List

I'm hearing a lot about AWS Lambda these days and "serverless" architectures. By "serverless", I mean the concept, not the product (here). Basically, AWS Lambda is computing power without management. You provide your code, AWS Lambda runs it. You have minimal setup and no responsibility for maintaining servers on which to run that code.

As with all new technology fads, there's a lot of buzz and a flurry of unorganized content. My intent with this entry is to keep a current list of relevant Lambda articles and categorize them. Hopefully, this will streamline your research if you're looking at using AWS Lambda at your organization.

I intend to update this list as new material comes to my attention. If you see articles that are worthy of mention, please add a comment providing a link. I'll take a look.

Getting Started
These articles provide overview material describing what AWS Lambda is.

  • "It’s Amazon’s way of delivering a microservices framework far ahead of its competitors."
A Walk in the Cloud with AWS Lambda (Slideshare)
  • Provides detailed overview and possible use cases.

What are the Business Benefits?
These articles highlight business benefits provided by AWS Lambda

  • Somewhat biased as it's an Amazon executive describing how they use Lambda internally.
  • “No more glue code. No more servers. Just run your code.”

Flies in the Ointment
These articles highlight limitations and issues with AWS Lambda.

  • "Lambda is a building block, not a tool"
  • "Lambda is not well documented"
  • "Lambda is terrible at error handling"
  • Pay attention to the discussion below. There's some resistance to the error handling points.
Vendor Lock-in
Many fear becoming relient on AWS Lambda as AWS can raise prices with just a simple edit.  The following products propose solutions to mitigate that risk.

Case Studies
This section details use of AWS Lambda in practice.

  • Embedded case study for, a AWS education company.
  • Also has material on the strategic reasons to consider AWS Lambda
AWS Case Study: VidRoll (VidRoll Blog)

Saturday, April 23, 2016

"Microservices for Java Architects" has been released!

Microservices for Java Architects is a companion for the Java EE Architect’s Handbook that details Microservices Architectures. This eBook will define microservices and provide an overview of costs and benefits. As with the handbook, I’ll go on to address architecture, design, and implementation concerns specific to microservices architecture.


Given the published success stories from Netflix, Amazon, and many organizations; companies are increasingly adopting microservices architecture. For organizations that are heavily invested in Java technologies, writing microservices using Java is a natural progression. This book is your guide to designing and implementing microservices in Java.

As Java architects, it’s important to understand what microservices are, when to use them, and the value they provide. We need to understand the design and implementation considerations for using microservices so we can guide a development team through microservice development efforts. We also need to understand the various pitfalls of using microservices and the common implementation mistakes that can erode the benefits of adopting microservices.

While this book is a companion to the Java EE Architect’s Handbook, it is also separate and self-contained. As with the handbook, I'll guide architects through the entire development process, including how to effectively design service contracts. I cover in detail the coding patterns that are needed to make microservice libraries effective showing you examples along the way. I also provide guidance on how to refactor large, complex web applications into smaller and more manageable microservices, and how to make microservices easier to maintain and support.

What you’ll learn

  • What microservices architecture is and the advantages and disadvantages of using it.
  • How to refactor large applications into a microservice library.
  • How to design microservice contracts effectively.
  • Design patterns that make microservices efficient and maintainable.
  • Best practices and common mistakes for implementing microservices.
  • Guidance on how to manage microservice development teams.

Who this book is for

  • Java EE application and enterprise architects interested in using microservices to provide business value.
  • Java EE technical leads implementing microservices.
  • Java EE senior Java developers working on microservice teams.

Saturday, March 26, 2016

Book Review: The Art of Business Value by Mark Schwartz

I just read the book "The Art of Business Value" by Mark Schwartz. I came upon this book as I follow Adrian Cockcroft on Twitter and he tweeted about it. Mr. Cockcroft also has a blurb for the book on the back cover.  That was enough to get me to buy and read the book.  As a technical author myself, Amazon does not let me post reviews on technical books. As a result, my review is here.

Even with agile methodologies, IT has trouble delivering business value. The book starts with some typical observations of difficulties IT departments have implementing scrum/agile. Technical debt payment and non-functional requirements don't seem to be adequately prioritized by product owners. Product owners make priority decisions that perplex the people implementing them sometimes. Even though developers follow product owner direction, IT departments still get complaints about the low value IT generally delivers.

Business value has a vague definition and is difficult to objectively measure. The book very effectively argues that the objective of product owner, which is to deliver business value, is a vaguely defined concept and is far from objectively measurable using the information product owners have at hand. More to the point, business value is difficult for anyone to measure. The book takes us on a discussion of how to measure value from a financial perspective.

Product owners are guessing about business value with they prioritize stories. Effectively, the business value assigned to stories isn't really known by product owners. Consequently, story prioritization is the product owners 'guess' and isn't really objectively determined. You can argue that a product owners guess might be more educated than an IT developers guess, but it's a guess all the same. 

As a result, we see the issues and problems IT is all too familiar with.  I'm summarizing the argument, but the book provides robust support for this argument drawing on financial valuation methods taught in many MBA programs.

The book has an excellent premise, but unsatisfying solution. While Schwartz starts with a very good premise, he doesn't really provide adequate answers for the problems associated with evaluating business value. Given the rigor with which he describes the problem, I expected to get the same level of rigor in his search for a solution. His prescription of developing a robust continuous delivery pipeline is very much disconnected for the valuation problem and almost seems to be arrived at out of the blue. Schwartz doesn't connect the dots for me as to how with a badly defined concept of business value as a target, building a delivery pipeline represents enables us to meet that target.

Schwartz is very concise and effectively communicates his points in 131 pages, which as a busy person, I greatly appreciate. The book is also cheap for books of this type.
Bottom line: this book is worth reading for the understanding of the issues we commonly face implementing agile methodologies. Understanding a problem is necessary for finding a solution. I just wish we got a solution too.

Bottom line: Three stars. Reading the premise was enlightening and worth the reading time. Just not the same analytic rigor when discussion solutions.

Wednesday, February 24, 2016

Using Maven to produce Docker imarges

Docker is fast becoming the deployable artifact of choice. There are good reasons for this. It standardizes deployables across all technical stacks. This has the effect of streamlining work for a DevOps team working to manage large numbers of deployables. It allows developers to control the technical stack under which they deploy without and any tooling able to deploy Docker containers will just work. The reasons go on and on.

As a result, I've begun adding Docker artifacts to my standard builds for Java microservices and web applications. I begun using the Spotify Maven docker plugin.  Quite nicely done, although there were a few rough patches in getting everything to work correctly.  While the plugin itself is documented quite nicely, there's gaps in the documentation for the touch points between Docker and the Spotify Maven Docker plugin that created some headaches. This is especially true if you've a Windows development environment, which many of us have. I'm documenting this in my blog for myself as much as anybody else.

This entry uses my microservice example Moneta and its Spring Boot deployment build that can optionally produce a Docker artifact.  In fact the specific build I'm using as an example can be found here.  Pay attention to the Profiles section for id=Docker.

Using the Spotify Maven Plugin

This section guides you through adding the Spotify Maven plugin to your project.  It assumes that you've used Spring Boot, Dropwizard, or some other product to package your service into an executable jar.

Add the Spotify Maven Plugin to the build

Here's an example of the plugin include from my Moneta project.

    <!-- <targetPath>/</targetPath> -->

The plugin does provide special tags so that you can effectively code the dockerfile in the POM.   I prefer to keep the dockerfile separate; more readable and easier to maintain. You do need to include all resources needed by your dockerfile.  As an example, I've incldued my Spring Boot application jar.  All resources get bundled and transmitted to the Docker host where the docker file is run. If you dockerfile has COPY statements, those resources need to be included as resources in the POM.

Create a Dockerfile

The Spotify plugin can accept your docker build information a couple ways. You can put tags in your maven build to specify your base image, exposed ports, and entry point. Or you can create a Dockerfile and specify the directory that contains it.  I prefer this method as developers already using Docker will be familiar with the Dockerfile.  Documentation for Dockerfile syntax can be found here.

Include COPY commands for any build artifacts you want to include in the Docker image. These artifacts must also be listed as a Resource in the docker plugin section of your build.  I'll talk more about that later.

Add Preparation Tasks

I have two preparation tasks.  The Dockerfile doesn't easily support Maven variables such as ${version}, hence I had to copy my Spring Boot jar artifact into a constant name without the version number.  For example, copying file moneta-springboot-${project.version}.jar to moneta-springboot.jar that's referenced in my dockerfile.

I also checked to make sure that the environment variables DOCKER_HOST and DOCKER_CERT_PATH were set. The advantage is that developers get a more meaningful error message than "connection refused" if they are not set.  Obtaining values for these variables isn't obvious if you're using a Windows environment; see section "Verifying Docker Environment" below for details on this.

Consider using Spring profiles to make production of your Docker artifact optional. Adding a docker deployable adds significantly. It might not be needed with all local development builds and integration tests. 

Verifying Docker Environment

You must set DOCKER_HOST and DOCKER_CERT_PATH environment variables for the Spotify Docker maven plugin to work.  Figuring out proper values for these variables wasn't obvious on a Windows environment. Well, it's easy once you've figured out where to look.

The Docker Quickstart Terminal is nice enough to give you the IP address of the default docker machine it creates, but not the port. To obtain the value specify in your DOCKER_HOST setting, use the command below.  My docker host setting from this output is tcp://
    docker-machine ls

You can obtain the DOCKER_CERT_PATH value to use by executing the command below in your Quickstart terminal.  The output is also shown below.

Wednesday, January 27, 2016

Considerations for adopting Canary releases

A canary release is a tactic for reducing deployment risk. The idea is to deploy a new release to one or two nodes in a service cluster, let them handle some portion of the work load, and see if any unexpected errors result. In effect, this is a kind of "testing" in production. The term comes from the practice of miners to bring canaries with them into the mines; if the canaries died, then the air wasn't safe to breathe and they should evacuate. Like all strategies, there are advantages and disadvantages to using this approach. This blog entry attempts to outline the more important considerations.

The reason this is attractive is that it mitigates risk without slowing down developer velocity as other forms of risk mitigation (e.g. additional testing before deployment) does. If you are adopting continuous delivery where there are far more deployments and changes are deployed at a greatly increased rate, risk mitigation techniques like this are attractive.

Canary deployments seek to mitigate risk as you'll have less impact should a defect accidentally be deployed. Fewer customers or end-users will be impacted by unintended defects making it to production as the portion of the load a canary deployment handles is a fraction of the total. For example, if the canary deployment only handles 2 percent of the load, that drastically reduces the impact of an unintended defect when compared to if the new release handled 100% of the load..

One problem is how to evaluate unintended defects. Logged exceptions, where the defect results in an exception, are easy to defect. Silent defects are defects that don't result in an exception (e.g. produce an incorrect answer or incorrect data). They are harder to catch and in fact might result in a derivative error in some other service that processes incorrect data. In fact, it might not be caught until an end user or customer complains. 

Canary deployments do not effectively mitigate the risk of silent defects. The reason is that they are often not caught until far after the defect occurs. Furthermore, they often are detected manually, which introduces a time lag in and of itself.

Canary deployment capabilities increase complexity associated with your automated deployment mechanism. There are a few strategies that can be used to manage canary deployments. The available strategies are a comp[lex enough subject that they should be addressed in a separate blog post. All of these strategies introduce additional complexity to automated deployments and back-outs.

Canary deployments require the ability to measure and compare metrics of the canary to metrics of other nodes in the cluster. To those operating at a larger scale, this metric comparison between the canary and baseline is automated. Those metrics differing substantially compared to baseline are automatically rolled back.

Some use canary deployments can be utilized instead of capacity testing. The idea is that load is transferred gradually from one release to another enabling administrators to spot resource consumption issues before 100% of the load is transferred over. With this idea, all deployments are canary deployments; it's just that the deployment happens gradually over a longer period of time.

Database changes can present a problem for canary deployments. If the database changes in a way that's not backward compatible. As an example, if tables and or columns are removed. This can be a large topic and should really be addressed in a separate blog post. 

Canary deployments are only possible in cases where there are no service contract changes. In an age where we're breaking up monolithic applications into larger number of smaller deployables, not every release will be a candidate for a canary deployment. Those deployments that introduce a contract breaking change will almost certainly require an all-or-nothing deployment of both the producing service and all consuming services.

Saturday, January 16, 2016

Exception Throwing Etiquette

Nothing irritates me more than investigating reported exceptions where the developer didn't take the time to record the context with the exception.  That is, you get an error message like "Field XXX value not valid" or something like that.  What value was provided that was invalid?  What are the valid values for that field?  What were the other arguments to the method being thrown so that it's quick and easy to construct a test case for fixing the bug?  

These are all items of information that should have been reported with the exception.  Had they been reported with the exception, the developer assigned would have likely spent much less time diagnosing the issue and providing a fix.

You might ask why application architects should be concerned with code-level items such as these.  The reason is that architects are partly responsible for designing applications that are easy to support and help the organization keep support costs at bay.  Promoting and enforcing good exception etiquette is a cheap way to do that. 

Exceptions should contain context information to facilitate support.  This speeds support and problem resolution because it saves developers time.  That time savings allows developers to spend more time on feature development rather than support.  If your enterprise separates development and support, it allows the organization to spend less on support entirely.

Providing context with exceptions is easy for developers to do now.  Tooling exists now that makes this easy and safe.  I think the reason that developers don't take the time to include needed information with exceptions is laziness.  I'm going to present three options I see used most frequently.  That said, I'm sure that there are other tooling options.

Don't write custom formatting code for exceptions.  We've all see the issue of the exception reported was a null pointer from code that was in the process of formatting an exception message.  In other words, the exception that got reported was derivative; not the root exception that needs to be diagnosed.  The tooling options I'll tell you about reduce the risk of including context information with your exceptions as well as making it easier.

Apache Commons Lang Validation

A common reason for throwing exceptions is to validate arguments on public and protected methods.  I'll save a discussion on why all arguments for public and protected methods should be validated for another post. One of my favorite tools for validating arguments is the Validate utility from Apache Commons Lang.  Validate makes it very easy to both perform the validation and include context information along with the exception.  Examples follow which should make usage obvious.

Validate.notEmpty(firstName, "firstName can't be null or blank.  accountId=%s, 
    accountType=%s", accountId, accountType);

Validate.notNull(accountType, "accountType is required.  Allowed values are %s", 

Validate.isTrue(startingBalance >= minimumBalance, "Starting balance must be larger than %s.  balance=%s, accountId=%s, accountType=%s", 
    minimumBalance, startingBalance, accountId, accountType);

Exception messages produced by the above examples are:

java.lang.NullPointerException: firstName can't be null or blank.  accountId=12345, accountType=PERSONAL

java.lang.NullPointerException: accountType is required.  Allowed values are {PERSONAL,BUSINESS}

java.lang.IllegalArgumentException: Starting balance must be larger than 1000.0.  balance=100.0, accountId=12345, accountType=PERSONAL

Validate is null safe, so you don't need to worry about derivative null pointer exceptions.

Some developers prefer the Javax Validator framework as it's more annotation based.  I provide information on that later in the post.

Apache Commons Lang Contexted Exceptions

Exceptions that are not the result of a validation, but a part of something more complex, such as a remote call, I use the ContextedRuntimeException from Apache Commons Lang.  Note that there is a checked ContextedException that operates exactly the same way.

The idea is that when you create the exception you also add context values that will automatically be present in the log when the exception is reported.  the exception is null safe, so you don't need to worry about passing it null context values.  An example should make usage obvious:

throw new ContextedRuntimeException("Error adding customer", rEx)
    .addContextValue("accountId", accountId)
    .addContextValue("accountType", accountType)
    .addContextValue("customerName", customerName);

When the exception is logged, you'll see something like this:

org.apache.commons.lang3.exception.ContextedRuntimeException: Error adding customer
Exception Context:
 at examples.commons.lang.context.ContextedExceptionTest.test(

Javax Validator Framework

The Java Validator framework is popular in some organizations.  It's annotation based and is handy for validating inputs for published services automatically.  One of it's advantages is that since it's annotation based, it doesn't add code to the code base that needs to be tested.

One of the disadvantages to the Javax Validator framework is that it's much more difficult to include context information along with the exception thrown.  Yes, developers can provide explicit messages that will be used when the validation fails.  An example of including such a message follows:

public class CustomerAccount {
    @NotBlank(message="accountId can't be blank")
    private String accountId;
    @NotNull(message="accountType can't be blank")
    private AccountType accountType;
    @NotBlank(message="customerName can't be blank")
    private String customerName;
    @DecimalMin(value="1000.00", message="startingBalance must be at least $1000")
    private BigDecimal startingBalance;

Notice that in the examples above, all messages are explicit and hardcoded.  There's no easy ability to add in additional context information about other fields in the bean being validated so that a developer could get additional context to assist with diagnosing the issue.  Note that you also don't get the invalid value that violated the exception.

This presents an interesting question.  Would it be possible to use the Javax Validator framework and provide context information?  At first glance, you would have to write a custom class that implemented the MessageInterpolator interface (details here) and bootstrap it in (details here).  Maybe, I'll write one someday if I see enough of my clients utilizing this framework.  

Saturday, January 9, 2016

How Apache Mesos Adds Business Value

The Apache Mesos product adds business value.  An incredible amount of it.  But the source of that value isn't what most people think.  Most think that the business value is simplification of deployment for developers so that they spend more time developing features instead of worrying about the deployment infrastructure.  There is value for developers, but there's more.

Mesos eliminates cloud vendor lock in.  Mesos slaves run on virtual machines with prerequisite software on it.  Those slaves can exist just as easily on Amazon AWS or Google Cloud, Microsoft Azure, or any other cloud vendor.  All cloud vendors provide VPN capability so that hybrid cloud (a mixture of cloud and on premise) infrastructures are possible; everyone wants that.  You can't exist in today's marketplace without it.  There's nothing getting in the way of Mesos masters running slaves in on premise our cloud infrastructures in any combination.  Somebody makes a financial decision add to where slaves (or Mesos masters, for that matter) are hosted.  

Mesos reduces switching costs between cloud vendors.  One of the cloud "fears" I hear from clients is vendor lock-in.  That is, they start with a cloud vendor, that vendor changes its pricing structure or the quality isn't what they need, then they have a large project to switch cloud vendors.  Mesos makes switching cloud vendors or using multiple cloud vendors much cheaper and easier.  Adopting a new cloud vendor consists of establishing a VPN and creating Mesos slaves in your newly consumed cloud. Through the VPN, those newly created slaves can communicate with their Mesos masters wherever they are hosted. From a business perspective, this feature eliminates business risk. Your application runtime environment is effectively decoupled from your cloud vendor.  

Mesos vendor agnostic platform introduces a least common denominator problem.  One of the advantages of using the cloud is to capitalize on cloud vendor R&D, which is much larger and often much more focused than what enterprises can do individually.  Cloud R&D is realized by features such as dynamic scaling, which automatically scales your deployment to match the load it's currently servicing.  The web is repleat with postings about difficulties in making Mesos dynamically scalable and utilizing other cloud features.  Cloud vendor features will always be ahead of Mesos' ability to utilize them and will always have features that those adopting Mesos won't easily be able to use.  

It should be noted that Mesos is popular enough to get the attention of cloud vendors. For instance, AWS is devoting time and energy into providing a way to dynamically scale Mesos slaves through the ECS Scheduler Driver.  Azure and Microsoft have an effort going to bring Mesos (and Docker) to the windows world (here).  A quick google search reveals that there are many other Mesos integration efforts with cloud vendors and there's every reason to expect this to continue.