Saturday, September 5, 2015

Management Benefits from using Microservice Architectures

Much has been written on technical costs and benefits of using microservice architectures.  Also, we’re starting to see more management topics, such as prerequisites for implementing microservice architectures.  For an excellent summary of needed prerequisites to implementing microservices, see Fowler’s “Microservice Prerequisites”.  Among the prerequisites for succeeding with microservices is a dev-ops culture where environment setups and deployments are automated.  You also need skills in establishing, defining, and documenting web service operations and contracts; specifying the inputs/outputs of each service.  I believe that there is more to the list of what’s been written about prerequisites, but that will be the subject of another blog post.  Not enough has been written about the management and business benefits to utilizing microservice architectures.  I’d like to change that.

You might question why a description of management/business benefits to using microservices architectures would appear on a blog targeted at java architects.  The reason is that it often falls to the architect's to 'sell' the idea to management.  That makes the topic relevant to architects and worth including on this blog.

Resource Staffing Flexibility and Insulation

Microservice development teams are small and focused.   Developers *only* work on one service at a time.  That service is supposed to have a firm contract those developers are coding to preferably with containerized deliverables (i.e. Docker) that.  The point is that they’re mainly heads-down development.   They shouldn’t be burdened (or have to be burdened) with communication overhead beyond with other developers working on that service.

They are working as an independent team, completely asynchronously from all other service teams.  Combined with the fact that they have low communication overhead and defined deliverables, they can be anyplace in the world in any time zone.  It doesn’t matter whether they are in-sourced in some corporate office somewhere or off-shore.  

Given that the work of the team is confined to the focused requirements for the service, worries about code quality, etc. are at least contained to the small service source code base.  In other words, there’s a limit to how much code that team can mess up and create maintenance issues.  
This means that for development for microservices services can be more easily outsourced.

This also means that you’re better insulated from rogue developers.  We’ve all managed developers who produce unmaintainable code.   As rogue developers are confined to one small and focused code base, there’s a limit to how much damage they can do to that code base.   Furthermore, that code base is small, focused, and well-defined; it’s more replaceable if need be.  You’re not letting rogue developers loose in a 500K line monolithic application to cause havoc on a wider scale.

Increased Development Velocity

Assuming that contracts for needed services are fully defined along with the dependencies between those services, it’s easier to bring larger number of development teams on for ongoing efforts.  The concern about bringing on additional developers is that that typically means more communication overhead.  As a result of that increased overhead, developer productivity has diminishing returns.  You get less and less productivity out of each additional developer.  

While individual developer productivity does increase in microservice teams as they don’t have as much communication overhead, there is a limit to how much productivity increase you’ll get with any individual developer.  Your velocity increase really comes from the ease with which new teams can be added.  Developers never see or are impacted by the size of the overall effort as they are heads-down development in a small, focused, team.   Consequently, it’s easier for management to scale development to larger numbers of developers.  Note that in addition to having all service contracts fully defined, you also have to have identified any dependencies between contracts; some services might require other services be completed first before work on a particular service can finish.  Note that any development team can stub dependent services and start work before dependent services are finished.

What really happens here is that the constraint to scaling development shifts to management and enterprise architects responsible for identifying individual services and specifying the service contracts.  With larger efforts, will come more coordination points.  As some services call other services, those dependent services need to be built first.  The order in which work flows down to development teams needs to be coordinated.

Technical Stack Freedom

In a microservices world, services are containerized.  That is, the deployment artifact has an operating system and other software that it needs (e.g. Java JVM, PHP, Python, GoLang, etc.).  Each service can easily operate at different levels of these technologies.  The corollary is that new technologies can be more easily introduced or upgraded.   That makes it easier for you to capitalize on newer technologies much more quickly than you can with traditional monolithic web applications.
With traditional web applications, the code base is much larger.  Therefore, the impact of upgrades will be much larger and require more extensive regression testing.  Moreover, switching development languages is much more difficult with the larger code base.

This is a technical benefit, but I believe it to be a management benefit as well.  Managers are always looking for new ways to deliver software faster.  One of the avenues for increased developer productivity is technology advances with the available tool sets.  Your ability to utilize newly developed tool sets and bring about productivity increases the business wants is better with microservice architectures than with traditional web deployments.

A Word about Microservice Prerequisites.

This blog post doesn’t address the prerequisites you need to have in place before you can succeed with microservice architectures on a large scale.  That list of prerequisites is a long litany.  However, there are benefits available to offset the costs.

Thursday, August 6, 2015

Exception Handling Issues for SOAP Faults with CXF Clients

The Apache CXF product is a wonderful product and for the most part it is easy to use.  It's very common to use CXF to generate clients for SOAP services.  Support for those clients for SOAP services that specify faults (the SOAP version of an exception) is not easy.

For example, when CXF generates exceptions for SOAP faults, which it does, it embeds details of the fault in a field on the exception that isn't reported in either the message or the stack trace.  This information is critical for support developers diagnosing problems and issues.  For example, I generated CXF clients on  a public SOAP service that utilizes faults.  It generated the following exception:

Generated Exception excerpt:
public class BatchException_Exception extends Exception { 
    private com.postini.pstn.soapapi.v2.automatedbatch.BatchException batchException;

public com.postini.pstn.soapapi.v2.automatedbatch.BatchException getFaultInfo() {
        return this.batchException;


The meat of the exception - the information you need to actually solve the issue - is in that BatchException class that contains a useful message.  Unfortunately, the generated CXF exception doesn't provide this information in either getMessage() or printSTackTrace().  Here's what the trace looks like; it doesn't contain the embedded CXF diagnostic information:

com.postini.pstn.soapapi.v2.automatedbatch.BatchException_Exception: exception message at org.force66.cxfutils.CxfSoapFaultRuntimeExceptionTest.testBasic( at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( at sun.reflect.DelegatingMethodAccessorImpl.invoke(

If you want this information, you have to programatically dig it out.  This isn't hard, but a typical SOAP service will have numerous exceptions. There's no consistent interface implemented, so it's hard to write generic code that will provide the useful fault information when errors occur.  It's a lot of custom code to handle each type of fault.  What a pain.  I've figured out a way to solve this problem.

Fortunately, CXF does allow you to specify the base exception it uses.  java.lang.Exception is the default.  I've created an exception that is meant to be used as a root exception by CXF.  It uses reflection to dig the embedded information out of the CXF exception and make sure it's in the exception message and printStackTrace.  In addition, I leverage the ContextedRuntimeException from Apache Commons Lang as that exception makes it easy to attach useful information to exceptions.  Here's a sample of the same exception output with the new base exception.  Note that the embedded CXF diagnostic information is present.

com.postini.pstn.soapapi.v2.automatedbatch.BatchException_Exception: exception message
Exception Context:
[1:cxfclient.message=embedded cxf info]
at org.force66.cxfutils.CxfSoapFaultRuntimeExceptionTest.testBasic(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(

at sun.reflect.DelegatingMethodAccessorImpl.invoke(

The exception that provides this functionality extends ContextedRuntimeException.  It then populates the exception context with the information CXF embeds so that it will appear in your logs without any manual programming effort.  

Information on how to specify the root exception CXF uses can be found here.

This code has been open sourced and is available on github here.  It also has been deployed to Maven and is easily includable in your projects.

 * Exception meant to be extended by Apache CXF when generating exceptions for
 * SOAP Faults. This exception will insure that embedded information CSF places
 * for exceptions will be logged.
 * @author D. Ashmore
abstract public class CxfSoapFaultRuntimeException extends
ContextedRuntimeException {

// constructors omitted for brevity

protected void checkExceptionContextInfo() {
if (!contextAdded) {
Object embeddedInfo = ReflectUtils.safeInvokeExactMethod(this,
ReflectUtils.reflectionAppendContextValues(this, embeddedInfo,
contextAdded = true;

public String getMessage() {
return super.getMessage();

public String getRawMessage() {
return super.getRawMessage();


Saturday, July 11, 2015

Writing Microservices in Java

I've been selected as a presenter at JavaOne in San Francisco (Oct 25-29) this year.  My session description is below.  I hope to see you there!

Tuesday, April 7, 2015

Making Bean Unit Testing Less Monotonous

Unit testing ordinary beans like value objects, transfer objects, or entities is the most boring coding one can be subjected to.  Yes, it's easy, but it’s extremely monotonous.  

The unit testing for beans like this do have value, but it’s limited.  If you're manually coding tests of accessors and mutators, you can make an argument that the benefit might not be worth the labor cost.  I've taken a different approach: let’s reduce the cost.  Delegate monotonous coding to a utility and only takes one line of code per bean in most cases.  At one line per bean, the cost is definitely worth the benefit.  This idea led me to create BeanTester

BeanTester gets this testing down to one line in most cases.  It works on reflection, not code generation, so tests automatically adapts to changes in your beans.  In addition to testing properties, it performs several other standard tests.  Here's the list:

  • All accessors (e.g. getXXX() methods) execute without exception
  • All mutators (e.g. setXXX() methods) execute without exception
  • accessors return the value assigned to them via their corresponding mutator (e.g. getXXX() returns the same value submitted with setXXX())
  • An instance equals() itself
  • hashcode() runs without exception
  • hashcode() for an instance is constant over time if no changes are made to it
  • hashcode() equals 0. The pit test product plugs 0 in to determine if you're properly testing hashcode()
  • clone() executes without exception (if Clonable is implemented)
  • clone() equals the instance from which it was produced
  • compareTo() executes without exception
  • toString() executes without exception
  • toString() is not null. The pit test product plugs null in to determine if you're properly testing
  • An instance compareTo() is 0 when compared to itself
  • An instance serializes and equals a reconstituted version of itself (if is implemented)

Benefits to a utility besides code reduction are numerous:

  • The test changes automatically as your beans do.
  • Some tests (i.e. Serialization) are often omitted with manually written tests.
  • Code is easy to use and a lot easier to understand.
  • Test framework independence; will work in Junit or any other Java testing framework.
  • No changes to production code needed to use
  • Published via maven for easy project incorporation

Here are a few usage examples.  The most simple example with default behavior:
BeanTester beanTester = new BeanTester();

Beans without null constructors can easily be tested, but you must provide construction arguments.

BeanTester beanTester = new BeanTester();

Fields can optionally be excluded.


Custom values can easily be added for property testing:

beanTester.addTestValues(PerformanceSummaryVO.class, new Object[]{custom1, custom2});

Please give BeanTester a field test (instructions and download here).  If you have trouble with the product, please let me know about it.

Monday, March 23, 2015

Using Docker to deploy Java Web Applications

Docker has fast become a favorite deployment format.  For those of you who haven't used Docker, Docker is "package once, run anywhere".  Docker does for software what shipping containers do for tangible goods.  It has standard connection mechanisms for disk and networking.  Additionally, it's easy to link Docker 'containers' together (eg link database Docker container with your application).

Advantages and Disadvantages

Not only is it a win for system administrators as it makes deployment management easier and more easily automated, but it's a win for Java developers and architects too.  As the applications environment is portable and is packaged with your application, environment-specific problems and defects are greatly reduced. What runs in production is more like what you test with.  In addition since your application runs with its own version of java and the operating system, you can upgrade your JDK and any native code you rely on with less fear of impacting other applications or even involving system administrators to make the native software change. 

Getting past the hype, there are costs and disadvantages to using Docker.  Running Docker and its associated tools is much easier on Linux; Windows has limited support.  This fact creates inconveniences in most corporate environments where most development occurs on Windows.  Yes, there is boot2docker, but that solution is far from complete or convenient.  Platform differences also present issues adding maven functionality to produce docker images; maven builds are supposed to be platform independent.  Additionally, Docker does have a learning curve.

Some may fear performance issues with an additional virtualization layer.  IBM did a good study showing the performance impact to be negligible.   All in all, the advantages of using Docker outweigh the costs.

A Java EE / Docker Example

As it turns out, producing a deployable docker image for Java Web application is easy.  I leverage a packaging product such as Dropwizard or Spring Boot to make such deployments even easier.  That said, Tomcat and many other containers have official Docker images available, so this step isn't strictly required.

As an example, I'll use a microservice I'm writing for a series of presentations called Moneta, named after the Greek goddess of memory.  Moneta provides a Restful web service interface to selected portions of a relational database.

Artifacts for the Moneta product include a Dropwizard deployment for which I've defined a Docker image, which I've open sourced (here).  I'll briefly go through the example as you might find it useful.

Sample Docker File

FROM java:7-jre

MAINTAINER Derek C. Ashmore

#  External volume definitions
RUN mkdir /jarlib
VOLUME /jarlib
RUN mkdir /config
VOLUME /config
RUN mkdir /logs
VOLUME /logs

RUN curl -SL "$MONETA_URL" -o moneta-dropwizard.jar

ENV CLASSPATH /config:/jarlib/*.jar:moneta-dropwizard.jar

EXPOSE 8080 8081

ENTRYPOINT ["java", "-classpath", "$CLASSPATH", "-jar", "moneta-dropwizard.jar", "server", "/config/moneta-dropwizard.yaml"]

Docker files are scripts that produce runnable Docker images.  A couple of items to note.  In a Docker file, you expose ports and disk volumes to the outside world.  I've exposed certain ports within the Docker container, but system administrators can map those ports differently when the container is run.  For example, admins might map 8080 to 9125.  In other words, the admins still need to coordinate ports; that is still managed outside Docker.

Additionally, I've created and exposed three file mount points.  Admins map those volumes to physical disk when they run the container.  I've elected to make the product configurable and extensible.  The configuration and even additional jars can be supplied when the container is run.  I've also exposed a file system for logs so they can be processed by a log management application, such as Splunk or Logsplash.

Docker Application Run Example

Bottom line, it's possible to run the same Docker container in multiple contexts.  I can run different instances of Moneta against different databases and can horizontally scale it ad infinitum.

These mappings become evident in the command used to run the Moneta image.

docker run -d -p 8080:8080 -p 8081:8081 \ 
-v /c/Users/TheAshmores/moneta/dropwizard/config:/config \ 
-v /c/Users/TheAshmores/moneta/dropwizard/logs:/logs \ 
-v /c/Users/TheAshmores/moneta/dropwizard/jarlib:/jarlib \ 

There are other possibilities.  Docker recently announced a product called 'Compose' (here) that allows you to specify and configure multi-container applications.  This makes coordination of multiple containers much easier.  A compose discussion is best left for another post.

Thursday, March 12, 2015

Project Estimation Tactics

I frequently asked for advice on how to estimate tasks or projects.  It’s a question architects get asked by project managers as we’re often in the best position to provide a realistic estimate.  Estimation is as much art as science, but there are some things you can do to reduce error over time.  Some may argue that it’s not the architect’s job to estimate; it’s the project manager’s job.  In my experience, it’s common to combine the architect and project manager roles, so the line between the two is blurry in some organizations.  

I typically determine estimates in terms of the number of labor hours or days required.  Calculating delivery dates and working different scenarios (e.g. different numbers of developers, altering the items delivered, etc.) is just math after that.

Tactic 1:  Keep a history log for past estimates

For the projects I manage as well as fill the architect role, I track hours spent on tasks as well as the estimate I provided for them.  For estimates that were largely different than the time actually taken, I do a private post-mortem.  This is valuable feedback that you can use to refine estimates for future projects.  I realize that tracking this information is boring.

Tactic 2:  Look for comparable projects

I price (a.k.a. estimate) projects the same way people price real estate.  When an assessor determines the value of your home, they measure the square footage and take various specifics (e.g. number of bedrooms, bathrooms, etc.) and compare them to recent sales in your area with about the same square footage and features.  They get a range of sales prices back and they use those to derive the value of your home.  We can use the same tactic for technology projects.

If you follow tip one, over time you will have a history record of labor used for different tasks and projects.  You can mine that library for “comparables” for the project or task you are estimating.  For example on a recent assignment, the average amount of work required to create, format, and deploy a new report took approximately 24-36 hours depending on the complexity of the report.  We’re I still at that assignment, that’s the time frame I used for the report components of projects I needed to estimate.  You can apply the same tactic to estimating other types of tasks.

Tactic 3:  Turn the “big” problem into several little problems

We apply this tactic when designing applications all the time.  It can help you with providing estimates as well.  
You might not have a comparable for the entire project.  However, if you separate that project into smaller portions, you might have history for comparable projects/tasks for at least some of the smaller components.  At the very least, you’ll have a better feeling for which portions of your estimate are likely to have the largest variance.

Sunday, March 8, 2015

Preparing Public Maven Repository Releases

As a mature product, it *should* be easier to prepare a product release bundle for Maven artifacts.  I had a hard time, both with a product that used Maven builds and one that did not.  However, the purpose of this blog entry isn’t to complain, it’s to document the process.  To be honest, my motivation to write this blog entry is really for my personal reference rather than public consumption.

Resource Requirements

Sonatype login to submit group creation requests (login available here) and deploy artifacts.
A GPG signer tool (I use Gpg4win).

One-time Workstation Set-up Tasks

Create a public / private key pair.  Instructions using Gpg4win tool Kleopatra here.

Project  Build Set-up (Maven project)

Add creation of the “sources” jar to your maven build.  Maven makes this easy via a plug-in.  Pom addition here:

Add creation of the “javadoc” jar to your maven build.  Maven makes this easy via a plug-in.  Pom addition here:

Execute your Maven build with the repository:bundle-create goal.  This will verify that you’ve provided all needed information for your project (e.g. description, SCM url, project url, etc.) in the pom file.  It will also create an unsigned deployment bundle (which is useless).

Add bundle creation and signing to your maven build.  My sample requires that you define an environment variable GPG_HOME that denotes the location of where you installed your GPG software (example C:\Software\GNU\GnuPG).  I use the ant plugin for Maven to accomplish this.  This really should be easier.  Pom addition here:

     <property environment="env" />
     <fail if="${env.GPG_HOME}" message="GPG_HOME environment variable not defined." />
     <mkdir dir="target/mavenrepo" />
     <copy file="pom.xml"
      tofile="target/mavenrepo/${}-${project.version}.pom" />
     <copy todir="target/mavenrepo">
      <fileset dir="target" includes="*.jar" />

     <exec executable="cmd" dir="target/mavenrepo">
      <env key="PATH" path="${env.GPG_HOME}" />
      <arg line="/c" />
      <arg line="gpg2.exe" />
      <arg line="-ab" />
       line="${}\mavenrepo\${}-${project.version}.pom" />
     <exec executable="cmd" dir="target/mavenrepo">
      <env key="PATH" path="${env.GPG_HOME}" />
      <arg line="/c" />
      <arg line="gpg2.exe" />
      <arg line="-ab" />
       line="${}\mavenrepo\${}-${project.version}.jar" />
     <exec executable="cmd" dir="target/mavenrepo">
      <env key="PATH" path="${env.GPG_HOME}" />
      <arg line="/c" />
      <arg line="gpg2.exe" />
      <arg line="-ab" />
       line="${}\mavenrepo\${}-${project.version}-javadoc.jar" />
     <exec executable="cmd" dir="target/mavenrepo">
      <env key="PATH" path="${env.GPG_HOME}" />
      <arg line="/c" />
      <arg line="gpg2.exe" />
      <arg line="-ab" />
       line="${}\mavenrepo\${}-${project.version}-sources.jar" />
     <jar destfile="target/${}-${project.version}-bundle.jar">
      <fileset dir="target/mavenrepo" includes="*.jar" />
      <fileset dir="target/mavenrepo" includes="*.pom" />
      <fileset dir="target/mavenrepo" includes="*.asc" />
Run you build and produce a signed bundle jar.  Example content of a properly created and sign bundle are as follows:

Submit a ticket on Sonatype (here) to setup your artifact.  An example ticket can be found here.

Manual Artifact Deployment

After successful project setup, you’ll receive an email notification.   Then you can issue deployments.   It’s possible to incorporate deployment of the bundle in your build script, but that’s a battle I haven’t fought; I deploy mine artifacts manually.  This is fine as I don’t need to deploy very often.

Log into Sonatype (here) to manually upload your deployment bundle and release the artifact , select the “Staging Upload” link, and select the “Artifact Bundle” upload mode.

Then you’re off to the races.   My hope is that this process gets *much* easier over time and that this post becomes useless and obsolete.  Please let me know if I can make any of these instructions clearer.