tag:blogger.com,1999:blog-28259356174435979492024-03-04T22:05:28.869-08:00The Cloud Architect's BlogDerek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.comBlogger54125tag:blogger.com,1999:blog-2825935617443597949.post-59375196599741700182022-12-26T06:26:00.002-08:002022-12-26T06:28:51.442-08:00Improve New Feature Delivery Rate with Value Stream Mapping<p style="text-align: justify;"> <span style="font-family: arial;">I constantly see a desire to deliver enhancements and additional business capabilities to end users at a faster rate. What I don't see is a methodical and data-driven approach to achieving a faster delivery rate. I typically use a tactic called value stream mapping to improve clients' speed to market. That tactic seems obvious to me but isn't used as widely as I think it should be.</span></p><p style="text-align: justify;"><span style="font-family: arial;">I'm going to define and illustrate value stream mapping for you in hopes that you see the value and understand the tactic well enough to apply it to your existing production processes and procedures. The concept applies to application features, infrastructure features, DevOps automation capabilities, and just about any type of information technology process I can think of. In fact, it applies to any business process I can think of, including those that aren't IT-related.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Example Application Delivery Value Stream</span></h2><p style="text-align: justify;"><span style="font-family: arial;">This is an example of the delivery process of a highly customized purchased application (COTS). As customizations were delivered by the vendor quite frequently, updates needed to be tested and deployed quite frequently. On average, the vendor supplied one to two updates per week. Each update required significant manual labor to test and deploy. The time and effort involved were costly and required tuning. We elected to do a value stream analysis. </span></p><p style="text-align: justify;"><span style="font-family: arial;">Below are components of the value stream, along with how long manual time was spent testing and deploying them. Note that given the length of outage required for deployments, significant coordination with the testing team and business users was necessary and often extended the lag between updates received from the vendor and getting those updates into the hands of end-users. </span></p><p style="text-align: justify;"><span style="font-family: arial;"></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgvG60ZEI7IZe8ZNM5BOr9EHuKMAxz3qXpCuleSa2eRWU_7XdtrTWRU3yaqjYpCtd2xNYPYW5lOi89BivNEA_66_XaMt5oiH7EzEtGxoHUaFHAjRxKsriu9KQVm4gaStAQV5KG9JTF77V1ETmHR0MJXI72fEpruS-TiSE9F79VnOXIi9Bp-fM7HASbh" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="457" data-original-width="1158" height="253" src="https://blogger.googleusercontent.com/img/a/AVvXsEgvG60ZEI7IZe8ZNM5BOr9EHuKMAxz3qXpCuleSa2eRWU_7XdtrTWRU3yaqjYpCtd2xNYPYW5lOi89BivNEA_66_XaMt5oiH7EzEtGxoHUaFHAjRxKsriu9KQVm4gaStAQV5KG9JTF77V1ETmHR0MJXI72fEpruS-TiSE9F79VnOXIi9Bp-fM7HASbh=w640-h253" width="640" /></a></span></div><span style="font-family: arial;"><br />We decided to automate the deployment process. The procedure given to us by the vendor was entirely manual. While the deployment process was only 32 clock-time hours of the total, decreasing that time to 4 hours allowed much greater flexibility in scheduling updates in the test environment as well as production. Now, updating installations for the test environment could be done off-hours without putting the testing team out of service. Additionally, production updates could be deployed off-hours and no longer require the weekend.</span><p></p><p style="text-align: justify;"><span style="font-family: arial;"></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi1gcfDslC22IbEIZ9bOjsHsR_0Ma6KQpbHZ4C31QQKuFukTfZAoNoKHorq85E7qQyk6KTnD_kClGOrRGe7IElIu9YROcJGJYE5AC3nARW46FHVa-CCCwtz6-WDC-tVCkvb37a4fjAc-0j0DR3nVfRagx7_OEjhqgr_nGl8yEshEbZ7wfNuORrK5qsy" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="473" data-original-width="1158" height="262" src="https://blogger.googleusercontent.com/img/a/AVvXsEi1gcfDslC22IbEIZ9bOjsHsR_0Ma6KQpbHZ4C31QQKuFukTfZAoNoKHorq85E7qQyk6KTnD_kClGOrRGe7IElIu9YROcJGJYE5AC3nARW46FHVa-CCCwtz6-WDC-tVCkvb37a4fjAc-0j0DR3nVfRagx7_OEjhqgr_nGl8yEshEbZ7wfNuORrK5qsy=w640-h262" width="640" /></a></span></div><span style="font-family: arial;"><br /></span><h2><span style="font-family: arial;">Lessons Learned</span></h2><p></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automating testing would be the logical next tuning step.</i></b> That said, automating tests for a COTS application is easier said than done. This particular application did not lend itself to easy UI testing in an automated fashion. As we didn't have access to product source code, testing service APIs wasn't an option either.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>The value stream tuning effort illustrated here works for custom applications just as well as it did for this COTS example.</i></b> The value stream tactic applies the tuning principle of optimizing the largest targets (those that take the most time) first. This principle is used when we tune CPU or memory consumption in applications. In fact, the principal can be used for non-IT processes as well, like budgets.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Value stream analysis should be an ongoing effort repeated periodically.</i></b> Over time, your deployment process changes. When that happens, the value stream will also change.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>As with other types of tuning efforts, it's important to identify a specific target. </i></b>Like other tuning efforts, it's always possible to keep improving. Having a target allows you to know when the tuning effort is "done" and effort can be directed elsewhere.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Value stream analysis often reveals business procedures and processes that are not optimal.</i></b> In another example from the field, I've seen DNS entries and firewall rule entries that are forced by organizational procedures to be manual and take significant amounts of lead time. It's important to track these activities in the value stream process as well. You need accurate information to make effective tuning decisions.</span></p><p style="text-align: justify;"><span style="font-family: arial;">Thanks for taking the time to read my article. I'm always interested in comments and suggestions. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><br /></span></p><p style="text-align: justify;"><span style="font-family: arial;"><br /></span></p><p style="text-align: justify;"><span style="font-family: arial;"><br /></span></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-43307478699915860312022-12-18T12:57:00.002-08:002022-12-18T12:57:58.882-08:00The Journey toward Continuous Delivery and Deployment: How to StartI work with teams who are nowhere close to achieving continuous delivery, let alone continuous deployment. Those teams are getting further and further behind where teams in other companies are on similar journeys. Often, they don't have automated testing. If they do, often they are unit tests and not integration tests. Often, they don't have application infrastructure code and can't easily create additional environments. Often, they work in long-lived feature branches and have teams working on disparate versions of the code. Consequently, the speed of new feature delivery to end users is abysmal. Hope is not lost. The journey is difficult but possible. Here are some initial steps to take. <div><div><br /></div><div><i><b>Establish a basic integration test suite if you don't already have one.</b> </i>There is not going to be continuous anything without automated integration testing. With automated integration testing, you can have confidence that changes, whether they are big fixes or feature enhancements, don't accidentally introduce new defects. Often legacy code bases aren't written to be easily testable at a unit level. Concentrate on integration testing the application at its consumption points. For web UIs, that means automated functional testing of the UI or at least the REST web service resources. For APIs consumed by other applications, it means integration testing the service endpoints. </div><div><br /></div><div>Unit testing is always important, and I never discourage it. Integration testing is more important as it tests end-user functionality, not just small sections of code. If you have no automated tests, start with integration testing first. You can implement unit tests for new features along the way. </div><div><br /></div><div>Ideally, the environment for integration testing should be established by infrastructure code before the tests and eliminated after. That said, I'm concentrating on the initial steps in this post and don't want to deviate.</div><div><br /></div><div><b><i>Management support is needed for funding automated testing.</i></b> That funding is both labor and tooling. Initial setup for automated testing has an upfront cost. Remember that there is a broad range between 0% and 100% coverage. The higher the percentage, the better. That said, higher percentages have diminishing returns. Don't let perfection be the enemy of progress. </div><div><br /></div><div><b><i>Establish a continuous integration pipeline (CI) for the main/master source code branch of you don't already have one.</i></b> Continuous integration is a firm requirement to continuous delivery. The CI pipeline should run automatically on check-in of code changes. Continuous integration identifies defects immediately after changes are checked in by executing all available unit and integration tests. The objective is to identify defects as early in the development process as possible. </div><div><br /></div><div><b><i>If continuous integration (CI) reveals an error, fixing the error is the highest priority.</i></b> Many pundits would say that this type of breakage should result in an "all hands on deck" type of emergency and should enlist all members of the team to fix it. As a practical matter, the developer who checked in the change that broke CI is usually the best person to fix it. They know what they did and are closer to the problem. The developer's tech lead can follow up with the developer at some point to identify any additional resources needed to fix the issue. </div><div><br /></div><div><b><i>Adopt trunk-based development and eliminate long-lived feature branches.</i></b> Trunk-based code management is a requirement for continuous anything. Trunk-based development ensures that developers are using the newest and most current code base. This minimizes the chances of "merge hell" and keeps all team members up to date and working on the current code.</div></div><div><br /></div><div>Depending on your CI/CD tooling, it might be easier to use a short-term feature branch and initiate CI on merge. These feature branches must be short-lived or you're not really adopting trunk-based development. As long as the short-term branch gets deleted after the merge, this isn't a bad tactic. </div><div><br /></div><div><b><i>Only start changes that can be completed in four hours or less.</i></b> Two hours is better. Break the change up into smaller pieces if it's longer than that. This reduces merge issues later. This also reduces the chance that another team member introduces a conflicting change. This practice provides an incentive to keep changes small and only make one change at a time. This is in keeping with the objective of delivering new features to users faster. All good.</div><div><br /></div><div><b><i>Eliminate the practice of reviewing pull requests. </i></b> Allow automated testing to catch defects. If a junior developer checks in code that isn't optimal, a more senior member of the team can refractor that change later and hopefully take the opportunity to educate the junior team member on the issue. Either way, if it didn't break CI, not much damage was done. </div><div><br /></div><div><b><i>Encouraging automated test coverage will change developer behavior. </i></b>If developers "know" they will be expected to produce automated tests for changes they make, they will code to make testing easier. It's enlightened self-interest. That also makes the code base cleaner and increases its quality.</div><div><br /></div><div><b><i>Continuous delivery is an ongoing, neverending process.</i></b> The beginning tactics I list here are just a start. </div>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-86119834145333798292022-09-14T04:35:00.009-07:002022-09-14T06:31:04.218-07:00Appropriate workloads for Kubernetes<p style="text-align: justify;"><span style="font-family: arial;">I've been asked about hosting cloud applications in Kubernetes. People seem to assume that Kubernetes is the best practice for hosting containerized workloads in all cases. While I love Kubernetes and have used it successfully in numerous applications, It shouldn't be the default, out-of-hand, hosting solution for containerized workloads. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Kubernetes is far from the simplest solution for hosting containerized workloads. </i></b>Even with cloud vendors making Kubernetes clusters more integrated and easier to manage, they are still very complex and take highly specialized administration skillsets. If you're using Azure, <a href="https://docs.microsoft.com/en-us/azure/container-instances/container-instances-overview" target="_blank">Containerized Instances</a> is a comparatively more straightforward method than AKS for deploying containerized workloads. In fact, Azure has <a href="https://www.atmosera.com/blog/all-the-ways-to-run-containers-on-azure/" target="_blank">several different ways</a> to deploy containerized workloads. Most are easier than AKS/Kubernetes.</span></p><p style="text-align: justify;"><span style="font-family: arial;">If you're using AWS, <a href="https://aws.amazon.com/getting-started/hands-on/deploy-docker-containers/" target="_blank">ECS </a>or <a href="https://docs.aws.amazon.com/lambda/latest/dg/images-create.html" target="_blank">Lambda</a> is comparatively easier than Kubernetes. In fact, AWS has at least <a href="https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/#:~:text=There%20may%20be%2050%20ways,to%20run%20containers%20on%20AWS." target="_blank">17 ways to deploy containerized workloads</a>. Incidentally, with any cloud, it's possible to create a virtual machine and run a containerized workload on it: I don't recommend this. Bottom line: Kubernetes AWS ECS, and Azure Containerized Instances are application runners.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>If you adopt Kubernetes as a hosting mechanism, the benefits should pay for the additional complexity.</i></b> Otherwise, the additional maintenance headaches and costs of Kubernetes are not a wise investment. That begs the question, what types of applications are most appropriate for Kubernetes? Which application types should adopt a more straightforward hosting mechanism?</span></p><p style="text-align: justify;"><span style="font-family: arial;">As an aside, containerizing your applications is the mechanism that separates the concern of hosting from the functionality of your application. Cloud vendors have numerous ways to deploy and run containerized applications. Vendor lock-in usually isn't an issue with containerization.<br /></span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Workloads Well-Suited for Kubernetes</span></h2><div><span style="font-family: arial;">This section details common types of applications that may benefit from Kubernetes hosting.</span></div><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Applications with dozens or hundreds of cloud-native services often benefit from Kubernetes.</i></b> Kubernetes can handle autoscaling and availability concerns among the different services with less set-up per service. Additionally, cloud vendors have integrated their security and monitoring frameworks in a way that generally makes management of Kubernetes-hosted services homogenous with the rest of your cloud footprint.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Applications servicing multiple customers that require single-tenant deployments often benefit from Kubernetes.</i></b> For these applications, there is one deployment of a set of services per "customer". For example, if there are 1000 customers, there will be 1000 deployments of the same set of services for each one. Given that the number of deployments grows and shrinks as customers come and go, Kubernetes streamlines the setup for each as well as provides constructs to keep each of the customer deployments separate.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Applications requiring custom Domain Name Services (DNS) resolution that can't be delegated to the enterprise custom DNS often benefit from Kubernetes.</i></b> This is because Kubernetes has configurable internal networking capabilities. Often this happens more as a result of organizational structure than technical reasons. In many enterprises, DNS is managed by infrastructure teams and not application teams.</span></p><p style="text-align: justify;"><b style="font-family: arial;"><i>Applications in enterprises where IP address conservation is necessary can benefit from Kubernetes. </i></b><span style="font-family: arial;">For applications with internal services that only Kubernetes-hosted services needs access to, Kubernetes internal networking model called </span><span style="font-family: courier;">kubenet </span><span style="font-family: arial;">provides an internal network not visible outside the cluster. For example, an application with hundreds of microservices may only need to expose a small fraction of those services outside the cluster. The </span><span style="font-family: courier;">kubenet </span><span style="font-family: arial;">networking model conserves IP addresses as internal services don't need IP addressability outside the cluster.</span></p><p style="text-align: justify;"><span style="font-family: arial;">In the cloud, we think of IP addresses as "free" and without charge. For firms with a large existing on premises network, IP address space is often not free. In most firms I've seen, on premises networks are tarballs without sensible IP address schemes. Often, nobody understands the entirety of what network CIDRs are in use and for what. Additionally, routing is often manually configured, making CIDR block additions labor-intensive.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Summary</span></h2><p style="text-align: justify;"><span style="font-family: arial;">Not every application should be hosted in Kubernetes. Simple applications with small transaction volume often don't require the additional complexity Kubernetes brings. </span></p><p style="text-align: justify;"><span style="font-family: arial;">Thanks for taking the time to read this post. I hope this helps.</span></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-90582046676645961102022-09-04T10:17:00.001-07:002022-09-04T14:51:08.052-07:00Policy-based Management Challenges and Solutions<p style="text-align: justify;"><span style="font-family: arial;">One of the most common best practices for managing security in the cloud is <b>policy-based management</b>. Policy-based management optimally prevents security breaches or at least alerts you to their presence. Additionally, it alieviates the need for as many manual reviews and approvals, which slow down development of new business capabilities. That said, policy-based management presents many challenges. This post details common challenges and tactics to overcome them. </span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Challenge #1: Introducing New or Changed Policies</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>New or changes to existing policies often break existing infrastructure code (IaC) supporting existing applications.</i></b> This occurs because at the time the IaC was constructed, the policy wasn't in place and the actions were allowed. This results in unplanned work for application teams and schedule disruptions. As policy makers are usually separate teams, they often don't pay the cost associated with the associated unplanned work.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Policy change announcements are often ineffective.</i></b> Partially, this is due to volume of announcements in most organizations. The announcement of an individual policy gets lost in a sea of other announcements. Additionally, sometimes IaC developers do not completely understand or see the ramifications of the policy change. </span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Challenge #2: Policies with Automatic Remediation</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Installing policies that have automatic remediations in them can actually break existing infrastructure and the applications that rely on it.</i></b> While automatic remediation for policies is appealing from a security perspective as it fixes an issue in a short time after a security hole is created, it really just kicks the can. Any resulting breakage will need to be repaired sometimes causing an outage for end users.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>The IaC that produced the invalid infrastructure will no longer match the infrastructure that physically exists and needs to be changed.</i></b> In other words, the automatic remediation causes unplanned work for other teams. Sometimes, new policies cause common IaC modules used by multiple teams to no longer work and not individual application infrastructure code.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Challenge #3: Adapting Policies to Advances in Technology </span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Many policy makers only consider legacy mutable infrastructure.</i></b> </span><span style="font-family: arial;">Mutable infrastructure is common on premises and consists of static virtual machines/servers that are created once and updated with new application releases when needed. Immutable infrastructure VMs are completely disposable. The VMs are still updated, but by updating the images they are created from and replacing the VMs in their entirety.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;">For example, it is common to place a policy that requires that automatic security updates be applied to virtual machines on a regular basis. The issue is that such policies assume that the VM has a long life as it would under a mutable infrastructure. Such a policy doesn't apply to immutable infrastructures. For immutable infrastructure, the base image needs security updates applied and any VMs built using it should be rebuilt and redeployed.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Cloud vendor technology changes at a rapid pace.</i></b> Keeping cloud policies up to date with current advances is a challenge. In practice, policy makers are often out of date and make invalid assumptions. Effects of this I commonly see are:</span></div><div style="text-align: justify;"><ul><li><span style="font-family: arial;">Assuming that cloud vendor capabilities for securing network access remains the same. Often, these capabilities advance.</span></li><li><span style="font-family: arial;">Assuming VM IP addresses are static can safely be used in firewall rules. In the cloud, IP addresses can change quite frequently.</span></li><li><span style="font-family: arial;">Assuming that VM images are changeable (vended provided images might not be)</span></li><li><span style="font-family: arial;">Assuming that there will be no needed exceptions to security policies</span></li></ul></div><h2 style="text-align: justify;"><span style="font-family: arial;">Tactics to Mitigate Challenges</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Always audit compliance to policies first before installing automatic remediation.</i></b> That is alert teams of new compliance issues before changing anything automatically. This allows teams to accommodate a security policy change proactively before change is forced. Additionally, a reasonable lead time needs to be provided so that teams have the opportunity to mitigate the additional work.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Test new or changed policies with any related enterprise-wide common IaC modules.</i></b> It is common for organizations with mature DevOps capabilities to centralize common IaC modules and reuse them for multiple applications. This allows organizations to leverage existing work instead of having multiple teams reinvent the wheel. </span><span style="font-family: arial;">For example, if a policy regarding AWS S3 buckets or Azure storage accounts is being changed, test any common IaC modules that use those constructs.</span><span style="font-family: arial;"> Make p</span><span style="font-family: arial;">olicy compliance part of the test. Note that these tests should be automated so they can easily be rerun.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Any policies with automatic remediation must provide an exception capability.</i></b> For example, if some VM images are purchased and not changeable according to the license. It is common for such images to be granted exceptions from related security policies. Additionally, I've seen exceptions granted for cloud vendor-provided Kubernetes clusters where underlying VMs don't and can't meet policy requirements. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>New policies should be deployed in lower environments first.</i></b> This increases the chance that any errors or issues will be identified before the policy is applied to production. Be sure to allow a reasonable period of time in lower environments to increase the likelyhood that issues will be identified and addressed.</span></div><div style="text-align: justify;"><br></div><h2 style="text-align: justify;"><span style="font-family: arial;">Summary</span></h2><div style="text-align: justify;"><span style="font-family: arial;">Policy-based management has challenges, but should still be considered best practice. Thanks for taking time to read this article. Please contact me if you have questions, concerns, or are experiencing challenges I've not listed here.</span></div><p style="text-align: justify;"><span style="font-family: arial;"><br></span></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-38864820292170527622022-08-08T02:35:00.000-07:002022-08-08T02:35:03.003-07:00Move your Network to the Cloud Too!<div style="text-align: justify;"><span style="font-family: arial;">Over the past year, I'm seeing indications of what will be a big trend in cloud consumption: let's move our network to the cloud along with data centers. I'm talking about the WAN network primarily which many enterprises maintain worldwide. Local offices will still need connectivity to the WAN; it's just that they will increasingly become on-ramps to the worldwide WAN hosted in the cloud. In other words, data centers will no longer be the "center" for all network access. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Graphically, the concept of moving the WAN to the cloud would look like figure 1 below. Notice how all data centers and offices are connected to the WAN that handles traffic between them. While the image doesn't describe it, the Cloud-based WAN is worldwide and can serve offices and data centers across the globe.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: center;"><span style="font-family: arial;"><b><u>Figure 1: Cloud-based WAN Network</u></b></span></div><div style="text-align: justify;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWslOulC9UaAsogZLy9vF2BGMZMC8WdMjx4-OGOhaFTCa_YM7Lu97_XlPBBXN58AGLDhOfKaQmoUNEsen2XcTkMlMvNFvKVridZWkVUBJARWmZCwVzWvxfYHvWf_Nb0WnQLsawB6Gjpk6lu-IY64yzifOEUK_MO0jMvFCIZQD374jS_DmlQ2_lReNd/s5708/Cloud-Based-WAN.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3563" data-original-width="5708" height="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWslOulC9UaAsogZLy9vF2BGMZMC8WdMjx4-OGOhaFTCa_YM7Lu97_XlPBBXN58AGLDhOfKaQmoUNEsen2XcTkMlMvNFvKVridZWkVUBJARWmZCwVzWvxfYHvWf_Nb0WnQLsawB6Gjpk6lu-IY64yzifOEUK_MO0jMvFCIZQD374jS_DmlQ2_lReNd/w400-h250/Cloud-Based-WAN.jpg" width="400" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Let's contrast this with figure 2 which depicts the WAN network topology common in enterprises today. Note that public cloud access typically routes via data centers making enterprise application access data center centric. Worldwide connectivity is managed by a custom MPLS network.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: center;"><span style="font-family: arial;"><b><u>Figure 2: Traditional Worldwide MPLS Network</u></b></span></div><div style="text-align: justify;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgp1bYRlJ72DB_zd0Eq9qRjieF2QQJYM6Z7HY5haQUEmfCK-82mycAKez88juLIfySzsE2yGxz1UfTlqCwAYtLMU8ndukZ01FJoOFm__8c9_xmanvvKAmQ1pFohje7RXYhHgc117RWFTtXdR8OGnsTa6l_dMHCjhRxtwgzW6CWHaXod2I9xSc6VUZ07/s5708/MPLS-Network.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4292" data-original-width="5708" height="301" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgp1bYRlJ72DB_zd0Eq9qRjieF2QQJYM6Z7HY5haQUEmfCK-82mycAKez88juLIfySzsE2yGxz1UfTlqCwAYtLMU8ndukZ01FJoOFm__8c9_xmanvvKAmQ1pFohje7RXYhHgc117RWFTtXdR8OGnsTa6l_dMHCjhRxtwgzW6CWHaXod2I9xSc6VUZ07/w400-h301/MPLS-Network.jpg" width="400" /></a></div><br /><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">I'm seeing several motivations for the change in thinking about how worldwide networks should be organized. I'll separate the reasoning into the following categories:</span></div><div style="text-align: justify;"><ul><li><span style="font-family: arial;">Complexity</span></li><li><span style="font-family: arial;">Performance</span></li><li><span style="font-family: arial;">Financial</span></li><li><span style="font-family: arial;">Speed to Market</span></li></ul></div><h2 style="text-align: justify;"><span style="font-family: arial;">Complexity</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>The complexity of non-Cloud MPLS networks, the base for most enterprise worldwide WANs, is tremendous.</i></b> MPLS networks typically require large amounts of hardware that needs to be upgraded and replaced regularly. They take a large networking staff. While some outsource that to an MSP provider, they are still necessary. Outsourcing a large portion of the network to cloud vendors outsources this complexity and associated maintenance to a large degree. They also tend to be replete with numerous vendor contracts.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>The complexity increases the business risk of change.</i></b> MPLS networks are rarely supported by testing sandbox environments and automation. Many still make changes manually leading to inevitable human error and outages for users. Utilizing cloud vendors makes it much easier to automate the WAN infrastructure and provides a sandbox environment to test networking-related changes. This decreases the business risk of changes to networking infrastructure. This is huge. For most enterprises, the WAN that integrates all data centers and offices is essential.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Simpler capacity planning requirements. </i></b>Hardware and vendor contacts needed for worldwide MPLS networks require sophisticated capacity planning due to long lead time requirements. This requirement is much simpler with cloud WAN implementations. Capacity planning still exists, but it is far simpler and is easily changeable and adaptable on the fly. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Performance</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Network latency is generally significantly lower (faster) using cloud-provided WAN networking than worldwide MPLS networks.</i></b> While your mileage will vary depending on your MPLS implementation, so much R&D goes into cloud-provided WANs that the likelihood that an enterprise will keep up any network performance advantages over time is low. Face it, most firms just can't compete.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Network latency is higher (slower) accessing resources that require networking between on premises and the cloud.</i></b> As more IT workloads move from on premises to the cloud, closer proximity to the cloud will yield better performance. To this end, I see more enterprises leveraging cloud VPN services, which are closer to most application workloads, yielding better performance.</span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Financial</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Converting networking hardware and infrastructure from capital expense (CapEx) to operational expense (OpEx) is appealing to many enterprises from an accounting perspective. </i></b>As with computing resources, you pay for what you use for cloud-based WANs without hardware expenditures and management.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Networking labor is expensive specialized labor.</i></b> Outsourcing that labor to cloud providers is definitively cheaper. Some enterprises mitigate this cost by enlisting a managed services provider (MSP), but outsourcing that labor to cloud vendors is cheaper as it capitalizes on the cloud's economy of scale advantages.</span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Speed to Market</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>No more long lead times for MPLS network upgrades and capacity increases.</i></b> Increasing capacity in a cloud-provided WAN is typically measured in hours, not months. Furthermore, cloud-provided WAN products benefit from the cloud's dynamic scaling capabilities. Increasing MPLS network capacity takes sophisticated capacity planning and typically long lead times due to additional hardware expenditures.</span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Additional Benefits</span></h2><div style="text-align: justify;"><span style="font-family: arial;"><b><i>The firm gets access to research and development advances made by cloud providers. </i></b>The R&D resources that cloud providers are investing in WAN technologies surpass what most enterprises are able or willing to invest in. This means that over time, any differences in functionality and performance are likely to appear in cloud vendors first.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>A cloud-based WAN is a natural partner when combined with a cloud-based VPN capability.</i></b> This makes sense especially if the cloud hosts a larger percentage of application compute resources. Consuming the cloud-providers VPN solution moves those compute resources closer to what users access. With that closer proximity, typically comes better performance.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>A cloud-based WAN is a natural partner for integrating multiple cloud providers. </i></b>That is, Your AWS footprint can be securely connected to your Azure or GCP footprint directly. This avoids the slower connection between the cloud providers through an on premises data center.</span></div><h2 style="text-align: justify;"><span style="font-family: arial;">Concluding Remarks</span></h2><div style="text-align: justify;"><span style="font-family: arial;">I'm reporting what I'm seeing at clients. This idea made no sense when many had a small fraction of their IT footprint in the cloud. Now that most firms now have most of their footprint in the cloud, thinking on how to provide worldwide access to internal users needs to evolve. And the time for that evolution has come.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">If you have thoughts or feedback, please contact me directly via <a href="https://www.linkedin.com/in/derekashmore/" target="_blank">LinkedIn </a>or <a href="mailto:derek.ashmore@asperitas.consulting">Email</a>. thanks for taking the time to read this article. </span></div><div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><br /></div></div></div>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-84389681205808300022022-08-03T08:34:00.008-07:002022-08-03T08:53:47.372-07:00Radical Idea: Let's do more Testing in Production<div style="text-align: justify;"><span style="font-family: arial;">There are many different types of application testing. This article is entirely about system-level testing, the most outer-level user experience testing. System-level testing is also the most difficult to automate.</span></div><div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div><div style="text-align: justify;"><span style="font-family: arial;">I'm talking about integration or testing of the application from an end-user perspective only. Many use the term <i>system-level testing</i> for this activity. Other types of testing such as unit, performance, exploratory, and usability, are not a part of this article. Other forms of testing are essential, but not the focus here.</span><span style="font-family: arial;"> </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>System-level automated testing has too much friction.</i></b> It can't keep up. It's the most challenging type of testing to automate. Because of that, many still perform system-level testing manually. The cost-benefit of automating these types of tests is elusive. This test automation certainly can't support high-performing DevOps teams' high-frequency change rates. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>System-level automated tests are fragile.</i></b> The slightest change or refactoring at the outer web layer breaks a large percentage of system-level tests. Automated testing at this level usually relies on labels programmers use for parameters and control identifiers. Programmers usually consider them free to refactor these labels for clarity without notice.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>The lack of automated system-level testing impedes the firm's ability to implement continuous delivery.</i></b> In turn, manual system-level testing lowers the lead-time, which is one of the <a href="https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance" target="_blank">DORA metrics</a>, we seem to be using these days.</span></div><h2 style="text-align: justify;"><span style="font-family: arial;">What's the Alternative?</span></h2><div style="text-align: justify;"><span style="font-family: arial;">Let's outsource system-level testing to end users. Rather, let's enlist a small percentage of end users to use a release candidate in production and measure their error rate. Additionally, those errors can be provided to the development team for remediation.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Instead of writing system-level tests, implement <a href="https://martinfowler.com/bliki/CanaryRelease.html" target="_blank">canary deployments</a> and provide a release candidate version that is considered production and uses production databases and resources.</i></b> The release candidate is production in every way, except that it's used by a small percentage of users. If the application is hosted in the cloud, it's possible to create a "sister" installation of an application in production that uses production resources in the same way the active version of the application does.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Remediate the release candidate until the error profile is acceptable for mainstream release.</i></b> In other words, fail forward, don't roll back when errors are discovered. At some point, the release candidate will be considered stable and is made active for 100% of users. At this point, a new release candidate is created for new features and changes to be tested in the same way. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>This solution avoids the problem of automating system-level tests and all its problems in terms of friction and fragility.</i></b> Sometimes, the winning move is not to play! What I propose doesn't skip testing. It just changes the paradigm under which that testing is conducted.</span></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>The testing that end users do is going to be more comprehensive than any test plan can provide.</i></b> Moreover, testing by end users will concentrate on the most frequently performed tasks. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>There are diminishing returns to increasing the number of users directed to the release candidate. </i></b>That is, you will discover more defects increasing the number of users on the release candidate from 0% to 2% than you will from 25% to 50%. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>If you monitor error rates on the application, automation can be built to support continuous delivery.</i></b> In other words, if the release candidate reveals no increase in error rates over the current live version, automation can make the switch based on thresholds you configure.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">This concept sounds scary, but is it functionally different from what we experience today? We all see defects deployed to production despite our best testing efforts. I'm just suggesting we use what we experience rather than pretend we can avoid it. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Thanks for taking the time to read this article. I'm always eager for feedback.</span></div><div style="text-align: justify;"><br /></div><div><br /></div><div><br /></div></div></div>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-49617864764979737012022-03-08T03:07:00.004-08:002022-03-08T03:07:54.765-08:00Infrastructure Code and the Shifting Sand Problem<div style="text-align: justify;"><span style="font-family: arial;">Infrastructure code (IaC) has a problem that application code does not: changes outside of infrastructure code impact its ability to function as intended. I call this the <b><i>shifting sand problem</i></b>. The goal is for infrastructure code executions (of the same version) to always produce the same result. In this sense, we want infrastructure code executions to be "repeatable" in the same way we strive to make application builds repeatable. When IaC executions aren't repeatable, unplanned work is the result. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">There are many sources of IaC shifting sand. I'll detail those sources and ways to mitigate the problem in this post. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Common IaC shifting sand sources are below. They are caused by a mixture of technology change and organization management procedures. </span></div><div style="text-align: justify;"><ul><li><span style="font-family: arial;">Automatic dependency updates</span></li><li><span style="font-family: arial;">Cloud backend updates</span></li><li><span style="font-family: arial;">Right-hand/Left-hand issues</span></li><li><span style="font-family: arial;">Managing cloud assets from multiple sources</span></li></ul></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Configuring automatic dependency updates is the practice of using the latest version of IaC software or common IaC code revisions.</i></b> Examples include Terraform and Cloud provider versions, common IaC code versions, virtual machine image versions, operating system updates, and many more. Most automatic updates are put in place as a convenience: developers want to avoid the additional work of upgrading the versions used. The problem is that breaking changes from these dependencies will cause IaC code not to function. Then unplanned work results and somebody will need to fix the issue, often at an inconvenient time with a looming deadline.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Some operating system and virtual machine image updates are security-related. Examples include anti-virus software updates, etc. This type of update is often unavoidable. Rather, the potential cost of delaying these updates can be considerable. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">For avoidable automatic updates (not security-related), the best mitigation is to explicitly specify the versions of dependencies used. Never use 'latest' or the newest available. Let upgrades of dependencies be driven by changes needed for planned enhancements for end-users. Then the upgrade work is planned and scheduled as opposed to inconveniently timed.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">For unavoidable automatic updates (e.g. security-related), early detection by automated testing is best. In addition, apply these updates in lower-level environments on a regular basis. Forcing such updates in lower-level environments first will increase the chance that issues not covered by automatic testing can be found before production. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Cloud backend updates are breaking changes introduced by cloud vendor software.</i></b> While Cloud vendors do attempt to make their changes backward compatible, they don't always succeed. Without naming names, I've been in support calls with Cloud vendor technical support teams and seen them blackout product changes or frantically release product fixes. In addition, I've frequently had to frantically upgrade IaC code to accommodate Cloud vendor backend changes. As with the automatic update problem, unplanned (and inconveniently timed) work results.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Scheduled testing in a sandbox environment is the best mitigation strategy we've found. This strategy increases the chance of detection early before changes are actually needed in real environments. Depending on the components used, sandbox testing can be expensive to run. The more often sandbox testing is run, the earlier problems like this will be detected. Unfortunately, increased run frequency drives up costs. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Right-hand/left-hand issues are created within the organization itself where one department makes changes that have ramifications they don't see in other departments. </i></b>One example I've seen frequently is a group in charge of making security policy changes effectively "breaks" IaC code maintained by other departments. In essence, tactics taken by that IaC code were no longer allowed. In this example, making the policy change is often necessary. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Early detection through scheduled sandbox testing (as described above) is the best mitigation strategy for the team maintaining IaC code for specific applications. The same tradeoff between frequency and cost applies.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Managing cloud assets from multiple sources occurs when something besides one IaC pipeline manages a cloud asset. </i></b>The most frequent example is manual change. When developers manually change assets that are managed by IaC, often drift results. That is, the cloud asset, in reality, differs from what is in the IaC pipeline. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">Another example is creating multiple IaC code bases to manage the same asset. I've seen this frequently in cloud asset tagging, which is frequently used for accounting charge-back purposes. Drift always results as multiple IaC code bases rarely come up with the same answer. As a result, all IaC code bases (except the one last executed) differ from reality. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;"><b><i>Bottom line: ensure that one and only one IaC code base manages each cloud asset.</i></b> This prevents configuration drift. It saves aggregation and labor in staff who are often left with the mystery of figuring out how that drift happened. This is a topic I might explore more completely in another article.</span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><span style="font-family: arial;">I hope this helps. I'm always interested in other types of problems you encounter maintaining IaC code. Thanks for taking the time to read this article. </span></div><div style="text-align: justify;"><span style="font-family: arial;"><br /></span></div><div style="text-align: justify;"><br /></div>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-78722576105520512222021-01-09T04:39:00.008-08:002021-01-28T12:01:53.797-08:00For DevOps Professionals: Evolutionary Terraform<p style="text-align: justify;"><span style="font-family: arial;">Organizations that use Terraform to manage cloud infrastructure often create and maintain Terraform modules as the code base grows. Inevitably, complexity increases with the introduction of reusable code. DevOps teams, I've worked with struggle with the level of modularization they should use and how to more easily manage it. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>I think of the modularization of Terraform as an evolutionary process.</i></b> The level of modularization needed when organizations first start out is different from what they need as they mature. This article will take you through a sensible evolutionary path that only increases code complexity when truly needed.</span></p><p style="text-align: justify;"><span style="font-family: arial;">Just to clarify my terminology, a <b><u>configuration </u></b>is a Terraform project that is used to directly manage cloud infrastructure. That is, create a virtual network and subnets for a specific development environment. A <b><u>module </u></b>is a Terraform project that is designed for reuse and is used by configurations. For instance, I usually have a module that creates a configured virtual network, all component subnets. This functionality is used for multiple virtual networks in multiple environments.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">In the Beginning</span></h2><div><span style="font-family: arial;">When new technologies are adopted, simplicity is and should always be the goal. Only accept complexity that is necessary and only when it becomes necessary. Terraform is no exception. Let's discuss some opening tactics.</span></div><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Use source control for all Terraform code.</i></b> This is easy and it should be used from the beginning. Repositories are easy and inexpensive these days.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Centrally manage Terraform state (e. g. back-end state).</i></b> By default, Terraform will store the Terraform state on the device where the configuration is executed. All cloud platform Terraform providers provide a way to store state in the cloud instead of on the device doing the execution. This is generally easy to set up and reduces the risk of loss of the current Terraform state. Here are setup instructions for <a href="https://www.terraform.io/docs/language/settings/backends/s3.html" target="_blank">AWS</a>, <a href="https://www.terraform.io/docs/language/settings/backends/azurerm.html" target="_blank">Azure</a>, and <a href="https://www.terraform.io/docs/language/settings/backends/gcs.html" target="_blank">GCP</a>.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Adopt a standard Terraform project structure that incorporates configurations and modules.</i></b> A typical directory structure for a Terraform repository looks like the following:</span></p><p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBM4RFDTJ1Z_iXasf9tZdQzrWEpQW8hT3SCSv54HmiCFyszBLlw4XIQceo_MM6zHS3QoHjvrtYa0oinhGrMpPzgPnpi2vHHl1K4qQK-gPp66DEbRJDyhD4-sb-epMMowrJsYxAWdJQcaQ/s706/Terraform-project-structure.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="164" data-original-width="706" height="93" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBM4RFDTJ1Z_iXasf9tZdQzrWEpQW8hT3SCSv54HmiCFyszBLlw4XIQceo_MM6zHS3QoHjvrtYa0oinhGrMpPzgPnpi2vHHl1K4qQK-gPp66DEbRJDyhD4-sb-epMMowrJsYxAWdJQcaQ/w400-h93/Terraform-project-structure.JPG" width="400" /></a></div><span style="font-family: arial;"><b><i><div><span style="font-family: arial;"><b><i><br /></i></b></span></div>Note that a standard project structure separates configurations from modules.</i></b> This makes reusable code easy for developers to identify. How configurations should be structured would be a great topic for another article. Briefly, I generally separate network infrastructure, common services for all applications, and application infrastructure into separate configurations to make the blast radius more manageable.</span><p></p><p style="text-align: justify;"><b><i><span style="font-family: arial;">Note that only configurations have environment </span><span style="font-family: courier;">tfvars </span></i></b><span style="font-family: arial;"><b><i>files. </i></b>As configurations are used to directly manage infrastructure, modules are more focused and do not need to be coupled with the concept of different environments. An illustration of where to place </span><span style="font-family: courier;">tfvars </span><span style="font-family: arial;">files follows:</span></p><p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7Q8pnBlvCOvSI21VBCnIkQRkdrPDzoEdQDly428kgx2Cja-YOwxmE6lHHWaHf9Q4Evg8DudkDYTlUDlgcw2gYzzNEdKr4asVGB5Aj6fiZJC41IWT8lfrX0KKGNn4CJoXvxi4Mf7X__vM/s255/Terraform-project-structure-with-tfvars.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="164" data-original-width="255" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7Q8pnBlvCOvSI21VBCnIkQRkdrPDzoEdQDly428kgx2Cja-YOwxmE6lHHWaHf9Q4Evg8DudkDYTlUDlgcw2gYzzNEdKr4asVGB5Aj6fiZJC41IWT8lfrX0KKGNn4CJoXvxi4Mf7X__vM/s0/Terraform-project-structure-with-tfvars.JPG" /></a></div><br /><p></p><h2 style="text-align: justify;"><span style="font-family: arial;">When the Number of Coders Grows</span></h2><div><span style="font-family: arial;">When the number of DevOps professionals on the team grows, it's common for changes from one person to accidentally conflict with changes others are making. This lengthens the time associated with changes and slows the team's velocity. </span></div><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Use feature branches to organize changes. </i></b>This allows developers to test their changes with less fear that another developer will accidentally interfere. I've addressed feature branch usage in detail in <a href="https://www.derekashmore.com/2020/10/best-practices-for-managing-feature.html" target="_blank">this post</a>.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Test feature branch changes in a sandbox environment.</i></b> A <b><u>sandbox </u></b>environment to me is an environment that can easily be destroyed and recreated if something goes awry. Do not run feature branches in any non-sandbox environment. This allows developers to test new code in isolation without fear of accidentally negatively impacting others.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Only apply changes from a CI/CD pipeline.</i></b> This provides an execution history. If something unexpected happens, execution history can provide information as to what was run when. It also removes any differences between the environments and access executing from individual devices.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Schedule CI/CD pipeline plan or validate operations for each configuration.</i></b> This will allow you to detect configuration drift. It also ensures that all configurations are at least correct as far as syntax and that a configuration hasn't been affected by a breaking change in one of the modules. </span></p><p style="text-align: justify;"><span style="font-family: arial;">Some organizations use <a href="https://terratest.gruntwork.io/" target="_blank">TerraTest </a>to automatically test Terraform configurations. While I support automated testing if you can do it, TerraTest requires GoLang knowledge that not all organizations have. Mandating TerraTest can be a big ask.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">When the Number of Configurations Grows</span></h2><p style="text-align: justify;"><span style="font-family: arial;">As the number of Terraform configurations grows, typically the blast radius for changes to modules also grows. The reason is that module usage also grows. With a small number of configurations, it's easier to test each configuration that uses the module that is being changed. The test effort grows with a growing number of configurations. Either velocity slows to accommodate the larger blast radius, or testing isn't as thorough, and accidental defects are released.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Ensure that you adopt module coding best practices.</i></b> This is a large topic and deserves its own article, but I summarize some key points in the <b>Module Coding Best Practices</b> section below. As the number of configurations grows, the opportunity for reuse increases, and the number of modules also grows.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Separate out all modules into a separate repository and formally release by version/tag.</i></b> This allows consuming configurations to insulate themselves from module changes. If configurations consume the latest release, they run the risk of not working if a breaking change was made to the modules they consume. In essence, consuming specific versions/tags converts "unplanned" work to "planned" work. Module upgrades can be scheduled with time allowed for it if needed. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Note that versioning modules reduce the risk of change for modules.</i></b> I've seen slightly different versions of modules that do much the same thing occur because people fear accidentally breaking configurations they know nothing about. Versioning eliminates this risk as the modified code will be published with a new version.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Once a version/tag is released, never change its content.</i></b> In this world, there should be no concept of forcing configurations to accept changes. Consuming configurations should always control and be able to plan for module upgrades. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><i><b>Only consume modules explicitly specifying a tag/version.</b> </i>Consuming the "latest" version increases the risk of unplanned work as discussed previously.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Module Coding Best Practices</span></h2><p style="text-align: justify;"><span style="font-family: arial;">These practices deserve their own article, but to summarize:</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Only create a module that has at least two consuming configurations.</i></b> Creating a module for use by only one configuration is classic YAGNI. It introduces complexity that isn't yet necessary.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Avoid data lookups in modules.</i></b> Pass needed information as input variables. The reason is subtle. Data lookups will error if the target is not found. As we're talking about modules, they don't (and shouldn't) understand configuration context. If the target of the lookup doesn't exist, the first plan for a configuration using the module will error out. Using data lookups in configurations are perfectly fine as they understand execution context. </span><span style="font-family: arial;">This is subtle.</span></p><p style="text-align: justify;"><span style="font-family: arial;">As an example, let's say the module </span><span style="font-family: courier;">virtual-machine</span><span style="font-family: arial;">, used by configuration </span><span style="font-family: courier;">app-fred</span><span style="font-family: arial;">, executes a data lookup for a specific subnet. Let's also say that configuration </span><span style="font-family: courier;">app-fred</span><span style="font-family: arial;"> <i><u>creates</u> </i>that subnet. Configuration </span><span style="font-family: courier;">app-fred</span><span style="font-family: arial;"> will not successfully plan because the subnet module virtual-machine is looking for doesn't exist yet on the first run. Bottom line - modules should not do data lookups because they don't (and shouldn't) understand the execution context.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Ensure that all modules are documented in Markdown with a README.</i></b> I usually include an example usage section with common input options. The objective is to make it quick and easy for developers to use the module. In my own README documentation for modules, I include the following sections:</span></p><p style="text-align: justify;"></p><p style="text-align: justify;"></p><p></p><p></p><p style="text-align: justify;"><span style="font-family: arial;"></span></p><ul><li><span style="font-family: arial;">A list of input variables and brief description if needed</span></li><li><span style="font-family: arial;">A common usage example that consumers can copy/paste/change to their own configurations.</span></li></ul><h2 style="text-align: justify;"><span style="font-family: arial;">Parting Ideas</span></h2><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Only assume complexity needed.</i></b> The later stages of evolution described here are not needed in the beginning. Avoid classic YAGNI (You Ain't Going to Need It).</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>You don't get away from change management.</i></b> While the practices described here reduce friction as your Terraform usage grows, change management is still needed. Somebody or group still needs to organize changes in a way that recognizes and accommodates dependencies.</span></p><p style="text-align: justify;"><span style="font-family: arial;">Thanks for reading this article. As always, please contact me or comment if you've alternative thoughts.</span></p><p style="text-align: justify;"><br /></p><p style="text-align: justify;"><span style="font-family: arial;"> </span></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-86213905862000034642020-12-26T08:58:00.020-08:002021-01-13T04:24:24.297-08:00For DevOps Professionals: Barriers to 100% Infrastructure as Code<p style="text-align: justify;"><span style="font-family: arial;">I was asked the other day why a particular part of the cloud infrastructure was added manually and not automated. It was a very small manual part and a one-time setup, but none-the-less I experienced déjà vu. It occurred to me that I've been asked that question at every client I've had since I got heavily into infrastructure code. We use the phrase "100% infrastructure as code" often. In fact, the overwhelmingly vast majority of cloud infrastructure is implemented via code. H</span><span style="font-family: arial;">owever, there is always some very tiny portion of the infrastructure that seems to be provided manually. The percentage is probably closer to 99.x% in most organizations I've had the privilege to do work for. Why is that? Why does the percentage never seem to be 100%? Let's make this more concrete and list some examples of automation I've encountered that wasn't 100% automated and why.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Examples of Automation Barriers</span></h2><p style="text-align: justify;"><span style="font-family: arial;"><b><i>High-bandwidth from the cloud to on premises data centers are rarely 100% automated.</i></b> This is the case for both Azure Express Routes and AWS Direct Connect connections. The reason is that a 3rd party firm controls access to the on-ramp or colocation device (e.g. CoreSite, Equinix, etc.). If the organization has access, it's usually manually controlled by a separate network infrastructure team. In other words, these devices aren't completely available to automation engineers that are needed for the development of infrastructure code. In essence, the cloud connectivity to the express route circuit can be automated, but that circuit's connectivity to the on-ramp is usually not.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automating DNS entries are problematic in many hybrid-cloud organizations.</i></b> This is an organizational barrier and not a technical barrier. It is common for DNS entries to be controlled by a separate team in a manual fashion. DNS authority is tightly controlled as there is effectively one DNS environment for the entire organization most of the time. The fear is that automation defects could negatively affect non-cloud entries or resources. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automating security policies and the assignment of those policies is problematic in many organizations.</i></b> Typically, security is handled by a separate team and usually a team without infrastructure automation skills. Consequently, I've seen automation engineers write code to establish security policies, but those policies are manually assigned by a separate team. In essence, the traditional test-edit cycle that automation engineers need for this type of development doesn't exist.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Frequently, the creation of AWS accounts or Azure subscriptions is not automated.</i></b> The reason is that in most organizations, that creation and their placement in the organizational tree is controlled by a separate team without automation coding skills. Furthermore, a sandbox environment for this type of automation code development doesn't exist.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Organizations define some resources to be central to the entire enterprise and don't have environment segregation.</i></b> Examples are Active Directory environments, DNS, and WANs. The problem with this is that changes to central resources such as this become "production" changes and are tightly controlled. When everything is a production change, the test-edit development cycle automation engineers need doesn't exist. </span></p><p style="text-align: justify;"><span style="font-family: arial;">After doing some introspection with these examples, I've identified several common barriers to implementing 100% infrastructure as code. It turns out that most of these limitations are not technology-based.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Environment limitations</span></h2><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Infrastructure code development requires support for test-edit loops.</i></b></span><span style="font-family: arial;"> That is, automation engineers need to be able to run, edit, and re-run infrastructure code to correct bugs. </span><span style="font-family: arial;">Writing infrastructure as code is just like application development in many ways.</span><span style="font-family: arial;"> Automation engineers need to be able to experience occasional failures without negatively impacting others. These requirements are usually accomplished by a "sandbox" environment that others are not using.</span></p><p style="text-align: justify;"><span style="font-family: arial;">The app developer part of me wants those tests and the verification of the result automated just like other types of application code. That said, the tooling to support automated testing of infrastructure code is sketchy at best and definitely not comprehensive. Automated testing is worth doing, but it is definitely not comprehensive. There are definitely limits to automated test coverage for infrastructure code.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>The sandbox environment used for infrastructure code development must support the add/change/delete of that environment without negative impact on others.</i></b> Like other types of development, infrastructure code doesn't always work as intended the first time it runs. In fact, you should assume that infrastructure code development might actually damage the sandbox environment in some way.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Sandbox environments should be viewed as completely disposable.</i></b> That is, they can be created or destroyed as needs require little effort. It needs to be easy for an automation engineer to create a new sandbox for infrastructure code development and destroy it afterward.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>It's common for sandbox environments to have limitations.</i></b> That is, it's difficult for sandbox environments to accommodate 100% of infrastructure code development. Dependency requirements (e.g. Active Directory, DNS, connectivity to SAS, or on premises environments) are primary examples. These limitations contribute to the small portion of the infrastructure that is at least partially maintained manually.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Organizational authority</span></h2><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automation engineers and the service accounts used for automation must have 100% control of the infrastructure maintained by code. </i></b>That is, one can't develop infrastructure code without the authority to do so. This code can't be completely developed and tested. Consequently, the portion of the infrastructure that directly interfaces such resources is often manual.</span></p><p style="text-align: justify;"><span style="font-family: arial;">Earlier in the post, I provided several examples that fit this category. For example, organizations using proprietary DNS products (e.g. Infoblox) often don't want to pay for additional licenses to support infrastructure code testing. Additionally, as DNS is often implemented in a one-environment paradigm (only production without separate development environments), organizations are hesitant to allow automation engineers security credentials needed to support infrastructure code support.</span></p><p style="text-align: justify;"><span style="font-family: arial;">Active Directory (A/D) environments also fit into this category in many organizations. As A/D is often used to grant security privileges, organizations are loath to grant automation engineers and automation service accounts needed privileges to create groups, edit group membership, and delete groups.</span></p><p style="text-align: justify;"><span style="font-family: arial;">All too often, the solution to these types of issues is to do a portion of infrastructure manually.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Low benefit/cost ratio</span></h2><p style="text-align: justify;"><span style="font-family: arial;"><b><i>For some types of infrastructure, organizations find the benefits obtained by a complete automated solution aren't worth the costs.</i></b> In other words, third-party costs (e. g. software licensing) make the "juice isn't worth the squeeze". Some infrastructure dependencies cost too much in money, labor, or time to dependency set-up costs to make the automation practical. Sometimes the manual labor involved in maintaining some infrastructure items is very small, making the cost of infrastructure code for those items not worth the effort.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Resources that are rarely updated and take an extremely long time to create/destroy often aren't worth the cost of automation.</i></b> As an example, the AWS Transit gateway is often an example. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Lack of DevOps team discipline can increase the cost of infrastructure automation and lower the benefit/cost ratio.</i></b> Without the good discipline to the development life-cycle for infrastructure code and good source control habits, it's common for development work by one automation engineer to negatively impact the work of others. This leads to an increase in manual work or a decrease in team velocity. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>The breadth of specialized skills needed for some types of infrastructure can lower the benefit/cost ratio.</i></b> As an example, work with one client required specialized networking and A/D skills to set up a test RRAS VPN target. If I didn't have a team member with these skills, I could never have tested that the cloud-side VPN infrastructure code worked - it would have been untested until use in one of the non-sandbox environments. I've seen other examples with regard to relational database administration skills and other types of specialized labor. The breadth of knowledge often needed by automation engineers is daunting.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Concluding Message</span></h2><p style="text-align: justify;"><span style="font-family: arial;">My acknowledgment that there are barriers to implementing 100% infrastructure as code should <b><i><u>not </u></i></b>be used as an excuse not to automate. Infrastructure code has produced some of the best productivity gains since we embarked on adopting cloud technologies. I'll never give up pressing for higher levels of infrastructure code automation. That said, when I recognize some of these non-technology barriers to infrastructure code, I'll feel a little less guilty. Yes, I'll try to craft workarounds, but recognize that it isn't always possible in every organization.</span></p><p style="text-align: justify;"><span style="font-family: arial;">Thanks for reading this post. I hope you find it useful.</span></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-46965723982219656392020-12-16T15:01:00.010-08:002021-01-29T10:01:58.396-08:00For Managers: Cloud Governance through Automation<p style="text-align: justify;"><span style="font-family: arial;">Cloud consumption and DevOps automation is not just a technology change. It is a paradigm shift that managers participate in as well, but don't always realize it. One of the paradigm shifts involves cloud governance. If managers apply governance tactics developed over the years, they risk many of the benefits obtained by cloud consumption including speed to market. Having seen this transformation at several organizations, I've some thoughts on the topic. Please take time to comment if you've thoughts that I haven't reflected here.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Place automated guardrails on cloud usage instead of manual review processes. </i></b>In short, when new policies are needed or existing policies modified, work with a cloud engineering team instead of adding manual review points. The benefits are:</span></p><p style="text-align: justify;"></p><ul><li><span style="font-family: arial;">Fewer review meetings</span></li><li><span style="font-family: arial;">Reduced manual labor with both management oversight and application team compliance</span></li><li><span style="font-family: arial;">Added security as enforcement is more consistent and comprehensive</span></li><li><span style="font-family: arial;">Evolves as your cloud usage grows and changes</span></li><li><span style="font-family: arial;">Allows decentralized management of cloud resources which frees application teams to innovate more.</span></li></ul><p></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>This is a paradigm shift over what was needed in data centers.</i></b> </span><span style="font-family: arial;">Hardware infrastructure found on premises makes governance and its enforcement manual. This leads to long lead times to acquire and configure additional infrastructure and makes governance a constraint to bringing additional technical capabilities to application teams and users. Manual approvals and reviews are needed costing time and management labor.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>In the cloud, infrastructure automation is possible because everything is now software.</i></b> Networking, infrastructure build-outs, security privileges/policies, and much more are now completely software configuration and don't involve hardware. </span><span style="font-family: arial;">The software nature of the cloud makes the automation of governance in the cloud possible. Once automated, governance is no longer manual. Governance is enforced automatically that will provide enterprise safety. As a consequence, the need for manual approvals decreases if not entirely eliminated. This frees application development teams to innovate at a faster pace.</span></p><h3 style="text-align: justify;"><span style="font-family: arial;">What types of automated guardrails are possible?</span></h3><div><span style="font-family: arial;">As the cloud is entirely software, the sky is the limit. That said, there are several guardrails that I see as implementation candidates.</span></div><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Whitelist cloud services application teams can use.</i></b> As an example, some organizations have legal requirements, such as HIPPA or FERPA, that need to be adhered to. These organizations usually have a need to whitelist services that are HIPPA or FERPA compliant. As another example, some organizations standardize on third-party CDN or security products. They commonly want to prohibit cloud-vendor based solutions that aren't a part of the standard solution.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Whitelist cloud geographic regions application teams can use.</i></b> Some organizations don't operate world-wide and want cloud assets existing only in specific regions.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automatically remediate or alert for security issues. </i></b>Most organizations have specific plans for publishing cloud assets on the internet. As an example, one of my clients automatically removes non-authorized published ports to all internet addresses (CIDR 0.0.0.0/0) within a few seconds after such a port is opened. Another example, a customer of mine provides alerts when people are provided security privileges in addition to non-security administration privileges.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automatically report and alert on underutilized cloud resources.</i></b> Underutilized resources often cost money to no benefit. These resources are generally computing resources such as virtual machines. Alerts like these provide ways to lower cloud spend as it's often possible to downsize the compute resources.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Automatically report and alert for unexpected cost increases.</i></b> Alerts like these need sensible thresholds. This alert usually prompts a review and possible remediation of the application causing the cost increase. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Schedule uptime for non-production resources to save money.</i></b> Often, organizations don't schedule downtime for non-production environments off-hours. Enterprises operating worldwide might not have this option as effectively there aren't "off-hours".</span></p><h3 style="text-align: justify;"><span style="font-family: arial;">How can automated guardrails avoid becoming a bottleneck?</span></h3><p style="text-align: justify;"><span style="font-family: arial;">Application teams do not like constraints of any type. Having been on application teams for many years, I understand their sentiment. There are ways to keep guardrail development from becoming a bottleneck.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Fund automated guardrail development and maintenance. </i></b>Like any other software produced by the enterprise, automated guardrails need development and support resources. Without adequate funding, they won't react to changing needs on a timely basis. Additionally, recognize that inadequate funding for automated guardrails will result in productivity losses for individual application teams across the enterprise.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Work with application development teams to identify and prioritize needed enhancements.</i></b> This provides visibility into the guardrail backlog. Additionally, application teams can participate in prioritizing enhancements. Make them part of the process.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><span style="text-align: left;"><b><i>As cloud platforms evolve and change, automated guardrail development and maintenance is an activity that never "ends". </i></b></span>Cloud governance is a continually evolving feedback loop. There must be a reasonable process for application teams to propose modifications to existing guardrails. As cloud technology changes over time, advances are made in current cloud services and new services invented. as an example, one of my clients must restrict cloud services used to those that are HIPPA compliant. As advances are made, that list grows over time and needs to be revisited.</span></p><h3 style="text-align: justify;"><span style="font-family: arial;">As a manager in charge of cloud governance, what does this change mean to me?</span></h3><p style="text-align: justify;"><span style="font-family: arial;"><b style="font-style: italic;">Declare "war" on manual approvals. </b>Instead of adding manual review/approval processes to govern cloud usage, engage a DevOps or cloud engineering team to enforce your desired behavior. A colleague of mine calls these "meat-gates". They slow everything down, both for management and application teams. They hamper delivering new features to end-users by slowing down application teams. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>DevOps automation is your friend and ally.</i></b> It allows you to set policy and not need to devote as much to enforcement. You specify "what" policies you want to be enforced. DevOps automation engineers construct and maintain the enforcement of the policies you choose. </span></p><h3 style="text-align: justify;"><span style="font-family: arial;">Conclusion</span></h3><p style="text-align: justify;"><span style="font-family: arial;">I hope you find these thoughts useful. I'm always open to additional thoughts. Thanks for reading this post and taking the time to comment.</span></p><p style="text-align: justify;"><br /></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-13721262243715416792020-11-14T12:00:00.002-08:002020-11-14T12:00:58.598-08:00When to execute ARM Templates with Terraform<p style="text-align: justify;"> </p><p style="text-align: justify;"><span style="font-family: arial; text-align: left;">ARM templates are the native automation mechanism for the Azure cloud platform. It is possible to execute ARM templates from Terraform using resource </span><span style="text-align: left;"><span style="font-family: courier;"><a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group_template_deployment" target="_blank">azurerm_resource_group_template_deployment</a></span></span><span style="font-family: arial; text-align: left;">. To Azure professionals with less Terraform experience, this is appealing. It allows them to use their existing skills and provides some short-term productivity gains. While I see the benefit, the tactic eliminates some of the benefits of using Terraform. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Don't use Terraform to run ARM templates unless you absolutely have to.</i></b> </span><span style="font-family: arial; text-align: left;">The template deployment resource represents an Azure Deployment, not the resources that deployment creates. For example, if you execute an ARM template that creates a VNet, Terraform will only understand changes made to the ARM template. Executing a Terraform plan will *not* report changes that will be affected to the underlying VNet. If somebody made a manual change to that VNet, Terraform will not sense the change and re-apply the ARM template. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Only use Terraform for ARM templates for new features that aren't in Terraform yet.</i></b> This is rare, but it does happen. Microsoft enhancements are reflected in the REST APIs, and thus the ARM template schema, before enhancements are incorporated in the SDK. Once new features are in the SDK, they commonly are reflected in Terraform very quickly. But there are enhancements (e. g. the VWAN additions) that take months to be completely incorporated in the SDKs.</span></p><p style="text-align: justify;"><span style="font-family: arial;">For example, at the time of this writing, Terraform resources do not yet exist for Virtual WAN VPN sites and VWAN VPN site-to-site connections. I recently used the template deployment resource to manage that infrastructure because there was no other choice from Terraform perspective. </span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Consider Terraform execution of ARM templates after Terraform resources exist for new features as technical debt.</i></b> That is, once Terraform formally supports the resources you need, you should enhance your Terraform to remove the ARM templates. This makes your Terraform more consistent and allows you to identify configuration drift. As with all technical debt, the work should be properly scheduled in light of the team's other priorities. to use my previous example, the ARM templates used to manage VWAN VPN sites and connections should be refactored once Terraform resources exist for those constructs.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>When an ARM template execution fails in Terraform, Terraform doesn't record the fact that the deployment was physically created in the state file.</i></b> Consequently, to rerun the terraform ARM template after corrections, you either need to manually delete the Azure deployment, to do a Terraform import for that deployment to re-execute the Terraform configuration. </span></p><p style="text-align: justify;"><span style="font-family: arial;">Some try to work around the deployment creation problem by generating unique deployment names: I consider this kludge paste. It creates a large number of deployments to sift through if you want to review details on an error. It also means that Terraform will re-run ARM templates unnecessarily when the configuration is executed.</span></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-84808713098481169162020-10-23T11:47:00.006-07:002020-10-24T12:05:32.716-07:00Best Practices for Managing Feature Branches<p style="text-align: justify;"><span style="font-family: arial;"><b>Feature branches</b> are a popular source code management tactic used to manage and coordinate changes made by development teams. Developers create a feature branch is created from the main branch (typically </span><span style="font-family: courier;">master</span><span style="font-family: arial;">) and then merge the changes made to that feature branch back to the main branch when they are complete. This isolates changes made for a specific feature and limits the effect of feature enhancements on other team members until the change is ready.</span></p><p style="text-align: justify;"><span style="font-family: arial;">When using feature branches, it's rare to directly develop using the master branch. In the example below, one developer might be working on a change called "feature 1" while another developer works on a separate enhancement "feature 2". Each developer writes/commits code in isolation in separate branches. When the enhancement is ready, that developer creates a pull request and merges the change back into master. The diagram below illustrates this example. Each bubble is a commit. </span></p><p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPYyOLMYqAOt-ssJhQZdwsiHlUPRKcZWc-OxE-giMWH-gLaX4tsX35fjmZ5z_C81IpqsZ7w1vhost3HRSkSxRagJJ6M1rioWyOcuxjwOEl9OkKUBPalOmxCbgU0n6vBUxwCrOtTtb_Pp8/s2048/FeatureBranches.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2048" data-original-width="1867" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPYyOLMYqAOt-ssJhQZdwsiHlUPRKcZWc-OxE-giMWH-gLaX4tsX35fjmZ5z_C81IpqsZ7w1vhost3HRSkSxRagJJ6M1rioWyOcuxjwOEl9OkKUBPalOmxCbgU0n6vBUxwCrOtTtb_Pp8/s320/FeatureBranches.jpg" /></a></div><br /><span style="font-family: arial;"><br /></span><p></p><h2 style="text-align: justify;"><span style="font-family: arial;">Observations</span></h2><p style="text-align: justify;"><span style="font-family: arial;"><b><i>The longer a feature branch lives, the higher the probability of integration problems when the feature branch is merged into master.</i></b> In the example above, the developer for feature 2 might make changes that conflict with the changes made for feature 1. The longer a feature branch lives, the higher the likelihood that change conflicts occur.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Feature branches work best if the branch contains one targeted enhancement.</i></b> Including multiple changes in a feature branch often lengthens the time the feature branch lives. It also makes code reviews more difficult as the change is more complicated.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>The more developers working on a codebase, the more discipline the team needs regarding source control and change management.</i></b> Even with feature branches, the chance of code integration issues on merge increases with each developer added. That is because the more developers making changes, the higher the probability of integration issues and code conflicts. The higher the probability that multiple developers are working on the same section of code at the same time. Yes, each developer is working in a separate branch, but those changes will be merged to </span><span style="font-family: courier;">master </span><span style="font-family: arial;">at some point.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Recommended Tactics</span></h2><div><span style="font-family: arial;"><b><i>Feature branches should have a short life. </i></b>Most of my feature branches live for less than one business day. They are narrow and targeted. If I need to make a "large change", I break it up into separate smaller changes using multiple feature branches.If a feature branch must live longer due to forces beyond my control, I rebase or merge in changes from master and address any needed merge issues.</span></div><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Feature branches should represent changes from one and only one developer. </i></b>When multiple developers make changes to the same feature branch, the chance of one developer of that feature branch negatively impacting other developers on that same feature branch greatly increases.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Feature branches should be removed after they are merged into master. </i></b>If you don't, the resulting branch pollution will become confusing. The list of existing branches will grow to a large number. It won't be obvious which branches are active and which are historical.</span></p><p style="text-align: justify;"><b><i><span style="font-family: arial;">Frequently </span><span style="font-family: courier;">rebase </span><span style="font-family: arial;">the feature branch</span><span style="font-family: courier;"> </span></i></b><span style="font-family: arial;"><b><i>against the master branch. </i></b>Definitely rebase before merging back into the </span><span style="font-family: courier;">master </span><span style="font-family: arial;">branch (or creating a pull request which will accomplish the merge when completed). More importantly, it will consolidate changes made for the feature branch in </span><span style="font-family: courier;">git </span><span style="font-family: arial;">history. It's common for developers to </span><span style="font-family: courier;">merge </span><span style="font-family: arial;">rather than </span><span style="font-family: courier;">rebase </span><span style="font-family: arial;">to incorporate new changes from master. Using </span><span style="font-family: courier;">merge </span><span style="font-family: arial;">is more intuitive. That said, </span><span style="font-family: courier;">rebase </span><span style="font-family: arial;">makes git commit history easier to interpret as feature branch commits will be consolidated. Additional information on rebasing can be found <a href="https://git-scm.com/book/en/v2/Git-Branching-Rebasing" target="_blank">here</a>.</span></p><p style="text-align: justify;"><span style="font-family: arial;">An example series of commands to accomplish this follows:</span></p><div style="text-align: justify;"><span style="background-color: white; font-family: courier;">git checkout master<br />git pull<br />git checkout feature_branch<br />git rebase master</span></div><p style="text-align: justify;"><b><i></i></b></p><div class="separator" style="clear: both; text-align: center;"><b><i><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7-UVC45M3RedmPXVO_yGMvtaA5ms8gmk32jQ4KUmlQh5u4yY-8WLk29OYjMJMUtJK5bhPKk7IB4wx_Pi_vo33sp27vHv4-IbdSyn32RxNa9nLg21Nl5XDt3m3MsLnGEVzZuJVcmqhdKE/s2048/RebaseVsMerge.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2048" data-original-width="1552" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7-UVC45M3RedmPXVO_yGMvtaA5ms8gmk32jQ4KUmlQh5u4yY-8WLk29OYjMJMUtJK5bhPKk7IB4wx_Pi_vo33sp27vHv4-IbdSyn32RxNa9nLg21Nl5XDt3m3MsLnGEVzZuJVcmqhdKE/s320/RebaseVsMerge.jpg" /></a></i></b></div><b><i><br /><span style="font-family: arial;"><br /></span></i></b><p></p><p style="text-align: justify;"><b><i><span style="font-family: arial;">Some teams prefer to squash commits when merging the feature branch into the </span><span style="font-family: courier;">master </span></i></b><span style="font-family: arial;"><b><i>branch.</i></b> This consolidates log history on the </span><span style="font-family: courier;">master </span><span style="font-family: arial;">branch as feature branches typically have multiple commits. For example, feature 1 with three committed changes can optionally be merged into master as one change. This makes git history more concise and easier to read. Squashing commits will lose commit history detail on the feature branch, however.</span></p><p style="text-align: justify;"><span style="font-family: arial;"><b><i>Promptly respond and participate in requested code reviews.</i></b> Most teams will use pull requests with code reviews as part of the process for merging feature branches into the master branch. It is common for developers to be slow to perform code reviews as they don't want the distraction. The trouble is that as long as the pull request is open, the feature branch it's associated with lives with it. The longer the feature branch lives, the greater the chance of integration issues.</span></p><h2 style="text-align: justify;"><span style="font-family: arial;">Conclusion</span></h2><p style="text-align: justify;"><span style="font-family: arial;">Thanks for taking the time to read this article. I'd love to hear your thoughts.</span></p><p style="text-align: justify;"><br /></p>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-32643816066252808622020-08-30T10:20:00.001-07:002020-10-24T12:18:19.733-07:00For Managers: DevOps Automation and Unintended Consequences<div style="text-align: justify;">
Most organizations adopting the cloud have adopted DevOps automation to some degree or another. The primary reason is that continued manual maintenance isn't possible with the same staffing level and increased demand for a faster change rate. Many aren't to the point of achieving 100% automation but are striving for it. By "automation", I refer to Infrastructure as Code (IaC), automated builds and deployments (CI / CD Pipelines), machine image creation, security enforcement functions, etc. Most organizations struggle with the unexpected and unintended effect on the technology silos most have. I've seen similar issues with most of my cloud adoption and DevOps/automation clients for the past few years.<br />
<br />
The goals most organizations have for consuming the cloud and adopting DevOps automation practices are several:<br />
<ul>
<li>Increased speed to market for application capabilities</li>
<li>Increased productivity for IT staff</li>
<li>Increased scalability and performance of applications</li>
<li>Cost-effectiveness as footprint can dynamically scale to load</li></ul></div>
<div style="text-align: justify;">
<span style="font-size: large;">Steel Copy of a Wooden Bridge</span></div>
<div style="text-align: justify;">
<b><i>All organizations initially view cloud adoption and DevOps automation as just a technology change.</i></b> Consequently, they adopt automation toolsets and keep all business management processes in place (e. g. request forms, manual approvals, the internal team structure that governs who does what, etc.). Unfortunately, the paradigm shift to cloud infrastructure and full automation doesn't really permit that with the same organization structure. The new world is just too different.<br />
<br />
Using existing business processes without change will make it difficult to achieve increased speed to market and consistency between environments.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b><i>Pre-automation business processes don't fit the cloud or DevOps automation. </i></b>DevOps automation is commonly introduced with cloud consumption. Typically, the business is looking for ways to provide additional business capabilities faster and more cost-effectively. Consequently, the number of applications and the number of supported infrastructures increases. For many organizations, the business processes in place either can't easily support a larger software footprint. They don't support the increased speed of change demanded by the business.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b><i>The structure of automation often doesn't match the existing organizational structure.</i></b> For example, setting up a cloud landing pad usually involves not only defining cloud networks, but configuring on premises connectivity, defining and enforcing security policies, defining and enforcing cloud service usage, and much more. From a strictly technology/coding perspective, the automation for these items is tightly coupled and a large portion of it usually belongs in the same automated source code project. Most organizations will have broken responsibility for these items into several teams, usually in separate departments, with people who don't usually work closely together. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As another example, it's typical for application developers to augment their responsibilities to include IaC automation to meet application needs. That is, the management of virtual machines, application subnets, allowed network ingress and egress to an application is managed by application development teams. Pre-automation, these items would have been managed by different application teams.<br />
<br />
<b><i>The implementation of infrastructure and application hosting drastically changes when consuming the cloud. </i></b>New cloud consumers quickly find out that the new world is different and consequently, existing business processes for allocating infrastructure and hosting applications in the cloud no longer apply. For example, existing business processes don't accommodate cloud vendors </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-size: large;">Patching the Steel Copy</span></div>
<div style="text-align: justify;">
On realizing the problems and organizational friction created by automation described above, most organizations attempt to "patch" their existing organization and supporting business processes. That is, they adopt a series of minor changes to mitigate some of the problems described above. Examples I've seen are:</div>
<div style="text-align: justify;">
</div>
<ul>
<li>Establish a manual review for security changes by the security team</li><li>Assume the cloud is "untrusted" and establish cumbersome firewall rules to guard on premises networks</li><li>Establish silos for networking and security changes</li>
<li>Establish tight restrictions for use of cloud options and services</li></ul>
<div style="text-align: justify;"><b><i>Any manual review will slow down velocity and productivity.</i></b> The perception is that this increases safety. However, manual reviews also slow everything down. To this extent, manual reviews through the baby out with the bathwater. A major benefit of DevOps and cloud consumption is increased speed to market. That is, both DevOps and cloud consumption should allow companies to make business capabilities available to end-users faster and increase competitive advantage. Manual reviews decrease if not eliminate this business benefit.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><b><i>Organizational silos and restrictions create process bottlenecks and discourage innovation.</i></b> The logic for silos is that it helps companies achieve economies of scale for specialized skillsets. The trouble is that these silos can't keep up with application team demand. Application teams recognize the bottlenecks and adjust their designs to accommodate and streamline silo navigation rather than use the design they would like. In other words, they are discouraged from using new techniques that don't fit how the silos operate. While most companies provide an "exception" process that allows for a review of new tools, techniques, or procedures; exception processes are often cumbersome and time-consuming. In the end, organizational silos and restrictions depress productivity and slow the release of new business capabilities to end-users.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><b><i>DevOps and cloud capabilities of companies often lag behind their needs.</i></b> It takes time to get up to speed on cloud capabilities and DevOps practices. Consequently, the following often happens:</div><div style="text-align: justify;"><ul><li>Initial environment set-ups and application deployments are much slower than expected.</li><li>Security vulnerabilities discovered at an increasing rate due to staff inexperience</li><li>The frequency of change for both management and staff is larger and more difficult than expected.</li></ul></div><div style="text-align: justify;">All the difficulties above depress productivity and reduce if not eliminate the benefits of DevOps and cloud consumption.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">By now the reader might be second-guessing their decisions to adopt DevOps practices and the cloud. That's not where I'm headed. They are definitely good decisions, but </div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="font-size: large;">Re-Write Management Processes from the Ground Up</span></div><div style="text-align: justify;">By now, it should be obvious that patching existing management oversight and procedures has limitations. In fact, it won't really work for anyone's satisfaction. DevOps and cloud consumption requires a management paradigm shift in many ways. Let's face it. Management oversight methods and procedures that worked for a smaller on premises footprint simply don't work well for DevOps and the cloud. This section will highlight many paradigm shifts managers face and highlight things that need to change.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><b><i>Acknowledge that DevOps and cloud consumption require a change in the way you think about management and oversight.</i></b> This is difficult for many to do and is resisted at first. Once the paradigm shift is recognized, it's much easier to objectively evaluate alternative means and methods. You won't achieve the benefits of consuming the cloud otherwise. It expands your footprint with existing management and oversight processes that don't easily scale. </div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><b><i>Automate management oversight for cloud assets.</i></b> Since everything in the cloud is "software", management oversight policies can be automated so that they no longer require manual oversight. Automated enforcement, once established, is much more consistent and doesn't require labor in the same way. Yes, this automation will require enhancement and maintenance just like any other software, but it increases the productivity of your security and cloud specialists exponentially. This is a body of work that will take planning and implementation effort - this isn't a costless option. That said, in the long run, this is the most cost-effective option available currently.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Management oversight automation will also allow the company to migrate to continuous deployment and continuous delivery someday. In fact, continuous delivery is not possible without automating approvals and eliminating manual steps.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><b><i>Don't try to transition to DevOps and the cloud without help.</i></b> Yes, you retain smart people and they will get make the transition eventually. That said, it will take them a lot longer and you will experience "rookie" mistakes and accrue technical debt along the way. Keep in mind that you need help from a strategy perspective at a management level in addition to ground-level skills. Companies that look at DevOps and cloud consumption as strictly a technology change have trouble from a management perspective that I've outlined above. </div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="font-size: large;">In Conclusion</span></div><div style="text-align: justify;">This article comes from my experiences in the field. I help companies consume cloud technology and adopt DevOps tactics on a daily basis. That said, I'm always interested in hearing about your experiences. I hope that you find this entry useful and hope for many insightful comments. Thanks for your time.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-27613751140122739902020-05-29T13:27:00.001-07:002020-05-29T13:29:09.081-07:00Design Patterns for Cloud Management and DevSecOps<div style="text-align: justify;">
With the cloud (it doesn't matter which cloud vendor), truly all infrastructure and application management is software-based now. Consequently, most organizations manage their cloud footprint through code. Some organizations are further along that path, but most strive to achieve 100% infrastructure as code. Additionally, application infrastructure and releases are also managed as code. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Having written code to manage cloud infrastructure, application infrastructure, and application build and release pipelines for years now; I frequently experience deja-vu. That is, I feel that I'm solving the same problem over and over again. Sometimes with different technologies or cloud vendors, but really repeating the same patterns over and over again.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It's time we start thinking of infrastructure code and the various forms of CI/CD pipelines in terms of software design patterns. Patterns that are repeatable and don't need to be "re-invented" for every application, every cloud vendor, or every enterprise.</div>
<h3 style="text-align: justify;">
What is a Software Design Pattern?</h3>
<div style="text-align: justify;">
This concept was invented and published in 1994 in a book entitled <a href="https://en.wikipedia.org/wiki/Design_Patterns" target="_blank">Design Patterns: Elements of Reusable Object-Oriented Software</a>. The book was written by four authors usually referred to as the "Gang of Four" (GOF). While the book originally targeted object-oriented software languages, the "pattern" concept was incredibly successful and has gone on to be applied to many other types of technologies. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidpz3pRw3D3CN4C3HuHa5me9LW_RqbmEZXoCFBLQwUZ0HfmSlhaWaOSq0WGiVzW1KLUmyXJrRTXdWIx7iIl6e-lpAMyBM6XoxOI_F9ZPF3Um2vC4A-dy9ZmaKY0uk8zJ95jsyf-RnNon0/s1600/Design_Patterns_cover.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="175" data-original-width="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidpz3pRw3D3CN4C3HuHa5me9LW_RqbmEZXoCFBLQwUZ0HfmSlhaWaOSq0WGiVzW1KLUmyXJrRTXdWIx7iIl6e-lpAMyBM6XoxOI_F9ZPF3Um2vC4A-dy9ZmaKY0uk8zJ95jsyf-RnNon0/s1600/Design_Patterns_cover.jpg" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Software design patterns usually have the following components:</div>
<div style="text-align: justify;">
</div>
<ul>
<li><b><i>Problem Statement</i></b> -- a description of the problem being solved</li>
<li><b><i>An Example</i></b> -- a real-world example to help explain the reason the pattern exists</li>
<li><b><i>Applicability Statement</i></b> -- a description of when this pattern should be considered</li>
<li><b><i>Structure </i></b>-- a description of the pattern in clear enough terms that somebody could implement it</li>
<li><b><i>Consequences </i></b>-- Listing of the advantages and disadvantages of using the pattern. This section also includes any limitations</li>
</ul>
The GOF book and many academic papers include some more sections and a more precise and detailed explanation for each component. I prefer a more practical approach.<br />
<h3 style="text-align: justify;">
What are the Design Patterns for Cloud Management and DevSecOps?</h3>
<div style="text-align: justify;">
I'm currently dividing patterns into these categories:</div>
<div style="text-align: justify;">
</div>
<ul>
<li>Build Patterns</li>
<li>Application Release Patterns</li>
<li>Infrastructure Patterns</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDaQ3Vil4ZkaAx6tn0P9XhX7ofsCUowUTMtTm2ef4cu7lu90Yvt6k-zl2cLHfq3CXXGGHok_uqcE8oMkEZMZvn-V_XtNswnTExL2Pvy4Bz5gw5L9TB8Q7Sv5tslCfVFa2b9BHj9nrm9UY/s1600/Pipeline+Patterns.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="227" data-original-width="766" height="94" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDaQ3Vil4ZkaAx6tn0P9XhX7ofsCUowUTMtTm2ef4cu7lu90Yvt6k-zl2cLHfq3CXXGGHok_uqcE8oMkEZMZvn-V_XtNswnTExL2Pvy4Bz5gw5L9TB8Q7Sv5tslCfVFa2b9BHj9nrm9UY/s320/Pipeline+Patterns.jpeg" width="320" /></a></div>
<div style="text-align: justify;">
<b><i><br /></i></b></div>
<div style="text-align: justify;">
<b><i>Build Patterns </i></b>describe how source code is compiled, packaged, and made available for release. Additionally, many organizations apply automated testing as well as gather quality metrics. Build patterns currently identified are:</div>
<div style="text-align: justify;">
</div>
<ul>
<li>Packaging --- Includes any needed compilation. The output is something that can be included in a software release.</li>
<li>Automated Testing -- Includes any unit and/or integration testing needed to validate packaged software.</li>
<li>Metric Analysis -- Includes and static code analysis that analyzes code quality and complexity. </li>
</ul>
<br />
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b><i>Application Release Patterns </i></b>are patterns used to safely deploy packaged software produced by a build pattern. Application release patterns currently identified are: </div>
<div style="text-align: justify;">
</div>
<ul>
<li>All at Once (Spray and Pray) -- Pattern to deploy software without concern for an outage</li>
<li>Rolling Deployment -- Pattern to deploy software incrementally to minimize user outage time.</li>
<li>Blue / Green -- Pattern to utilize cloud technologies to minimize user outage time and provide easy back-out.</li>
<li>Canary -- Variant of Blue/Green that incrementally directs users to a new version of software to minimize the impact of deployments with defects.</li>
</ul>
<b><i>Infrastructure Patterns </i></b>are patterns that create or update cloud infrastructure including networking, security policies, on premises connectivity, monitoring, logging, etc. Infrastructure patterns currently identified are:<br />
<div style="text-align: justify;">
</div>
<ul>
<li>Infrastructure Maintenance -- Includes network, security, monitoring, logging, infrastructure and much more</li>
<li>Image Production -- Create hardened virtual machine images often used by multiple applications or business units.</li>
<li>Mutable Infrastructure Maintenance -- Managing configuration updates for virtual machines that can't easily be destroyed and re-created at will.</li>
</ul>
<h3>
Next Steps</h3>
<br />
<div style="text-align: justify;">
Over the coming weeks, I'll document the patterns identified in this post. I'm always interested in patterns I might have missed. Please feel free to contact me with questions, comments, and suggestions. Thanks for reading.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
</div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-34104826003391974242019-11-16T12:05:00.000-08:002019-11-16T12:07:01.879-08:00Streamlining Tagging in Terraform projects.<span style="font-family: "arial" , "helvetica" , sans-serif;">Tagging resources in Azure or AWS Terraform projects used to be such a mind-numbing pain before the release of Terraform 0.12. For each resource in a Terraform project, the tag section was very verbose and very repetitive. Now, with new variable functionality that comes with Terraform 0.12, I've fallen into a much more streamlined way of maintaining tags.</span><br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Tagging before Terraform 0.12</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Before new features for Terraform 0.12, often a variable file had a large section dedicated to tag values and resources assigned those tags had many lines dedicated to tagging as well. An example of what life used to be like is below. This ritual was repeated for any other resoruces in the project that needed tags.</span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;">resource "aws_instance" "webServer" {</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> ami = "${data.aws_ami.linux_ami.id}"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> instance_type = "t2.micro"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> subnet_id = "${var.subnet_id}"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> key_name = "${var.key_pair}"</span></span><br />
<br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> tags {</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> "Name" = "${var.instance_name}"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> "Environment" = "${var.environment}"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> "ChargebackDept" = </span></span><span style="background-color: white; color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;">"${var.chargeback}"</span><br />
<span style="background-color: white; color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"> </span><span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;">"Business Priority" = </span></span><span style="background-color: white; color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;">"${var.priority}"</span><br />
<span style="background-color: white; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><b><span style="color: red;">Many more tags indeed ad nauseum!</span></b></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> }</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"><br /></span></span>
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;">}</span></span><br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Tagging after Terraform 0.12</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I've fallen into the habit of using Terraform's new variable typing construct along with the map variable type and the map and merge functions. Today, my tag references look like the following for all resources that need tags. The important part is <span style="background-color: yellow;">yellow</span>.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;">resource "aws_instance" "webServer" {</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> ami = "${data.aws_ami.linux_ami.id}"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> instance_type = "t2.micro"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> subnet_id = "${var.subnet_id}"</span></span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;"> key_name = "${var.key_pair}"</span></span><br />
<br />
<span style="background-color: yellow; color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"> tags {</span><br />
<span style="background-color: yellow; color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"> tags = "${merge(var.project_tags, var.environment_tags, map("Name", var.instance_name))}"</span><br />
<span style="background-color: yellow; color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"> }</span><br />
<span style="color: #24292e; font-family: , "blinkmacsystemfont" , "segoe ui" , "helvetica" , "arial" , sans-serif , "apple color emoji" , "segoe ui emoji"; font-size: xx-small;"><span style="background-color: white;">}</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Notice I use the </span><span style="font-family: "courier new" , "courier" , monospace;">merge </span><span style="font-family: "arial" , "helvetica" , sans-serif;">function to combine two maps together. The first is project-wide and the second contains environment-related tag names and values. I also use the </span><span style="font-family: "courier new" , "courier" , monospace;">map </span><span style="font-family: "arial" , "helvetica" , sans-serif;">function to add an entry for this specific resource.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In a variable file I typically call </span><span style="font-family: "courier new" , "courier" , monospace;">vars.tf</span><span style="font-family: "arial" , "helvetica" , sans-serif;">, there are two variables declared. One has project-wide tag entries and another I reserve for environment-related entries. Here's an example from </span><span style="font-family: "courier new" , "courier" , monospace;">vars.tf</span><span style="font-family: "arial" , "helvetica" , sans-serif;">:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">variable "project_tags" {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> type = map</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> default = {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> ChargebackDept = "dan_haberer@vfc.com"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> BusinessPriority = "VF Services"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> <span style="background-color: yellow;">Include any number of tags in a format easy to maintain</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">}</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">variable "environment_tags" {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> type = map</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">}</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Additionally, I typically use a </span><span style="font-family: "courier new" , "courier" , monospace;">tfvars </span><span style="font-family: "arial" , "helvetica" , sans-serif;">file for environment-related tags. Here's an example:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">environment_tags = {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> Environment = "Development"</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">}</span><br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Looking at the Pros and Cons</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I like this approach over the previous approach for several reasons.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Tag names and values are easy to maintain.</i></b> Most tag names and values are in one spot. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>There are not as many variables to declare.</i></b> While the variables contain larger values in the form of maps, I don't need to handle nearly as many of them in my Terraform projects.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>This solution is much less verbose. </i></b>I've cut hundreds of lines from some of my Terraform projects with this trick alone.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I hope you find this trick useful. I'm always open to ways to streamline Terraform code even further.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-240642004866368182018-07-17T02:12:00.001-07:002018-07-17T09:47:25.970-07:00Cloud Governance: Making DevOps Automation Effective<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I see cloud automation of all types implemented to control and/or secure cloud assets. Examples of this type of automation using Amazon Web Services (AWS) include the following:</span></div>
<div style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Preventing unauthorized entries in security groups allowing ingress from 0.0.0.0/0 (security)</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Alerts for Creating IAM users (possible security risk)</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Forwarding application logs to Splunk (operational effectiveness)</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Scheduling up-time for non-production assets (cost savings)</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif; text-align: justify;">While these examples are AWS-specific, the principles discussed in this article are equally applicable if you're using Azure, GCP, or another cloud. </span><br />
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Some of these examples (which are real-life examples, by the way) were very effective. Some were only marginally effective. For instance, the first two examples were extremely effective. If I defined a security group allowing unauthorized access publicly, that entry in the security group was automatically removed and an alert issued to the security team. There was a process to add an "authorized" security group allowing public access, but I had to justify the need. As for the second example regarding IAM user creation, i</span><span style="font-family: "arial" , "helvetica" , sans-serif;">f I created an IAM user, an alert was generated and sent to the security team. The new IAM user wasn't automatically removed, but a similar alert was generated and I was asked to justify the action.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The example regarding forwarding application logs to Splunk was absolutely horrible. It took many hours to implement. It was very fragile; new version deployments to applications broke the automation. The coding needed to automatically fix forwarding after an application deployment was cost prohibitive. Consequently, logs were not reliably forwarded.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Scheduling up-time for non-production assets using the </span><a href="https://docs.aws.amazon.com/solutions/latest/instance-scheduler/overview.html" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">AWS Instance Scheduler</a><span style="font-family: "arial" , "helvetica" , sans-serif;"> is marginally effective. The solution works. However, it depends on instance tags for functionality. Instances scheduled must have required tags if up-time for them is to be scheduled. Often, developers are inconsistent in the implementation of these tags. This dependency makes the solution marginally effective for the customer as the tags are often not consistently applied. This entire discussion leads to an obvious question.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: large;">What characteristics make DevOps Automation effective?</span></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The characteristics that make DevOps automation effective can be attributed to the following characteristics:</span></div>
<div style="text-align: justify;">
<br />
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">DevOps automation must be resilient</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">DevOps automation must be easy to install</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">DevOps automation must minimize ongoing manual labor.</span></li>
</ul>
</div>
<div style="text-align: justify;">
<b style="font-family: arial, helvetica, sans-serif;"><i>DevOps automation must be resilient to be effective.</i></b><span style="font-family: "arial" , "helvetica" , sans-serif;"> In other words, automation isn't effective if it breaks or doesn't work as intended when seemingly unrelated changes are made. Essentially, the benefit to the automation isn't realized. In the case of the fragile Splunk forwarding, broken forwarding meant that support could not trust the results of Splunk searches. Essentially, the benefit of having all application logs in one place was not realized. Developers had to go to the Cloudwatch source.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>DevOps automation must be easy to install to be effective.</i></b> Automation that's difficult to install is a classic "barrier to entry". In other words, installation difficulty increases the effective price of the automation. Automation that's not implemented doesn't provide benefit. The Splunk forwarding example was extremely difficult to install due to complexity as well as poor and inaccurate documentation. As a consequence, many teams didn't even attempt to implement the solution.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>DevOps automation must minimize or eliminate ongoing manual labor to be effective.</i></b> The purpose of automation is to eliminate manual labor. If the automation itself has manual labor requirements, that just negates some portion of the benefit. The AWS Instance Scheduler example requires manual labor in terms of tag maintenance. Consequently, the complete benefit of minimizing instance runtime costs is never realized. There's labor to pay for.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">How do I use these principles to improve the automation I create?</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Test your install documentation.</i></b> Write your documentation and let somebody, not familiar with your automation, install it. If they have questions, it means that your documentation has a defect. It might be that your documentation wasn't clear. It might be that you missed documenting something. Whatever the reason, take those questions as defect reports.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Support your automation when it breaks or doesn't work as intended.</i></b> If a consumer has trouble with your automation, help them fix. After you fix the issue, do a root cause analysis. Figure out why the automation broke and improve it to prevent that problem from repeating. It could be that your automation makes invalid assumptions about the environment. In which case, there is a code change to make. It could be that the user didn't understand how to properly use your automation. This can be fixed by improving your documentation. Whatever caused the issue, fix it.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Solicit feedback for ongoing care and feeding your automation requires.</i></b> Ongoing care and feeding required by your automation detract from the benefits your automation provides. If there are changes you can make that further minimize or eliminate that ongoing work, you should consider it. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let's consider the AWS Instance Scheduler example that requires ongoing work with regard to documenting instance schedules as tags on the instances themselves. If we wrote that automation, how could we further minimize that ongoing work? One way I can think of is to provide a way to specify a default schedule for an entire account. Tags on the individual instances would be optional.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Many companies use different AWS accounts for non-production and production resources. If I could provide a default schedule for all EC2 and RDS instances in the entire non-production account, individual tags on individual instances would be optional. There would still be a need for tagging options on the instance themselves for resources that need a custom schedule. But the default schedule would apply to all instances in an account without developers <i>remembering</i> to place scheduling tags. A large percentage of the ongoing work required by the AWS Instance Scheduler would go away. With that, the amount of money the organization saved using the automation would increase.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Consider leveraging the single installation model.</i></b> Most custom automation I've seen are written to work in their default region in the account in which they are installed. For example, if I install the automation in us-east-1 for account 1111111, that's the only place the automation is available. If I want to use that automation in other regions or other accounts belonging to the same organization, I need to install separate copies of that automation. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">It is possible to code automation so that it operates in multiple regions and multiple accounts. That said, the code within the automation would get more complicated and likely require cross-account roles. That automation would also require a centralized configuration where it runs. For example, the AWS Instance Scheduler supports the single installation model. If you use that feature, there's additional configuration to specify the accounts and regions the scheduler will operate in. Furthermore, cross-account roles are required to provide it access. Providing guidance on implementing the single installation model effectively is a separate topic and may be the subject of a future article.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">I hope this helps you with the automation you create. Comments and questions are welcome.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-18795763179277757382018-05-27T08:16:00.000-07:002018-05-29T06:24:57.957-07:00Tips and Tactics for Passing the AWS Solution Architect Certification Exams<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I've been using the AWS cloud platform since about 2010. When I embarked on the AWS certification path a couple of years ago, I knew it would be a challenge even with my experience. I knew professional level certs are some of the most challenging exams in IT. Having passed AWS Solution Architect Professional and Associate certification exams, I've been asked by several for tips on how to prepare and pass. The process is daunting given that the body of knowledge covered by the tests are incredibly broad. Many have blogged on this topic. This post describes tactics that worked for me; I hope they are of value to you.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Preparation and Study</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Memory only gets you so far. </i></b>The associate exam does have some questions on obscure service limits where memory skills help (e.g. How many VPCs can a user create by default?, What is the limit on the number of instances in a placement group?). The professional exam is purely story problems where each question is a scenario and you're expected to choose from different design options they present. You do need to learn how to apply the underlying principles AWS uses. You just can't memorize your way to success.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Networking skills should be considered a prerequisite.</i></b> Most of my experience is in the application development realm. Fortunately, I had experience as a system administrator at different points and knew networking concepts. I've friends who went down the same path and had a very difficult time as they didn't have networking experience. <u>You should be able to setup a VPC, public and private subnets in different availability zones including an internet gateway and NAT <b><i>from memory</i></b>.</u> Don't forget to install a couple of instances to make sure the public/private access works. </span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Purchase one of the online courses.</i></b> These courses whittle down the size of the haystack and in some cases include labs where you learn the different services by actually doing, not just reading. I've seen some bloggers recommend purchasing multiple courses, but I don't think this is wise. Not only does it cost more money, but takes more time.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I used the ACloud Guru courses for both the <a href="https://www.udemy.com/aws-certified-solutions-architect-associate/" target="_blank">associate</a> and <a href="https://acloud.guru/learn/aws-certified-solutions-architect-professional" target="_blank">professional</a> exams. Honestly, the associate course better prepared me for the exam than the professional course did. I still consider the professional course worth the money. I did use some lectures in the associate course as prep for the professional as it contained material on some services (like Redshift) that the professional course did not. The ACloud Guru courses include VPC labs that give you a chance to learn networking if that part of your experience is light. </span><br />
<span style="font-size: xx-small;"><b><span style="font-family: "arial" , "helvetica" , sans-serif;">Note: I</span><span style="font-family: "arial" , "helvetica" , sans-serif;">'m not affiliated with ACloud Guru other than being a customer.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></b></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Purchase at least one set of practice exams.</i></b> I used the tests from Whizlabs for the <a href="https://www.whizlabs.com/aws-solutions-architect-professional/" target="_blank">professional</a> exam. For the associate cert, I used the quizzes supplied with the ACloud Guru associate course and got to the point where I made 100% on those tests most of the time. Some of those questions literally appear on the exam word-for-word. I also purchased a set of practice tests for the associate exam from a vendor I lost track of. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif;">Practice exams not only give you confidence when you start doing well, but it optimize your time. It allows you to concentrate study on the portions you miss rather than the things you know well.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Take the AWS Practice Exam.</i></b> While the Whizlab tests are good, they aren't exactly like the AWS exams. The online experience with AWS is slightly different than Whizlabs. Also, AWS questions are more wordy, particularly in the professional exam. </span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Research and verify your AWS Practice Exam answers.</i></b> Capture screen shots of all questions and your answers. Rather than futz with screen shots during the timed test, I used <a href="https://www.techsmith.com/video-editor.html" target="_blank">Camtasia</a> to make a movie of me taking the test and clipped the screen shots after. This exercise forces you to check your assumptions and better acquaints you with their exam tricks. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>For those taking the professional SA exam, take the AWS sample questions (free) test (<a href="https://d1.awsstatic.com/Train%20&%20Cert/docs/AWS_certified_solutions_architect_professional_examsample.pdf" target="_blank">english version</a>).</i></b> There are six questions. There's a trick here: the exact same test is published in <a href="http://media.amazonwebservices.com/jp/certification/AWS_certified_solutions_architect_professional_examsample0701_08_final.pdf" target="_blank">Japanese </a>- with answers on the lower right for each question. You're welcome.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Test Taking Strategies</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This section details the strategies I used when taking the tests. </span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Eliminate wrong answers first.</i></b> Even if the question relies on facts you don't know, often you can eliminate at least one or two possible answers on the list. Even if you end up guessing, that can bump your chance of a correct answer to 50%. Sometimes, you even get to apply the Sherlock Holmes logic that if there's only one answer left after you eliminate the wrong answers, that answer must be correct even if you don't see why.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Often on the professional exam, an incorrect fact or tactic is often used in multiple answers.</i></b> If you spot the incorrect fact, makes it easy to eliminate several answers in one shot. Sometimes, there's more than one incorrect fact buried in multiple answers...</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>ReRead the one sentence that specifies how to choose among the various answers.</i></b> For example, some questions have you choose which strategy "<i><u>minimizes costs</u></i>" without any mention of other requirements you expect like minimizing latency or achieving high availability. They will try to trick you by putting in a strategy that does minimize cost but also increases the chance of data loss or latency time. Don't read into the question: select the strategy that minimizes costs even though other common objectives suffer.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Another tactic is to put the word "not" in the sentence. For example, which of these strategies does <i><u>not </u></i>improve latency. That three-letter word can send you down the wrong path if you miss it.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>There will be questions that rely on facts you don't know.</i></b> Obsessing over these questions when you're obviously reduced to a guess just wastes time you don't have. Eliminate what look like obvious wrong answers, take your best guess, and learn to live with the fact you might have missed that question.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Don't over use the Flag feature.</i></b> AWS exams have a 'flag' feature where you can mark specific questions and revisit them later. In the professional exam the other day, I flagged three of the 77 questions. If you flag most questions, it's the same as not having the feature at all as you won't have time to revisit all of them.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">For me to flag a question, it needed to pass the following criteria:</span></div>
<div style="text-align: justify;">
</div>
<ol>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">This question <i>can't</i> center on facts or a service I simply don't know -- I have to guess anyway.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Time will help improve my chances of getting the answer correct. (Be honest with yourself)</span></li>
</ol>
<span style="font-family: "arial" , "helvetica" , sans-serif;">If it doesn't pass both criteria, I don't flag it. Devoting more time to the question would be a waste. I take my best guess and move on.</span><br />
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Caffeine lovers -- Consider the expresso trick.</i></b> If you're a highly caffeinated individual like I am, you want caffeine for the test. However, you do not want to spend time in the rest room as it's a timed test. Expresso coffee, with a little cream to dilute the bitter taste, functions quite well for this without giving you unwanted liquid. I consumed four shots right before both tests. In all honesty, if I had it to do over again, I would have ordered six for the professional exam as it's so much longer.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I hope these thoughts help -- best of luck on the exams.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com2tag:blogger.com,1999:blog-2825935617443597949.post-72423154728908280952017-08-06T13:40:00.000-07:002017-08-06T13:40:24.605-07:00Making Cloud Code Testable<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Developers seem to assume that just because their application code may now interact with the cloud (e.g. read/write data from AWS S3 buckets, invoke AWS lambda functions, email via AWSD SES, send or receive AWS SMS messages, etc.) that it's no longer reasonable or easy to structure their code in a testable way without access to the cloud. Yes, my examples are AWS centric. The principles in this blog post apply to Azure or Google Cloud code as well.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">By "testable" I mean <b><i>unit tests</i></b> that don't require access to the cloud to run. The purpose of such tests is to test logic in your code; not to test that code has the ability to connect to outside resources. Testing the ability of your code to connect to outside resources is the purpose of <b><i>integration tests</i></b>. </span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Inject resources into application code that can't exist without the cloud, network, or external resources of any type.</i></b> When your application code creates/instantiates these resources, that code can no longer be run without those resources. A previous version of a <a href="https://github.com/Derek-Ashmore/AWSLambdaExamples/blob/master/aws-sdk-mock-examples/src/main/java/org/force66/aws/s3/ListS3Assets.java" target="_blank">class</a> in listing 1 violates this advice and isn't testable.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b><u>Listing 1: AntiPattern Example -- Don't do this at home!</u></b></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEumeIN0Tofe4SmckU1Q2ZNc7Y0xaBE_NkecbfLAewvhHg5RMfV-HaBWQInJGYbSJ0__T_-wtec_w2qQSXLw9OgN34wd5lkh4PBoswn7_auOHeLwWZK0Y4jQ5-7gPfNBuLlU30gX6NPTM/s1600/UntestableCloudCodeExample.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="307" data-original-width="540" height="361" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEumeIN0Tofe4SmckU1Q2ZNc7Y0xaBE_NkecbfLAewvhHg5RMfV-HaBWQInJGYbSJ0__T_-wtec_w2qQSXLw9OgN34wd5lkh4PBoswn7_auOHeLwWZK0Y4jQ5-7gPfNBuLlU30gX6NPTM/s640/UntestableCloudCodeExample.JPG" width="640" /></a></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Listing 1 isn't unit testable as it requires access to AWS to exist. By the way listing 1 is coded, any test on it's logic really is an <i>integration test.</i> A much better way to write this class can be found in listing 2.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b>Listing 2: A more testable way to write listing 1</b></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSCXn-c2Ry364XYAeOzyNNoQheyqxuS7Cl_1Ju2vnVtN35Limj4ijNVHwgtR0invGiK9yL2l7ChevZFCQxTpDKbB2dKCo3Is65w0GfJideVEvJiVX3dPPVtR3Tu2D29Cf4Ivsuu4qt7ws/s1600/TestableCloudCodeExample.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="132" data-original-width="510" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSCXn-c2Ry364XYAeOzyNNoQheyqxuS7Cl_1Ju2vnVtN35Limj4ijNVHwgtR0invGiK9yL2l7ChevZFCQxTpDKbB2dKCo3Is65w0GfJideVEvJiVX3dPPVtR3Tu2D29Cf4Ivsuu4qt7ws/s640/TestableCloudCodeExample.JPG" width="640" /></a></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Listing 2 is much more testable that listing 1 as it no longer depends on the environment in which the </span><span style="font-family: "courier new" , "courier" , monospace;">AmazonS3</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> instance was created and can easily be mocked. That instance can easily be mocked as I did in the <a href="https://github.com/Derek-Ashmore/AWSLambdaExamples/blob/master/aws-sdk-mock-examples/src/test/java/org/force66/aws/s3/ListS3AssetsMockitoTest.java" target="_blank">unit test</a> for this example class. I've a snippet of the <a href="http://site.mockito.org/" target="_blank">Mockito </a>code used to test this class in listing 3. In fact, test coverage is 100% for line, branch, and mutation coverage for this class.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b>Listing 3: Mocking an AWS resource for unit tests.</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjjLplmeZZ1VZTi6kVrFubO6fTMRD8CYukpWEX6ivbWq_97cRXtxBKefvtxKRhBkac2gmn-oKVGRT7xzmWij5S-qS6d69mzy8I6Qq_iVIPrRNx-0d4A0lJfqN9v4AfB0k8Rr7pu2Sh5iA/s1600/CloudUnitTestExample.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="354" data-original-width="729" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjjLplmeZZ1VZTi6kVrFubO6fTMRD8CYukpWEX6ivbWq_97cRXtxBKefvtxKRhBkac2gmn-oKVGRT7xzmWij5S-qS6d69mzy8I6Qq_iVIPrRNx-0d4A0lJfqN9v4AfB0k8Rr7pu2Sh5iA/s640/CloudUnitTestExample.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Incidentally, there are numerous ways to inject cloud-dependent resources. You don't need to inject it on construction as I did in my example. For instance, the Spring Framework or Guice can also do that injection for you.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Another way to look at listing 2 is that it uses the architectural principle of "<a href="https://en.wikipedia.org/wiki/Separation_of_concerns" target="_blank">separation of concerns</a>". Listing 2 separates the environmental concern of creating cloud dependent resources from the logic of listing the content of an AWS bucket. This makes the resulting classes much more focused and less complex. This concept begs a question: Where do you create the </span><span style="font-family: Courier New, Courier, monospace;">AmazonS3 </span><span style="font-family: "arial" , "helvetica" , sans-serif;">client and can you do that in a unit testable way? </span><span style="font-family: arial, helvetica, sans-serif;">The short answer is that you really can't. However, you can localize and minimize the amount of code that isn't unit testable.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Localize the creation of resources that are dependent on the cloud or any other external resource. Generally, my projects end up with a "factory" class that handles instantiations that aren't unit testable. To continue the S3 bucket list example, I'd envision a class like the factory presented in Listing 4.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b>Listing 4: AWS S3 Client Instantiation Example</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPVic1rHMYsDqkC0FICQNbn4J6ryCugxvXhT9o2K6p80yuBIdEU-8E9G8NiMgdvlB_4nMmxYSSNAgLdFUfvjvVYm6wObpoKxvR_0DsX_RhcAfbtGAY7PHxaOXBvQ5ZXqkgW1VHQIRQbZM/s1600/AWSFactoryClassExample.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="449" data-original-width="663" height="432" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPVic1rHMYsDqkC0FICQNbn4J6ryCugxvXhT9o2K6p80yuBIdEU-8E9G8NiMgdvlB_4nMmxYSSNAgLdFUfvjvVYm6wObpoKxvR_0DsX_RhcAfbtGAY7PHxaOXBvQ5ZXqkgW1VHQIRQbZM/s640/AWSFactoryClassExample.JPG" width="640" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This tactic minimizes the amount of code that isn't unit testable. Furthermore, this code doesn't generally contain complex logic leaving the more complex logic to classes that can be unit tested. </span></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Incidentally, I did also construct a unit test written using <a href="http://jmockit.org/" target="_blank">JMockIt </a>as a development team I'm working with had adopted it. As it happens, once JMockit decorates a class with stubbing code, it's not possible to use it directly anymore (bug documented <a href="https://groups.google.com/forum/#!topic/jmockit-users/iBhKe-N08_Q">here</a>). I commonly want to stub an </span><span style="font-family: "courier new" , "courier" , monospace;">InputStream </span><span style="font-family: "arial" , "helvetica" , sans-serif;">or </span><span style="font-family: "courier new" , "courier" , monospace;">OutputStream </span><span style="font-family: "arial" , "helvetica" , sans-serif;">only for the purpose of testing my code handling exceptions. It seems that with this bug, this isn't currently possible using JMockIt. the bug is very irritating and applies to a significant percentage of the unit tests I write. I'm personally sticking to <a href="http://site.mockito.org/" target="_blank">Mockito </a>for now.</span></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-21769830447168979962016-12-30T12:18:00.000-08:002016-12-30T14:09:09.115-08:00Making Technology Choices for Personal Growth.<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">When I started doing IT as a profession, the number of available, commercially used technologies was considerably less. Back then, most of IT ran on IBM mainframes. The number of tools you needed to learn to be marketable could be counted on one or two hands. Today, there are far too many technologies/products/frameworks and not nearly enough time to learn them all. Should I invest in AngularJS or React? Java8 or Scala? Amazon or Azure? Python or Node.js? The list goes on and on.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">If you're like me, <b><i>your time to learn new products or frameworks is spare time</i></b>. It's limited. There's just not enough spare time to learn everything, or even close. I often get asked whether they should learn this product or that framework. Very few ask about <b><i>how </i></b>they should be making their time investment choices. Let's face it, with so much educational material available on the internet for free or very low cash outlay, it's really a time investment we're talking about.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>I look at learning technologies the same way I look at investments.</i></b> Most software vendors, be they open source or commercial, publish documentation our even the ability to download and install most products you might consider for free. Thanks to <a href="https://www.udemy.com/">Udemy </a>and <a href="https://www.amazon.com/Kindle-eBooks/">Amazon</a>, who have pushed prices for online courses and eBooks to unbelievably low levels, cheap resources are available for the more popular choices. There's very little cash outlay for materials. However, it's an investment all the same. The investment is more time than cash. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>I believe time is money.</i></b> Time invested is time I can't spend consulting, writing, or anything else I do to make money. Consequently, I try to make good decisions as to which technologies I invest time in. That begs the important question: <b><i>how do you choose?</i></b> Furthermore, there's another dimension to this choice: <b><i>how much time do you invest in one of those choices?</i></b></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>The amount of time you spend learning a new product, technology, or framework has diminishing returns.</i></b> That is, the more time you invest, the less incremental payback you'll get for the investment. Like stocks, you don't necessarily buy a large portion to start out. Often, you invest a little and as the market develops, sometimes you buy more. Time investments in new technologies are the same way. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Like financial investments, investing your time has risk.</i></b> Any time spent learning a new technology, product, or framework that never takes off is time you'll never get back. Your objective is to consciously manage that risk. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Research Tactics</span></div>
<div dir="ltr" style="text-align: justify;">
<br /></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">There are several tactics I use to help me decide which technologies to invest in. Here are some of mine.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<b style="font-family: Arial, Helvetica, sans-serif;"><i>Ask a mentor or person whose opinion you value about technology choices you're considering.</i></b><span style="font-family: "arial" , "helvetica" , sans-serif;"> It's quick and easy. That person may have thought of issues or concerns that you haven't thought of so far. Often verbalizing your thoughts to another person helps you crystallize and more thoroughly think through your line of thought. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Look at what the market values. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The best way to do this is to use the <a href="https://www.indeed.com/jobtrends">Job Trends</a> site from indeed.com. Indeed is a job posting site where hunters post resumes and firms or recruiters post jobs. Indeed keeps history and lets you graph postings and hunter skills listed on their resume over time. As an example, I'm using this <a href="https://www.indeed.com/jobtrends/q-AWS-q-Azure-q-Google-Cloud.html">comparison </a>that compares top cloud vendors, perhaps for people looking at learning cloud technologies.</span></div>
<div dir="ltr" style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLrMoO2qAPVELEPg0QyoxU1H4T27K_cChoEyOtOEPShp6dspOtE-0qcfgpN2FKLfVA5xl_TmxUe-iKIzWWoGUk00Doye-F4aBkbh3h8Bai_u9qMlu8BQPJMad5dpnxHYugkd6a2XlugwU/s1600/cloud-vendor-job-postings-2016-12-30.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="356" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLrMoO2qAPVELEPg0QyoxU1H4T27K_cChoEyOtOEPShp6dspOtE-0qcfgpN2FKLfVA5xl_TmxUe-iKIzWWoGUk00Doye-F4aBkbh3h8Bai_u9qMlu8BQPJMad5dpnxHYugkd6a2XlugwU/s400/cloud-vendor-job-postings-2016-12-30.jpg" width="400" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<b style="font-family: Arial, Helvetica, sans-serif;"><i>You want technologies with an upward job posting trend.</i></b><span style="font-family: "arial" , "helvetica" , sans-serif;"> You want technologies that firms value and recruit for. Given the time you'll need to invest, you want a realistic chance of a payback for your investment. The market can change at any time and the landscape might look different in six months, but hard numbers are more attractive than gut feelings. I've posted the job posting comparison of cloud vendors </span><span style="font-family: "arial" , "helvetica" , sans-serif;">(taken on Dec. 30, 2016)</span><span style="font-family: "arial" , "helvetica" , sans-serif;">. I'm using Amazon Web Services, Azure, and Google Cloud in my comparison.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">You can see that Amazon AWS job postings are on a sharp uptrend followed by a slightly less uptrend for Azure postings. It would be safe to infer that AWS skills have more of a market right now than either Azure or Google Cloud. If you believe I've left off vendors that you care about, you should surf to the <a href="https://www.indeed.com/jobtrends/q-AWS-q-Azure-q-Google-Cloud.html">site </a>and change the vendors to your liking.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd7t_5OcL6YpdNfeAbFwnwP6KQM0VJ0pcjv0TxTA5Zt6Vzi2p8VtikM-JSUvR6Jze9wnzqqfFWXbJ0H2YkyqafHc9mnEYglKjxGyOc6Y1rFOyVAwK648W9UgOSLJRxUAyrL1ju5HSVMOo/s1600/cloud-vendor-seekers-vs-postings-2016-12-30.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd7t_5OcL6YpdNfeAbFwnwP6KQM0VJ0pcjv0TxTA5Zt6Vzi2p8VtikM-JSUvR6Jze9wnzqqfFWXbJ0H2YkyqafHc9mnEYglKjxGyOc6Y1rFOyVAwK648W9UgOSLJRxUAyrL1ju5HSVMOo/s400/cloud-vendor-seekers-vs-postings-2016-12-30.jpg" width="400" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>You want technologies with a postings vs. seeker interest ratio of at least 1.00.</i></b> Anything less means that there <i><b>might </b></i>be oversupply in the market and that can drive salaries/consulting rates down. I say "might" as not all job seekers are honest. Some look for jobs with skills they don't have. With the demand I hear about for AWS skills from recruiters, I believe the 1.02 seekers per posting ratio to be overstated and there are more postings per seeker. <i><u>Note: Pay attention to the description: "seekers per posting" vs. "Posting per seeker".</u></i></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Test your search criteria.</i></b> Toward the bottom of the comparison page, you're offered links that will list job postings for that particular search criteria. You should look at a sample to make sure your search term isn't picking something you're not interested in. For example, if one were to enter a term too general, you could be including irrelevant postings and seekers in your comparison.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>A few words of warning: Indeed data has limits.</i></b> Here are a few:</span></div>
<div dir="ltr" style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">This data doesn't account for current labor salaries/rates. For example, "HTML" looks hot until you figure out that market rates for that skill are really low.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The data presented is three months old and the current graph might be slightly different.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">This data works best when you enter competing products or frameworks. Comparing AWS to Java isn't useful; market drivers for those two are completely different and you'll get more double counting of postings (posts might have both terms).</span></li>
</ul>
<br />
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: large;">Look at what people are interested in</span></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Before technologies are listed in job postings or on resumes, they are searched for on the web. Google maintains <a href="https://www.google.com/trends/explore">search term history</a> that it graphs or allows you to download. Furthermore, as with the Indeed Job Trend site, you can compare search data for multiple searches. As an example, I've <a href="https://www.google.com/trends/explore?q=AWS,Azure,Google%20Cloud">graphed </a>the same search criteria we used above on the Indeed site.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC7-sZTxJBaESCNzbwHG3nPcc_h8KHfGgSFAqfjwgvngpdfkP1ZJn3NkQI1V8Ue0cuZUKy_xjbM4HVC7oiKY1z09vDkMjH9EgpEM_ypSDSHS7Ne4DCFrnKq37Dd4i-4p30yBiN0ePf_NA/s1600/web-search-interest-cloud-vendors-2016-12-30.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC7-sZTxJBaESCNzbwHG3nPcc_h8KHfGgSFAqfjwgvngpdfkP1ZJn3NkQI1V8Ue0cuZUKy_xjbM4HVC7oiKY1z09vDkMjH9EgpEM_ypSDSHS7Ne4DCFrnKq37Dd4i-4p30yBiN0ePf_NA/s400/web-search-interest-cloud-vendors-2016-12-30.jpg" width="400" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Internet search results appear to be congruent with what we see on the Indeed Job Trends site. The difference between AWS and Azure does appear to be less stark than what we see on the Indeed site. Here are some things to keep in mind:</span></div>
<div dir="ltr" style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">It's important to test your search results just like with indeed.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Trend data for technologies with common names will be meaningless.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Poor paying technologies might have rising interest too just like on the Indeed site.</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Deciding How Much Time to Invest</span><br />
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Deciding how much time to invest is more difficult. Everyone has different experience levels, talents, and abilities. Some technologies might take more investment than others. What amount I might need might be different than the amount you need. Here are some tactics I use.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>You <u>must</u> invest in something all the time.</i></b> Pick a commitment level; two hours a week, four hours a week, or even more. This should become a habit you don't even think about. If you don't you'll become stagnant. Your skills will get dated over time and your marketability will gradually decrease. Furthermore, you won't develop tactics for learning new things quickly. When you wake up and realize that you're very out of date, it'll take a long time to catch up. Unlike financial investments where you can adopt a cash position, it's too dangerous in a technology world to sit on the sidelines.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Time-box your investment.</i></b> That is, set a rough amount of time you'll spend on a technology choice up front and then re-assess what you're willing to spend after that learning period. I typically use two hours or four hours as an initial time frame limit. That said, over the past couple of decades, I've honed tactics that allow me to spend less time than many others. Two hours might not be enough for you or it might be too much. You'll need to decide the amount. It's important that you roughly track the time you spend. Also, feel free to quit early if you learn enough to make the decision that you're not going to invest additional time.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Time-boxing mitigates your investment risk.</i></b> What you want to avoid is going down a rabbit hole and spending boat loads of time unproductively. It also keeps the time you're spending at the forefront of your mind. You'll never get this time back.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Distinguish between "exploratory" learning and "objective" learning.</i></b> Objective learning has a defined purpose such as doing upcoming work at your current employer or gearing up for a new job search or interview. Exploratory learning is for general knowledge you can use with colleagues. Exploratory learning doesn't require as much depth. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">There's value in learning the basics and general advantages and disadvantages of a product. With general knowledge, you know whether or not you want to pursue work using this technology. You will know enough about a technology to participate intelligently in conversations with other developers or recruiters. You know enough to come back later and dig more deeply if the need arises (you decide to submit for a job that needs it). Some developers will resist this idea, but you don't need an advanced, in-depth knowledge of every product you become acquainted with.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Don't learn technologies in depth until there's a high probability you'll will use them at work or need them to support a job search.</i></b> Technology moves too quickly for that. Your newly learned in-depth knowledge becomes dated quickly. Looking at this in investment terms, don't invest until there's a reasonable probability of a payback. Another way to think about this is that classic <a href="https://en.wikipedia.org/wiki/You_aren't_gonna_need_it">YAGNI </a>applies. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Thanks for taking time to read this post. I would like to hear your thoughts on this topic.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-29844526523218195682016-12-11T09:25:00.000-08:002016-12-12T08:35:32.325-08:00Automated Integration Testing in a Microservices World.<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Everyone's dabbling with microservices these days. It turns out that writing distributed applications are difficult. They offer advantages to be sure. However, there is no free lunch. One of the difficulties is automated integration testing. That is, testing a microservice with all the external resources it needs including any other services it calls, the databases it uses, any queues it uses. Just setting up and maintaining these resources can be a daunting task that often takes specialized labor. All too often, integration testing is difficult enough that the task is sloughed off. Fortunately, <a href="https://docs.docker.com/engine/getstarted/">Docker </a>and it's companion product <a href="https://docs.docker.com/compose/overview/">Docker Compose</a> can make integration testing much easier.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Standardizing on Docker images for deployment artifacts makes integration testing easier. </i></b>Most organizations writing microservices seem to adopt Docker as a deployment artifact anyway as it greatly speeds up interaction between application developers and operations. It facilitates integration testing as well as you can deploy services you consume (or mocks for the services you consume) temporarily and run integration tests against them. Additionally, consumers for your service can temporarily deploy your docker image to perform their own integration tests. However, as most services also need databases, message queues, and possibly other resources to function properly, that isn't the end of the story. I've previously written about how to dockerize (is that a word?) your own services and applications <a href="http://www.derekashmore.com/2015/03/using-docker-to-deploy-java-web.html">here</a>.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Docker images already exist for most database software and message software.</i></b> It's possible to leverage these docker deployments for your own integration testing. In other words, the community has done part of your setup work for you. For example, if my service needs a PostgreSQL database to function, I leverage the official Docker <a href="https://hub.docker.com/_/postgres/">deployment </a>for my integration tests. As it turns out, the Postgres docker deployment makes their image very easy to consumer for integration testing. All I need to do is mount the directory '</span><span style="background-color: white; color: #333333; font-family: "consolas" , "liberation mono" , "menlo" , "courier" , monospace; font-size: 12px; white-space: pre;">/docker-entrypoint-initdb.d</span><span style="font-family: "arial" , "helvetica" , sans-serif;">' and make sure that directory has any SQL files and/or shell scripts I need run to set the database up for use by my application. The MySQL docker <a href="https://hub.docker.com/_/mysql/">deployment </a>does something similar. For messaging, similar docker distributions exist for <a href="https://hub.docker.com/_/rabbitmq/">RabbitMQ</a>, <a href="https://hub.docker.com/r/webcenter/activemq/">Active MQ</a>, and <a href="https://hub.docker.com/r/wurstmeister/kafka/">Kafka</a>. Note that ActiveMQ and Kafka aren't yet "official" docker deployments.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Docker Compose makes it very easy to assemble multiple images into a consistent and easily deployable environment.</i></b> Docker-compose configurations are YAML files. Detailed documentation can be found <a href="https://docs.docker.com/v1.7/compose/yml/">here</a>. It is out of scope for this blog entry to do a complete overview of Docker Compose, but I'll point you to an open source example and discuss a couple of snippets from the example as an illustration.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAbbqmcszN8CYvptFAVVdnhfqACvgBQOTklINKdrCpeq_Fiummib7CaB-UywRp7qttEsQsXmqhcpsTnOKczHoTkQoiMbt6WvUaLrcQqm8TZKfEhx0X8VmpgZrQhmyANvS-BOfT3kol9rQ/s1600/DockerComposeConfigExample.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAbbqmcszN8CYvptFAVVdnhfqACvgBQOTklINKdrCpeq_Fiummib7CaB-UywRp7qttEsQsXmqhcpsTnOKczHoTkQoiMbt6WvUaLrcQqm8TZKfEhx0X8VmpgZrQhmyANvS-BOfT3kol9rQ/s400/DockerComposeConfigExample.PNG" width="220" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The screen shot on the left contains a snippet of a docker-compose configuration. The full source is <a href="https://github.com/docker/example-voting-app/blob/master/docker-compose-simple.yml">here</a>. Note that each section under </span><span style="font-family: "courier new" , "courier" , monospace;">services </span><span style="font-family: "arial" , "helvetica" , sans-serif;">describes a docker image that's to be deployed and possibly built. In this snippet, images </span><span style="font-family: "courier new" , "courier" , monospace;">vote</span><span style="font-family: "arial" , "helvetica" , sans-serif;">, </span><span style="font-family: "courier new" , "courier" , monospace;">redis</span><span style="font-family: "arial" , "helvetica" , sans-serif;">, </span><span style="font-family: "courier new" , "courier" , monospace;">worker</span><span style="font-family: "arial" , "helvetica" , sans-serif;">, and </span><span style="font-family: "courier new" , "courier" , monospace;">db </span><span style="font-family: "arial" , "helvetica" , sans-serif;">are to be deployed. Note that </span><span style="font-family: "courier new" , "courier" , monospace;">vote </span><span style="font-family: "arial" , "helvetica" , sans-serif;">and </span><span style="font-family: "courier new" , "courier" , monospace;">worker </span><span style="font-family: "arial" , "helvetica" , sans-serif;">will be built (e.g. turned into a Docker image) before they are deployed. For images already built, it's only necessary to list the </span><span style="font-family: "courier new" , "courier" , monospace;">image </span><span style="font-family: "arial" , "helvetica" , sans-serif;">name.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Other common compose directives are as follows:</span></div>
<div dir="ltr" style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;"><u><b>volumes</b></u></span><span style="font-family: "arial" , "helvetica" , sans-serif;">-- links a directory in the real world to a directory inside the container</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;"><b><u>ports</u></b></span><span style="font-family: "arial" , "helvetica" , sans-serif;">-- links a port in the real world to a port on the inside of the container. For example, </span><span style="font-family: "courier new" , "courier" , monospace;">vote </span><span style="font-family: "arial" , "helvetica" , sans-serif;">links port 5000 on the outside to port 80 on the inside.</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;"><b><u>command</u></b></span><span style="font-family: "arial" , "helvetica" , sans-serif;">-- specifies the command within the Docker container that will be run at startup.</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;"><b><u>environment</u></b></span><span style="font-family: "arial" , "helvetica" , sans-serif;">-- (not illustrated here) allows you to set environment variables within the Docker container</span></li>
</ul>
<br />
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Assemble and maintain a Docker compose configuration for your services.</i></b> This is for your own use in integration tests and so that your consumers can easily know what resources you require in case they want to run integration tests of their own. It's also possible for them to use that compose configuration directly and include it when they set up for their own integration tests.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>The Docker environment for your integration tests should be started and shut down as part of the execution of the test.</i></b> This has many advantages over maintaining the environment separately in an "always on" state. When integration tests aren't needed, they son't consume resources regardless. Those integration tests, along with their environment, can be easily run by developers locally if they need to debug issues; debugging separate environments is always more problematic. Furthermore, integration tests can be easily and painlessly be hosted anywhere (e.g. on-premise, in the cloud) and are host agnostic.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">An Integration Test Example</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I would be remiss if I didn't pull these concepts together for an integration test example for you. For my example, I'm leveraging an integration test generic health check written to make sure that a RabbitMQ environment is up and functioning. The source for the check is <a href="https://github.com/BreakTheMonolith/btm-DropwizardHealthChecks/tree/master/btm-DropwizardHealthChecks-rabbitmq">here</a>, but we're more interested in its integration test today. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This test utilizes the <a href="https://github.com/BreakTheMonolith/DockerProcessAPI">DockerProcessAPI </a>toolset as I don't currently work in environments that require a docker-machine and the Docker Remote API (Linux or Windows 10 Pro/Enterprise). If your environment requires a docker-machine (e.g. it is a Mac or an earlier version of Windows), then I recommend the Spotify <a href="https://github.com/spotify/docker-client">docker-client</a> instead.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The <a href="https://github.com/BreakTheMonolith/btm-DropwizardHealthChecks/blob/master/btm-DropwizardHealthChecks-rabbitmq/src/test/java/guru/breakthemonolith/health/rabbitmq/RabbitMQHealthCheckTestIntegration.java">integration test</a> for the health check uses Docker to establish a RabbitMQ environment before the test and shut it down after the test. This part is written as a JUnit test using the </span><span style="font-family: "courier new" , "courier" , monospace;">@BeforeClass</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> and </span><span style="font-family: "courier new" , "courier" , monospace;">@AfterClass</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> annotations to bring the environment up once for the entire test and not for each test individually.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9AaaOjujIbzrKh2Z9utwr1DrGZyasanJe-hmwODUFEKKSClvrw70pzLNmKBJqWMk4VdXi1fFFQyh_IwEL6HJb1kspPhke-OXFP8nJxEUSsgJw22khZy7Jc2lF3q8VtxG8d9AilwUbjxk/s1600/DockerProcessAPIExample.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9AaaOjujIbzrKh2Z9utwr1DrGZyasanJe-hmwODUFEKKSClvrw70pzLNmKBJqWMk4VdXi1fFFQyh_IwEL6HJb1kspPhke-OXFP8nJxEUSsgJw22khZy7Jc2lF3q8VtxG8d9AilwUbjxk/s640/DockerProcessAPIExample.PNG" width="640" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this example, I first pull the latest RabbitMQ image (official distribution). I then map a port for RabbitMQ to use and start the container. I wait five seconds for the environment to initialize, then cause a logging for the current docker environment running.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">My log of what Docker containers are running isn't technically required. It does help sometimes if there are port conflicts where the test is running or other problems with a failed test that need to be investigated. As this test runs in a scheduled manner, I don't always know execution context.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">After the test completes, the </span><span style="font-family: "courier new" , "courier" , monospace;">@AfterClass</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> method will shut down the RabbitMQ instance I started and once again cause a container listing just in case something needs to be investigated.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">That's a very short example. Had the integration test environment been more complicated and I needed Docker Compose, that would have been relatively simple with the DockerProcessAPI as well. Here's an example of bringing up a Docker Compose environment given a compose configuration YAML:</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOZAO8awbF7bvYS9BTBcFqPBsvmwkXBa8Wd4kDsEUT1HHmr-0r2wJH1FcRj9tiQvIrwnAA1EqPA871OxlcavIugpwrjjN-ex8o4Zo5M5_H6Si7GweeSxLxdyIMwmTrZqgkfKFCXxXXhpQ/s1600/ComposeUpExample.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOZAO8awbF7bvYS9BTBcFqPBsvmwkXBa8Wd4kDsEUT1HHmr-0r2wJH1FcRj9tiQvIrwnAA1EqPA871OxlcavIugpwrjjN-ex8o4Zo5M5_H6Si7GweeSxLxdyIMwmTrZqgkfKFCXxXXhpQ/s640/ComposeUpExample.PNG" width="640" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Here's an example after the test of bringing that same environment back down:</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMOTrNrPm95bZV1JqTw-nJdpMR1jNhzISG3hony3i8_0MM4bWRGt8ubB4QW5TVV2ujynQsQpnNOYZT-x1VkafPeB62xkJw9XdB4BO_FKGpl16XEfRM8hY-ojovGxcjwiL0xPaYYNu4UzQ/s1600/ComposeDownExample.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="104" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMOTrNrPm95bZV1JqTw-nJdpMR1jNhzISG3hony3i8_0MM4bWRGt8ubB4QW5TVV2ujynQsQpnNOYZT-x1VkafPeB62xkJw9XdB4BO_FKGpl16XEfRM8hY-ojovGxcjwiL0xPaYYNu4UzQ/s640/ComposeDownExample.PNG" width="640" /></a></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In addition, there are additional convenience methods on the DockerProcessAPI that can log compose environments that are running for investigative purposes later.</span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Thanks for taking time to read this entry. Feel free to comment or contact me if you have questions. </span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Resources</span><br />
<br />
<ul>
<li><span style="font-family: arial, helvetica, sans-serif;"><a href="https://docs.docker.com/engine/getstarted/">Docker</a> and <a href="https://docs.docker.com/compose/overview/">Docker Compose</a> documentation</span></li>
<li><span style="font-family: arial, helvetica, sans-serif;"><a href="https://github.com/BreakTheMonolith/DockerProcessAPI">DockerProcessAPI</a> toolset</span></li>
<li><span style="font-family: arial, helvetica, sans-serif;">Spotify <a href="https://github.com/spotify/docker-client">Docker-Client</a></span></li>
<li><span style="font-family: arial, helvetica, sans-serif;"><a href="https://github.com/BreakTheMonolith/btm-DropwizardHealthChecks/blob/master/btm-DropwizardHealthChecks-rabbitmq/src/test/java/guru/breakthemonolith/health/rabbitmq/RabbitMQHealthCheckTestIntegration.java">Integration Test</a> Example from btm-DropwizardHealthChecks</span></li>
</ul>
</div>
<div dir="ltr" style="text-align: justify;">
<br /></div>
<div dir="ltr" style="text-align: justify;">
<br /></div>
<div dir="ltr" style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="text-align: justify;">
<br /></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com1tag:blogger.com,1999:blog-2825935617443597949.post-55389048569175471742016-11-21T02:23:00.001-08:002016-11-30T00:43:59.557-08:00Book Review: Reactive Services Architecture - Design Principles for Distributed Applications<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGmqwIADdGtGcjSF_igfYBmaHVG-JwfXJ_MSUu6SViZIS8SK3X9eXeewPKJz804NPHF_e0Q1W0jhH5gAdFPtrjHLVLSOLZX2yEoumav0_WVUriKpcZU9uu_B1aK182T1RGNPMozY3aVWM/s1600/COLL-ebook-Reactive-Microservices-Architecture.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGmqwIADdGtGcjSF_igfYBmaHVG-JwfXJ_MSUu6SViZIS8SK3X9eXeewPKJz804NPHF_e0Q1W0jhH5gAdFPtrjHLVLSOLZX2yEoumav0_WVUriKpcZU9uu_B1aK182T1RGNPMozY3aVWM/s320/COLL-ebook-Reactive-Microservices-Architecture.jpg" width="213" /></a></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">A friend of mine gave me a copy of the new book "<a href="https://info.lightbend.com/COLL-20XX-Reactive-Microservices-Architecture-RES-LP.html">Reactive Services Architecture - Design Principles for Distributed Applications</a>" by <a href="https://twitter.com/jboner">Jonas Bonér</a>. My friend was quite taken by the content and asked me what I thought of it. Having been working with microservices architecture and writing about them for a couple of years, I was intrigued and took up the gauntlet.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I was immediately struck by the word "Reactive" and wondered what was the <i><u>difference</u> </i>between a "reactive" microservice and a non-reactive microservice. I started reading the book with this question nagging me along the way.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Book Summary</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The book starts out by a traditional defining of the problem with monolithic applications; These are applications that have grown too large and complex to maintain and enhance with any reasonable ease and speed. It has a very well written introduction to the concept of microservices architecture and how it solves the problems presented by monolithic applications. The book describes basic principles of microservices architecture and also effectively compares microservices to traditional SOA in a manner that's very easy to read and understand. Principles described by this sections are:</span></div>
<div style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Microservices do one thing and do it well; they have a single functional (e.g. business) purpose.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Microservices most often evolve from monoliths; not often used for completely new applications</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The second chapter refines the single responsibility trait that microservices have and introduces us to several additional microservice principles. They are principles that are present in previous discussions of microservice architectures and aren't really new, but are explained very clearly and concisely. These principles are:</span><br />
<div style="text-align: justify;">
</div>
<ol>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Microservices act autonomously; they are context independent.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Microservices own their own state / data store.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Microservices should have location transparency; they use a service discovery mechanism so they can scale effectively and have clustering / resilience.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">In a microservice world, what <i style="text-decoration: underline;">can</i> be made asynchronous or non-blocking, <i style="text-decoration: underline;">should</i> be made asynchronous (some term this "eventually consistent").</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">In a microservice world, planning for failure is a necessity.</span></li>
</ol>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I did find the discussion on <b>isolation </b>at the beginning chapter two extremely interesting. Isolation is presented as a coupling of a service to time (when execution occurs) </span><span style="font-family: "arial" , "helvetica" , sans-serif;">and space </span><span style="font-family: "arial" , "helvetica" , sans-serif;">(where the service is hosted)</span><span style="font-family: "arial" , "helvetica" , sans-serif;">. Removing this coupling (i.e. isolating services) is a prerequisite to adhere to principles 1, 3, 4, and 5 above. I had never explicitly identified this type of coupling previously in my writing and consider it insightful. </span><br />
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The author does discuss reliance on messaging technologies as a part of the definition of a "reactive" microservice. I look at messaging as one of many techniques for making service calls asynchronous. I see messaging as a technique for supporting principle 4 and mitigating 5 and not really fundamental to the architecture. Usually, architecture definitions are principle based and not reliant on a specific implementation tactic. I agree with the author in that designers usually "assume" real-time needs instead of challenging them and making more portions of a system asynchronous. The author seems to assume the opposite. To be fair on pp 36-8 (chapter 3), the author does acknowledge that there are portions of a system that must be synchronous.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">There are reasons for using ("persistent") messaging to make service calls asynchronous rather than spawning the work in a separate thread. Work in a spawned thread will be lost if a service instance dies where it won't be if persistent messaging is used. The author really doesn't discuss this point or address why "messaging" (persistent or not) is the only option he presents for making work asynchronous.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Chapter three discusses several cross-cutting concerns that frequently come up with people newly introduced to microservice architectures. They are:</span></div>
<div style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Service discovery as a means to support location transparency.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">API Gateways as a way to manage evolving contracts over time.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Messaging as a way to support several of the principles outlined above.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Security management </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Minimizing data coupling and coordination costs</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The author does point to <a href="https://en.wikipedia.org/wiki/Event-driven_architecture">event driven architecture</a> and <a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">conflict-free replicated data types </a>(CRDTs) as being natural complements to microservice architectures. Essentially, the event-log becomes the source of truth for a microservice and the underlying database is simply a convenient "cache" of that truth. These concepts are touched on, but not really explored in depth. To be fair, these are weighty topics and likely deserve books of their own; declaring in-depth discussion of them "out of scope" for this book is reasonable.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Reviewer Summary -- Rates 4 Stars out of 5</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>This book is a great summary for those looking for an overview of microservices architecture. </i></b>All concepts are explained concisely, clearly, and in an easy to understand writing style. As Lightbend is handing out <a href="https://info.lightbend.com/COLL-20XX-Reactive-Microservices-Architecture-RES-LP.html">free </a>copies (at the time of this writing), it's certainly cost-effective.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>This book respects the readers time.</i></b> Time is a scarce resource for me as I'm sure it is for many. At 47 pages, it can be easily be read in one or two sittings.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>This book doesn't try to sell products.</i></b> When I saw the company name "Lightbend" on the cover, I was a little nervous that there would be plugs for Lightbend products. There isn't. Akka is mentioned as an implementation option as well as Apache Camel and other products. Nothing, however, that's overt marketing.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<b style="font-family: Arial, Helvetica, sans-serif;"><i>I'm not convinced that a "reactive" microservice is any different than a normal microservice.</i></b><span style="font-family: "arial" , "helvetica" , sans-serif;"> While I agree that all service calls that can be non-blocking/asynchronous</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> should be, that's not really new or different. It is a microservice design best practice to be sure, but not really a different architecture style. The book does contain mentions of </span><a href="https://en.wikipedia.org/wiki/Reactive_programming" style="font-family: Arial, Helvetica, sans-serif;">reactive programming</a><span style="font-family: "arial" , "helvetica" , sans-serif;"> and the </span><a href="http://www.reactivemanifesto.org/" style="font-family: Arial, Helvetica, sans-serif;">Reactive Manifesto</a><span style="font-family: "arial" , "helvetica" , sans-serif;">. But not really enough to link the books content specifically to those constructs for me. In the end, it's this point that kept my rating out of the five-star category.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">You might wonder why I published this review on my blog instead of on Amazon. As a book author myself in this genre, Amazon will not publish my reviews on other computer-related books. Thanks for reading this article.</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com2tag:blogger.com,1999:blog-2825935617443597949.post-77662495375319285142016-09-23T03:17:00.002-07:002016-11-30T00:56:46.546-08:00Using Java Thread Dumps to Diagnose Application Performance<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">On a holiday weekend last year, I got an emergency call from a client. One of their Java EE applications would freeze and stop servicing users within an hour after container start-up. I was called in to help investigate. I started off by requesting a thread dump and memory dump of the container once it had stopped accepting requests. It's the thread dump that I'm focusing on today. That Java thread-dump is <a href="https://github.com/BreakTheMonolith/StackWise/blob/master/StackWise-core/src/test/dumps/Thread-dump-4.txt" target="_blank">here</a> (package scrubbed to protect the client).</span></div>
<div dir="ltr">
<br /></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">During that exercise, I noticed that when I analyze thread dumps, I look for the same things. Whether it's a performance issue or some sort of freezing issue, I manually scanned the thread dump for the same types of conditions. This year, working for another client, I'm faced with a performance tuning exercise that will likely require analysis of numerous thread dumps and wasn't looking forward to the busy work. That prospect got me to do some introspection and figure out <b><i><u>exactly</u></i></b> what I look for and find or build a product that does this.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Threads that block other threads</span></h2>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Most developers know the syntax behind using Java's </span><span style="font-family: "courier new" , "courier" , monospace;">synchronized</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> keyword and that it's used to ensure that one and only one thread executes a section of code or uses a given Java resource </span><span style="font-family: "arial" , "helvetica" , sans-serif;">at one time</span><span style="font-family: "arial" , "helvetica" , sans-serif;">. I'm not going to digress into a discussion of lock monitors and coding issues; if you need a refresher, please see this <a href="https://docs.oracle.com/javase/tutorial/essential/concurrency/sync.html" target="_blank">article</a>. Most experienced developers use synchronization with extreme care as bugs with synchronization are intermittent and extremely hard to diagnose and fix.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Frequent symptoms of </span><span style="font-family: "arial" , "helvetica" , sans-serif;">synchronization</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> issues are performance problems or cases where applications freeze and no longer accept client requests. Essentially, synchronization causes other threads servicing other client requests to wait until the needed Java resource is available for use. Those waiting client threads are typically in a <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.State.html" target="_blank">BLOCKED</a> state. I immediately suspected this type of issue in diagnosing the issue I was investigating over the holiday last weekend. Here's a sample of a BLOCKED thread entry in a thread dump:</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">"http-bio-10.13.0.222-30105-exec-427" daemon prio=6 tid=0x000000001cf24000 nid=0x2054 waiting for monitor entry [0x0000000022f9c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.beans.Introspector.getBeanInfo(Introspector.java:160)
- waiting to lock <0x0000000680440048> (a com.sun.beans.WeakCache)
at org.apache.axis.utils.BeanUtils$1.run(BeanUtils.java:92)
at java.security.AccessController.doPrivileged(Native Method)</span></pre>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note that the dump explicitly lists that it's waiting on resource </span><span style="white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;">0x0000000680440048</span></span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"> which is owned by this thread:</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span></div>
<pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">"http-bio-10.13.0.222-30105-exec-289" daemon prio=6 tid=0x000000001bf06000 nid=0x21b0 runnable [0x000000002e0dd000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.privateGetPublicMethods(Class.java:2641)
at java.lang.Class.getMethods(Class.java:1457)
at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1280)
- locked <0x0000000680440048> (a com.sun.beans.WeakCache)
at java.beans.Introspector.internalFindMethod(Introspector.java:1309)</span></pre>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">It turns out that more than one thread was waiting on this resource.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span></div>
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">IO bound threads</span></h2>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">One frequent source of application performance issues are threads that are waiting on Input/Output to occur. This often takes the form of a database read or write or a service call of some type. Many developers <i><u>assume</u></i> that most performance issues are caused by slow database access and start looking at SQL queries. I do not make this assumption. Furthermore, if it is a database tuning issue, you need to identify the specific SQL that needs to be tuned. At any rate, if the source of your performance issue is IO, thread dumps can help you identify where in your code the issue is taking place. </span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Here is an example thread that's IO-bound:</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">"QuartzScheduler_Worker-2" prio=6 tid=0x000000001abc7000 nid=0x2208 runnable [0x000000001df3e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:850)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)</span></pre>
<pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">... (many thread stack entries omitted for brevity)</span></pre>
<pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> at com.jmu.scholar.dao.FormRuleDAO.findByFormId(FormRuleDAO.java:50)</span></pre>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note that within the thread stack, there's an explicit reference to application code that's initiating the IO. In this case, IO is being initiate by a database query in a specific application method. </span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note that just because the dump caught this <i>one</i> occurrence of this query, doesn't mean it's a performance issue. If, however, a large percentage of running threads are IO-bound in the same method, then this database access would become a tuning target. To tune this specific database access, a developer can focus on this one query instead of looking at all queries within the application. How to tun the database access is out of scope for this blog entry.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Performance Hot Spots</span></h2>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Most developers upon being asked how to tune an application in an interview will tell you to use a Java profiler. That answer misses the point. A profiler helps you tune a specific section of code <i><u>after you've identified the section of code that needs to be tuned</u></i>. Often performance issues show up in production and it's not possible to run a profiler on your container in production. </span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">A thread dump taken in production on an active application can help you identify which section of code needs to be tuned, perhaps with a profiler. Furthermore, thread dumps are unintrusive enough that you <i>can</i> take them in production without material impact to users. </span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">To see how dumps help, let's review how a profiler works. A profiler works by taking a thread dump periodically, perhaps every 5 milliseconds. That thread dump specifies where in your code you're spending time. For example, if your test causes the profiler to take 100 samples and method </span><span style="font-family: "courier new" , "courier" , monospace;">Work.do()</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> appears in 33 of them, then your spending 33% of your time in that method. If that's the method with the highest percentage, that is where you'll often start tuning.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In fact, thread dump data is better than profiler data in several ways:</span></div>
<div dir="ltr">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">It measures what's actually happening in production vs. a profile of a specific business process or unit test case.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">It includes any synchronization issues between threads that won't show up in a profile of one thread in a unit test case (there are no other threads to contend with).</span></li>
</ul>
<br />
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The problem is collecting the data. Yes, thread dumps are easy to collect. Counting occurrences of method references in running threads is laborious, tedious, and annoyingly time consuming.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Introducing StackWise</span></h2>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The first thing I did was scan for products that analyze thread dumps for these types of issues. There are many products that analyze thread dumps, but they tend to be interactive tools that summarize threads and allow you to selectively expand and contract them. A couple categorized threads by state (e.g. RUNNING, BLOCKED, etc.). However, none of these really looked for the three items I look for. Hence, I created the product <a href="https://github.com/BreakTheMonolith/StackWise" target="_blank">StackWise</a>, which is open source and freely available for you to use.</span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">StackWise will analyze a thread dump and report useful information on all three of these conditions. A sample of StackWise output can be found <a href="https://github.com/BreakTheMonolith/StackWise#sample-output" target="_blank">here</a>. Note that you get the following items of information:</span></div>
<div dir="ltr">
</div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The percentage of threads that are IO bound and a summary of those IO-bound threads.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Threads that are locking resources needed by other threads.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">A list of application method reference counts listed in descending order. </span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In interpreting performance hot spots, StackWise will report application methods in which you're spending the most time. Methods belonging to </span><span style="font-family: "courier new" , "courier" , monospace;">ServletFilter</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> classes can be ignored as they are often listed in all running threads. Other method mentions, however, are possible tuning targets.</span><br />
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: "arial" , "helvetica" , sans-serif;">If you analyze thread dumps in ways other than what StackWise already covers, I'd like to hear your ideas. Thanks for reading this entry.</span></div>
<div dir="ltr">
<br /></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-42283774377335716412016-09-04T07:23:00.000-07:002016-11-30T00:43:59.552-08:00All Monoliths Are Not Created Equal<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">All monoliths are not created equal. That is, not all monoliths <i>became</i> monoliths the same way nor do they have the same root causes. With this post, I'll identify different types of monoliths I've seen. This is useful as the tactics I use to break the monolith into smaller, more manageable, pieces is different depending on which category of monolith I'm dealing with. It's worth noting that some times elements of these different categories of monoliths are used in combination.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">A monolith is an application that has grown too large to effectively manage. That is, it's expensive to enhance and fix. Change to monoliths often come with unintended consequences; you fix something and other things break. Monoliths effectively marry you to a technical stack, making it difficult to grow with new technologies as they evolve and mature. Due to the size and complexity of underlying monolithic code, it's often expensive and time-consuming to bring on new developers (there's so much to learn). The same labor characteristic limits the business in that outsourcing change often isn't an option. </span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Feature-Bloated Web Application</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This type of monolith is specific to web applications. It becomes larger and more complex as new features are added. Moreover, some of those new features weren't considered when the application was originally designed. Consequently, those new features are often inelegantly "tacked on" and don't really conform to the initial design of the application. Often, there's no time and budget to revamp the application design to add new features in ways that are easy to maintain and support. </span><span style="font-family: "arial" , "helvetica" , sans-serif;">The consequence for this is often technical debt in the form of unwanted coupling.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>New features add complexity to all parts of the application. </i></b>New features impact not only the user interface but server-side backing code that supports that interface and the underlying database. Often that server-side backing code and database is tightly coupled with the user interface and "assumes" that the business processes currently implemented by the interface. Often, it is assumed that these business processes are static and rarely change. Unfortunately, that is often not the case. Developers react to the additional complexity in predictable ways.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Beware of undocumented developer assumptions.</i></b> These <i>assumptions</i> are often made for developer convenience. Those assumptions make the new feature easier to code. I don't blame them; the extremely large feature set is often too difficult to keep in one person's head. Developers, as their assumptions produce initially more streamlined and easy to read code, believe that they are making a positive contribution. Unfortunately, they are also inadvertently creating land-mines for other developers who aren't aware of those underlying assumptions and need to change that code for some other feature addition. As most of these assumptions go undocumented or even unidentified, other developers working on that section of code are often unaware of all the consequences of their changes.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Most monoliths don't measure feature usage. </i></b>As a consequence, features, once implemented, never get removed. It would be logical to remove features that aren't used or aren't used often as the complexity for having that code remain will negatively impact your ability to add new features down the road. Sadly, getting budget for removing features is hard to do in most organizations as the benefit to doing so is hard to measure and too intangible.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"> It's worth noting that often the design of the underlying database often becomes bloated along with the web application. Due to the large scope of business it supports, the underlying database often becomes an Achilles heel for the organization. Not only is database refactoring much more costly and difficult than refactoring code, it's not uncommon for that database to be used by multiple applications. The implication here is that change to the underlying database often has consequences for more than the monolithic application it was originally created for.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Data Store Strangle</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Despite the length of time that databases (relational and non-relational) have existed, database design skills are woefully inadequate at most organizations. As a result, you often get a process-oriented database structure. That is, a database structure that is specific to processes currently implemented by the application that database supports. Usually, the mindset is that the database can be 'refactored' along with code as new features are implemented.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i style="font-weight: bold;">Database design impacts application code. </i>At the risk of stating the obvious, applications have code dedicated to reading and writing to the database. Hopefully, database code is restricted to a data access layer within the application, but not all developers adopt the practice of separating data access concerns. However, the readability of that data access code is directly impacted by the quality of the design of the underlying database. If the database structure is complex to the point that its not understood, chances are the data access code that uses that database will be just as hard to understand. Furthermore, if the application uses a different domain mode than was implemented in the database, some type of data structure translation is present in data access code. The short story is that the database design used can greatly increase the size and complexity of data access code.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Refactoring databases is harder and more time-consuming than refactoring code.</i></b> There are a couple of false assumptions buried in that line of thinking. First, it is assumed that refactoring a database is as easy as refactoring code. Not true. Any database refactoring often must take place while supporting existing code. Also, any database refactoring also must have some type of data conversion as a part of it. </span><span style="font-family: "arial" , "helvetica" , sans-serif;">Another way to look at it is that code only has state that's transient. That is, its state exists only at run time. Database state is persistent. Changes to that database must include converting the data that was stored into its new format.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Refactoring databases has more risk than refactoring code. </i></b>If a code refactoring is discovered to have a defect, backing out that change is usually an option. Database refactoring changes usually don't have this safety net. Yes, you can restore a version of the database that was taken before the refactoring was implemented, but users will loose any data entered while that refactoring was in place. Unless you code and test a reverse conversion, implementing database refactoring changes commits you (pun definitely intended) past the point of no return. </span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">The Over-Engineered Tarball Component</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Some developers can't resist over-engineering internal components they write.</i></b> Often that happens because they are bored and looking for something more interesting to do. I can't fault them for that motivation. Unfortunately, when those mental musings get implemented in applications, additional complexity often results. This contributes to an application becoming a monolith.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I've seen a case where such a component was written to validate cash transfers from account to account. While laymen to this business process might look at such a transfer action as simple, those of you deeply embedded in bank know differently. The tax ramifications and considerations are incredible depending on how that transfer is recorded. In fact, we had over 100,000 permutations and combinations of transfers that needed to be verified and tested. Anyway, the component in question utilized the same type of bit-switching algorithm that UNIX uses for file permissions, but with many more dimensions than just user, group, and all permissions. Anyway, when that person left, nobody could understand the component nor could they effectively change the rules it applied. Long story short: the application was held hostage by this embedded, but critical component.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Overly complex components have a way of creating complexity in the surrounding parts of the application.</i></b> In my example, we had code to correct the validation verdict in places where it came up with the incorrect answer. This was needed as nobody knew how to change the overly-complex component handling validation.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>The business process implemented by the component is often not understood.</i></b> That is, replacing such a component is often more difficult as the business rules it implements are often not understood. It's hard to code a replacement without a specific target.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">The Distributed Monolith</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Separate services are supposed to increase developer velocity by reducing the size and complexity of code that needs to be changed.</i></b> We all know that this doesn't always work out. We've all seen situations where feature additions require coordinated changes across multiple components. We've all seen that even though these services are theoretically <i>decoupled </i>and should be able to be changed/enhanced/deployed independently, that this doesn't seem to happen much of the time. <i>The reason is that these services are not truly decoupled.</i> Let me explain.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Developers have a desire to consolidate code.</i></b> We've incented this in developers for years by promoting DRY (Don't Repeat Yourself). Carrying this idea forward, developers consolidate repeated code whenever they find it. Invariably, business code gets consolidated into common libraries. It starts with implementing a service contract in code, making a library out of that, and sharing it across the publishing service and all consumers. It often progresses from there to include common business code that is listed as a dependency for multiple services. this common business c</span><span style="font-family: "arial" , "helvetica" , sans-serif;">ode, which usually is used for manipulating common business data, tends to also be consolidated. Perhaps by publishing that logic in common libraries, and is used by several services.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Developers are often quite proud of the code consolidation. Unfortunately, whenever that common code changes, it forces a coordinated redeployment of all consumers along with the producer. In this world, there's no such thing as a non-breaking change. </span><span style="font-family: "arial" , "helvetica" , sans-serif;">In essence, </span><span style="font-family: "arial" , "helvetica" , sans-serif;">DRY, if used for business logic including service contracts, increases coupling between services. If that common code is purely technical and services are completely free to decide when they upgrade to new versions, then there's no coupling issue.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Distributed monoliths are caused by coupling between services that forces change across multiple services. </i></b>In these cases, you're not getting the benefit of breaking the world up into smaller services. In fact, just the opposite. You get all the disadvantages of a highly coupled monolith along with the additional overhead of managing a larger number of services. In other words, you forgo all advantages of implementing microservices and pay more for the privilege.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i><br /></i></b></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Separate common business functionality into its own service instead.</i></b> Deploy it once. call it from whatever services need it. Only maintain it once. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Don't create common code that represents your service contracts.</i></b> Consumers usually don't need all attributes in the request or response for a REST service call. Let's take an example of a REST call that returns customer information. For each customer, there might be 50 attributes. Of that, possibly a consumer only needs five or six. That consumer call should not break if new attributes are added or if an attribute that the consumer ignores is changed or withdrawn. If that contract is represented by common code, that common code will necessarily list all the attributes that the consumer ignores and such changes will be a breaking change. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><i>Services should always be able to choose when they upgrade included libraries.</i></b> Always - no exceptions. </span><span style="font-family: "arial" , "helvetica" , sans-serif;">To get the benefits of breaking the monolith up into several services, you cannot permit any common code that could force a deployment of a service.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;">Trailer</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In forthcoming blog entries, I'll suggest tactics for breaking each of these types of monoliths into more manageable applications. Thanks for reading.</span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-42862913464776784172016-08-27T06:38:00.002-07:002016-11-30T00:44:23.232-08:00A Pleasurable Journey into Text Translation using ANTLR4<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">For one of my clients, I needed to spec out a REST service API for developers. Although they had standardized on the <a href="https://apiblueprint.org/">API Blueprint</a> product, I feel compelled to look for alternatives for specking out future REST APIs. The input needed for API Blueprint is just too verbose and requires far too much time. If I expected to spend less time specking out REST APIs, this might not be an issue. Or at least not as much of an issue. </span><span style="font-family: Arial, Helvetica, sans-serif;">I also checked out <a href="http://swagger.io/">Swagger</a> and <a href="http://raml.org/">RAML</a> and found the same verbosity issue; I just don't have time for that on an ongoing basis. There's a good review of these products that I found particularly useful <a href="https://restful.io/a-review-of-all-most-common-api-editors-6a720dc4f4e6#.fs4jkzxj4">here</a>.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">My thoughts turned to a syntax for specking REST APIs that would be far more streamlined and concise. Within an hour, I had a draft of a syntax (example below) that would be far less verbose and that I could increase my productivity significantly per API. The problem would be writing something that could interpret specifications in this format and could then generate the verbose XML , Markdown, or YAML syntax for one of the other API designer products mentioned above. It turns out that there are products that specialize in text translation. <a href="http://www.antlr.org/">ANTLR</a> is the most popular of these products right now.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">The way ANTLR works is that you specify a <i><b>grammar</b></i> that describes the text you want to translate. ANTLR uses that grammar to generate Java code that can take text in that format and interpret it for you in the terms that you specified in the grammar. For instance, I define a REST Resource block in my grammar and told it about the syntax for different operations and the json arguments they accept as input or emit as output. ANTLR generated code will analyze the specifications I write and convert that to a Java object format that I can more easily read, interpret through code, and generate useful translated output with the help of a templating technology, such as <a href="http://freemarker.org/">Freemarker</a>. An example grammar for the Java language as an example can be found <a href="https://github.com/antlr/grammars-v4/blob/master/java/Java.g4">here</a>.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">It turns out that specifying a grammar wasn't as easy as I thought it would be going in. What, in my mind, is a simple context is actually quite complex when you break it down into constructs that a product like ANTLR can understand. Essentially, you need to specify all whitespace (characters to ignore) and comments if the syntax is to support them. All special keywords and rules that govern when those keywords are expected to be used also need to be specified. At this point, I should have just backed down from this idea and suffered through one of the more verbose solutions. However, by this point, I'm far too interested in how structured text gets specified and what's possible by interpreting it through code to stop.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">I'm part way through this project and will open source it once complete. For those taking similar text translating journeys with ANTLR, I have ferreted out some techniques that helped me immensely.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><b><i>Write and test the Lexer portion of the grammar first.</i></b></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">ANTLR breaks up grammars into two pieces: a "Lexer" and a "Parser". A "lexer" understands what characters and keywords are important for what you're doing and skipping any unneeded whitespace. It also formats those characters/keywords internally as "Tokens" so that it can be used for more sophisticated translation later on. </span><span style="font-family: Arial, Helvetica, sans-serif;">A "Parser" applies rules to important characters and keywords to interpret context. For example, a REST resource definition doesn't make sense in the data type structure section of my proposed REST API specification syntax.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">As the parser uses lexer output; it's important to make sure the lexer portion of your grammar tests out first. Any testing of the parser at this point is premature. Assertions in your lexer test should be:</span></div>
<div style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Make sure all characters and keywords are recognized.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Make sure that the lexer identifies characters and keywords correctly. For instance, I had a bug early on where the keyword 'Resource' was recognized as a string literal. In my syntax, 'Resource' has a special context and meaning.</span></li>
</ul>
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">You can test the lexer generated from your grammer by iterating through the Tokens generated. Any unrecognized tokens shouild cause a test failure. If the lexer doesn't recognize your special characters and keywords (e.g. doesn't identify the correct number of keyword 'Resource' from your test sample), then it should also cause a test failure. </span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><b><i>Write Parser rules iteratively from general rules to more specific rules.</i></b></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">Parser rules apply context to the tokens identified by the Lexer. I found it much easier to start with very general parser rules and get those working. For example, my syntax has two main sections: a bounded context section that describes resources and operations and a Types section that describes all data types used by the API. My first iteration of the parser rules just identified the two sections. That isn't enough to do what I need, but I didn't leave it there. </span><span style="font-family: Arial, Helvetica, sans-serif;">Over time, I specified the portions of both sections and progressively describe them in more detail.</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">In other words, parser rules describe a section of your input text. </span><span style="font-family: Arial, Helvetica, sans-serif;">The first test for parser rules can be simple; just test the start line/column position and end line/column position for each parser rule. If those are correct, then you can describe more specific rules that carves up the larger sections in the first iteration. Each parser rule you write has a value object specifically generated for it. That value object has the starting and ending token for the section it covers (you can get the starting and ending positions from those tokens).</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">There are a few points that aren't obvious about the ANTLR product to remember.</span></div>
<div style="text-align: justify;">
</div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Lexer rules have UPPER_CASE names. Parser rules are lower case</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">At least one parser rule should apply to the entire document (minus skipped whitespace).</span></li>
</ul>
<span style="font-family: Arial, Helvetica, sans-serif;">I'll post additional reports and publish the resulting work via Github when complete. I'm still midway through this effort.</span><br />
<h2 style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">An Example REST API Specification Syntax</span></h2>
<table class="highlight tab-size js-file-line-container" data-tab-size="8" style="background-color: white; border-collapse: collapse; border-spacing: 0px; box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; line-height: 21px; tab-size: 8;"><tbody style="box-sizing: border-box;">
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="2" id="L2" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC2" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">#
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="3" id="L3" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC3" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">#student.spec - Student REST API
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="4" id="L4" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC4" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">#
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="5" id="L5" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC5" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="6" id="L6" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC6" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">Bounded Context: Student Information {
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="7" id="L7" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC7" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> Resource: Student // Everything about current and past students
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="8" id="L8" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC8" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> Operation: /student - POST, json, student //Creates a student
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="9" id="L9" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC9" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> httpStatus: 201,400
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="10" id="L10" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC10" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> return: json, studentId
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="11" id="L11" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC11" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> Operation: /student/{studentId} - PATCH, json, student //Update student (only those attributes provided)
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="12" id="L12" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC12" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> httpStatus: 200,400,404,405
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="13" id="L13" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC13" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> Operation: /student/{studentId} - DELETE //Delete student
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="14" id="L14" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC14" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> httpStatus: 200,400,404,405
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="15" id="L15" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC15" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> Operation: /student - GET //Finds a list of students by status
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="16" id="L16" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC16" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> parms:
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="17" id="L17" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC17" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> status - string[] // Status values to search by
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="18" id="L18" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC18" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> httpStatus: 200,400
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="19" id="L19" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC19" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> return: json, student
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="20" id="L20" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC20" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> Operation: /student/{studentId} - GET //Finds a student by their id
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="21" id="L21" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC21" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> httpStatus: 200,400,404
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="22" id="L22" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC22" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> return: json, student
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="23" id="L23" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC23" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">}
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="24" id="L24" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC24" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">Types: {
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="25" id="L25" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC25" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> student {
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="26" id="L26" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC26" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> studentId - required, string // student Identifer that uniquely identifes a student
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="27" id="L27" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC27" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> firstName - required, string $$Bill
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="28" id="L28" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC28" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> middleName - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="29" id="L29" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC29" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> lastName - required, string $$Williamson
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="30" id="L30" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC30" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> title - enum{Mr, Ms, Mrs}
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="31" id="L31" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC31" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> birthDate - required, date
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="32" id="L32" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC32" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> primaryAddress - required, address
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="33" id="L33" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC33" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> schoolAddress - address
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="34" id="L34" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC34" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> primaryPhone - required, phone
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="35" id="L35" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC35" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> cellPhone - phone
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="36" id="L36" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC36" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> emailAddress
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="37" id="L37" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC37" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> status - enum{Applied, Accepted, Active, NonActive}
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="38" id="L38" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC38" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> }
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="39" id="L39" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC39" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> address {
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="40" id="L40" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC40" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> streetAddress1 - string $$123 Testing Lane
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="41" id="L41" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC41" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> streetAddress2 - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="42" id="L42" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC42" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> City - string $$Somewhere
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="43" id="L43" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC43" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> StateCode - string(2) $$IL
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="44" id="L44" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC44" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> zipCode - int(5)
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="45" id="L45" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC45" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> zipCodeExt - int(4)
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="46" id="L46" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC46" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> }
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="47" id="L47" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC47" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> phone {
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="48" id="L48" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC48" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> countryCode - int(2)
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="49" id="L49" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC49" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> areaCode - int
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="50" id="L50" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC50" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> prefix - int
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="51" id="L51" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC51" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> line - int
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="52" id="L52" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC52" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> extension - int
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="53" id="L53" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC53" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> }
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="54" id="L54" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC54" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> classSection {
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="55" id="L55" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC55" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> title - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="56" id="L56" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC56" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> discipline - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="57" id="L57" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC57" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> courseNbr - int
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="58" id="L58" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC58" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> building - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="59" id="L59" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC59" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> room - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="60" id="L60" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC60" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> time - string
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="61" id="L61" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC61" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;"> }
</td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="62" id="L62" style="-webkit-user-select: none; border-color: rgb(238, 238, 238); border-style: solid; border-width: 0px 1px 0px 0px; box-sizing: border-box; color: rgba(0, 0, 0, 0.298039); cursor: pointer; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC62" style="box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre; word-wrap: normal;">}</td></tr>
</tbody></table>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0tag:blogger.com,1999:blog-2825935617443597949.post-86100976579256918452016-07-20T06:43:00.002-07:002016-11-30T00:47:33.346-08:00A Cute Trick with Generics to Avoid Unwanted Casts<div style="text-align: justify;">
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">I ran
into a cute trick while coding a unit test on a Spring application the other
day that I'd like to share.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">Consider
these lines of Java code.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDBCffd4cTc7ujALaYVgGZ5i6kUh9LX401WN-Vi2I8BfJHSKZZumMS_JG3mM7yqtLrJygPziorQ058mCQhb2PSSbTLuDwoDph09YRNd2cy_4_SUMV8rGBgWnvBvuXG7QGRvXDaGpoWdVU/s1600/codelines1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="43" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDBCffd4cTc7ujALaYVgGZ5i6kUh9LX401WN-Vi2I8BfJHSKZZumMS_JG3mM7yqtLrJygPziorQ058mCQhb2PSSbTLuDwoDph09YRNd2cy_4_SUMV8rGBgWnvBvuXG7QGRvXDaGpoWdVU/s400/codelines1.png" width="400" /></a></div>
</div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">The first
line requires a cast. the reason is that the method returns an<span class="apple-converted-space"> </span></span><span style="font-family: "courier new"; font-size: 13.5pt;">Object</span><span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">. While the cast
isn't the most labor intensive construct, but it makes code less clear and is
inconvenient to developers.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">Note that
the second line does *not* require the cast making it much more convenient for
developers. the secret is that Spring's ReflectionTestUtils uses generics.
take a peak at how<span class="apple-converted-space"> </span></span><span style="font-family: "courier new"; font-size: 13.5pt;">invokeMethod</span><span class="apple-converted-space"><span style="font-family: "arial" , sans-serif; font-size: 13.5pt;"> </span></span><span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">is defined.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_ISesSXEieOBb0vsTaP-0fHUhN0K8er9xBH8FbuXjelQFvysHAWd6MsCIe8cRdlOBXwtMiVAKDCrWLa5bkuB3gb7NXZC7KeOBJ6SxYahhnIptyMqJKB9F9LG8CbRNSrc5WQPR-jcdpgQ/s1600/codelines2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="12" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_ISesSXEieOBb0vsTaP-0fHUhN0K8er9xBH8FbuXjelQFvysHAWd6MsCIe8cRdlOBXwtMiVAKDCrWLa5bkuB3gb7NXZC7KeOBJ6SxYahhnIptyMqJKB9F9LG8CbRNSrc5WQPR-jcdpgQ/s400/codelines2.png" width="400" /></a></div>
<div align="center" class="separator" style="margin: 0in 0in 0.0001pt; text-align: center;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">In
effect, Java infers the type of the value returned by the definition of the
variable it is assigned to. In this case, it's a<span class="apple-converted-space"> </span></span><span style="font-family: "courier new"; font-size: 13.5pt;">String</span><span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">This is convenient
for developers in that there's less to type, but also more easily read.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">You
should not that the developer is<span class="apple-converted-space"> </span><b><i>required<span class="apple-converted-space"> </span></i></b>to know what type of value is
returned. The following statement will generate a<span class="apple-converted-space"> </span></span><span style="font-family: "courier new"; font-size: 13.5pt;">ClassCastException</span><span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcZXU1pxccS1IMVDn1xak4j-xx9CJQN6V-mIfz45ceyv7IQzXcHE3ayhKBttnPSYCcOLayVyJ_SYLR6GrCHjZcae1YHvxHdSr0ZqXH0p9fLUR2-Z4ar99JQPbpq1CF8I6LVZ7mVc9QjRI/s1600/codelines3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="33" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcZXU1pxccS1IMVDn1xak4j-xx9CJQN6V-mIfz45ceyv7IQzXcHE3ayhKBttnPSYCcOLayVyJ_SYLR6GrCHjZcae1YHvxHdSr0ZqXH0p9fLUR2-Z4ar99JQPbpq1CF8I6LVZ7mVc9QjRI/s400/codelines3.png" width="400" /></a></div>
<div align="center" class="separator" style="margin: 0in 0in 0.0001pt; text-align: center;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">This
might seem like a disadvantage to using generics in this way. I don't think so.
In either case, the developer needs to understand the data type that will
actually be returned.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<br />
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: "arial" , sans-serif; font-size: 13.5pt;">Just
thought I'd pass this tid bit along. I'll certainly think more about using
generics in this way in APIs I write.</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
</div>
Derek C. Ashmorehttp://www.blogger.com/profile/02379227156744675210noreply@blogger.com0