Outline :

• • Constraints and origins of Traditional SW Development
  • Roots of SW Development Agility
  • An overview of Infrastructure as Code (IaC) & Infrastructure Automation
  • From DevOps to Cloud Native w Microservices & Containers
  • Today’s DevOps Enabling Technology & Benefits (Ansible, Chef, Puppet, Terraform, Docker, Kubernetes)
  • Level setting & Challenges of today’s FOSS & DevOps movements
  • Progression towards the HolyGrail of CI/CD (Continuous Integration/Deployment with Jenkins / Spinnaker)

Constraints and Origins of Traditional SW Development

The past, tried-and-true, Software Development “Waterfall” Model is where most Enterprise Application Environments have been born :

WHAT was MISSING before the late ~1990’s+ ?

(noting that Silo’d deployments and dedicated HW was best practice)

• No Wide-spread access to the Internet via HTML/Web Browser – Which wasn’t available until after the 1994 NCSA (National Center for SuperComputer Applications) release of Mosaic, 1994’s Netscape release of Mozilla, and 1995’s Microsoft release of Internet Explorer.
• No/little Re-Use of assets via large public repositories (Apache, Github, javascript/ npm, Container/ VM images, etc) to leverage elsewhere for Development.
• Minimal FOSS (Free / Open Source Software) w/o SW Licensing limitations/restrictions (GNU, .. pre-Apache).    Don’t forget.. that the term “Open Source” wasn’t even coined until 1998 !!
• Many enabling technologies were missing (such as Java’s cross-platform portability) along with a variety of tools to offer low-cost, Rapid/Agile capabilities (Polyglot dev w Python/ Javascript/ Node.js/ JSON.., Virtualization/ OS Containers, etc) restricted and constrained SW Development within “Silo’d” environments bounded within platforms.  (for more details see my article Cloud Architecture 101)
• Continuous Improvement wasn’t yet in vogue (see my article regarding Enterprise Architecture).
• Automation of Testing and promotion from Dev -> Test -> Prod .. didn’t exist beyond ~C/ “makefiles”.

The roots of SW Development “Agility”

(this section is taken from our article : Enterprise Architecture 101 : Part 1 – From Frameworks & Methodologies.. to Agile Cloud Enablement *)

2001 – The “Agile Manifesto”, aka the “Manifesto for Agile SW Development“

As described in the History of Agile, the Spring of 2000 initiated talks at a ski resort ( ), where several in the SW development community gathered and started to come to a shared emphasis to promote/progress ideas in “Light methodologies, such as Extreme Programming, Adaptive Software Development, Crystal, and SCRUM“.   While this initial meeting didn’t result in one formal consensus, it did result in several articles that started conversations.

.. Then on February 11, 2001 at the lodge in Snowbird (ski resort), seventeen members from this extended community did align their objectives to become the “Agile Alliance”, and ultimately together wrote the Agile Manifesto and the 12 principles of Agile SW Development.

Note that the 12 principles of the Agile Manifesto came before and is distinctly different from the 12 Factor Application methodology that several Heroku developers contributed to and published in 2011 by the Heroku Co-Founder Adam Wiggins.    The latter (12 Factor App) lends itself toward more prescriptive SW development as a service or “stateless” microservice guidelines for Application Architecture (stateless, don’t hard-code config info but store locally, isolate dependencies, scale horizontally, and build-deploy-run in an iterative model..).   The 12FactorApp principles come more into context when discussing the post-DevOps (post 2008) movement where “Cloud Native” development and concepts flourished (see the next section).   Note that while stateless was the initial focus, the reality within the Enterprise space has begun to extend the need for “Stateful” cloud-native architectures and capabilities, where data locality contributes to enterprise business critical requirements & SLA’s (latency, availabilty, regulatory compliance, security, etc).

As you can see from the above example, the founders of Agile came from the SW development world, but adopted and incorporated MANY ideas and leveraged the best of several models (from several disciplines beyond SW development) to develop Agile methodology (Continuous Improvement, Small teams, customer directed discovery/req’s, end-to-end ownership of SW, iterative/regular readouts on progress, etc).

Ironically, several years prior (1994), while leading my company’s end-to-end SW re-architecture and development efforts, I published my MBA dissertation on agile SW Development through Continuous Improvement and TQM that incorporated most of these concepts from a real-world vantage point.   Though I never published it beyond my University, I was on the right track and got my MBA (since then it’s been a part of required course materials).    Since that point in time, these themes and principles have become a part of my DNA 😉 .

Taking a step back and being fair, Agile (more holistically than only within the SW development specific context) is neither a Methodology, nor a specific Framework or process, but more so a collection of principles and a philosophy of deploying and reducing wasted cycles.. offering more rapid continuous value.

To be fair, Agile is neither a Methodology, nor a Framework or process, but more so a collection of principles and a philosophy of deploying and reducing wasted cycles.. offering more rapid continuous value.

*As such, we need to acknowledge that Agile & Continuous Improvement themes came from and built upon movements, as we discuss in our article : Enterprise Architecture 101 : Part 1 – From Frameworks & Methodologies.. to Agile Cloud Enablement **

SDLC with Agile Methods :

SDLC with Agile is a more traditional “Enterprise” SW development methodology applying “Agile” methods :

Infrastructure as Code (IaC)

Upon the arrival of tools like Ansible, Chef,  Puppet, and the more comprehensive Terraform (one of many Hashicorp tools), the concept of Infrastructure as Code was born (aka automation of Operations and Configuration Management tasks).   Together these tools enabled developers / sysadmins to script & REUSE the former manual command-line driven process of maintaining/configuring infrastructure.    Upon use and re-use/extensions of these tools / techniques, the PROCESS of building & maintaining systems and infrastructure (deployment environments) began to closely resemble the way that SW developers build/ maintain their applications and source code.    Note in the diagram below from edureka!, that these tools and frameworks typically come as either a “push” or “pull” (poll) model.

Infrastructure Automation gains acceptance and comes in many forms

At the same time that IaC was gaining traction, several advancements and many competing efforts, tools and frameworks to automate and centralizing the Management of these many virtualized commodity environments grew rapidly.

Many ad-hoc tools began to fill the void in the Application Container space (today’s Docker and competing CoreOS rkt) offering a lighter-weight way to virtualize, image, and rapidly deploy services without full-fledged heavy weight VM’s/ Hypervisors.

Given that consolidating workloads (or constituent services) within Containers ultimately shares ONE underlying OS Kernel, if/when that OS goes down or fails, all of the Containers that rely upon it will also go DOWN.     This is one of the many architectural considerations that need to be considered up-front when doing Cloud-Native Development, or Modernizing / Re-Factoring Legacy Applications.   See my other Cloud and Application Architecture articles for an in-depth understanding.

Hence, when deploying Applications/Services to Containers, one must also factor in the RAS (Reliability Availability Scalability) capabilities, in addition to Orchestration, Scheduling, and Monitoring (for faults, etc).    This is the realm that Kubernetes has become the standard for managing.

Kubernetesalso offers you an additional layer of abstraction to manage above the underlying hardware (or IaaS Infrastructure as a Service layer) that ultimately host the VM’s or bare-metal infrastructure running the container workloads. 

You can think of Kubernetesas analogous to a basic subset of VMware’s cVenter VRA/ VRO (vRealize Automation and Orchestration packages for managing VM’s), but targeting Container centric management (and with super-set components like Knative/ Istio, a micro-services ecosystem for serving, building, and eventing).    While Kubernetes(also abrevited K8’s) can provide additional structure, orchestration, and management mechanisms for container-based deployments, the amount of time and effort required to manually configure & deployment K8s (and ensure it’s secure and extensible) can be a very daunting process.

For this reason, many have moved to “canned” K8s frameworks such as PKS, OpenShift, Heptio(acquired by VMware), Rancher, etc.. or considered utilizing integrated Hybrid/ Public cloud platform services (eg.  AWS EKS managed control plane, Google GCP’s GKE, or Azure’s AKS services).

While VMware has been very aggressive in extending it’s local/private customer SDDC/VCF deployments into the public cloud space (eg. VMware cloud on AWS), it should also be noted that VMware has acquired several companies recently for container/ kubernetes capabilities (Heptio, including 2 of the 3 Kubernetes co-founders), and will be offering Kubernetes and Container management within vCenter out of the box (via VMware’s Project Pacifica and Tanzu) in the near future.

Related to VMware Kubernetes/Containers futures and Dev/Ops orchestration (as presented at VMworld’19), with VMware’s recent acquisition of Bitfusion, you can most likely expect to see integrated GPUaaS/FPGAaaS capabilities not too far down the road.    This can be thought of like how you provision and align fractional CPU’s (cores) to VM’s today (or as noted above to containers w K8’s coming), you will also be able to provision and spin up new fractional GPU/ FPGA cores to VM’s (and/or containers) within this same managed framework over the network.

Abstraction is alive and well, and continuing the consolidation and simplification of managing several disparate components/systems/environments from fewer and fewer interfaces, where a unified SPOG (single pane of glass) interface will be the preference for most.

Coining the term “DevOps”

.. Means going from Isolated Developers & Operations teams (separate tool-sets, environments, and systems)  :

… To a UNIFIED approach of Integrating the 2 Environments and Applying “Continuous Improvement” principles

How does Agile “DevOps” Compare to Traditional Development ?

When Patrick DuBois and Andrew Clay Schafer met at the Agile2008 Conference for “Agile Infrastructure”, the connection to DevOps was born.    Patrick later coined the term “DevOps”, to represent the merging of Development and Operations.

As a contrast to more traditional Development models, today’s DevOps is intended to bypass the serial nature of Development and disjointed Operations.. Integrating them together and focusing around “Automation” of Testing / Integration, decreasing the time to Develop-> Deploy.

In terms of the contrast between a traditional “Waterfall” development model vs. Agile SW Dev, see the diagrams below :

An Overview of today’s DevOps :  Enabling Technology & BENEFITS :

At the same time that technology was rapidly advancing, we need to look back and recognize the impact and benefits of the economic down-turn that simultaneously hit all with several years of Global Recessions & recovery (that we still feel the remnants of today).   We need to realize that in part, this down-turn in Corporate spending led to many of today’s movements toward Efficiency, and doing More with Less  :

• FOSS (Free and Open Source Software & Frameworks, as opposed to non-OSS alternatives such as MS .NET-ASP.NET, WebAPI, SQL Server, C#). Stated another way.. FOSS = NO Licen$ing fees nor single-vendor lock-in.
  • LAMP (Linux, ApacheWeb, MySQL DB, and PHP [and/or commonly Python/Perl]), not to mention frameworks built upon these technologies dominating Content Mgmt (WordPress), IoT (Kafka), BigData (Hadoop), etc..
  • MEAN (MongoDB, Express.js, Angular, Node.js), which offers a framework for dynamic Javascript web app development.
  • + MANY Other OSS Stacks & Frameworks.. such as Pyramid, Django, etc..

• SW Development adoption of lighter-weight “Polyglot” (blended multi-language) scripting in Python, Javascript,.. (vs. J2EE/C , SOA, for Apps)

• Application Container Deployment models/ frameworks (Docker/ Kubernetes).

• Agile Project Management / SCRUM / Kanban   (doing more with less.. more rapidly, Continuously improving = better products/services getting to market FASTER !)

• Dev + Ops :   Continuous Improvement & Delivery Mechanisms (which has led to “Agile” development tools offering more independent developer “ownership” of not only Development, but also the end-to-end lifecycle Operational management aspects from Dev -> Test -> Prod ).  (eg. integrated DevOps toolsets such as : Jenkins, Wercker, etc..).

• The movement to “Cloud Native” Development :  from “Monolithic” (heavy-weight) Enterprise Application Stacks (SOA, ESB, App Servers w shared JVM’s) to the independent rapid/automated Dev/Test/Ops (CI/CD) of many smaller “Micro-Services” that can run as stand-alone services, as well as the leap to Fn /FaaS (Function as a Service) or “Serverless” Architecture options (offer the most pure from of Utility, low cost computing.. IF/ When/ Where it can be utilized).

• API Repositories (beyond FOSS packages/frameworks and Docker Images) enabling more rapid development cycles by re-using already existing “standard” API’s, as shown in the following diagram from Bessemer Venture Partners :

Today’s top FOSS Developer Tools & Frameworks for Automation :

• Chef is a ruby based configuration management tool. You might have come across the term ” infrastructure as code”, which means configuration management. Chef has the concept of cookbooks where you code your infrastructure in DSL (domain specific language) and with a little bit of programming. Chef provisions virtual machines and configures them according to the rules mentioned in the cookbooks. An agent would be running on all the servers which have to be configured. The agent will pull the cookbooks from the chef master server and runs those configurations on the server to reach its desired state.

• Puppet is also a ruby based configuration management tool like chef. The configuration code is written using puppet DSL’s and wrapped in modules. While chef cookbooks are more developer-centric while puppet is developed by keeping system administrators in mind. Puppet runs a puppet agent on all server to be configured and it pulls the compiled module from the puppet server and installs required software packages specified in the module.

• SW Repositories & Tools (git/ Github, GoogleCloud Source Repositories, Altassian bitbucket, npm / javascript repo, etc.)

Given that github is among the top repo’s, no DevOps discussion is complete without an overview of what it offers and some of it’s key concepts and capabilities :

• Application Containers  (eg. Docker, CoreOS rkt) vs. traditional VM’s
  

For more information, ​Seehttps://www.docker.com/what-docker

• • Jenkins is a java based Continuous Integration tool for faster delivery of applications.  Jenkins has to be associated with a version control system like GitHub or SVN.  Whenever new code is pushed to a code repository, Jenkins server can build and test the new code and notify the team for with the results and changes.   While Jenkins is no longer simply a CI build tool, that is it’s origin of core strengths and capabilities.   Jenkins is been used as an orchestrationtool by building pipelines for the application provisioning and deployment.  Its new pipeline as code functionality lets you keep the CI/CD pipelines as a complete code.  **(See below for Jenkins integration examples within the CI/CD sections) **