Discover faster, more efficient performance monitoring with an enterprise APM product learning from your apps. Take the AppDynamics APM Guided Tour!
Summary
With more than a decade of experience working on DevOps practices, AppDynamics’ consultant Sean Stanford has witnessed first-hand how organizational practices, philosophies and tools can impact a company’s ability to deliver applications and services. Here’s his cautionary tale of one company that had not yet embraced DevOps, the pain it endured as a result, and how it eventually uncovered the source of its problems.
For planning new software projects and maintaining existing deployments, DevOps helps organizations creates stronger ties between all stakeholders throughout the development lifecycle. So it’s no surprise that DevOps is rapidly gaining popularity around the world. In the most recent Global Development Survey from Evans Data, 72% of developers reported working in a formal DevOps environment, ranging from specific development operations interactions to enterprise-wide implementation.
DevOps practices may vary by organization, of course (for a quick overview, check out our AppDynamics primer).
In my experience, organizations that fail to embrace DevOps do so at considerable risk. Not long ago, a major real estate developer reached out to AppDynamics for help. The company’s .NET installation was having problems with a third-party web asset management library, which had specific write-to-disk configuration requirements.
These requirements were configured properly in the development environment, but not in production. And because because Dev was siloed off from Prod—with no process for keeping these environments in sync—the company was unaware of the oversight.
This end result: ongoing performance problems in production. And because IIS auto-restart settings recycle the application pool after a time interval, the company was unable to identify the root cause of ongoing application instance crashes.
Finding the Cause
The company contacted AppDynamics, asking if we could figure out why the application kept crashing. We instrumented their system, which hadn’t run APM software before. Upon reviewing the AppDynamics system logs, we immediately found the source of the problem: the application didn’t have permission to write to disk in the production environment.
This discovery spurred the company to undertake a lengthy internal testing and environment comparison process, which uncovered numerous configuration differences between Dev and Prod.
Lessons Learned
On a foundational level, a dysfunctional culture was largely responsible for the company’s production woes.
For instance, there was a lot of friction—including major arguments at times—between various stakeholders, including management, operations and development. This was a classic case of “code being thrown over the wall” from Dev to Prod. Communication between these factions was so poor, in fact, that a contractor was the primary liaison between Dev, Ops and management.
In addition, there was a loss of tribal knowledge whenever a technical practitioner left the company. When AppD arrived on the scene, none of the Devs knew anything about the troublesome third-party tool, nor how it was being used. Even the aforementioned contractor—the sole link between the siloed factions—was unaware of the problematic utility and the critical role it played.
Avoiding Pain through DevOps
Communication and knowledge-transfer between teams is critical. Each environment—Dev, Prod, and so on—should be as similar as possible for continuous integration and continuous deployment (CI/CD). Major configuration differences between environments often leads to negative outcomes.
A key tenet of DevOps is to standardize your environments. If you build code on one Dev’s machine, it should run everywhere the exact same way.
The software industry is moving in this direction. “Today, most developers state that some portion of their development team includes operations engineers.” the Evans Data study found. “The movement toward DevOps has not only impacted stakeholders’ roles, it has also influenced development team composition.”
Implementing DevOps practices, combined with a solid APM solution, will enable your organization to avoid pain and gain deeper insights into the performance of your code.
Learn more about how AppDynamics can help your own organization on its DevOps journey.
Discover faster, more efficient performance monitoring with an enterprise APM product learning from your apps. Take the AppDynamics APM Guided Tour!
