Charlene: Well, good afternoon, good morning, or good evening, depending on where you are in this world. Welcome to today’s DevOps.com webinar, How to Really Achieve DevOps Transformation. I’m Charlene O’Hanlon, I’m the moderator for today’s event. As you can see, I’m surrounded by some really interesting, exciting and very, very intelligent people. We have Robert Stroud from Forrester, we have Sanil Pillai from Apexon, we have Brian Dawson from CloudBees, and we have Brad Hart from Perfecto. Thank you, gentlemen, for joining me today. I am so excited about today’s webinar.
Rob: Thanks for having us.
Brian and Brad: Thanks for having us.
Charlene: Great. Well, we’re going to have hopefully, a spirited conversation about really how companies can go about achieving DevOps transformation. I know all of you have much experience in the field. I’m hoping that today our audience members will be able to partake in your knowledge and really walk away with some useful information. So, why don’t we go ahead and get started? It’s going to be a question and answer session, obviously. I really want everybody to get involved in the conversation. I also want to remind the audience that we are taking questions for any of our panelists today.
At any time during the presentation, if you have any questions for anybody, just please feel free to use that control panel there and get your questions in. I also do want to remind the audience that we are recording today’s session so if you miss any or all of our today’s webinar, feel free to take a look at it about 24 hours of post event on devops.com. With that, we’ll go ahead and get things started. Rob, you are my first victim today. How are you doing?
Rob: Good. Happy to be a victim.
Charlene: Great. All right, let’s get things kicked off with this question about really where we are in the DevOps transformation, especially within enterprises. I mean, really where do you see it happening now? What do you see is kind of the hurdles that are maybe slowing things down?
Rob: A multi-pack question to get going. I think DevOps has fundamentally now reached what I call escape velocity. We’re seeing what was the perview of development organizations almost totally and unicorns, Netflix’s of the worlds, etcetera and Etsy has really become mainstream and it’s part of the fuel of the business transformation that’s going on in the industry today. Recent DevOps Heat Map survey that we did show that the retail banking and finance, believe it or not, is actually leading the pack right now in terms of the DevOps transformation.
We’re seeing massive investments, especially in their mobile applications, their web applications and their net new products and go to market as I look to transform the way they develop and develop in a significantly fast manner in terms of more velocity. The other aspect that’s really interesting in our research shows that there’s two maturity levels of DevOps right now. There’s one of the, what we call that the people have been doing it for a couple of years and they’re getting quite good at it. They’re doing quite well in terms of maybe being in maturity level two or three, depending on how you rate it.
There’s a lot of people who are just starting. It’s interesting, the increase have really changed lately from Ops and Devs professionals to enterprise architects and CIOs who are looking to actually transform their organizations and scale it. One of the things I’ll just leave my opening remarks with is that, the biggest problem we see at the moment is scaling. However, it takes something that’s in a small team group, or an application, or a division, or in an end-user department and scale that across the whole company. That’s a good problem to have, one that I’m happy very to be talking about.
Charlene: Okay. Great. Well, thanks very much. I’m sorry, guys, we’re having a little bit of a technical difficulty with the videos here but we’re going to get that taken care of for you really quickly. Sanil, I’d like to turn things over to you but before we do, I want to make sure that none of our panelists have anything that they would like to add to what Rob had said.
Sanil I think one thing just came to my mind at what Rob was talking is that when we deal with some of the engagements with the client, what we’ve seen is the DevOps is now front and center when you think of setting up teams and thinking about what development should be and processes should be. While in the past, it was something that was added on to existing teams. I think that is the major shift that we are witnessing across the board and I think that’s great actually.
Charlene: Excellent, anybody else?
Brian: Nothing to add here.
Charlene: Okay. All right, great. Well then, we’ll go ahead and move along. Sanil, thank you for being part of this today. This is, like I said before, such a great topic and I think the audience is really going to get a lot out of it. My question to you is how do companies know where they are in their maturity and what really they should be doing to advance?
Sanil: Sure. First of all, I’m really happy to be here and it’s exciting to be a part of this group. Maturity and DevOps has been something that a lot of companies struggle with and I think it’s a very, very pertinent question. I think the important part to remember that customers would never be at a single point in the maturity curve. It’s because DevOps really transcends a lot of things, it goes all the way from culture to tools to talk about test automation, the infrastructure, build processes, deployment processes.
Often what we’ve seen is that customers are pretty mature in some couple of these areas and they are pretty much at the initiation state in some other areas. In order to really look at a holistic view of where they are on a maturity curve, what we suggest is that basically they have to look at a maturity matrix across all these different vectors, and then run some an assessment tool to really see at what point they are in each of these areas. The reason for doing that is because the next steps that customers have to take really to mature could take one or two different dimensions.
One is, they could really say that, “Okay, there certain areas where we really need to do a lot of advancement then we put a lot of budgets and resources on that.” But in certain areas, we do really good but I think we could really get a lot of bang for the buck but by really optimizing it. So, using the maturity curve along with an assessment, a maturity model around with the assessment, what customers get to see is what are the immediate holes they can plug to really advance where they are and what should be the long-term strategy in kind of reactivating projects, processes, etcetera. That’s how I think they can really leverage the maturity monitoring model.
Charlene: Okay, great. Anybody have anything that they’d like to maybe disagree with Sanil on?
Brian: I absolutely disagree with everything Sanil said. [laughs] No, no, no, I think Sanil is spot-on. A couple of things that I would add are, being that DevOps is really about a culture, a set of tenants that you subscribe to and aspire to. To a certain extent, when you talk about the cultural aspects of DevOps, in front of the people in culture aspect, you could say that maturity can be somewhat subjective. A couple of things that I would add, well, it’s important to try to quantify what level of maturity you’re at, what level of maturity you’re trying to achieve and where you are along the road.
It is also important to understand that there really isn’t a one-size-fits-all recipe for DevOps, so you have to approach your maturity somewhat subjectively. Also that actually, Sanil and I have worked together on one of the CloudBees concepts of something called The Four Quadrants of DevOps Maturity.
Right now, the approach with this is, not to say that it’s a maturity model that’s going to give you specific steps to execute, which is absolutely required in webinar, it starts at a higher level and says, “Okay, how do we talk across teams in the software delivery lifecycle? How do we start by considering what Enterprise DevOps is as opposed to implementation on a single team? And how do we gain initial alignment on our overall maturity and overall objectives?”
On summary, I think absolutely, measurement is important, measurement is key to this. Keep in mind that those objectives and that measurement may be somewhat subjective and it does help to simplifying at a high level of maturity as you’re starting your transformation.
Charlene: Okay, all right.
Rob: If we look at that for a minute, one of the things we see in most DevOps executions, let’s call them an execution, they’re there to support some aspect of business transformation, we’re going to start probably in one team but I think we’re only just starting to see organizations take an enterprise view first, that’s probably a change in the last six months from the majority of calls, it didn’t exist six months ago. Would that be true of everybody’s experience or is it just my little corner of customer inquiries?
Brian: No, I agree and if– Where we focus on scaling, Rob, I’d say it’s a pretty key thing and ultimately, you’re going to implement it across the enterprise as you start to build out from teams all along the way. You’re thinking about what an enterprise implementation looks like or else you’re going to get a sealing or inhibitors that aren’t across.
Brad: That’s something that we see when we saw large enterprises. A typical project isn’t one team, it could be up to 50 teams that some work on the front-end, the back-end, third party client services, APIs, etcetera and they’re all at different levels and they don’t all have the same requirements. Your front-end team might be super agile, all plugged in the DevOps, daily deployments, but the back-end team might still waterfall. That’s where we see our real challenges, as the different groups who still have to come together at the end to make a cohesive product, they need to be assessed independently and they each need help along this journey in different ways.
Charlene: Okay, all right. Excellent. Brian, it’d be interesting to hear from your perspective when companies are really starting down that DevOps journey, what do you see is the benchmarks for them? And then really keeping that in mind, what are some of the principles that you think teams need to follow to really achieve that successful DevOps implementation?
Brian: I’ll try to parse this question, I see this being two slightly different things but I’ll try to address it. Let’s start with principles because they tend to be more qualitative, often times less quantitative than DevOps metrics. Overall, this is going to sound cliché frankly but I believe that you should be interviewing with a principle of collaboration, and that’s collaboration early and often. And that collaboration is then going to lead to what is really required for speed repeatability and reliability and that is a focus on automation and ultimately, because the path to success is never a linear path, it’s never a clean one. You also have to be prepared to embrace experimentation.
Brad: The thing is, if you think about the way software was traditionally developed, developers would work, they would do their pre-commit validation, get code, they’d be building, working, collaborating together, but then they don’t really know it’s done, it’s good until maybe they give QA a drop every week, every two weeks. That’s a very common pattern that we see and that’s too long, that’s the opposite of continuous.
There’s a couple of big problems there because one, you can’t move forward. If people strive to go from back of the day releasing quarterly and then, releasing monthly, and weekly, and daily, and they want to go multiple times a day, you need to know what you’ve done is good. The developer takes code, they’ve been given a feature or bug fix and they submit that code, they lose their little finger tips, they think they’re good because they never write bad code.
But we know that that’s reality. Maybe they forgot to check something in, maybe they didn’t implement it right or it didn’t fix the problem. The idea is that continuous feedback, it’s so critical because you don’t want to have to wait because you don’t want to defer decisions, you want to know you’re good to go now.
The other problem that comes in is one of the things that we’ve been trying to work a lot with customers on, we’ve seen really help, is the faster that the developers get feedback on what the changes they’ve done, it saves them time overall. I could be a developer, submit a defect fix, I’m in a certain area code, I submit my change, and if I don’t find out for two weeks that it didn’t fix the problem or it broke something else, not only did we delay our ability to release that code by two weeks.
But as a developer, if you told me within 10 minutes because I ran a build and I ran it in CI and I got feedback right away that it was broken, I can maybe fix it in 10 minutes, I already written in the code, I’ve got my environment, I’m not context-switching. If you take somebody who was working here and then you don’t tell them for two weeks and they’re already working over here, it’s too hard for the developer to switch back mentally, to get their environment back, to get their mindset back. The overall time to fix a problem is a lot longer.
The idea of getting continuous feedback to the developers, to the team leaders so they can get instantaneous per build, nightly feedback on exactly what the health in the state of their DevOps project is, it’s going to make things go exponentially faster. It’s not just waiting a certain amount of time to test, it’s being able to avoid that context-switching. So, it’s very important and it allows you to make good decisions, actionable decisions quicker, real time. That’s the goal, is to get real time.
Charlene: Anybody disagree?
Sanil: I think one of the great examples that we have seen exactly to what Brad mentioning is, if we have a customer we work with and the earliest– They would know when a developer broke something is during the nightly visit so typically, it could be like the whole day. What they realized after they kind of implemented DevOps and continuous feedback is, not only are you getting– Continuously, we’re getting timely feedback and what that has changed, that has changed developer mentality. They want to check in more often now because they really want to know things are going faster.
Earlier it was like, “You know, I’m just going to go and test and test internally before I even push anything out.” That is a side effect of having this process where the developer is not ready to push and code faster, wait for feedback, and then pushing more. I think all our productivity as a result definitely, it answers. Yes, it’s pretty very important to get continuous and timely feedback.
Charlene: Rob, I feel like you want to say something.
Rob: Many things. Continuous feedback is a key part of this. We’re talking about the developer’s side a lot here and it does increase developer productivity. It’s very interesting, one of the stats we pull is that the average developer spends 20% of their time writing code, one day out of five. I’d like to come to work one day out of five, that’ll be good.
One of the value props of this continuous feedback is, if we can just get that developer to spend one and a half days out of five writing code, that’s a 50% productivity increase. That’s one of the value props of what we’re talking about here, is we want to act actually give them the continuous feedback so that they can be more productive, ideate more and take ideas more. Ultimately, this is what we’re after, this is the objective.
We talked about KPIs just a question ago and one of the questions is, if I can generate more lines of code, more lines of value, you’re going to make mistakes, that’s okay. You’re going to tell me about them sooner but when I’m done, I’ll moved to the next thing and I can focus on it. This is a really key part of the value prop we often lose.
Charlene: Okay, it’s actually a really great segment into my next question which has to do with your most recent blog. You talked about how DevOps is actually transitioning from kind of those unicorn companies to be companywide initiatives, can you give us some ideas or share a little bit of what you know about how some of this larger companies are implementing in leveraging DevOps?
Rob: Yes, we’re definitely seeing a transition, it’s not just the Netflix’s and the Etsys and Amazons anymore, it’s most companies. We’re seeing this happen across the board, most industries with exception of the of government. They’re a bit slower but it’s still happening. What they’re actually looking to do is, they’re looking to change the way they work. A large global bank will use that to keep their name protected] but they’re really transitioning now into a position where they started off just in their mobile division, their web division, where they did mobile payments, and their web app, and they got that right and they implemented product teams.
Now, they have a product team that owns each capability within their organization and now, they’ve rolled that out very successfully, the product teams work together. Originally, they pick and choose around tools and that did lead to wonderful tool proliferation. They had 480 different tools by the way in the Devops tool chains and that’s good or bad, we’re not giving merits of it either way.
But what they actually found is over time as they worked together and shared knowledge and collaborated, the ethical point where product teams are being formed across the organization. They are now transitioning whether it is through intend ownership of the products and the services and the components and teams are actually cross-skilling, they’re getting really good to my knowledge, they’re starting to form relationships, they can cover for each other, they know each other’s weaknesses and strengths.
That’s now led to a situation where they actually don’t want to release daily or hourly, they want to release weekly and weekly is a significant advance on where they were just six months ago when they released every six months. This is not unusual, right? And if you look at that velocity increase, it’s really being reflected now in smaller pieces of change happening more regularly and it requires less documentation, less retraining, less education of the consumers.
This is a mainstream financial services org, now they had to do this to compete with the FinTechs that were coming out there. You look at another manufacturing organization where they’re actually changing the way they do manufacturing now, where they’re using DevOps and they’ve actually been using forever by doing discrete manufacturing. They’ve been doing it to have their production line set the order lines so they can change things, they’re quite the same thing in their code now that they deliver, and that’s really allowing them to transition.
But the biggest headache that each of them have had is breaking down the functional silos. Functional silos exist where, “I’m the Ops person, don’t come near me. Don’t touch my production system, I’ll kill you if you do. ” [laughs] The reality of it is that they stand behind compliance, and governance, and risk, and climb these walls that protect them. The reality of what we’re going to talk about I’m sure more is the fact that if you put good automation in place, end to end automation in place, you can actually overcome a lot of those issues. That’s one of the things when learning now, is we take DevOps from Dev and no Ops to DevOps now which is what we’re really doing.
We’re actually saying to shift testing, performance testing, user performance testing, CX validation, we’re shifting all that layer so that when the code gets to production, it breaks less. The recent reports show that we’re seeing more availability and more uptime. And this is what I really think, all organizations now are doing this. The final point I’ll end on is, I just finished doing a piece of work with another financial services org where they’re now going to scale DevOps across their whole organization, Mainframe, as well as Distributed.
Now, Mainframe is becoming an equal citizen in the DevOps agenda so they’re actually not just writing stuff on Distributed or Cloud, they’re actually going back and writing code on Mainframe. They’ve actually set the situation up now so they can release a whole release at once, all together, all dependencies noted, all pieces working. They don’t have to wait until midnight, Sunday night to do it anymore. They can actually do it three o’clock on Wednesday afternoon if they want and typically they know the risk in impact and if it fails, they can back out in seconds.
It’s amazing to see and it’s really changed some perspectives in terms of the industry. This is at scale, this is not just the product team sitting over in the web app anymore. It’s the whole organization.
Charlene: It’s pretty amazing to think about because I remember when a lot of companies were starting to transition to just even using Cloud services, a lot of companies were using the excuse that they would be out of compliance or they had a whole bevy of excuses that they were using to not move to the Cloud. Now, it sounds like a lot of those same excuses are being used for DevOps. It’s kind of interesting, it’s almost like a one-size-fits-all excuse for some of these companies but I’m glad to see that they know slowly but surely the veil is being lifted. These companies are realizing that they’re not going to be out of compliance if they use the DevOps kind of culture methodology. It’s actually helpful to their organization such as, in the same way that the Cloud is.
Rob: Absolutely. And compliance is interesting and I know some of my other colleagues left some comments but there are some rules that are rigid that say that the same people who write the code shouldn’t implement the code. But it doesn’t say that you actually have to have another human touch it and we’ve seen good automation leverage and they used to leverage that role out in a successful manner with full audit and compliance. It requires some thinking, it requires some process and actually, you put your pipeline process under change management to conform to these rules like you would a piece of code. Everything should be code these days, everything should be treated as code.
Charlene: Anybody else have any thoughts on the compliance issue and anything else that might be kind of slowing down the DevOps?
Brian: Yes, I actually have a lot I’d like to add to what Rob said [laughs]. Actually, those are very good questions for you. Maybe we’ll grab a beer, a remote beer offline. But I think you brought up a few things when we talk about compliance, right? We really talked about a software delivery process that is organized with discretely different functional groups each sort of heroically and arguably, selfishly protecting their own domain because they know best for that part of the process. That has led its way to compliance.
I’m QA, I know quality, I know how the software should function, just give it to me and I’ll figure out. I’m operations at the end of the day. They’re going to come to me. The security groups are going to come to me and kick my butt when we’ve had a vulnerable component come in and land on our server, so I’m going to protect that because at the end of the day, it’s me on the line. But shift left because we’re looking to rotate this data. I think Rob, you bring up something really important that we often times don’t dare to say to properly implement DevOps. You ultimately, maybe gradually over time have to look at reorganizing whether it’s Furman or whether it’s matrix. We are now cutting across.
That starts to speak to the shared ownership tenant of DevOps. You really can’t get fast flow, rapid delivery, and rapid recovery until you rotate tear down those silos. What I often like to say is we look at processes in terms of being risk averse of the time all the way back when we used to put program code on punch cards or on tape, or we print it up a hundred thousand disks.
The recovery for an error was very, very expensive. We now have systems in place, Cloud technology, automation, more robust and flexible languages, environments, and architecture. We have the capability to recover from errors fast but yet we’re still– Let’s wait. Let’s inspect everything up. Something that the agile movement first brought in and DevOps accepts is, errors are inevitable.
And from Brad’s plate from earlier talking about feedback, we’re better than trying to do everything we can do to prevent the inevitable. Let’s accept the inevitable is going to happen and position ourselves to respond quickly, and Rob, what you’ve talked about with the reorganization across functional groups enables that.
Charlene: Great. Perfect. Another thing, Rob, that you brought up was the idea that the Mainframes are very much a part of the DevOps ecosystem these days and it’s actually something I’ve been hearing a lot more about but that’s in and out of itself is a great segue for the question I have for Sanil. When you’re talking about the different kinds of tools and everything that you need to implement DevOps, do you have to actually retire those legacy systems and those legacy tools to implement DevOps? Or can they use both legacy and new tools or–? What’s the strategy overall that companies really need to think about?
Sanil: That is the 10 million dollar question.
Charlene: Congratulations. You get that question.
Sanil: I was reading this the other day and it really stuck with me, it said that there are no legacy systems, it’s only a legacy thinking. I think that is very important instances to this question. When you look at legacy tools for DevOps, we cannot take them disassociated with the legacy applications of what are really there. So, when you talk about legacy tool, we have to really think about the legacy application. Trying to modernize your tool chain without trying to really look at the application that’s supposed to be serving is really not a strategy.
So, really going to think of legacy tools, I really try to look at the application sensors and see what can be done within those existing tool sets and tool chain for those legacy application without having to really talk about it sufficiently is complete. For example, if you look at current legacy processes, often a lot of manual steps that happen, either on deployment, unfilled, full checkout, etcetera and that by itself, is a lot of optimization that you can do and there is not even need a report of your existing tool set.
The existing tool set can probably aid in that process but they can really work with conjunction with your existing tool sets in the tool chain that you really are already using. But a lot of automation of manual step itself can give you a lot of runway in the legacy system. Then, you can look at infrastructure. So you have infrastructure, you would think them on Cloud, maybe move to the hybrid Cloud so that you also serve some of the constituents you have in your legacy system.
Third part is automation. You can look at your test automation for a model. You then, look at your test automation, see what is the coverage that you have right now. Often we find, when we work with our clients is that there is a lot of optimization that can be done just with legacy application, just on the test automation site, but that gives you another set of runways.
When you look at– My take with legacy tools essentially is that the modern tool sets are designed for DevOPs from the ground up but it is often important to look at your legacy tool set and see– Although they’re not built to be resilient sometimes, right? They are not built to find all the different EMC running, but they can always be retrofitted to fix things automatically when things break. That is one side of approach to in a kind of way.
In summary, our approach always has been from Apexon and for me, when I’m going to talk to customers is that, I’m trying to see what optimizations that you can do with their existing tool set, and then there are different design patterns to slowly migrate over from one tool set to the other.
One of the clients that we did a major migration project for their work, from old set of tools to newer, we spend a lot of time thinking about the migration strategy itself. Because there are points of time during the migration of one to the other when you have both systems actually work in time and that is very important for a successful move from legacy to systems, right? It is not a none or all strategy. There is always a middle ground there, and there are ways to continue that.
Charlene: Okay. Makes total sense to me. Great. Anybody have anything they’d like to add to Sanil’s comments?
Rob: Yes, I’ll just add one thing, one thing Sanil made great points there. One of the challenges though with DevOps, the data we have clearly shows that one of the weaknesses we have is the pipeline. Our pipelines are typically automated even in the DevOps world in silos of automation. In fact, less than 30% of organizations have end to end automated pipelines. The challenge is– It’s okay if we’ve done integrations to have those different tools integrated but the reality is most companies are doing manual handoffs. Actually, this is the biggest failure of DevOps from a practical perspective, is that this lack of integrated automation, lack of using tools– I don’t don’t think there new at all.
But the fact we’ve actually don’t use technology to automate the handoffs is really introducing human error again further down the train after we’ve done a good work we talked about the first part of this call. I think it’s one of the opportunities and challenges we’ve got to really work on as an industry.
Charlene: Well taken. Okay. Anyone else? Okay. Brian, back to you. I’ve got just– Something that’s been coming up in conversations lately and that’s the idea of DevOps as a service. What do you think of that? What do you think it is and can you explain it to everybody?
Brian: DevOps as a service is a fallacy. It doesn’t exist.
Charlene: All right then.
Brian: That was a controversial statement. And there’s a couple of ways to look at DevOPs as a service that I think what some people intend by it is real. Many people may think of DevOps as a service, DevOps in a box. How do I go out and buy DevOps? How do I give my team DevOps and make it something that they can just go through like a vending machine, put a couple of requisition dollars in, and now they get out of that box process?
I think we’ve all said here at some point, some people may have heard me say or I’ve said to you guys, DevOps is not– Actually to borrow from a colleague of mine, DevOps is not something you can do. You don’t do DevOps. It’s not something you can package, it’s not something you can sell. It’s something you can achieve and you continuously work to improve.
The other take I see on DevOps as a service though is being that underlying component to go back to the tools and automation conversation. Is it– I’d say commoditization and packaging of that tools and automation to implement a CD process which teams can rapidly adopt. That I think is achievable, that I think is real, and that I think is invaluable as organizations start to take an enterprise approach to implementing DevOps.
In the past, DevOps used to start these change initiatives, would start by a forward-looking Dev Lead Manager, individual contributor that takes it upon themselves to start to bring in some improvements and innovation. Now, what we have to start to do is look to architect this so it’s something that will stand the test of time, it’s something that will scale across teams, it’s something that will meet enterprise objectives.
What this requires is that we have to balance the flexibility of individual team and technology’s needs and the need to have some level of standardization and shared implementation so we can more rapidly adopt this without burning individual teams with undue administration overhead, be that of your delivery tools, be that of your target systems, be that of your reporting, and analytics, databases, et cetera.
Now we ask, what is DevOps as a service? DevOps as a service doesn’t exist. You can provide CD as a service, CD as a service is about identifying shared tools, shared systems, and shared environments so that you can reliably implement DevOps at scale.
Charlene: Okay. Anybody have anything they’d like to add or contradict?
Sanil: Not contradict but what we have seen as– We have seen companies who have centered teams providing CICD, as Brian mentioned, as a service to other development teams and that is to the extent that we’ve seen practical implementations of the concept. What happens with DevOps as a service, essentially, as Brian mentioned, there’re different connotations, one is a SAS based model that you can develop from. It really means that from an organization perspective, it’s almost like a black box where you get the CICD services you can really leverage. That is I think the real value of CICD as a service the real value of what this is all about. Otherwise, I agree with Brian.
Brian: Well, I think you just disagreed with me Sanil when you said DevOps
Brian: but yes, I like how quickly is that there are a couple of connotations and one of the things we’re seeing at CloudBees is people that are not quite ready to go to SAS, I think Rob would note with some stats and probably others as Brad is in here, organizations are rapidly getting comfortable with moving the majority of these things to the Cloud in particular, the CICD pipeline.
In fact, over 50% of the Jenkins workload is run on AWS. The majority of workloads for private SAS implementation are CD tool. That sort speaks to two different things and I’ll try to get through this quick. You can provide not DevOps as a service, CD as a service in the Cloud using a number of providers or what you can look to do is build out sort of centrally managed, internally managed CD as a service for your teams. Both of them achieve those end objectives that I talked about, scalability and repeatability for taking this across your organization.
Charlene: Well, that makes sense. I mean, such an important part of DevOps’ culture and I guess to me, I didn’t know how somebody could do DevOps as a service, it’s not like you can provide culture as a service. I agree with you there.
Brad: One thing’s to carry onto is we’ve seen traditionally, some of our largest customers are in finance and insurance, and these guys would never even consider using Cloud services for anything, even for email, not even two years ago and we’re seeing them shift. Part of the thing is what the service based offerings do is they allow people they don’t have to reinvent the wheel over and over again and spend all that overhead, building things that already exist.
These service companies, CloudBees, allow people to leverage best practices that they’re the expert at, right? So, that this financial institution could worry about managing the organizational change and the structural change and focus on building their product not re-implementing a CICD architecture. People are finally getting the value that it’s okay to use tools and frameworks that were built by other people. It’s helping them go a lot faster and achieve their goal a lot faster.
Charlene: I think that there’s a real place in DevOps for that but I do agree with Brian that the terminology DevOps as a service is a misnomer and it actually provides a disservice to the concept as a whole because it really does commoditize the idea of “You can just buy this particular types of tools and platforms and voila, you’re doing DevOps.” Obviously, as we all know that’s not the case. Interesting stuff. I’m so glad that we have these conversations.
I do want to remind the audience that if you have a question for any of our panelists, there is still time. We’re going to take about maybe one or two more questions that we have prepared for our panelist and then, we’re going to move on to the audience questions. We have a ton already but if you have a question, please don’t hesitate to get it in. If we don’t get to your question during the webinar, I’m sure these guys would be more than happy to follow up with you.
Brad, I don’t want to leave you out. [laughs] Let’s go ahead and move on to your question which is that whole concept of shifting left when in DevOps, how can a company do that and really do their testing earlier in that DevOps cycle? I know it’s an odd concept for some companies to grasp.
Brad: Let’s first start with the why. Shift left is a buzzword but why? Why is this important? I’ll give you an example of a customer working within the financial industry. I’m not going to say the name of the bank but they’re in America. [laughs] Very large. What they did is they did an analysis on the cost of fixing a defect in production versus fixing it when it was found in QA versus found in development. The cost in production of finding a defect, in resolving it, and the impact it has in the business is measured in the tens of thousands of dollars, if not more. Whereas in QA, it might be $800, $900. I forget the exact numbers. In development, it’s like a hundred bucks. There’s a real cost.
If you find all your defects in production, it’s going to cost you millions and millions of dollars. You dramatically reduce those costs if you find them earlier in development. Not only are you saving yourself money, you’re also having a happier customer base and you’re releasing quicker. I gave some examples earlier with– This is all part of the continuous feedback. Continuous feedback means your shifting the feedback left of the development teams but it’s also helping put some of the onus on those teams in their environments of the definition of done, that’s an agile term. The Dev Team needs to know. They don’t just throw it over the wall anymore.
Part of shifting left is to achieve it, you just can’t take– We’ve seen some organizations take a stab at it and they were told to shift left. So, what they try to do is they try to take their end-to-end suites and jam them into the development CI environment. It doesn’t work because all it does is create tons and tons of noise. Having 10,000 automated tests that you run every time somebody a does check in and have it always be read is no good. A bunch of read tests all the time doesn’t help anybody. It’s noise and it just causes a distraction.
We work with organizations, try to help them. The ones that we’ve seen go down this path are shifting left the best is they don’t try to run their whole thing left. What they do is figure out what’s smart. What do we do at each stage? They model the development pipeline from, what are we going to do pre-commit? What unit tests are developers going to do? What are they going to do for pre-commit validation before they submit their code? Do something at the developer workstation as an individual and then expand that. They do their commit and it kicks off Jenkins, and it does a build. There’s a smoke test. Something a little bit smaller but always green.
You want tests to be always green unless there’s a real problem. Then it goes. We’re going to do scheduled tests, they do a nightly test, regression, eventually, it works its way to the QA person. The idea is that you want to save human time. By the time a code drop gets to the QA and end people– I don’t care if you have QA people sitting on the Scrum teams, there is going to be a fan base, code’s going to go through some QA organization and end kind of tests as all the team codes come together.
But by the time it gets in their hands, it should have gone through multiple stages of testing that has been part of the shift left process with more and more gradual maturity being built into the code by the time it gets to a human. Therefore, they’re not spending their time finding defects that could’ve been caught in CI. They’re spending their time validating. Does this look the way we thought it would look? A business analyst can say, “Does this meet the customer’s need?” Not dealing with things that are broken because we didn’t test long enough.
The process is critical. It reduces cost, it dramatically increases efficiency, it increases velocity, but you have to do it smart. You have to think and that is part of what I spent a lot of time working with customers as what do I test and when. How do I build the level of testing so we don’t slow down development but yet we don’t also introduce problems by the time it gets to the QA team. That’s a real strategy that you need to incorporate.
Charlene: Okay. Anybody having anything they want to add to that? That was a great explanation there.
Rob: I just had one point, Brad. A case study I’m just running at the moment, the organization had a 100% of defects standing by the end users, just a few years ago.
Charlene It’s not a good way we do it.
Rob: Yes, but that was normal software development for over six years ago, right? It was– Basically, put it out there and we’ll see what happens. That’s why enterprises never went to version one, so it is that version, second version. They are now at a point where I think from memory, it’s almost 90% of all defects are found in development, and they have now so much time back, they had dramatically increased their velocity of features to production.
It’s just an amazing step that if you stop finding those bugs by your end users, and also upgrade, that’s the other advantage. Every almost all the users are on the current version. I mean, it’s huge turnaround in just a very short time, and that’s the value we’re talking about, Brad.
Brad Yes, that’s amazing data.
Charlene I think mobile has had a lot to do with that, change that mindset. I do want to switch gears now and move over to audience questions because when I say we’ve got a ton, I mean, we’ve got a ton here, folks, and we only have about 10 minutes left. So, let’s go ahead and maybe we’ll do some lightning round things. First one is for Rob, the question is that really, what distinguishes DevOps culture change from maybe Agile and Scrum?
Rob: Agile and Scrum pretty much impact the development organization changing and the requirements. The real transition of DevOps is its organization wide. It actually tackles everything from requirements ideation right through to deployment to production, and the ongoing success and happiness of the products and services being run. That’s the lightning round answer.
Charlene: Perfect. All right, great. Moving on. Sanil, I’ve got a question for you. For organizations that are just starting to assess their DevOps maturity, is there an industry standard or recommendations for a DevOps maturity model or an assessment tool that could be used as a starting point? Any suggestions?
Sanil: There isn’t an industry standard but as Brian mentioned, infrastructure and Cloud, we’ve worked together to create an assessment model to work against the maturity assessment tool against the maturity model and that is a great tool to really assess where an organization is. But there isn’t a real industry with standard because of the wide spectrum that DevOps bring.
Charlene: Okay. All right, great. Brian, is DevOps the next iteration or evolution in iterative development?
Brian: Can I answer Sanil’s question instead?
Charlene: You can answer both of them.
Brian: We’re quite likely round so I’ll walk back it. Yes, I don’t know that it’s so much the next iteration innovative development, but to go back to something to what Rob said about the difference between DevOps, and Agile, and Scrum, and tie it back to what Brad said. What we’re really doing is we’re extending the definition of done to be what is really required to be confident at the time that you have gone through at iteration, you have delivered what is expected and with through automation and end to end systems, enabling us to get that feedback that we need to continue and ensure we’re delivering value through an iterative development cycle.
I will say that when people have found us, they optimize the left-hand side in terms of project planning and execution practices, but they ended up fighting yet they still have legacy and waterfall delivery practices on the right-hand side, they did not get the expected value from their implementation in Agile and Scrum on the left-hand side so it’s bringing them together. I’ll go faster next time.
Charlene: No, that’s fine. That’s fine. Okay. Brad, got a question for you. You were talking about continuous feedback before. To get continuous feedback, is it mandatory to have continuous testing?
Brad: Certainly. Especially given who I work for.
Brad: You have to have feedback on something. The feedback comes in more than just automated testing of my application. If I’m running through my CICD stack, doing deployments, what does that look like? Feedback around performance data, how we’re performing in different geographies. It’s user validation. Feedback comes from many, many, many different sources. Continuous testing is very important but it is only one key part of it. It’s required, but it’s not the only thing to focus on. There’s a whole variability of things that you need to consider when you look at the overall feedback. User ratings, everything.
Charlene: Excellent. Anybody else that want to add on to that or we’re good?
Rob: We’re good.
Charlene: Okay. Great. I have another question here for you, Rob. What are some recommendations to start integrating previously siloed operations team to the development or delivery teams?
Rob: It’s a great question, one that we’ve been asked a lot lately. The first successful method I’ve seen is where– First, we put infrastructure as code in place, where we start developing consistent infrastructure deployment and we apply the DevOps methodology to delivering what our infrastructure’s decks look like. That then, allows you to redeploy resources in your Ops team to put as part of product teams.
You actually include the Ops person as product team, this is what I see happening in a lot of organizations now where they come in and become part of it. They ensure that the Ops requirements of monitoring customer experience, appropriate incident triage, documentation where it’s required, and all become part of the development organization and all become part of the team.
When the code is no longer tossed over the fence but transition through the process, those requirements are met day one and the results are really simple. If you do it right, your incident count reduces dramatically, meaning you need less people to actually resolve incidents. It’s really an interesting statistic we’re starting to see. If the team owns the code, they actually have to go and fix it, which is even better.
Another thing we’ve just started to see implemented is most sprint teams, they now got a requirement for the technical debt as a percentage of user stories. The number that we’re starting to see used now is 15%. That’s a 15% of every sprint must be technical debt retirement. Kind of interesting.
Charlene: That is interesting.
Brian: I’d like to add something, if I may.
Charlene: Sure.
Brian: What I’ve seen is I have– Oftentimes my time in the field, I’ve had to deal with organizations who either couldn’t rapidly re-order their teams to be product-based or feature-based teams or they would. We’d start with the incremental transformation. To your point, Rob, one of the things I’ve recommended and I’ve seen implemented successfully is even if you still have a legacy operations team that is maybe on the path to implementing infrastructure score but is not fully there, let’s still at minimal, get Ops representation and traditional QA representation on the Scrum team.
So that you’re actually participating in your story creation up front and worst case, you’re getting advance insights, you’re providing the team feedback on decisions that are being made in the story and how they impact downstream, but you also can look forward into your traditional transfer information, into your traditional operations team backlog, to have them prepare, to address what’s coming down the pipe and give them visibility. So, Rob, big plus one on getting that operations participation up front in the planning.
Brad: Yes, Brian. I concur. The most highly functional organizations we’ve seen have the Ops and QA as part of the actual teams whether they dotted line or whatever, it rolls both ways. Just a decision-making happens so much faster and you make better quality decisions as you go forward. It’s a great way to do it.
Charlene: Well, you guys are giving me segs left and right because I think we have time for one more question here and I’ve got it for Sanil. The question is should DevOps teams be part of Scrum teams or should they be separate teams? And what is the best organization structure for a successful DevOps transformation? It sounds like we’ve already started that conversation.
Sanil: Yes. I think, exactly, just to add on to what Rob, Brian, and Brad just mentioned, DevOps obviously, they are kind of to work together as teams. I think the more important question here is, for to be a successful Scrum team, you need to have a DevOps, an automation system in place, the CIC. It’s more important to have that laid out so that the teams themselves are pretty self-sustaining in terms of leveraging DevOps across their entire product development cycle.
It’s not that important that you really have an Ops person consistently embedded with your team. It is important to have them available and have them provide the right infrastructure for the team to be successful. I think that, to me, is the right way to look at that question.
Charlene: Okay. Anybody have anything they want to add the last 60 seconds or so?
Rob: I was still learning. DevOps is still new. What we’re talking about now, we may contradict ourselves in six months.
Brian: That’s not right.
Charlene: I hope we’re not contradicting ourselves in six months, but I agree
Sanil: and you’ve disagreed enough today.
Charlene: Okay, great. Well, that is all the time we have for the questions. If we didn’t get to your questions, first of all, I apologize. Time just ran away from us but certainly, I’m sure these guys will be more than happy to address your questions offline. With that, it just leads me to thank Robert Stroud from Forrester, Sanil Pillai from Apexon, Brian Dawson from CloudBees, and Brad Hart from Perfecto. Thank you all for joining me today. It has been such a great conversation and I hope we can do it again soon.
Rob: I hope so.
Brian: Thanks, Charlene. Thanks, guys.
Sanil: Thank you.
Charlene: All right. Have a great day, everybody.
