July 21, 2014

Continuous Delivery: The Power of Embedded Teams

Continuous Delivery

continuous delivery

In my last article, "Why is Perforce Adopting Continuous Delivery?" I talked about why the Engineering group at Perforce is migrating to Continuous Delivery (CD) and how we are going about it. In this post I’ll dig into the first step of that migration, which was the creation of embedded product teams. What are embedded teams, what were the challenges in moving to them, and what benefits did we expect?

So what’s an embedded product team? Our definition is:

An embedded product team consists of all the people and skills needed to 
independently take product all the way from requirements to delivery.

The Engineering structure at Perforce is typical of most companies: Multiple product teams (each consisting of Product Management, Development, UX, QA, and Project Management) sharing the build, packaging, documentation and release services provided by a separate “Engineering Services (ES)” organization. As release milestones approach, ownership passes back and forth between the product team and the ES team as the product moves through to delivery.

Moving to fully embedded teams required the following changes:

  • It was first necessary to change the charter of the ES organization from “services” to “infrastructure development.” We formed new teams to build out the build, package, and delivery infrastructures needed to provide a “self-service” model to the product teams. Secondly, the “service” resources were added to the existing product teams so that each team had all the skills and resources needed to take ownership for their own delivery.
  • We committed to a long-term investment in developing and supporting the self-serve infrastructures needed by product teams. To do so, we first had to free up engineering time in the ES organization by deprioritizing many typical service requests and by transferring some responsibilities that were historically acquired but best pushed back out to the correct owners.
  • Once enough of the self-serve infrastructure was in place, we then transferred ownership for builds, tests, packaging, etc. out from the ES organization and into the product teams. In some cases this transfer of ownership came with the resource that used to do the work in ES, but in most cases an existing member of the product team was trained up to take on the transferred responsibility.
    This move created two concerns within the Engineering team. Firstly, product team leaders worried that having existing product team members take on the additional build, package, and delivery responsibilities would reduce the time available for new feature development. Secondly, there was concern that product quality would drop due to the transfer of delivery ownership and due to changes in the “tried and tested” tools and techniques used in the past.
  • We empowered each product team to deliver snapshot, patch, and feature releases by themselves on a schedule of their choosing.
  • With all roles now embedded into the various product teams, it was important to create the position of ‘role architect’ or ‘role consultant,’ who would work closely with each team to ensure consistent use of tools and processes, as well as mentor/coach on best practices, train on the use of the infrastructure etc.

The migration to embedded teams is still a work in progress for us in 2014. However, early feedback is very positive, such as:

  • Engineering are pleased to see long-overdue upgrades in our build, test, and packaging infrastructure materialize.
  • Build and packaging bottlenecks that used to exist when services were shared are now gone. Product teams can make the needed changes and fixes by themselves. In addition, delivery throughput times have significantly reduced as a result of the improvement test and delivery automation.
  • Teams that have migrated to CD enjoy their new-found control over their product development and delivery.

The concerns around product quality have not materialized. There were some teething issues early on, but they were fixed quickly. Our QA metrics showed that the move to CD caused no degradation in quality as seen by the customer. We expect quality to improve with more testing and delivery automation. The impact on a product teams’ available development time is observed as being around 10%, but that was expected as part of the investment in CD. We expect that overhead to reduce back into the ‘noise’ once our tools and techniques are refined and fully automated.





To hear about the experiences of some of our customers, take a look at our on-demand webinar - Expert Panel: Continuous Delivery Best Practices Revealed.