Establishing a customer-friendly pace of change without sacrificing DevOps practices — the AlayaCare story

Author: Jean-Philippe Boudreault, Director of Product and Engineering, AlayaCare

This article discusses AlayaCare’s journey and demonstrates how it is possible to slow down releases without compromising DevOps practices. While every situation is unique, we hope that other companies can relate to AlayaCare’s challenges and apply the general principles outlined in this article.

The internet is not lacking in case studies highlighting the benefits that quick feedback loops and fast innovation bring to software companies. For nearly a decade, the State of DevOps report has underscored the value of practices like CI/CD.

Transitioning to continuous deployment is tumultuous

Many teams face executive and customer frustration when transitioning their product to continuous deployment. This challenge often arises when moving from a monolithic to a microservice architecture. The monolith typically has a well-established release schedule, complete with extensive sets of automated and manual tests. However, when teams begin to ship services independently, even a few mistakes can attract scrutiny and pressure to revert to the old way of releasing software. Unfortunately, this situation often leads to increased complexity and results in a distributed monolith.

AlayaCare’s growth has been substantial, and its monolith served the company well for years. However, as time went on, it became evident that teams could not evolve the monolith quickly enough to meet business needs. Changes had unpredictable impacts, updating dependencies was an arduous task, and removing performance bottlenecks seemed impossible. Empowering care providers with transformative technology to achieve better health outcomes was not feasible without a shift in architecture.

Monolithic regression tests gave an impression of safety

In early 2023, AlayaCare’s monolith was deployed roughly every week. A large feature release occurred every four weeks, interspersed with smaller weekly patches. Despite these regular release schedules, teams still deployed hotfixes for customer customizations and urgent bug fixes, sometimes daily. Regression testing was a lengthy process, particularly for the larger releases. Customers had access to these major releases a week in advance on pre-production environments, along with the corresponding documentation.

By mid-year, around ten teams were deploying their microservices autonomously, and the product documentation team struggled to keep the release notes up to date. Unsurprisingly, the monthly feature release cadence could not keep up with the constant deployments of the multiple elements part of the AlayaCare platform.

Customers were dissatisfied with the platform’s quality and found it challenging to internalize changes quickly enough to train their staff. Internally, teams became increasingly frustrated with the coordination required to update the monolith. Adding and populating new database tables had to be staggered over multiple deployments to prevent downtime, and housekeeping tasks like removing obsolete code flags were delayed by the weekly deployment schedule. Monthly releases became a significant source of stress, leading to increased pressure to slow down the release pace to once every 2-3 months.

Slow pace of change or quick innovation?

We had reached a breaking point; no software company would deploy every few months and innovate. But were business & engineering needs at odds?

These needs are not at odds if you successfully separate the concepts of deployment and release. The term “gentle deployment,” coined by the Silicon Valley Product Group, resonated with AlayaCare and was adopted within the organization.

The agreement that business and engineering came to at AlayaCare was: teams can deploy software at any time as long as it is a non-event for customers.

This approach allowed teams to deploy code related to customer events at any moment, provided the changes were not activated. In fact, all activations were bundled as part of a monthly “release” event that customers could preview. A focus group, consisting of representatives from product, engineering, and customer-facing teams, was tasked with determining which changes constituted customer events and which did not.

Standardized definition of customer events

At AlayaCare, we synchronize all software releases for our platform through a monthly event. While new versions of the iOS, Android, web, and Family Portal apps may be deployed at various times, the release activation occurs simultaneously for our customers.

Each company’s context is unique, so there is no one-size-fits-all approach. It is crucial for companies to establish their policies around the versioning capabilities of their technologies, especially for items like webhooks and APIs. While some trade-offs are inevitable, the key is to start somewhere and maintain consistency.

The table below summarizes the high-level elements established by the focus group.

Non-events don’t exempt teams from performing due diligence in terms of quality and monitoring. In fact, teams often activate non-events using individual activation flags.

It was crucial to empower the product team to innovate and iterate quickly based on customer feedback. This is the primary reason why changes to custom flows and features in early access are considered non-events. Best practices dictate that product managers collaborate with product marketing and customer-facing teams to manage changes for early adopters and custom flows, working directly with impacted customers.

Managing feature flags with LaunchDarkly

A good feature flag management system greatly aids engineering teams in performing smooth deployments. While AlayaCare adopted LaunchDarkly in late 2019, we only recently began leveraging its prerequisites and workflows to manage release events with staggered activation.

At a high level, all customer event flags are linked to a parent flag. We call these parent flags “monthly release” flags. They are associated with a specific workflow that activates them in customer preview environments during the first week of the month and staggers the activation across our customer regions the following week. Because parent flags act as prerequisites, customer-event flags are activated in advance.

Teams manage their standalone flags autonomously for non-events and use LaunchDarkly segments presets to test in production and reduce blast radius if a change is not performing as it should.

Perfect is the enemy of good

Gentle deployment practices aren’t meant to be immutable, and as a company matures, changes are expected. At AlayaCare, we embarked on a continuous improvement journey and established a monthly review panel with customer-facing teams to reflect on each month. We didn’t wait to build complex tooling around flag management or automated release notes documentation. 

This decision proved to be wise, as we quickly saw improvements in release quality and gained insights into the tools that would make the most significant difference for our teams and customers. For example, we realized the need to provide teams with more ownership over their product documentation and streamline flag management. For our customers, we recognized the importance of incorporating release documentation within the app and offering more testing time in UAT. It is not unrealistic to envision AlayaCare adopting a more flexible release model in the coming years. 

Our mission is to empower care providers to achieve better health outcomes by delivering transformative technology and data insights, allowing them to focus on what truly matters. This can only be achieved if technology evolves at a pace manageable by care providers.