How We Deploy Code

Can you describe your guiding principles or philosophies for deploying code, and how do you ensure they're upheld throughout the process at Rotate?

As with any engineering team, we adhere to core engineering principles and standards, including KISS (Keep It Simple Simon), Separation of Concerns, and IAC (Infrastructure as Code). Additionally, we uphold cultural principles such as shipping fast and iterating, continuously improving, promoting constant learning, and fostering a blame-free environment. Our deployment process serves as the backbone for enforcing and promoting these standards.

To ensure that our deployments align with our goals, we initiate the process long before code is written. We meticulously define acceptance criteria and cover aspects such as testing, alerting, monitoring, and split testing. These considerations guide our deployment process, emphasising processes such as code reviews, pipeline health, and rapid feedback to ensure a seamless deployment into production and continued learning.

What tools or methodologies do you use to automate and streamline the deployment pipeline?

We empower our team with a suite of tools, including CircleCI, Doppler, and various hosting platforms like Netlify, Edgio, Heroku, Vercel, AWS, Cloudflare, and Hookdeck. This diverse toolkit ensures we have the best resources for every task at hand.

To facilitate rapid iteration and avoid large releases, we employ trunk-based development and utilise feature flags for controlled deployments. Our standardised pipeline incorporates a minimum of linting, auditing, and testing across all projects. Metrics generated from these pipelines and deployments are then seamlessly integrated into a Retool dashboard, providing a quick and up to date overview of all our projects health.

How do you ensure the reliability and scalability of the deployment infrastructure?

Reliability and scalability are foundational to our operations, ingrained in our culture and WoW (Ways of Working). Culturally, we prioritise:

• Enhancing code stability for confidence in releases.

• Improving readability for developers.

• Facilitating agility for goal achievement.

• Implementing analytics and quick feedback loops for learning.

Mitigating bottlenecks across pipelines, architectures, and WoW is crucial for scalability. This is why we advocate for cross-departmental review processes to learn from diverse project backgrounds. While aiming for foresight, we recognise the need for adaptable planning is always required thats why cultivating agility and continuous learning is key.

We prioritise top-tier tooling and open access to empower our team, fostering autonomy and excellence. This is fuelled by recruitment philosophy of only senior professionals.

Could you walk us through a typical deployment workflow at Rotate?

As emphasised, the cornerstone of successful production deployment lies in establishing alignment regarding the work and its goals from the outset. This alignment is achieved through a comprehensive understanding of the “Definition of Ready” and its associated elements. Central to this process is the establishment of clear and measurable goals and objectives.

With these foundational elements in place, we are equipped to develop robust metrics, monitor key performance indicators (KPIs), and devise an effective test plan. This comprehensive approach enables us to determine the appropriate testing procedures, logging mechanisms, and alerting systems necessary to streamline feedback collection and minimise risk.

Once alignment is established and preparations are complete, development can proceed seamlessly. By ensuring developers can focus solely on task resolution without distraction or impediment, we maintain a high velocity throughout the development cycle.

Following completion of development, the work undergoes thorough validation through a collaborative QA process involving all relevant departments. Upon successful validation, the deployment proceeds, and the code is released to production, where it is diligently monitored.

How do you handle rollbacks or incidents during the deployment process?

In our deployment process, efficiently managing rollbacks or incidents is crucial. We employ several strategies for smooth recovery and ongoing enhancement.

Firstly, our tooling framework supports atomic deployments, allowing seamless rollbacks in case of issues. We prioritise 'fix forward,' ensuring clean pipelines after rollbacks for subsequent deployments.

We also utilise feature flags to mitigate risks. By toggling specific features, we control the rollout of new functionalities, gradually introducing changes and swiftly responding to complications.

Moreover, split testing features enable us to validate changes in a controlled environment before full deployment. By deploying new features to a subset of users, we gather real-time feedback and address any issues before widespread implementation.

Incidents are however inevitable, therefore we have an extensive incident management process. Thorough post-incident analysis helps identify root causes and insights for future prevention, fostering continuous learning and improvement. Each incident becomes an opportunity to fortify our systems against future disruptions.

How do you balance the need for speed in deploying code changes with the importance of stability and security?

Balancing speed in code deployment with stability and security requires a comprehensive approach. Here are the strategies we employ:

1. Automation: Automation is central to achieving both speed and stability. By automating tasks like building, testing, and deployment, we minimise errors and ensure consistency. Continuous integration and deployment pipelines automate the entire delivery process, incorporating robust testing.

2. Core Team: A dedicated team with expertise in development and operations maintains the balance. They design and optimise deployment processes, identifying and addressing issues early.

3. Auditing: Rigorous monitoring safeguards stability and security. Automated tools and manual reviews track performance and detect anomalies. Regular audits and vulnerability assessments mitigate risks, with logging providing valuable insights for continuous improvement.

4. Testing: Testing is fundamental, guarding against regressions and vulnerabilities. We employ various tests, including unit, integration, and security testing, executed rapidly through automation. Techniques like chaos engineering and canary deployments help identify and address issues proactively.

What are the key benefits of our deployment process for Rotate° clients?

Stability & Maintainability: Our deployment process prioritises stability, ensuring consistent and reliable performance for Rotate°'s clients. Robust testing, automated pipelines, and vigilant monitoring minimise disruptions, allowing clients to focus on their core objectives without technical worries.

Fast Iterations and Learning: Our process supports rapid iterations and continuous learning through agile methodologies and DevOps practices. Quick turnaround times for enhancements and bug fixes enable swift responses to client feedback, improving the user experience dynamically.

Diversity of Skillset and Knowledge: We foster a culture of collaboration and knowledge sharing, empowering individuals from diverse backgrounds to contribute meaningfully. This approach ensures innovative solutions that leverage a wide range of perspectives, while our commitment to continuous learning enriches the value we deliver to clients.

What role does testing play in your deployment strategy, and how do you ensure comprehensive test coverage?

Testing is fundamental to a deployment strategy, anchoring the success of our team and the dependability of our products. We customise our approach to align with diverse client needs, ensuring value that surpasses expectations. Testing also serves as documentation, facilitating efficient troubleshooting and fuelling long-term velocity.

Guided by timeless principles, including the 7 Principles of Testing such as the inevitability of defects and the context-dependent nature of testing, we integrate testing early, proactively identifying and addressing issues to elevate software quality. Embracing the concept of shifting testing left, we involve testers from the outset, minimising defects and enhancing overall software quality.

Robust testing criteria, like Gherkin, ensure clarity and consistency in defining test cases. In summary, testing embodies a commitment to excellence that permeates every facet of our work. By prioritising testing and embracing established methodologies, we uphold the highest standards of quality, reliability, and performance.

How do you collaborate with other disciplines during the deployment process?

Collaboration across disciplines is crucial for successful deployment, ensuring alignment, efficiency, and high-quality products. Our proactive approach emphasises early involvement and ongoing engagement.

We implement the 'shift left' strategy, involving stakeholders from various disciplines early in the software development lifecycle to foster shared understanding and mitigate risks. Transparency and open communication are prioritised, enabling stakeholders to track progress and address challenges effectively through regular meetings and collaborative tools.

We also promote a culture of continuous collaboration, encouraging feedback and knowledge sharing among team members through workshops and joint planning sessions, driving innovation and improvement.

Are there any key learnings you would mention to other engineering teams on how they improve their deployment process?

Reflecting on our deployment journey, we've distilled key insights that can aid other engineering teams in refining their deployment processes:

1. Perspective on Deployments: Recognise deployments as part of a broader process, not the final goal. This mindset promotes continuous improvement across the software delivery lifecycle.

2. Embrace DevOps Principles: Prioritise the 'three ways' of DevOps: flow, feedback, and continual learning. By fostering collaboration, streamlining workflows, and automating processes, teams can achieve faster, more reliable deployments.

3. Continuous Improvement: Cultivate a culture of learning and iteration. Regularly assess and refine deployment processes to adapt to changing requirements and technologies, driving efficiency and innovation.

In summary, by adopting a holistic view, embracing DevOps, and committing to continuous improvement, engineering teams can optimise deployment processes for enhanced productivity, reliability, and agility.