workshop,

Michael

DevOps automation

DevOps automation

related articles

Cloud

related documents

qrsolve-flyer qrsolve-whitepaper

tags

devops

Share

Michael
Written by Michael
Infrastructure & Automation master

For our team automation is a starting point, we always design automated development process. Let’s find out how we do this.

When we take care about the automation on the first stages of the project we can simply save a lot of time and money on the nest stages of the project. Our goal is to automate everything and don’t use human resources for repeatable and expensive activities like deploying, creating infrastructure, testing and checking our software. Nowadays we have many of great tools and techniques to achieve this goal providing and maintaining software with possibly the best quality.

Source code projects

We used to use GitLab as a source code repository based on Git technology. Starting new project we are preparing the right projects and groups structure depending on project requirements.

Projects structure

projects structure
Figure 1. Projects

Our example project has structure like below:

  • device - Sub group for IoT projects, deployed to electronic boards

  • internal - Sub group for UX, project informations, documentations

  • dashboard - Sub group for dashboard application, UI part, microservices and infrastructures creation

  • public - Sub group for public resources like web page and infrastructure creation for this area

Of course the naming convention is very important and we need to remember that bad names on the beginning can has big impact later to understanding by other team members and customers. Good practice is to prepare and maintain example projects to speed up new project creation by forking base projects.

projects
Figure 2. Base project structure

Project sources

Project structure needs to be compliant with the standard to be easily extended by another team or developer.

project structure
Figure 3. Project sources

Every project has to have own README documentation and CHANGELOG file to easily identify how to run and maintain source code. Keeping our projects always stable and well documented we can easily avoid misunderstanding and mess.

project readme
Figure 4. Project readme

Pipelines

This is definitely a heart of our projects ecosystem. Well designed and used can be a solution to keep all projects and systems stable, secure and maintenance-free. On the market there are various CI/CD systems, we used to use GitLab CI/CD in our projects.

Development flow

Every framework is different, we need to provide suitable continuous integration process for every project an application.

Java microservice pipeline

For example our Java microservice pipeline can look like below:

service pipeline
Figure 5. Service pipeline
  • Build - compile Java source code, make JUnit tests and prepare JAR file

  • Test - module tests with TestContainers framework

  • Deploy - prepare Docker image, deploy docker to AWS EKS registry and deploy new ECS TaskDefinition to ECS container

We can easily extend our pipeline and add another stages like

  • Anchor security scanner verification

  • Sonar Quality Gate verification

Infrastructure pipeline

Another pipeline example is Terraform project automation based on CI/CD scripts.

infrastructure pipeline
Figure 6. Infrastructure pipeline
  • Plan - create a plan for Terraform scripts. Comparing with current infrastructure and prepare ready to run Terraform plan

  • Apply - make changes for AWS infrastructure

  • Start and stop - optional step to start ond stop our test environment in the cloud

Test results

Every pipeline can export test results and provide this for users and simply show in our merge requests.

service tests
Figure 7. Test results

Group variables

To not repeat variables between projects is a good idea to gather all variables under group.

group variables
Figure 8. Group variables

Cloud Automation

Cloud automation is strictly related with Infrastructure as Code pattern where we can reflect our whole architecture on source code. Good used to use Terraform framework to onboard complete environments to the cloud. Terraform scripts can te integrated with CI/CD scripts and infrastructure creation can be also automated process without human activities.

GitFlow

GitFlow is a current standard for developing and maintaining projects by development teams. Everybody uses it to plan and release software without conflicts and problems due to concurrency.

gitflow
Figure 9. Git Flow

This methodology can be used to drive our pipeline flow. The most common use cases:

  • develop branch - flow dedicated to compile and deploy our changes merged from feature branches

  • feature branch - flow dedicated to compile developer changes ad hoc

  • release branch - flow dedicated to provide our build to stage environment

  • master branch - flow dedicated to production environment

  • merge requests - flow dedicated to compile, check and test developer changes from feature branches

We can easily design our own behavior for the above cases.

Summary

CI/CD is a very large topic, as always a good idea is to stay close the standards, track Technology Radars and practice ideas on real projects. Because of current architecture trends like microservices and cloud - DevOps is a key skill for good working development team.