Using Terraform and Azure DevOps Work Items and Pipelines to automate creation of environments
It is common to every IT department to receive regularly requests to setup a new, more or less temporary, environment for a test lab, or a demo, or a trial, or a training, etc…
In the old days this meant to setup hardware in the needed place, being a classroom or a server room, install and configure operating systems and application, configure user access and data.
With virtualisation the hardware part was more or less gone (pretending to forget the need to operate capacity management on the privately hosted hypervisor service).
Then it came the Cloud and we all were able to finally really forget about hardware provisioning but still we had to operate virtual resource provisioning and setup.
We all discovered Infrastructure as Code, the usage of Orchestrators, templates, automation, deployment pipelines and so provisioning and setup became automated steps, highly reducing time and effort and surface for errors when a new environment of a known setup was requested.
We could even became able to delegate the execution of the pipeline or to link it to actions performed in the GIT repository and so development teams gained velocity and independence with the introduction of a self-service model.
But what if the environment to be created is requested by non-technical people? What if there is no GIT repository, no code commit, no development team? What if, for example, a teacher is asking for a laboratory for 50 students to complete an exam?
The problem
One of our customers had exactly this kind of need. Creation of a self-service process to allow non technical people to trigger the creation of an environment on a Cloud, selecting from a list of possible configurations and with a variable number of virtual desks.
To complete the picture, this result had to be achieved without introducing new custom code to be maintained and with the lowest costs as possible and introducing the smaller number as possible of new tools to be trained for.
The solution
Our customer had already a Microsoft Azure contract in place and was also already using Azure DevOps Service for the internal development projects. We also already helped them to setup Terraform templates and to use them in Azure DevOps Pipelines to operate provisioning of resources in Azure Cloud for those internal projects.
Considering the existing context we defined the following solution:
- Usage of Azure DevOps Boards to define a “Service Request” Work Item as only interface to be used by the non-technical requestors
- Usage of Terraform templates to define each possible configuration
- Usage of Azure DevOps Pipeline to execute the Terraform templates and to operate the provisioning of the environments as per requestors’ requests
The workflow
The requestor selects a link in the company intranet to access directly the form, in Azure DevOps Boards, to create a new Work Item of the specific kind.
The requestor compiles the form, in some case with free text, in other cases selecting from predefined list of options and selecting dates.
A pipeline evaluates if there are pending requests to setup an environment in the current day.
A request for an environment in the current date is used by the pipeline to retrieve the values to populate the variables defined and execute the Terraform environment definition by replacing placeholders with variables values.
At the end of the environment creation, the pipeline updates the Work Item containing the requests.
The requestor receives an email to inform her\him about the change of status of the request.
The requestor follow the link in the email and access the Work Item to read the details of the setup operated to be able to autonomously start to use the environment requested.
How we achieved the result
Using Azure DevOps Organization Settings we defined a custom process containing a specifically designed Work Item kind to represent the environment creation request, with introduction of custom fields to collect every information needed by the execution of the Terraform templates to create the required environment, as for example the date the environment is required, the software solution and its version to be deployment, the number of client machines, the dataset to be used, etc…
For each custom field the type is defined and in some cases a predefined list of possible values is configured, to allow selection from the known supported configurations.
The Work Item workflow definition includes only three possible states: “Requested”, “In Progress”, “Completed”.
The requestors are invited as Stakeholders in the Azure DevOps Organization.
A specific project in Azure DevOps Organization has been created, linking it with the newly defined process. The new process does include one unique backlog with one unique kind of Work Item to make it easy and avoid any confusion on the requestors in case they would decide not to use the direct link to the creation of the new Work Item but to navigate via the DevOps portal.
Using the Project Settings in the specifically created project in Azure DevOps, a security group is created for the requestors, to remove any access, if not to view the project-level information. Each requestor is inserted in the project as member of that specific security group.
In the boards configuration for the project one unique iteration and one unique area is defined and the requestor security group is assigned the minimal rights to the area.
We defined a shared query in the Azure DevOps Boards section, to report any request in status “Requested” having as desired date a date in the past or the current day.
Terraform templates had been created to deploy each of the possible environments, with options for a series of pre-defined solutions and datasets, using various Azure resources, including custom images to deploy Virtual Machines to be used not only as servers but also as clients for laboratories and online training classrooms. Those templates composed a project stored into the Git repository of Azure DevOps. It is not in the scope of current story to describe the usage of Terraform and the templates created, but it is clear a list of variables has been defined. I just want to report the attention to the definition of the output section, in order to have a readable set of “name — value” to describe also to the requestor what deployed, as for example the list of Virtual Machines used as clients, with their names and IPs.
A Release Pipeline had been defined in Azure DevOps for the actual execution of the deployments. The trigger for the execution of the Pipeline had been set to be based on a schedule, due to specific business rules, with an execution every hour during each working day.
A Pre-Deployment Gate had been configured. In case no request is returned by the query, the execution of the Pipeline is automatically aborted; while it continue in case at least one request is present.
The fist step executed by the Pipeline is a Powershell script to retrieved the values from the request ticket (the Work Item) and to populate the Pipelines variables with the result.
The second step, executed only in case of success of the first step, updates the Work Item status as “In Progress” using a Powershell script. A similar script is used, at the end of the execution of the steps involving the Terraform templates, to update the status as “Completed” and to add to the Description the output of the Terraform execution.
The notification mechanism linked with the Azure DevOps Board allow the requestor to receive an email with a link to go and see the details of her\his request updated by the Pipeline execution. The Work Item is now reporting all the details to access and use the environment requested.
Conclusion
We were finally able to accomplish what needed by our customer, extending the usage of the Self-Service model for Cloud resources provisioning also to non-technical staff, people not having access or rights to operate on the Web Portal or APIs of the Cloud Service provider, people not able to translate the terms they use to express their business need into technical elements composing an actual deployment.
We did it without creating new software components and tools, but just adapting, using some scripting and templating, the integrations between services they were already using, removing any need for further internal training to operate the solution or the need for software\licenses purchase.
Usage of Cloud Services, declarative deployment models, templates based orchestrators, scripting to consume existing APIs: it all provides a powerful combination of elements to move from automation to full Self-Service provisioning beyond the barrier of access for technicians only.
P.S.: for compliancy with data confidentiality, the images reported in this story are all taken from an internal sample setup and not from the actual customer solution
P.S.2: As a note I would just add that I do not like to insert directly in the Terraform templates the placeholders to be replaced by the deployment pipeline, but I prefer to structure my templates with a file per logical resource, using a dedicated Terraform variable file where to insert in one unique file all the placeholders that the Pipeline will replace at run time. I also prefer to create a dedicated file to define all the outputs, instead of having them defined into each resource file.
