Auto DevOps

Introduced in GitLab 10.0.

Generally available on GitLab 11.0.

Auto DevOps provides pre-defined CI/CD configuration which allows you to automatically detect, build, test, deploy, and monitor your applications. Leveraging CI/CD best practices and tools, Auto DevOps aims to simplify the setup and execution of a mature & modern software development lifecycle.

Overview

With Auto DevOps, the software development process becomes easier to set up as every project can have a complete workflow from verification to monitoring with minimal configuration. Just push your code and GitLab takes care of everything else. This makes it easier to start new projects and brings consistency to how applications are set up throughout a company.

For an introduction to Auto DevOps, watch AutoDevOps in GitLab 11.0.

Enabled by default

Introduced in GitLab 11.3.

Auto DevOps is enabled by default for all projects and will attempt to run on all pipelines in each project. This default can be enabled or disabled by an instance administrator in the Auto DevOps settings. It will be automatically disabled in individual projects on their first pipeline failure, if it has not been explicitly enabled for the project.

Since GitLab 12.7, Auto DevOps will run on pipelines automatically only if a Dockerfile or matching buildpack exists.

If a CI/CD configuration file is present in the project, it will continue to be used, whether or not Auto DevOps is enabled.

Quick start

If you are using GitLab.com, see the quick start guide for how to use Auto DevOps with GitLab.com and a Kubernetes cluster on Google Kubernetes Engine (GKE).

If you are using a self-managed instance of GitLab, you will need to configure the Google OAuth2 OmniAuth Provider before you can configure a cluster on GKE. Once this is set up, you can follow the steps on the quick start guide to get started.

Comparison to application platforms and PaaS

Auto DevOps provides functionality that is often included in an application platform or a Platform as a Service (PaaS). It takes inspiration from the innovative work done by Heroku and goes beyond it in multiple ways:

Auto DevOps works with any Kubernetes cluster; you're not limited to running on GitLab's infrastructure. (Note that many features also work without Kubernetes).
There is no additional cost (no markup on the infrastructure costs), and you can use a Kubernetes cluster you host or Containers as a Service on any public cloud (for example, Google Kubernetes Engine).
Auto DevOps has more features including security testing, performance testing, and code quality testing.
Auto DevOps offers an incremental graduation path. If you need advanced customizations, you can start modifying the templates without having to start over on a completely different platform. Review the customizing documentation for more information.

Features

Comprised of a set of stages, Auto DevOps brings these best practices to your project in a simple and automatic way:

Auto Build
Auto Test
Auto Code Quality (STARTER)
Auto SAST (Static Application Security Testing) (ULTIMATE)
Auto Dependency Scanning (ULTIMATE)
Auto License Compliance (ULTIMATE)
Auto Container Scanning (ULTIMATE)
Auto Review Apps
Auto DAST (Dynamic Application Security Testing) (ULTIMATE)
Auto Deploy
Auto Browser Performance Testing (PREMIUM)
Auto Monitoring

As Auto DevOps relies on many different components, it's good to have a basic knowledge of the following:

Auto DevOps provides great defaults for all the stages; you can, however, customize almost everything to your needs.

For an overview on the creation of Auto DevOps, read more in this blog post.

NOTE: Note Kubernetes clusters can be used without Auto DevOps.

Requirements

To make full use of Auto DevOps, you will need:

Kubernetes (for Auto Review Apps, Auto Deploy, and Auto Monitoring)

To enable deployments, you will need:
1. A Kubernetes 1.12+ cluster for the project. The easiest way is to create a new cluster using the GitLab UI. For Kubernetes 1.16+ clusters, there is some additional configuration for Auto Deploy for Kubernetes 1.16+.
2. NGINX Ingress. You can deploy it to your Kubernetes cluster by installing the GitLab-managed app for Ingress, once you have configured GitLab's Kubernetes integration in the previous step.
  
  Alternatively, you can use the nginx-ingress Helm chart to install Ingress manually.
  
  NOTE: Note: If you are using your own Ingress instead of the one provided by GitLab's managed apps, ensure you are running at least version 0.9.0 of NGINX Ingress and enable Prometheus metrics in order for the response metrics to appear. You will also have to annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254".
Base domain (for Auto Review Apps, Auto Deploy, and Auto Monitoring)

You will need a domain configured with wildcard DNS which is going to be used by all of your Auto DevOps applications. If you're using the GitLab-managed app for Ingress, the URL endpoint will be automatically configured for you.

You will then need to specify the Auto DevOps base domain.
GitLab Runner (for all stages)

Your Runner needs to be configured to be able to run Docker. Generally this means using either the Docker or Kubernetes executors, with privileged mode enabled. The Runners do not need to be installed in the Kubernetes cluster, but the Kubernetes executor is easy to use and is automatically autoscaling. Docker-based Runners can be configured to autoscale as well, using Docker Machine.

If you have configured GitLab's Kubernetes integration in the first step, you can deploy it to your cluster by installing the GitLab-managed app for GitLab Runner.

Runners should be registered as shared Runners for the entire GitLab instance, or specific Runners that are assigned to specific projects (the default if you have installed the GitLab Runner managed application).
Prometheus (for Auto Monitoring)

To enable Auto Monitoring, you will need Prometheus installed somewhere (inside or outside your cluster) and configured to scrape your Kubernetes cluster. If you have configured GitLab's Kubernetes integration, you can deploy it to your cluster by installing the GitLab-managed app for Prometheus.

The Prometheus service integration needs to be enabled for the project (or enabled as a default service template for the entire GitLab installation).

To get response metrics (in addition to system metrics), you need to configure Prometheus to monitor NGINX.
cert-manager (optional, for TLS/HTTPS)

To enable HTTPS endpoints for your application, you need to install cert-manager, a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster will issue a certificate by Let’s Encrypt and ensure that certificates are valid and up-to-date. If you have configured GitLab's Kubernetes integration, you can deploy it to your cluster by installing the GitLab-managed app for cert-manager.

If you do not have Kubernetes or Prometheus installed, then Auto Review Apps, Auto Deploy, and Auto Monitoring will be silently skipped.

One all requirements are met, you can go ahead and enable Auto DevOps.

Auto DevOps base domain

The Auto DevOps base domain is required if you want to make use of Auto Review Apps, Auto Deploy, and Auto Monitoring. It can be defined in any of the following places:

either under the cluster's settings, whether for projects or groups
or in instance-wide settings in the Admin Area > Settings under the "Continuous Integration and Delivery" section
or at the project level as a variable: KUBE_INGRESS_BASE_DOMAIN
or at the group level as a variable: KUBE_INGRESS_BASE_DOMAIN.

The base domain variable KUBE_INGRESS_BASE_DOMAIN follows the same order of precedence as other environment variables.

TIP: Tip: If you're using the GitLab managed app for Ingress, the URL endpoint should be automatically configured for you. All you have to do is use its value for the KUBE_INGRESS_BASE_DOMAIN variable.

NOTE: Note: AUTO_DEVOPS_DOMAIN was deprecated in GitLab 11.8 and replaced with KUBE_INGRESS_BASE_DOMAIN. It was removed in GitLab 12.0.

A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of example.com, you'd need a DNS entry like:

*.example.com   3600     A     1.2.3.4

In this case, example.com is the domain name under which the deployed apps will be served, and 1.2.3.4 is the IP address of your load balancer; generally NGINX (see requirements). How to set up the DNS record is beyond the scope of this document; you should check with your DNS provider.

Alternatively you can use free public services like nip.io which provide automatic wildcard DNS without any configuration. Just set the Auto DevOps base domain to 1.2.3.4.nip.io.

Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).

Enabling/Disabling Auto DevOps

When first using Auto DevOps, review the requirements to ensure all necessary components to make full use of Auto DevOps are available. If this is your fist time, we recommend you follow the quick start guide.

GitLab.com users can enable/disable Auto DevOps at the project-level only. Self-managed users can enable/disable Auto DevOps at the project-level, group-level or instance-level.

At the project level

If enabling, check that your project doesn't have a .gitlab-ci.yml, or if one exists, remove it.

Go to your project's Settings > CI/CD > Auto DevOps.
Toggle the Default to Auto DevOps pipeline checkbox (checked to enable, unchecked to disable)
When enabling, it's optional but recommended to add in the base domain that will be used by Auto DevOps to deploy your application and choose the deployment strategy.
Click Save changes for the changes to take effect.

When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch.

At the group level

Introduced in GitLab 11.10.

Only administrators and group owners can enable or disable Auto DevOps at the group level.

To enable or disable Auto DevOps at the group-level:

Go to group's Settings > CI/CD > Auto DevOps page.
Toggle the Default to Auto DevOps pipeline checkbox (checked to enable, unchecked to disable).
Click Save changes button for the changes to take effect.

When enabling or disabling Auto DevOps at group-level, group configuration will be implicitly used for the subgroups and projects inside that group, unless Auto DevOps is specifically enabled or disabled on the subgroup or project.

At the instance level (Administrators only)

Even when disabled at the instance level, group owners and project maintainers can still enable Auto DevOps at the group and project level, respectively.

Go to Admin Area > Settings > Continuous Integration and Deployment.
Toggle the checkbox labeled Default to Auto DevOps pipeline for all projects.
If enabling, optionally set up the Auto DevOps base domain which will be used for Auto Deploy and Auto Review Apps.
Click Save changes for the changes to take effect.

Enable for a percentage of projects

There is also a feature flag to enable Auto DevOps by default to your chosen percentage of projects.

This can be enabled from the console with the following, which uses the example of 10%:

Feature.get(:force_autodevops_on_by_default).enable_percentage_of_actors(10)

Deployment strategy

Introduced in GitLab 11.0.

You can change the deployment strategy used by Auto DevOps by going to your project's Settings > CI/CD > Auto DevOps.

The available options are:

Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.
Continuous deployment to production using timed incremental rollout: Sets the INCREMENTAL_ROLLOUT_MODE variable to timed, and production deployment will be executed with a 5 minute delay between each increment in rollout.
Automatic deployment to staging, manual deployment to production: Sets the STAGING_ENABLED and INCREMENTAL_ROLLOUT_MODE variables to 1 and manual. This means:
- master branch is directly deployed to staging.
- Manual actions are provided for incremental rollout to production.

Using multiple Kubernetes clusters (PREMIUM)

When using Auto DevOps, you may want to deploy different environments to different Kubernetes clusters. This is possible due to the 1:1 connection that exists between them.

In the Auto DevOps template (used behind the scenes by Auto DevOps), there are currently 3 defined environment names that you need to know:

review/ (every environment starting with review/)
staging
production

Those environments are tied to jobs that use Auto Deploy, so except for the environment scope, they would also need to have a different domain they would be deployed to. This is why you need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.

The following table is an example of how the three different clusters would be configured.

Cluster name	Cluster environment scope	`KUBE_INGRESS_BASE_DOMAIN` variable value	Variable environment scope	Notes
review	`review/*`	`review.example.com`	`review/*`	The review cluster which will run all Review Apps. `*` is a wildcard, which means it will be used by every environment name starting with `review/`.
staging	`staging`	`staging.example.com`	`staging`	(Optional) The staging cluster which will run the deployments of the staging environments. You need to enable it first.
production	`production`	`example.com`	`production`	The production cluster which will run the deployments of the production environment. You can use incremental rollouts.

To add a different cluster for each environment:

Navigate to your project's Operations > Kubernetes and create the Kubernetes clusters with their respective environment scope as described from the table above.
After the clusters are created, navigate to each one and install Helm Tiller and Ingress. Wait for the Ingress IP address to be assigned.
Make sure you have configured your DNS with the specified Auto DevOps domains.
Navigate to each cluster's page, through Operations > Kubernetes, and add the domain based on its Ingress IP address.

Now that all is configured, you can test your setup by creating a merge request and verifying that your app is deployed as a review app in the Kubernetes cluster with the review/* environment scope. Similarly, you can check the other environments.

Currently supported languages

Note that not all buildpacks support Auto Test yet, as it's a relatively new enhancement. All of Heroku's officially supported languages support it, and some third-party buildpacks as well e.g., Go, Node, Java, PHP, Python, Ruby, Gradle, Scala, and Elixir all support Auto Test, but notably the multi-buildpack does not.

As of GitLab 10.0, the supported buildpacks are:

- heroku-buildpack-multi     v1.0.0
- heroku-buildpack-ruby      v168
- heroku-buildpack-nodejs    v99
- heroku-buildpack-clojure   v77
- heroku-buildpack-python    v99
- heroku-buildpack-java      v53
- heroku-buildpack-gradle    v23
- heroku-buildpack-scala     v78
- heroku-buildpack-play      v26
- heroku-buildpack-php       v122
- heroku-buildpack-go        v72
- heroku-buildpack-erlang    fa17af9
- buildpack-nginx            v8

Limitations

The following restrictions apply.

Private registry support

There is no documented way of using private container registry with Auto DevOps. We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.

Installing Helm behind a proxy

GitLab does not yet support installing Helm as a GitLab-managed App when behind a proxy. Users who wish to do so must inject their proxy settings into the installation pods at runtime, for example by using a PodPreset:

apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: gitlab-managed-apps-default-proxy
  namespace: gitlab-managed-apps
spec:
   env:
    - name: http_proxy
      value: "PUT_YOUR_HTTP_PROXY_HERE"
    - name: https_proxy
      value: "PUT_YOUR_HTTPS_PROXY_HERE"

Troubleshooting

Auto Build and Auto Test may fail to detect your language or framework with the following error:
```
Step 5/11 : RUN /bin/herokuish buildpack build
 ---> Running in eb468cd46085
    -----> Unable to select a buildpack
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 1
```
The following are possible reasons:
- Your application may be missing the key files the buildpack is looking for. For example, for Ruby applications you must have a Gemfile to be properly detected, even though it is possible to write a Ruby app without a Gemfile.
- There may be no buildpack for your application. Try specifying a custom buildpack.
Auto Test may fail because of a mismatch between testing frameworks. In this case, you may need to customize your .gitlab-ci.yml with your test commands.
Auto Deploy will fail if GitLab can not create a Kubernetes namespace and service account for your project. For help debugging this issue, see Troubleshooting failed deployment jobs.

Development guides

Development guide for Auto DevOps