Terrakube is an open source collaboration platform for running remote infrastructure as code operations using Terraform or OpenTofu that aims to be a complete replacement for close source tools like Terraform Enterprise, Scalr or Env0.

The platform can be easily installed in any kubernetes cluster, you can easily customize the platform using other open source tools already available for terraform (Example: terratag, infracost, terrascan, etc) and you can integrate the tool with different authentication providers like Azure Active Directory, Google Cloud Identity, Github, Gitlab or any other provider supported by Dex.

Terrakube provides different features like the following:

• Terraform module protocol:

  • This allows Terrakube to expose an open source terraform registry that is protected and is private by default using different Dex connectors for the authentication with different providers.

• Terraform provider protocol:

  • This allows Terrakube to expose an open source terraform registry where you can have create your own private terraform providers or create a mirror of the current terraform providers available in the public Hashicorp registry.

• Handle you infrastructure as code using Organization like closed source tools like Terraform Enterprise and Scalr.

• Easily extended by default integrating with many open source tools available today for terraform CLI.

• Support to run the platform using any RDBMS supported by liquibase (mysql, postgres, oracle, sql server, etc).

• Plugins support to extends functionality using different cloud providers and open source tools using the Terrakube Configuration Languague (TCL).

• Easily extends the platform using common technologies like react js, java, spring, elide, liquibase, groovy, bash, go.

• Native integration with the most popular VCS like:

  • Github.

  • Azure DevOps

  • Bitbucket

  • Gitlab

• Support for SSH key (RSA and ED25519)

• Handle Personal Access Token

• Extends and create custom flows when running terraform remote operations with custom logics like:

  • Approvals

  • Budget reviews

  • Security reviews

  • Custom logic that you can build with groovy and bash scripts

General security

Terrakube security is based organizations and groups.

An organization can have one or multiple groups and each group have different kind of access to manage the following options:

• Module

  • Manage terraform modules inside an organization

• VCS

  • Manage private connections to different VCS like Github, Bitbucket, Azure DevOps and Gitlab and handle SSH keys

• Template

  • Manage the custom flows written in Terrakube Configuration Language when running any job inside the platform

• Workspaces

  • Manage the terraform workspaces to run remote terraform operations.

• Providers

  • Manage the terraform providers available inside the platform

Administrator group

There is one special group inside Terrakube called TERRAKUBE_ADMIN, this is the only group that has access to create organizations and grant access to a teams to manage different organization features, you can also customize the group name if you want to use a different name depending on which Dex connector you are using when running Terrakube.

Terrakube can be installed in minikube as a sandbox environment to test, to use it please follow this:

Setup Helm Repository

helm repo add terrakube-repo https://AzBuilder.github.io/terrakube-helm-chart
helm repo update

Setup Minikube

minikube start
minikube addons enable ingress
minikube addons enable storage-provisioner
minikube ip 

Setup Terrakube namespace

kubectl create namespace terrakube

Setup DNS records

sudo nano /etc/hosts

192.168.59.100 terrakube-ui.minikube.net
192.168.59.100 terrakube-api.minikube.net
192.168.59.100 terrakube-reg.minikube.net

Install Terrakube

helm install terrakube terrakube-repo/terrakube -n terrakube

allow-snippet-annotations: "true"

Reference: https://github.com/AzBuilder/terrakube/issues/618#issuecomment-1838980451

The environment has some users, groups and sample data so you can test it quickly.

Visit http://terrakube-ui.minikube.net and login using admin@example.com with password admin

You can login using the following user and passwords

Select the "simple" organization and the "sample_simple" workspace and run a job.

The docker images are generated using cloud native buildpacks with the maven plugin, more information can be found in these links:

• https://docs.spring.io/spring-boot/docs/current/reference/html/container-images.html#container-images.buildpacks

• https://docs.spring.io/spring-boot/docs/3.1.5/maven-plugin/reference/htmlsingle/#build-image

Alpaquita Linux (Experimental)

From Terrakube 2.17.0 we generate 3 additional docker images for the API, Registry and Executor components that are based on Alpaquita Linux that are designed and optimized for Spring Boot applications.

docker pull azbuilder/api-server:2.17.0-alpaquita
docker pull azbuilder/open-registry:2.17.0-alpaquita
docker pull azbuilder/executor:2.17.0-alpaquita

module "consul" {
  source = "git@github.com:hashicorp/example.git"
}

The easiest way to deploy Terrakube is using our Helm Chart, to learn more about this please review the following:

You can deploy Terrakube using different authentication providers for more information check

We use Gitpod to develop the platform. You can have a complete development environment to test all the components and features with one click using the following button.

Running Terrakube in Gitpod is like running the platform in any Kubernetes distribution in Azure, Aws or GCP, so you will be able to use all the features available with one single click without installing any software in your computer and the environment should be ready in only 1 or 2 minute.

Terrakube API uses an in memory database, it will be deleted and preloaded with default information in every startup.

Gitpod Environment Information.

A file called GITPOD.md will have the information for the entire workspace

The configuration is generated dynamically using the following bash script. This is executed by Gitpod in the environment startup process.

./scripts/setupDevelopmentEnvironment.sh

If for some reason we need to execute the script again, it needs to be executed form the /workspace/terrakube folder

The development environment will have the following information:

Start Development Environment

To run all the Terrakube components run the following task:

After a couple of seconds all the components will be running, you can start/stop/restart any component as needed.

Login Development Environment

After the application startup is completed, we can login to the development environment, to get the URL you can execute the following command:

gp url 3000

The Terrakube login will look like this example:

Dex Authentication.

The Gitpod environment is using an OpenLDAP tha is connected to a Dex using the LDAP connector. The OpenLDAP configuration(users and groups) can be found in the following file:

scripts/setup/dex/config-ldap.ldif

The Dex configuration file is generated dynamically and will be available in the following file:

scripts/setup/dex/config-ldap.yaml

The template that is used to generate the Dex configuration file can be found in this path:

scripts/template/dex/template-config-ldap.yaml

Dex login page should look like this example:

Sample Organizations

The development environment will be preloaded with 4 organizations(azure, gcp, aws and simple), this information is loaded inside the API startup and the scripts to load the information can be found in the following xml files

api/src/main/resources/db/changelog/demo-data/aws.xml
api/src/main/resources/db/changelog/demo-data/gcp.xml
api/src/main/resources/db/changelog/demo-data/azure.xml
api/src/main/resources/db/changelog/demo-data/simple.xml

Sample Teams

Depending of the selected organization and user you will see different information available.

Sample Modules

Each organization is preloaded with some modules that can be found in Github like the following:

Templates

Each organization is preloaded with the following templates to run Terrakube jobs:

For more information of how to use templates please refer to the following repository

Workspaces

All the organization have different empty workspaces, the Simple organization can be used for testing because the workspace is using a basic terraform file with the following resources:

• null_resource

• time_sleep

Other workspaces can be created but they will be deleted on each restart unless are added inside the initialization scripts:

api/src/main/resources/db/changelog/demo-data/simple.xml

API Testing

Gitpod has the thunder-client installed to easily test the API without using the UI.

All the environment variables for the collection are generated on each Gitpod workspace startup The environment variables template can be found in the following directory:

scripts/template/thunder-tests/thunderEnvironment.json

To authenticate the thunder client you can use Dex Device Code Authentication using the following request

The response should look like the following and you can use the verification_uri_complete to finish the device authentication.

{
  "device_code": "htlj4w73ftjfplinifqntp7ept6xaratvgauu2nj3nldlcgxjym",
  "user_code": "HLMH-WKXX",
  "verification_uri": "https://5556-azbuilder-terrakube-XXXXX.ws-us54.gitpod.io/dex/device",
  "verification_uri_complete": "https://5556-azbuilder-terrakube-XXXX.ws-us54.gitpod.io/dex/device?user_code=HLMH-WKXX",
  "expires_in": 300,
  "interval": 5
}

Once the Dex device code authentication is completed you can use the following request to get the access token.

Now you can call any method available in the API without the UI

Terraform Login Protocol

Terraform login protocol can be tested using the thunder-client collection:

The response should look similar to the following:

Terraform Module Protocol

There is one example of terraform module protocol inside the thunder-client collection that you can be used for testing purposes executing the following two request.

The response should look like the following:

• Getting versions for the module

• Getting zip file for the module

OpenAPI Spec

The specification can be obtained using the following request:

This is the high level architecture of the platform

Component descriptions:

• Terrakube API:

  • Expose a JSON:API or GraphQL API providing endpoints to handle:

    • Organizations.

    • Workspace API with the following support:

      • Terraform state history

      • Terraform output history

      • Terraform Variables (public/secrets)

      • Environment Variables (public/secrets)

      • Cron like support to schedule the jobs execution programmatically

    • Job.

      • Custom terraform flows based on Terrakube Configuration Language

    • Modules

    • Providers

    • Teams

    • Teamplate

• Terrakube Jobs:

  • Automatic process that check for any pending job operations in any workspace to trigger a custom logic flow based on Terrakube Configuration Language

• Terrakube Executor:

  • Service that executes the Terrakube job operations written in Terrakube Configuration Language, handle the terraform state and outputs using cloud storage providers like Azure Storage Account

• Terrakube Open Registry:

  • This component allows to expose an open source private repository protected by Dex that you can use to handle your company private terraform modules or providers.

• Cloud Storage:

  • Cloud storage to handle terraform state, outputs and terraform modules used by terraform CLI

• RDBMS:

  • The platform can be used with any database supported by the Liquibase project.

• Security:

  • All authentication and authorization is handle using Dex.

• Terrakube CLI:

  • Go based CLI that can communicate with the Terrakube API and execute operation for organizations, workspaces, jobs, templates, modules or providers

• Terrakube UI:

  • React based frontend to handle all Terrakube Operations.

You can deploy Terrakube to any Kubernetes cluster using the helm chart available in the following repository:

Helm Repository

To use the repository do the following:

May 2024 (2.21.0)

Welcome to the 2.21.0 release from Terrakube! As we reach the midpoint of the year, we’re excited to introduce several new features and enhancements that improve the overall user experience and functionality. Let’s dive into the key highlights:

Terrakube Actions

Terrakube actions allow you to extend the UI in Workspaces, functioning similarly to plugins. These actions enable you to add new functions to your Workspace, transforming it into a hub where you can monitor and manage your infrastructure and gain valuable insights. With Terrakube actions, you can quickly observe your infrastructure and apply necessary actions.

In the below demo, you can view how to use a couple of the new actions to and .

Authenticate Providers with Dynamic Credentials

Terrakube's dynamic provider credentials feature allows you to establish a secure trust relationship between Terraform/OpenTofu and your cloud provider. By using unique, short-lived credentials for each job, this feature significantly reduces the risk of compromised credentials. We are excited to announce support for dynamic credentials for the following cloud providers:

This enhancement ensures that your infrastructure management is both secure and efficient, minimizing potential security vulnerabilities while maintaining seamless integration with your cloud environments.

Support for --auto-approve Flag

We are pleased to announce the addition of support for the --auto-approve flag in the CLI-driven workflow. This feature allows for the automatic approval of Terraform and OpenTofu apply operations.

UI Enhancements

We have made several enhancements to the UI, focusing on improving the overall user experience:

• Buttons: Basic rounded corner adjustments.

• Global Shadow Optimization: Adjusted from three layers of shadows to two layers, used in various components.

• Loading Spinners: Added spinners for loading main pages instead of static loading text.

• Field Validations: Added validations for fields to improve form submission accuracy.

These updates aim to make the Terrakube interface more intuitive and visually appealing.

Bug fixes

We have made some improvements and fixes based on the feedback from the community and the security analysis. You can find the full list of changes for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.21.0

March 2024 (2.20.0)

Welcome to the 2.20.0 release from Terrakube! In our second update of the year, we're excited to share several key highlights with you:

Workspace Importer

Enhanced API Token Management

Organization Default Execution Mode Update

This new feature allows you to set the default execution mode—either local or remote—at the organization level.

Enhanced Support for Monorepos in Private Registry

Responding to community feedback regarding the management of Terraform modules within a monorepo, we've made significant enhancements to our Private Registry. Now, you have the capability to specify the path of the module within the monorepo and set a tag prefix when creating a module.

Agents Support

We've expanded Terrakube capabilities with the introduction of multiple terrakube executor agents. Now, you have the flexibility to select the specific agent you wish to use for each workspace. This new feature enables a higher level of execution isolation based on workspaces, offering improved security and customization to suit your project's needs.

Prefix Support for Remote Backend

If you are using the remote backend this enhancement allows you to perform operations across multiple workspaces under the same prefix, improving workspace management and operation execution.

Breaking change for POSTGRESQL user

if you are using POSTGRESQL and using the helm chart 3.16.0 it will require you to add the database port using the property

Bug fixes

We have made some improvements and fixes based on the feedback from the community and the security analysis. You can find the full list of changes for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.20.0

January 2024 (2.19.0)

As we step into the new year, we are thrilled to announce the release of Terrakube 2.19.0, a significant update that continues to push the boundaries of Infrastructure as Code (IaC) management. Here are some key highlights of this release:

OpenTofu Now Available on Terrakube

With OpenTofu now in General Availability (GA), we've integrated it as an alternative within Terrakube. Now, when you create workspaces, you have the option to select either Terraform or OpenTofu. Terrakube will then tailor the user experience specifically to the tool you choose. Additionally, you can switch your existing workspaces to OpenTofu via the Workspace settings, allowing for a smooth transition between tools.

In the future we are planning to introduce specific features for each Terraform and Opentofu based in their evolution. Also we will introduce support for more Iac tools later.

Enhanced Workspace Overview Page

We've upgraded the Workspace Overview Page to enrich your interaction with your infrastructure. Now, you'll find a detailed count of resources and outputs. Each resource is represented by an icon indicating its type, with current support for AWS and Azure icons. Clicking on a resource name unfolds a detailed view of all its attributes and dependencies, and provides direct access to the resource's documentation.

Webhook Improvements

Now, you have the flexibility to select the default template that will execute for each new push in your Version Control System (VCS) at the workspace level. By default, Terrakube performs Plan and Apply operations, but you can now choose a different template tailored to your specific use case. This enhancement allows for the implementation of customized workflows across different organizations or workspaces.

Additionally, Terrakube now checks for the configured directory in the Workspace and will only run the job when a file changes in the specified directory. This feature is particularly useful for monorepositories, where you want to manage different workspaces from a single repository. This functionality has been implemented for GitLab, GitHub, and Bitbucket.

CLI Driven workflow

Here's an example without the refresh flag:

Example setting refresh to false:

API Token Revocation

Bug fixes

We have made some improvements and fixes based on the feedback from the community and the security analysis. You can find the full list of changes for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.19.0

December 2023 (2.18.0)

Greetings! As the year comes to a close, we are excited to present you the 2.18.0 release of Terrakube. This version brings you some features and enhancements that we hope you’ll enjoy, some of the key highlights include:

AWS icons in State Diagram

The state diagram now includes all the AWS architecture icons, so you can easily recognize each resource type. This feature will help you visualize your Terraform state in a more intuitive way. We are also working on improving the diagram to group the resources by VPC, location, etc which will give you a closer look at how your architecture is designed using the Terraform state.

Visual State Diagram export option

Following the state diagram enhancements, we have also added the option to save the diagram in different formats. You can export a high-quality image in png, svg or jpeg formats by using the Download and selecting the preferred format. This feature will help you use the diagram for your documentation or presentations.

Self Hosted VCS Providers Support

We have added support for Github Enterprise, GitLab Enterprise Edition and GitLab Community Edition. This means that you can now integrate your workspaces and modules using your self-hosted VCS providers. This is a major milestone in our vision of providing an enterprise-grade experience with Terrakube. In the upcoming releases we will add Bitbucket Server and Azure DevOps Server. For more details in how to use this feature check the following links.

Webhooks support for GitLab (Cloud, EE and CE), BitBucket and Github (Cloud and Enterprise)

We have introduced a new feature that will make your workflow more efficient and convenient. Now, Terrakube will automatically trigger a new Plan and Apply job for each new push in the repository that is linked to the workspace. You no longer have to run the jobs manually using the UI or the terraform CLI. This feature will help you keep your infrastructure in sync with your code changes.

Bug fixes

We have made some improvements and fixes based on the feedback from the community and the security analysis. You can find the full list of changes for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.18.0

November 2023 (2.17.0)

Welcome to the 2.17.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Submodules view

You can now view the submodules of your Terraform Module in the open registry. When you select a submodule, you will see its details, such as inputs, outputs and resources.

Workspace Overview

We have enhanced the workspace overview to give you more information at a glance. You can now see the Terraform version, the repository, and the list of resources and outputs for each workspace.

Team API Tokens

Terraform provider for Terrakube

Bug fixes

We’ve addressed some issues reported by the community and fixed some vulnerabilities. You can see the full change log for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.17.0

October 2023 (2.16.0)

Welcome to the 2.16.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Improvement in Workspaces List

We have added more filters to workspaces, so you can easily find the one you need. You can now use the tags or description of the workspaces to narrow down your search. Additionally, you can see the tags of each workspace on its card, without having to enter it. This way, you can quickly identify the workspaces that are relevant to you.

Team API Tokens

Bug fixes and Performance

We’ve addressed some issues reported by the community regarding performance,bugs and security. You can see the full change log for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.16.0

August 2023 (2.15.0)

Welcome to the 2.15.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Local Execution Mode

In the previous version, all executions were run remotely in Terrakube. However, starting from this version, you can choose to define the execution mode as either local or remote. By default, all workspaces run remotely. If you want to disable remote execution for a workspace, you can change its execution mode to “Local”. This mode allows you to perform Terraform runs locally using the CLI-driven run workflow.

Accessing State from Other Workspaces

Support Cloud Block with Tags

Search modules

We’ve added a search function to the Modules pages, making it easier to find modules.

Bug fixes

We’ve addressed some issues reported by the community, so Terrakube is becoming more stable. You can see the full change log for this version here https://github.com/AzBuilder/terrakube/releases/tag/2.15.0

Thanks for all the community support and contributions. Special thanks to @klinux for implementing the search feature inside Modules.

Jun 2023 (2.14.0)

Welcome to the 2.14.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Terraform cloud block support

Enhanced UI error messages for Authorization errors

We have enhanced the UI to handle authorization errors more gracefully.

Workspace Tags

Overview Workspace page

With Terrakube’s new workspace Overview page, you can access all the essential information for each of your workspaces in a simple and convenient panel. Initially we are including the last run and workspace tags. We will add more information in upcoming releases.

Improved CLI run workflow logs

From this version onwards, you can view the logs in real time when you use the Terraform CLI remote backend or the cloud block.

SSH key injection

For the full changelog for this version please check https://github.com/AzBuilder/terrakube/releases/tag/2.14.0

May 2023 (2.13.0)

Welcome to the 2.13.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Terraform init -migrate-state support

Now you can easily migrate your Terraform state from other tools like TFE or Terraform Cloud to Terrakube. The migration command is supported by using the terraform CLI.

Improved Log streaming in the UI

Starting with this version, you can see the logs in the Terrakube UI in almost real time, so you can easily follow up on your Terraform changes.

Also we optimized the code and updated most libs to the latest versions For the full changelog for this version please check https://github.com/AzBuilder/terrakube/releases/tag/2.13.0

Mar 2023 (2.12.0)

Welcome to the 2.12.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Search options in Workspaces

Use the new controls in the workspaces list to filter by status or search by name.

Simplified process to add a new VCS provider

You can now connect to a vcs provider in one step, instead of creating it first and then updating the call back url.

Open Telemetry

Improvement to Templates

• In the previous version we introduced the use of importCommands, and now improve reutilization we are adding inputsTerraform and inputsEnv so basically now you can send parameters to your templates in an easy way. For example:

• And if you want to create a schedule for a template, you can do it easily now with scheduleTemplates.

Restrict Terraform Versions or Provide your own Terraform build

Terrakube Installation

Thanks for the community contribution, specially thanks to @Diliz and @jstewart612 for their contributions in the Helm Chart. For the full changelog for this version please check https://github.com/AzBuilder/terrakube/releases/tag/2.12.0

And if you have any idea or suggestion don't hesitate to let us know.

February 2023 (2.11.0)

Welcome to the 2.11.0 release of Terrakube. There are a couple of updates in this version that we hope you'll like, some of the key highlights include:

Adding Global Variables to Organization Settings

Adding CLI and API Driven workflows to workspaces

For CLI or API driven workspaces you will see the steps directly in the overview tab:

And in the list of workspaces you will see a badge so you can easily identify CLI and API driven workspaces

Improved error handling for invalid templates

If you have an error in your template yaml syntax will be easy to identify it. You will see a more clear message about the job execution failure so you can easily see if the error is related with the execution or with the template configuration.

Import scripts inside the templates

Adding UI templates support in workspaces

UI templates extend the idea of templates to the UI, so you can customize how each step will look like, in your custom flow inside Terrakube. Basically you can use standard HTML to build the way in which Terrakube will present the specific step

For example you can present a more user friendly information when extracting data from infracost result. And in the UI if you have a custom Template for that step you will se the Structured UI by default, but you can switch the view to the standard terminal output to see the full logs for that step.

Check some examples:

UI templates will allow to create better UI experiences. And brings new ways to customize the way you use Terrakube in your organization. In the upcoming releases we will create some standards templates that you can reuse for the terraform plan and apply. And also we will provide a Terrakube extensions that allows to create UI templates easily.

Improving organization selection on initial login

When you login to Terrakube for the first time and if you have multiple organizations assigned, then you will see a screen to select your organization.

For the full changelog for this version please check https://github.com/AzBuilder/terrakube/releases/tag/2.11.0

And if you have any idea or suggestion don't hesitate to let us know.

Local DNS entries

Update the /etc/hosts file adding the following entries:

Running Terrakube Locally.

Terrakube will be available in the following URL:

• http://terrakube-ui:3000

  • Username: admin@example.com

  • Password: admin

Docker Compose with Open Telemetry Setup

You can also test Terrakube using the Open Telemetry example setup to see how the components are working internally and check some internal logs

The default ingress configuration for terrakube can be customize using a terrakube.yaml like the following:

Now you can install terrakube using the command

To authenticate users Terrakube implement so you can authenticate using differente providers using like the following:

• Azure Active Direcory

• Google Cloud Identity

• Amazon Cognito

• Github Authentication

• Gitlab Authentictaion

• OIDC

• LDAP

• Keycloak

• etc.

The Terrakube is using DEX as dependency, so you can quickly implement it using any exiting dex configuration.

To customize the DEX setup just create a simple terrakube.yaml and update the configuration like the following example:

The first step will be to create one s3 bucket with private access

Once the s3 bucket is created you will need to get the following:

• access key

• secret key

• bucket name

• region

Now you have all the information we will need to create a terrakube.yaml for our terrakube deployment with the following content:

## Terrakube Storage
storage:
  defaultStorage: false
  aws:
    accessKey: "rqerqw"
    secretKey: "sadfasfdq"
    bucketName: "qerqw"
    region: "us-east-1"

Now you can install terrakube using the command:

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

The first step will be to create one azure storage account with the folling containers:

• content

• registry

• tfoutput

• tfstate

Once the storage account is created you will have to get the "Access Key"

Now you have all the information we will need to create a terrakube.yaml for our terrakube deployment with the following content:

## Terrakube Storage
storage:
  defaultStorage: false
  azure:
    storageAccountName: "<<STORAGE ACCOUNT NAME>>"
    storageAccountResourceGroup: "<< RESOURCE GROUP >>"
    storageAccountAccessKey: "<< STORAGE ACCOUNT ACCESS KEY"

Now you can install terrakube using the command

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

The first step will be to deploy a minio instance inside minikube in the terrakube namespace

Create the file minio-setup.yaml that we can use to create the default user and buckets

auth:
  rootUser: "admin"
  rootPassword: "superadmin"
defaultBuckets: "terrakube"

kubectl install --values minio-setup.yaml miniostorage bitnami/minio -n terrakube

Once the minio storage is installed lets get the service name.

kubectl get svc -o wide -n terrakube

The service name for the minio storage should be "miniostorage"

Once minio is installed with a bucket you will need to get the following:

• access key

• secret key

• bucket name

• endpoint (http://miniostorage:9000)

Now you have all the information we will need to create a terrakube.yaml for our terrakube deployment with the following content:

## Terrakube Storage
storage:
  defaultStorage: false
  minio:
    accessKey: "admin"
    secretKey: "superadmin"
    bucketName: "terrakube"
    endpoint: "http://miniostorage:9000"

Now you can install terrakube using the command:

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

The first step will be to create one google storage bucket with private access

Once the google storage bucket is created you will need to get the following:

• project id

• bucket name

• JSON GCP credentials file with access to the storage bucket

Now you have all the information we will need to create a terrakube.yaml for our terrakube deployment with the following content:

## Terrakube Storage
storage:
  defaultStorage: false
  gcp:
    projectId: "sample project"
    bucketName: "sampledata"
    credentials: |
      {
        "type": "service_account",
        "project_id": "XXXXXXX",
        ......
      }

Now you can install terrakube using the command:

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

WIP

To use a SQL Azure with your Terrakube deployment create a terrakube.yaml file with the following content:

## Terrakube API properties
api:
  defaultDatabase: false
  loadSampleData: false
  properties:
    databaseType: "SQL_AZURE"
    databaseHostname: "server_name.database.windows.net"
    databaseName: "database_name"
    databaseUser: "database_user"
    databasePassword: "database_password"
    databaseSchema: "dbo"

Now you can install terrakube using the command.

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

To use a H2 with your Terrakube deployment create a terrakube.yaml file with the following content:INT

## Terrakube API properties
api:
  defaultDatabase: false
  loadSampleData: true
  properties:
    databaseType: "H2"
    databaseHostname: ""
    databaseName: ""
    databaseUser: ""
    databasePassword: ""

Now you can install terrakube using the command.

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

Terrakube will support terraform cli custom builds or custom cli mirrors, the only requirement is to expose an endpoint with the following structure:

Example Endpoint:

• https://eov1ys4sxa1bfy9.m.pipedream.net/

{
   "name":"terraform",
   "versions":{
      "1.3.9":{
         "builds":[
            {
               "arch":"amd64",
               "filename":"terraform_1.3.9_linux_amd64.zip",
               "name":"terraform",
               "os":"linux",
               "url":"https://releases.hashicorp.com/terraform/1.3.9/terraform_1.3.9_linux_amd64.zip",
               "version":"1.3.9"
            }
         ],
         "name":"terraform",
         "shasums":"terraform_1.3.9_SHA256SUMS",
         "shasums_signature":"terraform_1.3.9_SHA256SUMS.sig",
         "shasums_signatures":[
            "terraform_1.3.9_SHA256SUMS.72D7468F.sig",
            "terraform_1.3.9_SHA256SUMS.sig"
         ],
         "version":"1.3.9"
      }
   }
}

To support custom terrafom cli releases when using the helm chart us the following:

api:
  version: "2.12.0"
  terraformReleasesUrl: "https://eov1ys4sxa1bfy9.m.pipedream.net/"

executor:
  version: "2.12.0"

To use a PostgreSQL with your Terrakube deployment create a terrakube.yaml file with the following content:

## Terrakube API properties
api:
  defaultDatabase: false
  loadSampleData: false
  properties:
    databaseType: "POSTGRESQL"
    databaseHostname: "server_name.postgres.database.com"
    databaseName: "database_name"
    databaseUser: "database_user"
    databasePassword: "database_password"
    databaseSslMode: "disable"
    databasePort: "5432"

Now you can install terrakube using the command.

helm install --values terrakube.yaml terrakube terrakube-repo/terrakube -n terrakube

Terrakube can be installed in minikube as a sandbox environment with HTTPS, using terrakube with HTTPS will allow to use the Terraform registry and the Terraform remote state backend to be used locally without any issue.

Please follow these instructions:

Requirements:

• Install mkcert to generate the local certificates.

We will be using following domains in our test installation:

terrakube-api.minikube.net
terrakube-ui.minikube.net
terrakube-reg.minikube.net

Generate local CA certificate

mkcert -install
Created a new local CA 💥
The local CA is now installed in the system trust store! ⚡️
The local CA is now installed in the Firefox trust store (requires browser restart)! 🦊

Local CA certificate path

mkcert -CAROOT
/home/myuser/.local/share/mkcert

Local CA certificate content

cat /home/myuser/.local/share/mkcert/rootCA.pem

Generate certificate for *.minikube.net

These command will generate two files key.pem and cert.pem that we will be using later.

mkcert -key-file key.pem -cert-file cert.pem minikube.net *.minikube.net

Created a new certificate valid for the following names 📜
 - "minikube.net"
 - "*.minikube.net"

Reminder: X.509 wildcards only go one level deep, so this won't match a.b.minikube.net ℹ️

The certificate is at "cert.pem" and the key at "key.pem" ✅

It will expire on 19 January 2026 🗓

Now we have all the necessary to install Terrakube with HTTPS

Setup Helm Repository

helm repo add terrakube-repo https://AzBuilder.github.io/terrakube-helm-chart
helm repo update

Setup Minikube

minikube start
minikube addons enable ingress
minikube addons enable storage-provisioner
minikube ip 

Create Kubernetes secret with local certificate

kubectl -n kube-system create secret tls mkcert --key key.pem --cert cert.pem

Setup Terrakube namespace

kubectl create namespace terrakube

Setup DNS records

sudo nano /etc/hosts

192.168.59.100 terrakube-ui.minikube.net
192.168.59.100 terrakube-api.minikube.net
192.168.59.100 terrakube-reg.minikube.net

Setup Ingress Certificates

minikube addons configure ingress

-- Enter custom cert(format is "namespace/secret"): kube-system/mkcert
✅  ingress was successfully configured

minikube addons disable ingress

🌑  "The 'ingress' addon is disabled

minikube addons enable ingress

🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

Helm Values.

We need to generate a file called value.yaml with the following using the content of our rootCa.pem from the previous step:

security:
  caCerts:
    rootCA.pem: |
      -----BEGIN CERTIFICATE-----
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      -----END CERTIFICATE-----

## API properties
api:
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "rootCA.pem"
          path: "rootCA.pem"
        - key: "type"
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true
  properties:
    databaseType: "H2"

executor:
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "rootCA.pem"
          path: "rootCA.pem"
        - key: "type"
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true

## Registry properties
registry:
  enabled: true
  replicaCount: "1"
  serviceType: "ClusterIP"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "rootCA.pem"
          path: "rootCA.pem"
        - key: "type"
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true

dex:
  config:
    issuer: https://terrakube-api.minikube.net/dex

    storage:
      type: memory
    web:
      http: 0.0.0.0:5556
      allowedOrigins: ['*']
      skipApprovalScreen: true
    oauth2:
      responseTypes: ["code", "token", "id_token"]

    connectors:
    - type: ldap
      name: OpenLDAP
      id: ldap
      config:
        # The following configurations seem to work with OpenLDAP:
        #
        # 1) Plain LDAP, without TLS:
        host: terrakube-openldap-service:1389
        insecureNoSSL: true
        #
        # 2) LDAPS without certificate validation:
        #host: localhost:636
        #insecureNoSSL: false
        #insecureSkipVerify: true
        #
        # 3) LDAPS with certificate validation:
        #host: YOUR-HOSTNAME:636
        #insecureNoSSL: false
        #insecureSkipVerify: false
        #rootCAData: 'CERT'
        # ...where CERT="$( base64 -w 0 your-cert.crt )"

        # This would normally be a read-only user.
        bindDN: cn=admin,dc=example,dc=org
        bindPW: admin

        usernamePrompt: Email Address

        userSearch:
          baseDN: ou=users,dc=example,dc=org
          filter: "(objectClass=person)"
          username: mail
          # "DN" (case sensitive) is a special attribute name. It indicates that
          # this value should be taken from the entity's DN not an attribute on
          # the entity.
          idAttr: DN
          emailAttr: mail
          nameAttr: cn

        groupSearch:
          baseDN: ou=Groups,dc=example,dc=org
          filter: "(objectClass=groupOfNames)"

          userMatchers:
            # A user is a member of a group when their DN matches
            # the value of a "member" attribute on the group entity.
          - userAttr: DN
            groupAttr: member

          # The group name should be the "cn" value.
          nameAttr: cn

    staticClients:
    - id: example-app
      redirectURIs:
      - 'https://terrakube-ui.minikube.net'
      - '/device/callback'
      - 'http://localhost:10000/login'
      - 'http://localhost:10001/login'
      name: 'example-app'
      public: true

## Ingress properties
ingress:
  useTls: true
  includeTlsHosts: true
  ui:
    enabled: true
    domain: "terrakube-ui.minikube.net"
    path: "/"
    pathType: "Prefix"
    tlsSecretName: tls-secret-ui-terrakube
    annotations:
      nginx.ingress.kubernetes.io/use-regex: "true"
  api:
    enabled: true
    domain: "terrakube-api.minikube.net"
    path: "/"
    pathType: "Prefix"
    tlsSecretName: tls-secret-api-terrakube
    annotations:
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
  registry:
    enabled: true
    domain: "terrakube-reg.minikube.net"
    path: "/"
    pathType: "Prefix"
    tlsSecretName: tls-secret-reg-terrakube
    annotations:
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
  dex:
    enabled: true
    path: "/dex/"
    pathType: "Prefix"
    annotations:
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"

Install Terrakube with local HTTPS support

helm install terrakube terrakube-repo/terrakube -n terrakube --values values.yaml 

allow-snippet-annotations: "true"

Reference: https://github.com/AzBuilder/terrakube/issues/618#issuecomment-1838980

Using Terrakube

The environment has some users, groups and sample data so you can test it quickly.

Visit https://terrakube-ui.minikube.net and login using admin@example.com with password admin

We should be able to use the UI using a valid certificate.

Connect to Terrakube

Using terraform we can connect to the terrakube api and the private registry using the following command with the same credentials that we used to login to the UI.

terraform login terrakube-api.minikube.net
terraform login terrakube-reg.minikube.net

Terraform Remote State.

Lets create a simple Terraform file.

terraform {
  cloud {
    organization = "simple"
    hostname = "terrakube-api.minikube.net"

    workspaces {
      tags = ["myplayground", "example"]
    }
  }
}

# This resource will destroy (potentially immediately) after null_resource.next
resource "null_resource" "previous" {}

resource "time_sleep" "wait_5_seconds" {
  depends_on = [null_resource.previous]

  create_duration = "5s"
}

# This resource will create (at least) 30 seconds after null_resource.previous
resource "null_resource" "next" {
  depends_on = [time_sleep.wait_5_seconds]
}

Execute Terraform Remotely

Terraform Init:

$ terraform init

Initializing Terraform Cloud...

No workspaces found.
  
  There are no workspaces with the configured tags (example, myplayground)
  in your Terraform Cloud organization. To finish initializing, Terraform needs at
  least one workspace available.
  
  Terraform can create a properly tagged workspace for you now. Please enter a
  name to create a new Terraform Cloud workspace.

  Enter a value: simple


Initializing provider plugins...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/time...
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)
- Installing hashicorp/time v0.9.1...
- Installed hashicorp/time v0.9.1 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform apply:

$ terraform apply

Running apply in Terraform Cloud. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://terrakube-api.minikube.net/app/simple/simple/runs/40

Waiting for the plan to start...

***************************************
Running Terraform PLAN
***************************************

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.next will be created
  + resource "null_resource" "next" {
      + id = (known after apply)
    }

  # null_resource.previous will be created
  + resource "null_resource" "previous" {
      + id = (known after apply)
    }

  # time_sleep.wait_5_seconds will be created
  + resource "time_sleep" "wait_5_seconds" {
      + create_duration = "5s"
      + id              = (known after apply)
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + creation_time = "5s"

Do you want to perform these actions in workspace "simple"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.previous: Creating...
null_resource.previous: Creation complete after 0s [id=6182221597368620892]
time_sleep.wait_5_seconds: Creating...
time_sleep.wait_5_seconds: Creation complete after 5s [id=2023-10-19T20:25:22Z]
null_resource.next: Creating...
null_resource.next: Creation complete after 0s [id=9093191930998774410]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Checking the UI will show:

Terrakube allow to have one or multiple agents to run jobs, you can have as many agents as you want for a single organization.

To use this feature you could deploy a single executor component using the following values:

# Executor should be enable but we need to customize the apiServiceUrl
executor:
  enabled: true
  apiServiceUrl: "http://terrakube-api-service.terrakube:8080" ## The API is in another namespace called "terrakube"

# We need to disable the default openLdap but we need to provide the internal secret
# so the executor can authenticat with the API and the Registry
security:
  useOpenLDAP: false
  internalSecret: "AxxPdgpCi72f8WhMXCTGhtfMRp6AuBfj"

# We need to disable dex in this deployment
dex:
  enabled: false

# We need to disable default storage MINIO and set some custom values 
# in this example will be deploying like using an external MINIO
#(other backend storage could be used too)
storage:
  defaultStorage: false
  minio:
    accessKey: "admin"
    secretKey: "superadmin"
    bucketName: "terrakube"
    endpoint: "http://terrakube-minio.terrakube:9000" ## MINIO is in another namespace called "terrakube"

# We need to disable API, the default redis and default postgresql database
# But we need to provide some properties like the redis connection
api:
  enabled: false
  defaultRedis: false
  defaultDatabase: false
  properties:
    redisHostname: "terrakube-redis-master.terrakube" ## REDIS is in another namespace called "terrakube"
    redisPassword: "7p9iWVeRV4S944"

# We need to disable registry deployment
registry:
  enabled: false

# We need to disable ui deployment
ui:
  enabled: false

# We need to disable the ingress configuration
# but we need to specify the api and registry URL 
ingress:
  useTls: false
  includeTlsHosts: false
  ui:
    enabled: false
  api:
    enabled: false
    domain: "terrakube-api.minikube.net"
  registry:
    enabled: false
    domain: "terrakube-reg.minikube.net"
  dex:
    enabled: false

Now that we have our values.yaml we can use the following helm command:

helm install --debug --values ./your-values.yaml terrakube terrakube-repo/terrakube -n self-hosted-executor

Now we have a single executor component ready to accept jobs or we could change the number or replicas to have multiple replicas like a pool of agent:

Adding certificates at runtime

Terrakube componentes (api, registry and executor) are using buildpacks to create the docker images

When using buildpack to add a custom CA certificate at runtime you need to do the following:

Provide the following environment variable to the container:

SERVICE_BINDING_ROOT: /mnt/platform/bindings

Inside the path there is a folder call "ca-certificates"

cnb@terrakube-api-678cb68d5b-ns5gt:/mnt/platform/bindings$ ls
ca-certificates

We need to mount some information to that path

/mnt/platform/bindings/ca-certificates

Inside this folder we should put out custom PEM CA certs and one additional file call type

cnb@terrakube-api-678cb68d5b-ns5gt:/mnt/platform/bindings/ca-certificates$ ls
terrakubeDemo1.pem  terrakubeDemo2.pem  type

The content of the file type is just the text "ca-certificates"

cnb@terrakube-api-678cb68d5b-ns5gt:/mnt/platform/bindings/ca-certificates$ cat type
ca-certificates

Finally your helm terrakube.yaml should look something like this because we are mounting out CA certs and the file called type in the following path " /mnt/platform/bindings/ca-certificates"

## Terrakube Security
security:
  caCerts:
    terrakubeDemo1.pem: |
      -----BEGIN CERTIFICATE-----
      MIIDZTCCA.........

      -----END CERTIFICATE-----
    terrakubeDemo2.pem: |
      -----BEGIN CERTIFICATE-----
      MIIDZTCCA.....
      

      -----END CERTIFICATE-----

## API properties
api:
  version: "2.11.2"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type'
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true

Checking the terrakube component two additional ca certs are added inside the sytem truststore

Added 2 additional CA certificate(s) to system truststore
Setting Active Processor Count to 2
Calculating JVM memory based on 5791152K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx5033128K -XX:MaxMetaspaceSize=246023K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 5791152K, Thread Count: 250, Loaded Class Count: 41022, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 126 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=2 -XX:MaxDirectMemorySize=10M -Xmx5033128K -XX:MaxMetaspaceSize=246023K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.8)

Additinal information about buildpacks can be found in this link:

• https://blog.dahanne.net/2021/02/06/customizing-cloud-native-buildpacks-practical-examples/#Add_certificates_binding_at_runtime:~:text=ca%2Dcertificates-,Add%20certificates%20binding%20at%20runtime,-If%20your%20image

• https://github.com/paketo-buildpacks/ca-certificates

• https://github.com/paketo-buildpacks/spring-boot

Adding certificate at build time

Terrakube allow to add the certs when building the application, to use this option use the following:

git clone https://github.com/AzBuilder/terrakube
cd terrakube
git checkout <<TERRAKUBE-VERSION>>
mv EXAMPLE.pem bindings/ca-certificates

# This script should be run from the root folder
./scripts/build/terrakubeBuild.sh

The certs will be added at runtime as the following image.

The following will explain how to run the executor component in"ephemeral" mode.

These environment variables can be used to customize the API component:

• ExecutorEphemeralNamespace (Default value: "terrakube")

• ExecutorEphemeralImage (Defatul value: "azbuilder/executor:2.22.0" )

• ExecutorEphemeralSecret (Default value: "terrakube-executor-secrets" )

The above is basically to control where the job will be created and executed and to mount the secrets required by the executor component

Internally the Executor component will use the following to run in "ephemeral" :

• EphemeralFlagBatch (Default value: "false")

• EphemeralJobData, this contains all the data that the executor need to run.

Requirements

To use Ephemeral executors we need to create the following configuration:

Service Account Creation

Role Creation

Role Binding Creation

Helm Chart Configuration

Once the above configuration is created we can deploy the Terrakube API like the following example:

Workspace Configuration

Add the environment variable TERRAKUBE_ENABLE_EPHEMERAL_EXECUTOR=1 like the image below

Workspace Execution

Now when the job is running internally Terrakube will create a K8S job and will execute each step of the job in a "ephemeral executor"

Internal Kubernetes Job Example:

Plan Running in a pod:

Apply Running in a different pod:

Node Selector.

If required you can specify the node selector configuration where the pod will be created using something like the following:

The above will be the equivalent to use the Kubernetes YAML like:

When Terrrakube needs to run behind a corporate proxy the following environment variable can be used in each container:

When using the official helm chart the environment variable setting can be set like the following:

To use a MySQL with your Terrakube deployment create a terrakube.yaml file with the following content:

Now you can install terrakube using the command.

Terrakube use two secrets internally to sign the personal access token and one internal token for intercomponent comunnication when using the helm deployment you can change this values using the following keys:

You can handle Terrakube users with differen authentication providers like the following

Terrakube components support by default to enable effective observability.

To enable telemetry inside the Terrakube components please add the following environment variable:

Once the open telemetry agent is enable we can use other environment variables to setup the monitoring for our application for example to enable jaeger we could add the following using addtional environment variables:

Now we can go the jaeger ui to see if everything is working as expected.

There are several differente configuration options for example:

Jaeger exporter

Zipkin exporter

Prometheus exporter

Open Telemetry Example

One small example to show how to use open telemetry with docker compose can be found in the following URL:

Requirements

To run Terrakube in Docker Desktop you wil need the following:

• Create Github Organization

• Create Github Teams and add some members

• Create a team called TERRAKUBE_ADMIN and add the members

• Install Docker Desktop

• Nginx Ingress for Docker Desktop

• Azure Storage Account/AWS S3 Bucket/GCP Storage Bucket

To get more information about the Dex Configuration for Github you can check this link

Setup

• Create a new Github Oauth application with the authorization callback URL "http://host.docker.internal/dex/callback" in this link

• Copy the values for Client Id and Client Secret

• Update the HOSTS file adding the following

# Linux Path /etc/hosts
# Windows Path c:\Windows\System32\Drivers\etc\hosts


127.0.0.1 ui.terrakube.docker.com
127.0.0.1 registry.terrakube.docker.com

YAML Example

Replace <<CHANGE_THIS>> with the real values, create the values.yaml file and run the helm install

## Global Name
name: "terrakube"

## Terrakube Security
security:
  adminGroup: "<<CHANGE_THIS>>" # This should be your Github team the format is OrganizationName:TeamName (Example: MyOrg:TERRAKUBE_ADMIN)
  patSecret: "<<CHANGE_THIS>>"  # Sample Key 32 characters z6QHX!y@Nep2QDT!53vgH43^PjRXyC3X 
  internalSecret: "<<CHANGE_THIS>>" # Sample Key 32 characters Kb^8cMerPNZV6hS!9!kcD*KuUPUBa^B3 
  dexClientId: "github"
  dexClientScope: "email openid profile offline_access groups"
  dexIssuerUri: "http://host.docker.internal/dex" # Change for your real domain

## Terraform Storage
storage:
  # SELECT THE TYPE OF STORAGE THAT YOU WANT TO USE AND REPLACE THE VALUES
  
  #azure:
  #  storageAccountName: "<<CHANGE_THIS>>"
  #  storageAccountResourceGroup: "<<CHANGE_THIS>>"
  #  storageAccountAccessKey: "<<CHANGE_THIS>>"
  #aws:
  #  accessKey: "<<CHANGE_THIS>>"
  #  secretKey: "<<CHANGE_THIS>>"
  #  bucketName: "<<CHANGE_THIS>>"
  #  region: "<<CHANGE_THIS>>"
  #gcp:
  #  projectId: "<<CHANGE_THIS>>"
  #  bucketName: "<<CHANGE_THIS>>"
  #  credentials: |
  #    ## GCP JSON CREDENTIALS for service account with access to read/write to the storage bucket
  #    {
  #      "type": "service_account",
  #      "project_id": "",
  #      "private_key_id": "",
  #      "private_key": "",
  #      "client_email": "",
  #      "client_id": "",
  #      "auth_uri": "",
  #      "token_uri": "",
  #      "auth_provider_x509_cert_url": "",
  #      "client_x509_cert_url": ""
  #    } 

## Dex
dex:
  enabled: true
  version: "v2.32.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  resources:
    limits:
      cpu: 512m
      memory: 256Mi
    requests:
      cpu: 256m
      memory: 128Mi
  properties:
    config:
      issuer: http://host.docker.internal/dex
      storage:
        type: memory
      oauth2:
        responseTypes: ["code", "token", "id_token"] 
        skipApprovalScreen: true
      web:
        allowedOrigins: ["*"]
  
      staticClients:
      - id: github
        redirectURIs:
        - 'http://ui.terrakube.docker.com'
        - 'http://localhost:10001/login'
        - 'http://localhost:10000/login'
        - '/device/callback'
        name: 'github'
        public: true

      connectors:
      - type: github
        id: github
        name: gitHub
        config:
          clientID: "<<CHANGE_THIS>>" 
          clientSecret: "<<CHANGE_THIS>>"
          redirectURI: "http://host.docker.internal/dex/callback"
          loadAllGroups: true

## API properties
api:
  enabled: true
  version: "2.6.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    databaseType: "H2"

## Executor properties
executor:
  enabled: true
  version: "2.6.0"  
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"

## Registry properties
registry:
  enabled: true
  version: "2.6.0"
  replicaCount: "1"
  serviceType: "ClusterIP"

## UI Properties
ui:
  enabled: true
  version: "2.6.0"
  replicaCount: "1"
  serviceType: "ClusterIP"

## Ingress properties
ingress:
  useTls: false
  ui:
    enabled: true
    domain: "ui.terrakube.docker.com"
    path: "/(.*)"
    pathType: "Prefix" 
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      cert-manager.io/cluster-issuer: letsencrypt
  api:
    enabled: true
    domain: "host.docker.internal"
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
  registry:
    enabled: true
    domain: "registry.terrakube.docker.com"
    path: "/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"
  dex:
    enabled: true
    path: "/dex/(.*)"
    pathType: "Prefix"
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: "proxy_set_header Authorization $http_authorization;"