How to Create a New Entra ID Enterprise Application and Configure Custom Attributes for SAML Login for AWS Cognito

Tue, 08 Oct 2024 00:00:00 +0000

When integrating Entra ID (formerly Azure AD) with AWS Cognito for SAML login, it’s important to use a unique attribute to identify users. In this guide, we’ll walk you through the steps to create a new Enterprise Application in Entra ID and configure a custom attribute named user.objectid. This attribute ensures that user identities remain consistent even if other attributes, such as last names, change.

Why use `user.objectid`?

The user.objectid attribute is unique to each user in Entra ID and does not change, even if other user attributes are updated. This is particularly important for scenarios where an employee changes their last name, as other attributes will be updated with the new value. Using user.objectid prevents the creation of a new local user in your application and ensures that existing user data is preserved.

Steps to Create a New Enterprise Application in Entra ID

Create a New Application

Navigate to the Azure Portal
- Open the Azure Portal and go to the Entra ID service.
Create a New Application
- Go to Enterprise applications and select New application.
- Choose Create your own application.
- Select the option to Integrate any other application you don’t find in the gallery (Non-gallery).
- Type the name of your new application and create it.

Create the connection between Entra ID and your application by setting the login URL and the identity of your application.

Select Single Sign-On Method
- In your Enterprise application, select the Single Sign-On option from the left menu.
- Click on SAML as the single sign-on method.
Edit Basic SAML Configuration
- Edit the Basic SAML Configuration.
- Add the Identifier (Entity ID) and Reply URL (Assertion Consumer Service URL).
  - The Identifier (Entity ID) should follow the format: urn:amazon:cognito:sp:cognito_userpool_id.
  - The Reply URL (Assertion Consumer Service URL) should follow the format: https://cognito_domain_url/saml2/idpresponse.
  - Save the changes to the Basic SAML Configuration.
Save the changes to the Basic SAML Configuration.

Assign the users and groups that should have permissions to log in to your application.

Assign Users and Groups
- Select the Users and groups option from the Enterprise application options.
- Click on Add user/group.
- Select the users and/or groups who should have access to your application.
- Confirm your selections and save.

Configure which Entra ID attributes should be used to log in to your application.

Edit User Attributes & Claims
- From the Single Sign-On option for your Enterprise application, edit the User Attributes & Claims.
Set Unique User Identifier
- Select the Unique User Identifier (Name ID) claim to edit it.
- In the Source attribute, set the value to user.objectid.
- Save the Changes

Update the AWS Cognito userpool

Once you have defined all the claim mappings on the Entra ID side, it is time to connect the dots on AWS’s side.

Retrieve SAML Federation Metadata

This is the intermediate step between configuring Entra ID and Cognito.

Get the Federation Metadata URL
- In your Entra ID Enterprise application, navigate to the Single Sign-On section.
- Locate the App Federation Metadata Url.
- Copy this URL, as it will be needed in AWS Cognito.

Enable the IdP

After all configurations are done on Entra ID side, you need to update the configuration in Cognito.

Enable the IdP
- In AWS Cognito, select the User Pool and go to the Sign-in experience tab.
- Under the Federated identity provider sign-in section, click on Add identity provider.
- Select the SAML type.
- Enter a name under Provider name.
- Under Identifiers, add the IdpIdentifier value.
- Use the recommended setting Require SP-initiated SAML assertions for the IdP-initiated SAML sign-in setting.
- Enter the metadata document endpoint URL saved previously.

This is a better solution than uploading the XML file because Cognito refreshes the metadata every 6 hours or before the metadata expires. This way, you don’t have to manually refresh the metadata XML every time the Entra ID SSL certificates expire or any other change occurs on the Entra ID side that would impact the federation authentication.

Configure Attribute Mapping

Configure the attributes that are stored in Entra ID and are mapped via the SAML schema in AWS Cognito. Here is a copy-and-paste friendly table for easier usage:

SAML Attribute	User Pool Attribute
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn	Profile
http://schemas.xmlsoap.org/claims/CommonName	Preferred User Name
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname	Given Name
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname	Family Name
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress	Email

Enable the External IdP for App Clients

Now that you have an IdP using the Entra ID configuration, you need to assign it to your application created in the Cognito userpool.

Enable the IdP for App Clients
- In AWS Cognito, navigate to the App integration tab, App client list section.
- Select the App client you want to configure and edit the Hosted UI section.
- From the Identity providers dropdown, select your newly created IdP (e.g., EntraID) and save the changes.

Test the Configuration

Click on the View Hosted UI button to quickly test your changes

Final Steps

After a user has successfully authenticated via the external IdP, it will automatically be created in your Cognito userpool with the “Enabled” status. The confirmation status will be set to EXTERNAL_PROVIDER.

The user attribute identities will store the metadata relating to the external IdP that “owns” this identity. This includes the user’s ID in the external IdP’s attribute, in our case, the “Identifier (Entity ID)”.

These fields will be updated on each successful authentication, so you can rely on the fact that the fields you receive via JWT attributes will be up-to-date.

Conclusion

By following these steps, you will have a fully functioning solution offering federated authentication with an external Entra ID. This setup ensures that user identities remain consistent and up-to-date, even when user attributes change.

Lazy Loading in C# (and not only)

Mon, 16 Sep 2024 00:00:00 +0000

Introduction

Lazy loading is a technique used to delay the execution of code for later on. There are a couple of reasons as to why you’d want to do that, and we’ll enumerate some:

Speed – the best code, is also the fastest code, the most secure and most maintainable… namely no code at all. However, for obvious reasons, we do need at times to write code, but while we can’t avoid writing it, we could delay or even, possibly, completely avoid executing it.
Memory footprint – RAM is still important, avoiding loading a heavy object in memory does not just result in increased speed, but also in a more efficient system, memory-wise.

Hands-on

While the theory sounds great, there still remains the question of how do you actually avoid calling code? While there are a couple of ways to do that, I’d like to focus on two of them:

Singletons

First, when it comes the the singleton pattern, objects (if we’re talking in an OOP context, but this goes for other paradigms as well) are often statically allocated. This affects us even more, as static code, often get initialized early on. For example in .NET Framework static fields get initialized before the constructor of the class is called. This is also the case for other platforms, like Java. Here an if check is usually employed for lazy loading:

   class Singleton 
   {
      private static Singleton _instance = null; // redundant, for explicity
      
      private Singleton { ... }
      
      public Singleton Instance()
      {
          if (Singleton._instance == null)
          {
             _instance = new Singleton();
          }
          return _instance;
      }
   }

In the above example ☝️ we can see how an if statement is used to check if our singleton has ever been instantiated before, if so, we just return that instance. However, if this is the first time, we create a first instance. This saves us some time and memory, because we might not get to use the class at all (depending on the use case) or we might just delay the execution.

Notice, the more moder .NET platform actually uses lazy loading by default. Static fields only get initialized before being used, and so, .NET does this for us, as opposed to the older .NET Framework.

Instance fields

The more interesting and common use case is when we’d like to delay the initialization of an instance field. For this, both frameworks (.NET and .NET Framework) come with build-in support, namely: Lazy<T>, a generic class which can wrap objects of any type and delay their initialization. We’ll explore a simple implementation of such an idea, and see how we can achieve this in pretty much any programming language.

Frist imagine the scenario of two classes:

public class Foo { ... }

and

public class Bar { ... }

with a composition relationship of type:

public class Foo 
{
   Bar bar = new();
   ...
}

Our goal is to delay the initialization of the field bar in any Foo instance.

From the get go, our solution needs to address any possible type, not just Bar. This is why generics are needed. So our solution begins to look as such:

   public class Lazy<T> { … }

Second, we’d need to know all about how exactly to create this bar object, and yet, delay the process… This sounds like a producer – a method that produces such and object, encorporates the how – and a callback (lambda or higher-order function), a method passed as argument, to be called when needed. This is exactly what we need! In .NET the Func<T> type holds a method (or function for you functional programmers) that takes no parameters and returns a T – Func comes in a buch of variations that also take arguments, the last one always representing the return value, such as Func<T,TResult>, Func<T,T,TResult> and so on.

We can use is as such:

   public class Lazy<T> 
    {
        private Func<T> _generator;
        public Lazy(Func<T> generator) => _generator = generator;

        public T Load() => this._generator();
        
    }

All that we’re providing to the constructor of the Lazy class is a function, that describes how to return a T, and whenever we want to actually return that T, we just call Load.

Let’s see this in action:

public class Foo 
{
   // old code: Bar bar = new();
   Lazy<Bar> bar_lazy = new Lazy<Bar>(() => new());
   ...
}

This looks great! Now, whenever we want our instance, we can just call Load on bar_lazy, but the beauty of warping things in a function is that, even if our Bar constructor required parameters, this would be fixed with just a small adjustment:

public class Foo 
{
    //ctor injection, provide `lazy_bar` as a dependency
    Foo(Lazy<Bar> lazy_bar)
    ...
}

// calling code
lazy_bar = new Lazy<Bar> (() => new Bar(name, age, other_param))
new Foo(lazy_bar);
...

It’s true that now we can’t really do an in-line initialization, but that would have been true even without our static wrapper, unless, of-course, we use static variables.

Improving security in AWS Cognito

Fri, 12 May 2023 00:00:00 +0000

A call for debate on how to better secure AWS Cognito

AWS Cognito is an identity management service for users who sign-up directly and for federated users who sign-in with external identity providers. It grants the ability to control access to web and mobile applications.

The user handling is being done via the User Pools while the identities and assign permissions for users are configured inside the Identity Pools

Over time, we found out that some Cognito user pools were deployed in their default configuration, making them possible honeypots.

Updating users via the public Cognito API:

We discovered that the app client configuration attributes are writable by default. As a result, if a user obtains a token with the aws.cognito.signin.user.admin scope, they can modify a local user’s attributes via the Cognito public endpoint (https://cognito-idp.REGION.amazonaws.com) using the x-amz-target header with the “CognitoIdentityProvider.UpdateUserAttributes” value. Moreover, a user can also delete its attributes or its account using the same approach.

Imagine that some developers are not aware of this fact and use custom attributes for tenant separation or RBAC. An attacker can breach the tenant separation, access foreign user data, etc. by modifying these attributes (e.g. by changing attribute “custom:tenant: companyA” to “custom:tenant: companyB”).

The generic solution

Remove the aws.cognito.signin.user.admin from the app client scopes. Keep in mind that this does not solve the issue for public clients. When authenticating directly against the Cognito public endpoint (initiateAuth) with a user and password flow (or others), you always get a token with the aws.cognito.signin.user.admin scope.
Remove the write-access permission in the app client configuration. Consider that this breaks in the case of federation with an external IdP because when a user signs in, Cognito updates the mapped attributes with the latest information from the IdP, even if its current value already matches the latest information. This happens automatically in Cognito’s backend involving no public APIs. Due to this nature, your SAML logins won’t be affected by blocking the API calls discussed below.

The custom solution

Block undesired calls towards the Cognito public API using AWS WAF rules. Basically, you need to block API calls using the x-amz-target header containing the “AWSCognitoIdentityProviderService.API_ACTION” string. A list of all APIs can be found here: https://docs.aws.amazon.com/cognito/latest/developerguide/user-pools-API-operations.html#user-pool-apis-auth-unauth-token-auth

You can read more about AWS WAF and Cognito here: https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-waf.html

You might be thinking how do you know which API calls should you allow and which should you block. You’re in luck because we also faced the same issue. Our suggestion is to initially create a WAF rule in count mode which tracks all API calls made by your Cognito userpool towards the public endpoint, centralize the data and afterwards build a new WAF rule that blocks all API calls except the ones tracked by the first rule.

Below is an example of a WAF rule that counts all the AWSCognitoIdentityProviderService calls via the “x-amz-target” header:

{
  "Name": "Cognito-counting-calls-to-public-api",
  "Priority": 0,
  "Statement": {
    "ByteMatchStatement": {
      "SearchString": "AWSCognitoIdentityProviderService",
      "FieldToMatch": {
        "SingleHeader": {
          "Name": "x-amz-target"
        }
      },
      "TextTransformations": [
        {
          "Priority": 0,
          "Type": "NONE"
        }
      ],
      "PositionalConstraint": "CONTAINS"
    }
  },
  "Action": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "Cognito-counting-calls-to-public-api"
  }
}

The next step is to create a WAF Regex pattern set where you define the allowed calls tracked with the above rule. For this example we allowed only the ^AWSCognitoIdentityProviderService.InitiateAuth$ and ^AWSCognitoIdentityProviderService.GetUser$ patterns:

Further you create the WAF rule that blocks all API calls initiated by your userpool towards the public API, except the patterns defined in the regex:

{
  "Name": "BLOCK-all-public-apis",
  "Priority": 1,
  "Statement": {
    "AndStatement": {
      "Statements": [
        {
          "NotStatement": {
            "Statement": {
              "RegexPatternSetReferenceStatement": {
                "ARN": "arn:aws:wafv2:REGION:ACCOUNT_NUMBER:regional/regexpatternset/REGEX_NAME/REGEX_ID",
                "FieldToMatch": {
                  "SingleHeader": {
                    "Name": "x-amz-target"
                  }
                },
                "TextTransformations": [
                  {
                    "Priority": 0,
                    "Type": "NONE"
                  }
                ]
              }
            }
          }
        },
        {
          "SizeConstraintStatement": {
            "FieldToMatch": {
              "SingleHeader": {
                "Name": "x-amz-target"
              }
            },
            "ComparisonOperator": "GT",
            "Size": 0,
            "TextTransformations": [
              {
                "Priority": 0,
                "Type": "NONE"
              }
            ]
          }
        }
      ]
    }
  },
  "Action": {
    "Block": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "BLOCK-all-public-apis"
  }
}

Besides the regex reference statement, we added an AND condition using the “Field to match” for the x-amz-target header greater than 0. This is useful for scenarios where the x-amz-target header is missing, allowing the calls to bypass our WAF block rule (example: SAML login where the Hosted UI is used).

Account takeover via unverified email/phone

Most of the user pools are configured with multiple login options, including email, username or phone. By default, the user pool option “Keep original attribute value active when an update is pending” is turned on. Make sure it stays like this to be protected.

If the option “Keep original attribute value active when an update is pending” is not turned on and your application consuming a Cognito issued token does not check the email_verified attribute but uses it directly to load the data/identify of a user, it will be exposed to a possbile takeover.

An attacker can change the email attribute value of its own user to impersonate a victim’s email address, then login to an application using an alternative login option like username. The application processing the Cognito issued token will see the victim’s email address and use the unverified email attribute to load data/identify of the user.

The solution

Make sure the Keep original attribute value active when an update is pending setting is on in all your Cognito userpools.
Modify your applications to respect the email_verified and phone_number_verified claims.
If possible, modify your applications not to rely on modifiable attributes like email, username, etc. Instead use the combination of immutable subject (sub) and issuer (iss) claims to identify a user. You can find extra details about the Cognito ID Token Payload here

Have you faced any of the scenarios above? How did you mitigate the issues? What else caught your attention when it comes to securing Cognito?

Configure Centralised S3 bucket replication from multiple S3 bucket sources

Wed, 11 Jan 2023 00:00:00 +0000

In Haufe we are using AWS Organizations service with hundreds of accounts and multiple OUs, therefore it was a challenge for us to offer a centralised backup solution for files.

AWS does not offer an out-of-box backup service for your files, so we needed to be creative.

Considering this requirement, we realised that S3 bucket replication can be a good candidate in order to achive our goal. Of course, having a one-to-one replicated bucket solution does not scale, therefore we were thinking to create ONLY one in the centralised replicated S3 bucket, which stores all the files from multiple S3 buckets sources.

Amazon S3 replication enables automatic, asynchronous copying of objects across Amazon S3 buckets. Buckets that are configured for object replication can be owned by the same AWS account or by different accounts.

There was one last challenge: how do we organise the centralised S3 bucket, in order to have a well structured folder/prefix for each source S3 bucket. The solution came from the centralised S3 bucket permission policy, where we used the ${aws:PrincipalAccount} context key.

Prerequisites

In order to keep data encrypted at rest, you have to create at least 2 KMS keys, used by s3 to encrypt data :

one in centralised account
one in each source account

From AWS console -> KMS -> Customer-managed keys -> Create key. One important aspect is to create a valid kms policy, which allows AWS services to encrypt/decrypt data. Below you can see the kms policy for each type of key:

KMS for centralised bucket, contains a condition based on OrganizationID, which allows all AWS accounts in our AWS Organization to use it

KMS for each source AWS account

Create S3 Buckets

We have to create 2 buckets one in each source account and another one in the centralised one. From AWS console, go to S3 service and select Create bucket

Write the name of the source/centralised bucket
Enable Bucket version
Enable server side encryption and specify kms key created in the previous step

The next step is valid ONLY for centralised S3 bucket, in order to have a valid policy based on aws:PrincipalAccount and including Organization ID condition:

Replication is configured via rules.

You have to create a rule in each source S3 bucket to replicate objects to the centralised S3 bucket. The bellow step is done only in sources AWS accounts:

Go to the Amazon S3 console
Click on the name of the source S3 bucket
Click on the Management tab
Click Create replication rule
Specify Replication rule name
Leave Status set to enabled
Choose a rule scope select, Limit the scope of this rule using one or more filters
- Specify for prefix/folder the account id value, in order to be sync with centralised S3 bucked policy based on ${aws:PrincipalAccount}
  - eg. 123456789123 <- account id
For Destination select Specify a bucket in another account
- Specify the account id of the centralised S3 bucket
- Specify the centralised S3 bucket name created in the prerequisites chapter
- Check box Change object ownership to destination bucket owner
For IAM Role leave Choose from existing IAM roles selected, and select Create a new role from the search results box
- A new IAM role will be created, which has privileges to both S3 buckets and to KMS keys in both accounts
For Encryption, choose Replicate objects encrypted with KMS keys and select Enter AWS KMS Key ARN
- For source objects check box the KMS key name created in the prerequisite step in source account
- For destination objects get the KMS key ARN from the centralised account and paste in the field
For Destination Storage Class, you can specify a destination storage class for objects replicated in centralised S3 bucket.
For Additional replication options, I recommend to check box only Replication Time Control, and do not select Delete marker replication, since Delete markers created by S3 delete operations will be replicated in centralised S3 bucket
Click Save

Test replication

To test this rule you will upload an object into the source S3 bucket in account id prefix and observe that it is replicated into the centralised S3 bucket. For this step you will need a test file

As seen, the object uploaded to source S3 bucket in the account_id prefix is replicated in centralised S3 bucket to the same prefix(account_id). This is the way you can configure a centralised S3 bucket for multiple source S3 buckets, splitting the source buckets based on the account_id prefix.

AWS Gameday

Mon, 10 Oct 2022 00:00:00 +0000

A few weeks ago our company hosted an AWS Gameday event throughout our sites. There aren’t too many details online about it so I decided to give you a short heads-up.

If you get the chance to take part in it, don’t miss out. It’s really fun and in the end everyone agreed they learned something new.

The teams will be created based on the individual AWS experience of the participants, trying to keep a balance and a sense of competitiveness. In our case, each team had 3 members passing from juniors to experienced users.

Half an hour before the start you get all the details about the actual game, the entire “show” will be about 4-5 hours with no actual breaks. You can take a snack or a coffee break at any time but this will impact your teams ranking 😄

During the activity there will be a live ranking system displayed on a local TV/large monitor where you can see how your team is performing. This might give you a sense of being on the Wall Street…

You’ll get access to AWS accounts with deployed infrastructure so prepairing with templates/scripts won’t help, it will just consume your time and you won’t have much, those 5 hours fly really fast.

There are two ways to win points during the game. One is to configure and tweak your resources to be available as much as possible for other teams to use. The other is to consume data from other teams and this adds an extra spice to the entire gamble.

Every now and then, some changes will be done to your existing infrastructure which will cause a bit of chaos. This is a great moment to be on your toes and climb the rakings.

Try to delegate separate tasks to each member because details matter… 👀

I’m not going to mention the services used because it would be a major spoiler but don’t worry, they are really common ones, so you’ll be just fine.

I wish you all the fun a geek can have and enjoy it!

Aws Sso From Adconnect To Azure Ad

Thu, 01 Sep 2022 00:00:00 +0000

Secure your website access with Kubernetes NGINX Ingress Controller, OAuth2 and Azure AD

Mon, 22 Aug 2022 00:00:00 +0000

Protecting resources behind a Kubernetes Ingress, is often NOT an easy task. Many applications do not provide built-in authentication out-of-the-box. Therefore, creating an awesome user experience, including a single sign-on solution across all applications, can quickly become a tedious task. In Haufe, we are using Kubernetes for a while now. One of our common practices with Kubernetes is using NGINX Ingress Controller for the main ingress controller whether we deploy it using EKS or AKS.

How does it work

We use the NGINX Ingress Controller built-in functionality to define an external service to provide authentication for the ingress. Before passing a request to your app, the ingress will check whether the user is logged in or not by sending a request to proxy /oauth2/auth endpoint and depending on its response it will pass the request or redirect the user to /oauth2/start?rd=$escaped_request_uri endpoint.

metadata:
  name: application
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"

NGINX Ingress Controller can be combined with oauth2_proxy to enable many OAuth providers like Google, AzureAD, GitHub and others. I am going to use OAuth2 Proxy together with the NGINX Ingress Controller to authenticate my Azure AD account against the Kibana website.

Prerequisites

EKS cluster (or any other type of Kubernetes cluster)
Global Administrator, Cloud Application Administrator or Application Administrator permissions into Azure AD in order to be able to create a new App registration.
ElasticSearch cluster and Kibana
Helm3
kubectl

Register an Application

Before you start, you need to register an application in Azure AD and create a client secret for it. You will need this for OAuth2 Proxy later, so write down the Application Client Id for later and make sure you enter a Redirect URI. The callback URI for the OAuth2 Proxy is of the form https://<your_company_domain>/oauth2/callback.

In the next steps you will have to create a client secret and write it down for later use as well.

Install Oauth2 Proxy

The OAuth2 Proxy deployment manifest oauth2-proxy.yaml needs some placeholders to be replaced using the values from the previous step:

    - name: OAUTH2_PROXY_CLIENT_ID
      value: <Application Client ID> # replace with client id
    - name: OAUTH2_PROXY_CLIENT_SECRET
      value: <Client Secret> # replace with client secret
    - name: OAUTH2_PROXY_COOKIE_SECRET
      value: <Random Secret> # replace with value of: python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode("ascii"))'

Also, the <Tenant ID> placeholder needs to be replaced with your Azure AD tenant id.

After you defined the variables with the correct values, you can create the oauth2-proxy deployment

# Create namespace
kubectl create namespace demo-oauth2

# Make sure the variables above are available in the shell
kubectl apply -f oauth2-proxy.yaml

Installing the NGINX Ingress Controller

You need to install NGINX Ingress-controller, in order to allow Kubernetes to understand Ingress objects. Since I mentioned earlier that I used EKS from AWS, I need to setup NGINX Ingress Controller k8s service to use Network Load Balancer.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Installs NGINX Ingress Controller
helm install nginx ingress-nginx/ingress-nginx \
    --namespace demo-oauth2 \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="nlb" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-cross-zone-load-balancing-enabled"="true"

Prepare ElasticSearch cluster and Kibana

In order to have a valid example, you need to have a website, where you can verify authentication. My option was a Kibana application, but feel free to choose what fits your need. Going further with this option, you have to install ElasticSearch and Kibana using helm charts

# Add the Elastic Helm charts repo
helm repo add elastic https://helm.elastic.co

# Install ElasticSearch 
helm install elasticsearch-demo-outh2 elastic/elasticsearch \
--namespace demo-oauth2 \ 
--set clusterName=demo-oauth2 \
--set replicas=1  \
--set minimumMasterNodes=1


# Install Kibana with Ingress object configured
helm install kibana-demo-outh2 elastic/kibana \
--namespace demo-oauth2 \
--set elasticsearchHosts="http://demo-oauth2-master:9200"

Deploy Ingress objects

No, it was not a typo in the step title: you need to deploy 2 Ingress objects, both of them pointing to the same host.

First Ingress object needs to be annotated in such a way that it requires the user to authenticate against the second Ingress’s endpoint and can redirect 401s to the same endpoint.

The second Ingress objects exposes the oauth2-proxy service via the /oauth2 path and handles the actual authentication.

You can use the kibana-ingress.yaml example, but remember to replace the placeholders (for Ingress host, service name and port) with your own values.

# Deploy ingress objects
kubectl apply -f kibana-ingress.yaml

Test your Setup

Once your NGINX endpoints come online, test the configuration by navigating to specific url in your web browser Login with your credentials

Access granted

Recommendation

Of course if you would like to setup a proper environment, I recommend to install cert-manager helm chart, in order to generate valid certificates and activate HTTPS endpoints for your FQDN.

How to: login into Kibana/AWS OpenSearch using Azure AD

Fri, 17 Sep 2021 00:00:00 +0000

AWS released native SAML authentication support for Kibana back in October 2020 so we decided to give it a try and drop “the legacy” Cognito & ADFS integration. Our goal is to enable fine-grained access control into Kibana roles based on Azure AD groups.

Prerequisites

a deployed AWS OpenSearch stack
Global Administrator, Cloud Application Administrator or Application Administrator permissions into Azure AD in order to be able to create a new Enterprise application.

Step 1

Go to the Azure Portal, open the Azure AD service, go to Enterprise applications and select New application (Create your own application). The type of your new application should be “Integrate any other application you don’t find in the gallery (Non-gallery)”, type the name of your new application and create it (in this example, my app is called “Kibana login with Azure AD”).

Step 2

Login into your AWS account, go to the OpenSearch service, select the Actions drop-down button, click on Modify authentication and select the Enable SAML authentication. From here, you will use the “Service provider entity ID” & “SP-initiated SSO URL” information in the next step. Scroll down to the “SAML master username (optional)” section and put your AD username (HawkeyeP@example.com) so you will be able to login as the master user and do the role mappings in Kibana, later at step 7.

Step 3

Go back in Azure AD to your Enterprise application, select the Single Sign-On option from the left menu and click on SAML as the single sign-on method. Edit the “Basic SAML Configuration” and add as the “Identifier (Entity ID)” the “Service provider entity ID” taken previously from OpenSearch service, then add as the “Reply URL (Assertion Consumer Service URL)” the “SP-initiated SSO URL” taken previously from OpenSearch service.

Step 4

Configure the users/groups who have permissions to access your Enterprise application and then federate into your OpenSearch instance. Select the “Users and groups” option from the Enterprise application options, click on “Add user/group” and select who should have access to your application. In our example, we want to add an Azure AD group which will later be mapped into a role in Kibana (eg: active_directory_admin_group).

Step 5

From the Single Sign-On option for your Enterprise application, edit the “User Attributes & Claims”, click on “Add a group claim”, select as claim the “Groups assigned to the application” option and the “Source attribute” as “sAMAccountName”.

This will send to your OpenSearch instance, as an attribute, the Azure AD groups you configured to have access to your Enterprise application. This information will then be used in “Step 6” to map the group name into a Kibana backend role.

The last step in Azure AD is to download the Federation Metadata XML file and to upload it into your SAML configuration from the OpenSearch service.

REMEMBER: the Enterprise application uses a selfsigned certificate which is valid for 3 years - keep a record of this, in case you are running the setup in production.

Step 6

Login into your AWS account, go to the OpenSearch service, select the Actions drop-down button, click on Modify authentication and in the “SAML authentication for OpenSearch Dashboards/Kibana” section, click on “Import from XML file” and select the file you just downloaded in the previous step. You will see that the “IdP entity ID” section will auto update with the Azure Enterprise application ID. The last step is to add in the “Optional SAML settings - Roles key” section the “http://schemas.microsoft.com/ws/2008/06/identity/claims/groups” value.

Step 7

With the Azure AD user you added as “SAML master username (optional)” in Step 2, login into your OpenSearch application and map to a backend role, the Azure AD group you allowed to login into your Enterprise application in Step 4.

Congrats, you are now able to login into Kibana using an Azure AD group as a backend role.

Getting full transparency throughout your DevOps process

Mon, 07 Dec 2020 00:00:00 +0000

Getting full transparency throughout your DevOps process

DevOps is one of those terms, which is interpreted differently across our profession. Sometimes as a set of tools used for deployment automation, sometimes as the two guys in the basement, who administer our Kubernetes cluster. Much too often, we can observe the same pattern, as we did - and still do - with Agile: Using new terms while doing old things. Which leads to both, not gaining the benefits of the new approach, while keeping the known drawbacks from the old one.

As it was for Agile, DevOps should much rather be seen as a holistic approach, which only unleashes its full potential when its underlying ideas, principals, and mindset are understood and applied in our daily work¹. So I rather consider DevOps a consequent evolution of Agile: Empowering self-responsible teams to creating products in a customer-centric way. By eliminating time-consuming hand-overs or endless review processes before the first release to production. That is done leveraging a shift-left approach. Shift-left describes shifting both, responsibility and empowerment from the right side of a traditional software development workflow - manual user acceptance testing, security audits, deployment, monitoring, user-behavior analytics - to the left: an empowered development team, having proper skills to embed all those steps within its regular software development process.

Principal DevOps metrics

One of the key principles behind DevOps is bridging the gap between development and production. Increasing transparency across all stages of your product- and software development lifecycle can vastly help to build these bridges.

This article concentrates on showing how to integrate metrics, which cover that whole lifecycle on one central dashboard, every team member has easy access to. This might be by having it on a big screen on the way to the coffee machine.

The most central abstract DevOps metrics are the Delivery Lead Time, Deployment Frequency, Mean Time to Repair (MTTR), and Change Fail Rate (rep², hdb³, acc⁴).

These Metrics are reflected and/or influenced by several other metrics. One example would be open merge- or pull requests, which influence the first three of them. The same would be for the time you need to fully build and deploy your product (end-to-end time of your CI/CD pipeline). Whereas plenty of other metrics, which concentrate on your code’s quality, are likely to correlate to your Change Fail Rate.

The following overview depicts the whole lifecycle, we are talking about. This article is about creating a central dashboard, which shows key metrics from all of these stages. Helping to improve the high-level overall DevOps metrics, described above.

I am going to write some further articles, each covering some topics from this overview.

Systematic presentation of all stages throughout the software development lifecycle. From writing code to running it in production.

One Dashboard for the whole cycle

Our dashboard combines data from these areas and sources:

Open Merge Requests (GitLab)
Software metrics: Code Quality and SAST (SonarQube)
Status (and duration) of build pipelines for multiple repos (GitLab)
Product version, which is deployed to any environment (custom API)
Estimated duration of a job queue (Jenkins)
Status and health of an Elasticsearch cluster (Elasticsearch)
APM: Requests per minute, Apdex score, Error rate, … (NewRelic)
Analytics: Active users (also divided by instance / tenant) (NewRelic)

The complete dashboard.

All elements on the dashboard aim at creating transparency about the key stages across the whole product lifecycle. The overview of open merge requests, for instance, shall encourage developers to tackle them. Since code, which waits in this phase, does not add any value to our customers.

The software metrics, including potential vulnerabilities, help to identify issues at the earliest stage possible. Since the earlier an issue is spotted, the quicker and cheaper it can be handled (as opposed to having QA, or worse, the customer find and report it). So these metrics shorten the feedback cycle.

Development

Instead of focusing on reducing the MTBF (Mean Time between Failures) by extensive up-front testing and lengthy review processes, modern software development rather tends to focus on minimizing the MTTR (Mean Time to Repair). No matter how well your (manual) tests are, eventually your software will fail. Especially since not all of the real-world scenarios can be simulated during tests. So it might be wiser to “prepare for the worst” instead of “hoping for the best“⁵.

Being able to quickly respond to errors in production is the most valuable asset. To address this, we visualize several metrics concerning our build pipelines: what is their success rate (reliability), how long do they take (speed) and how is that speed evolving over time (to prevent losing speed without noticing it).

Development related figures: Software metrics coming from static code analysis, open merge requests and memory consumption of our Kubernetes clusters, where most of our build pipelines run into.

Build

Having several services running and aiming at a high deployment rate increases the need to know which version of your services are running at a given time. For that, we added a widget, which displays the currently deployed version across all our services and stages.

After covering delivering new value as reliably and quickly as possible, you are interested in two things: first, is it working as expected and has no negative side effects on your application and, second, do your customer like it.

To address these two questions, we added visualizations for response times and error rates. Allowing to quickly react, before any customer needs to pick up the phone to call the help desk. Besides, we added some analytics to see the number of users currently using the application.

Success rates of the last five builds and build duration over time. The color changes to yellow or red if certain thresholds are met.

Operation

By the way, this also helps to visualize the impact we as a DevOps team have. We see actual users using what we have built. And we see how that affects response times and error rates. Opposed to implementing a feature, then having it to pass through several stages, including manual QA, eventually being deployed by the ops department. The next time we hear from it might be when pulling some associated bug ticket from the backlog some weeks later.

Allowing developers to have impact and making this impact transparent, thus, is a great means to boost the team’s satisfaction and motivation. And there, the cycle closes. Since having a self-motivated and self-responsible team are the best preconditions to build great software.

This part of the dashboard covers the application running in production: Number of users and requests, the Apdex Score and the error rate.

Our dashboard contains several views / pages. This comes in handy to prevent overloading a single page. The dashboard automatically switches between these pages after a defined period (e.g. every 30 seconds). We will see how this is implemented in the next section.

We used this mechanism in combination with the possibility to embed foreign Dashboards: We already had dashboards in various other tools, for instance Kibana. Whereas we also query data from the underlying Elastic Stack to show them in one of the shown widgets, it would not make sense to try to recreate all existing dashboards again. Instead, we are integrating them as they are.

This one shows several aggregations for our production logs: Number of messages grouped by level, within each level, we aggregate for the classes, the messages come from. This easily enables us to see which classes produce the most errors or warnings.

The date histogram quickly reveals unusual behavior, which helps us to react to possible issues before they hit the customers - or even better: react to warnings before they become problems at all.

Integrated Kibana dashboard: Smashing has a special widget, which allows embedding iframes. Here we leverage this to embed a full-page Kibana dashboard.

Technical implementation

As mentioned, since these metrics refer to different areas of the product lifecycle, they typically need to be gathered from different systems. So we were not looking for a holistic monitoring solution, which contains all needed data (like e.g. Prometheus/Grafana), but a dashboard, which can integrate data from different services and present them in a holistic view. Without permanently storing these data itself. This allows having an almost stateless tool, where we do not need to bother providing and maintaining some persistence solution.

The dashboard we ended up with, is Smashing, the successor of the no longer maintained Dashing (thanks to Thomas Doerr, who introduced me to Dashing a while ago).

Smashing allows placing widgets on a board, which fetch their data from arbitrary sources but presenting them in a unified way and a unified layout. There is a whole bunch of widgets available to fetch data from certain services (e.g. GitLab, GitHub, Google Analytics, NewRelic, Jira, …). Nevertheless, it is quite easy to write widgets to present data from any service, which provides some kind of API access.

Connecting a custom source

Connecting to a custom data source is quite easy. Dashing/Smashing uses so-called Jobs to periodically fetch data from various sources and display the metrics on the board. These Jobs are Ruby files and are located within your dashboard’s jobs subfolder.

The relevant parts of our job, which fetches data from SonarQube looks like this:

Since we have different jobs for different repositories / applications, we extracted some commonly used methods into sonar_methods.rb. These methods are included in our actual job:

require_relative 'sonar_methods'

Afterward, we read some configuration parameters from the job’s configuration sonar.cfg. For instance, the SonarQube server URL, the widget’s id, where we will show the results and the interval in which the data gets refreshed.

# Load configuration file
config_path = File.expand_path(File.join(File.dirname(__FILE__), "sonar.cfg"))
configuration = Hash[File.read(config_path).scan(/(\S+)\s*=\s*"([^"]+)/)]

# Read parameters
server = "#{configuration['server']}".strip
id = "#{configuration['widget_id']}".strip
interval = "#{configuration['interval']}".strip
# ...

Sensitive configuration parameters like credentials are read from the environment resp. a local .env file to not have them checked into source control:

unless ENV['SONARQUBE_ACCESS_TOKEN']
    abort("MISSING SONARQUBE_ACCESS_TOKEN variable (check your .env file)!")
end

accessToken = ENV['SONARQUBE_ACCESS_TOKEN'].strip

TIP: In an upcoming blog post, I am going to show how to keep Secrets-as-Code, having them properly encrypted using SOPS.

The job gets executed within the Scheduler:

SCHEDULER.every interval, :first_in => 0 do |job|

    # Get data
    data = getIssueCountBySeverities(server, accessToken, groupId, artifactId, severities)
    data.concat(getFurtherMetrics(server, accessToken, groupId, artifactId))

    # Pass the retrieved data to a widget on the dashboard
    send_event(id, { items: data })
end

The send_event publishes the data to the widget, referenced by its id.

That’s all related to the general setup, scheduling, and execution of the job. How to fetch the data from the corresponding services differs according to their APIs. Here are the methods we use to fetch data from SonarQube (the included sonar_methods.rb):

require 'net/http'
require 'json'
require 'openssl'
require 'uri'

@metricLabels = { 
    "bugs" => "BUGS",
    "vulnerabilities" => "VULNERABILITIES",
    "code_smells" => "SMELLS",
    "coverage" => "COVERAGE (%)",
    "duplicated_lines_density" => "CODE DUPLICATION (%)"
}

def getIssueCountBySeverities(baseUri, accessToken, groupId, artifactId, severities)

    issuesBaseUri = "#{baseUri}/api/issues/search?componentKeys=#{groupId}%3A#{artifactId}&statuses=OPEN&severities="

    severitiesArray = severities.split(',')
    data = []

    severitiesArray.each do |severity|
        uri = URI(issuesBaseUri + severity)
        req = Net::HTTP::Get.new uri.request_uri
        req.basic_auth accessToken, ""
        res = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https', :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
            http.request(req)
        end
        data.push({:label => severity, :value => JSON[res.body]['total'] })
    end
    return data
end

def getFurtherMetrics(baseUri, accessToken, groupId, artifactId)
    
    uri = URI("#{baseUri}/api/measures/component_tree?component=#{groupId}%3A#{artifactId}&metricKeys=bugs,code_smells,coverage,vulnerabilities,duplicated_lines_density")
    req = Net::HTTP::Get.new uri.request_uri
    req.basic_auth accessToken, ""

    res = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https', :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
        http.request(req)
    end

    metricData = {}
    data = []
    
    JSON[res.body]['baseComponent']['measures'].each do |metrics|
        metricData[metrics['metric']] = metrics['value']
    end

    @metricLabels.each do |key, label|
        data.push({:label => label, :value => metricData[key] })
    end

    return data
end

Embedding already rendered widgets

Besides the possibility to render widgets from source data, Smashing also allows integrating already rendered widgets. This is done using the data-view="Iframe". We are using this to embed a full page Kibana dashboard (using max values for data-sizex and data-sizey):

<div class="gridster">
  <ul>
    <li data-row="1" data-col="1" data-sizex="7" data-sizey="4">
      <div data-id="logs_grouped" data-view="Iframe" data-src="https://uri/app/kibana#/dashboard/baba891a-b0a9-11ey-879b-7de326663eab?embed=true"></div>
    </li>
  </ul>
</div>

TIP: The embed=true URL parameter leads to hiding unnecessary control elements, like the search bar. So it automatically opens it in full-screen mode.

Automatically browse through several dashboards

As shown above, one quickly has more meters than fit on one screen. To keep your dashboards neat and clear - allowing spotting the most important elements at first glance - you can consider splitting the meters to various dashboards.

There is a plugin for Smashing, which automatically browses through a defined list of dashboards in a defined period.

Having the plugin installed and configured, you can start the browsing mode by appending /_cylce to your dashboard’s URL. The period can be passed via an URL parameter. This example would switch the board every 30 seconds:

https://your.dashboard:3030/_cycle?duration=30

Securing access

Depending on where you run the dashboard and which data it contains, you may want to restrict access to it. That can be done i.e. via BasicAuth. Within Dashing’s configuration (config.ru) you’ll find a placeholder for it:

configure do

  # [...]

  helpers do
    def protected!
      # Put any authentication code you want in here.
      # This method is run before accessing any resource.
    end
  end
end

To activate BasicAuth, using credentials USER_NAME and USER_PASS from the environment, adjust it like this:

require 'dashing-contrib'
require 'dashing'

DashingContrib.configure do

  # [...]

  helpers do
    def protected!
      unless authorized?
        response['WWW-Authenticate'] = %(Basic realm="Restricted Area")
        throw(:halt, [401, "Not authorized\n"])
      end
    end

    def authorized?
      @auth ||=  Rack::Auth::Basic::Request.new(request.env)
      @auth.provided? && @auth.basic? && @auth.credentials && @auth.credentials == [ENV['USER_NAME'].strip, ENV['USER_PASS'].strip]
    end
  end

end

Dockerize the Dashboard

Finally, we dockerized the dashboard. Allowing us to deploy it to a local workstation, which is attached to a big screen, or to deploy it to some cloud service - which for sure is needed if you have distributed teams. Or, like these days, home-office becomes more important.

Make technology and culture reinforce each other

Making the complete software development lifecycle transparent to the whole team brings a lot of benefits. Besides spotting possible bottlenecks, quality-, security- or performance issues, it shows the team the impact it has.

That’s a great way of supporting a team’s satisfaction, self-awareness and self-responsibility.

By that, technology helps to support the culture, whereas culture leads to improved technology. This mutually reinforcing cycle is the best basis for building great software.

Endnotes and Literature

1: Aiming in that direction, Jez Humble, author of Continuous Delivery (2010), felt the urge starting to work on a DevOps Manifesto, which stresses DevOps to be rather a “philosophy” and a “cultural, professional movement with attitude and values” rather than “a role, a set of tools” or “a prescriptive process”.

2: rep: State of DevOps Report

3: hdb: Gene Kim, Patrick Debois, Jez Humble, John Willes: The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, 2016.

4: acc: Gene Kim, Jez Humble, Nicole Forsgren: Accelerate. The Science of Lean Software and DevOps. Building and Scaling High Performing Technology Organizations, 2018.

5: elev: Gregor Hohpe: The Software Architect Elevator: Redefining the Architect’s Role in the Digital Enterprise, 2020.

Let's Scale - Part 2!

Sat, 04 Jul 2020 00:00:00 +0000

Part 2: Keep your Infrastructure Code Clean!

Retro

In part-1 we took a short look on Haufe`s cloud journey and how Terragrunt can boost your Terraform modularization by use DRY Terraform code. In the this part of the series, we take a look which tools from the Terraform community could support your daily work to keep your Terraform code clean, smiliar to keeping your app code clean.

Generate automatically Documentation with Terraform-Doc

What I get from it?

TF-Docs generates and updates documentation for Terraform modules automatically, for input and output variables. This helps when you decide to split Terraform code ( something you can do with Terragrunt ) into a lot of small composable modules. You can choose between various output formats like markdown text or tables. Therefore you can just add it to your readme.md files in your repository for each module. With that TF-Docs is a consistent shortcut to keep your module documentation up to date on the same place where the code is, without writing it.

This is how it looks like as a sample markdown table:

How does it work?

TF-Docs is a utility tool written in Golang which is available by MIT license. After local download you can run it via bash, or how we see later, add it via pre-commit to your git workflow.

It supports terraform v0.12.

Install it with brew or linuxbrew*:

brew install terraform-docs

If you use Windows, WSL is an easy way to use it.

Run it this way to get a markdown text:

terraform-docs markdown .

Run it this way to get markdown table:

terraform-docs markdown table .

TF-Docs prints the description of your variables and outputs, as following sample shows:

This code …:

variable "environment" {
  description = "Environment tag, e.g prod"
}

variable "region" {
  description = "WS Region, e.g eu-central as default"
  default = "eu-central-1"
}

output "public_ip" {
  value = "${aws_eip.bastion.public_ip}"
  description = "Bastion hosts external IP address"
}

… outputs to this documentation:

# This is a Sample Readme

Lorem ipsum dolor sit amet, cum sint soluta instructior ut, eleifend efficiantur eam in. Cu tota splendide sed, mel discere appellantur cu.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|:----:|:-----:|:-----:|
| environment | Environment tag, e.g prod | string | - | yes |
| region | AWS Region, e.g eu-central as default |eu-central-1 | string | - | yes |

## Outputs

| Name | Description |
|------|-------------|
| public_ip | Bastion hosts external IP address.|

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

What I have to consider?

I personally think it is a best practise to add it as a githook to your git workflow, to use it in a consistent way. If you are a small team, you may not benefit as much from it as you would if you are platform team which shares the code with multiple teams. Also you could say, Terraform for itself is always an always-up-to-date document. Nevertheless with .md integration it works nicely!

Run Security Scans with Terraform-Sec

What I get out of it

TF-Sec performs vulnerability tests for your Terrafrom configuration in the following way:

Checks for sensitive data inclusion across all providers
Checks for violations of AWS, Azure and Google security best practice recommendations
Scans modules (currently only local modules are supported)
Evaluates expressions as well as literal values

The complete provider specific tests are listed here.

How does it work?

The tool is written in Golang and supports terraform v0.12. Install it with brew or linuxbrew. If you use Windows the WSL is a easy way to run bash commands:

brew tap liamg/tfsec
brew install liamg/tfsec/tfsec

It works by scanning your directory recursively by running it on your terminal or the terminal from your ci/cd tool. You can choose between varrious output formats like JSON or jUnit.

Below you will find an example issue with S3. TF-Sec gives you a list of all findings, an explination per finding, the issue id and a link to get more information about each listed issue.

Terraform Code:

resource "aws_s3_bucket" "my_secure_bucket" {
    bucket = "my_bucket" 
}

TFSEC Report:

Problem 1
      
[AWS017][ERROR] Resource 'aws_s3_bucket.my_secure_bucket' defines an unencrypted S3 bucket (missing server_side_encryption_configuration block).

      /tfsec_test/s3_bucket.tf:1-3

           1 | resource "aws_s3_bucket" "my_secure_bucket" {
           2 |   bucket = "my_bucket"
           3 | }
           4 | 

      See https://github.com/liamg/tfsec/wiki/AWS017 for more information.
      

You can also choose to ignore specific issues by adding a flag to your Terraform code.

resource "aws_s3_bucket" "my_secure_bucket" {
    bucket = "my_bucket" 
     #tfsec:ignore:AWS017
}

Additionally, you can also disable specific checks for all modules.

What I have to consider?

If you run this tool and have fixed the promoted issues, it does not mean you are fully protected afterwards, which is clear. But you can add it as an additional step to your CI/CD pipeline with the single command “tfsec .” and you then have then made the world a bit safer place ;)

Keep your Terraform Code Clean with Terraform-Lint

What I get from it?

On your IDE or in an editor like VS Code, you can snap in Terraform-Lint for several programming languages as well as for IaC syntaxes like the Hashicorp language HCL linters. The good thing about TF-LINT is, it is a part of the other open source tools shown here in the article, and you can integrate it as a githook the same way as the other tools. Main advantagie is that it validates provider specific issues which terraform/terragrunt validate, plan or apply doesn`t find.

How does it work?

No suprise, this is like all other tools written in Golang and supports terraform v0.12. You can install it with brew or linuxbrew:

brew install tflint

TF-Lint inspects all configurations under the current directory by default:

tflint .

It has rules for Azure as well as for AWS: AWS rules have been further developed by the community. Here you find a list of the rules. Below, you get a sample for how it works when you run it:

Terraform Code:

resource "aws_route" "my_route_table" {
  route_table_id         = "rtb_for_something"
  destination_cidr_block = "10.0.1.0/16"
}

TFLINT Report:

1 issue(s) found:

Error: The routing target is not specified, each aws_route must contain either egress_only_gateway_id, gateway_id, instance_id, nat_gateway_id, network_interface_id, transit_gateway_id, or vpc_peering_connection_id. (aws_route_not_specified_target)

  on template.tf line 1:
   1: resource "aws_route" "foo" {

Reference: https://github.com/terraform-linters/tflint/blob/v0.11.0/docs/rules/aws_route_not_specified_target.md

You can run it as well as all the other tools locally, or in your pipeline as an additional stage like following blueprint example shows:

.gitlab-ci.yaml:

stages:
- lint

terralint:
  stage: lint
  image:
    name: "$CI_REGISTRY/my_cicd_image"
  script:
    - tflint .
  tags:
    - my-runner

What I have to consider?

TF-Sec support specific TF versions in certain ways.

Integrate all the Terraform Helpers in One to your Git Workflow with Pre-Commit

What I get from it?

Pre-commit enables you to add all the listed utility tools to your git workflow. More Terraform integrated tools can be found as ready-to-use githooks on github within this project pre-commit-terraform. Besides the tools I already explained, pre-commit-terraform also uses the default Terraform function with “terraform fmt” for formatting and “terraform validate” for validatation as well as the same for Terragrunt.

Here you see the complete list of general available githooks.

How does it work?

To install the pre-commit tool and all the Terrform helpers as dependencies run:

brew install pre-commit gawk terraform-docs tflint tfsec coreutils

To install the pure pre-commit run following command:

brew install pre-commit

After installation you can add a ‘.pre-commit-config.yaml’ for pre-commit configuration to your local git root and configure each tool. Following sample lists the tools configuration from this article:

---

repos:
- repo: git://github.com/antonbabenko/pre-commit-terraform
  rev: v1.30.0 # Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases
  hooks:
      - id: terraform_docs
        name: Terraform docs
        description: Inserts input and output documentation into README.md (using terraform-docs).
        require_serial: true
        entry: terraform_docs.sh
        args: [--args=--with-aggregate-type-defaults]
        language: script
        files: (\.tf)$
        exclude: \.terraform\/.*$

      - id: terraform_tfsec
        name: Terraform validate with tfsec
        description: Static analysis of Terraform templates to spot potential security issues.
        entry: terraform_tfsec.sh
        language: script

      - id: terraform_tflint
        name: Terraform validate with tflint
        description: Validates all Terraform configuration files with TFLint.
        entry: terraform_tflint.sh
        language: script
        files: (\.tf|\.tfvars)$
        exclude: \.terraform\/.*$

Afterwards, you can run pre-commit with the specific command in your terminal as a test. It is also executed in the background before you commit or push as soon as you have added it as a githook. Below you see a sample screenshot of how the report looks like after execution:

What I have to consider?

You need to like to work with terminal tools and the git command line: I really believe that you can boost your collaboration for IaC, by using these little helpers.

Testing your Terraform Modules as Units

I haven’t had time to take a deep look into it yet, but if you heavily rely on Terraform, the community ecosystem also has Unit-Test-Frameworks to offer. Terraform offers to write Golang unit tests for custom plugins, Terratest enables you to write Golang unit tests for your Terraform code and include Docker, Helm, and more. Terratest can also be added as a githook[https://github.com/gruntwork-io/terratest/blob/master/.pre-commit-config.yaml] or to your ci/cd. As an alternative, kitchen-terraform is a Ruby based test framework.

Summary

In this series, we have seen how Terraform community tools can expand your IaC ecosystem and boost your scalability. Terragrunt helps you to get DRY code for your Terraform modules, which can be useful if you think about sharing, reusing and providing infrastructure. Pre-commit-terraform gives you the ability to enhance security, linting, testing and documentation within your git workflow for your team or teams. In addition, the tools are easy to add to your ci/cd pipeline, like we saw with the example of Terragrunt and Terraform-Lint, by adding a Docker Container to your pipeline and running a single bash command.

If you are responsible for one monolthic product with only 2 environments like dev and prod, Terragrunt has less impact on your scalability. It is different, if you share IaC for multiple environments, services, teams and accounts. If and how much these tools helping you to scale also depends on how heavily you work with Terraform and whether you view IaC as a critical part of your software development skill portfolio.

How to Create a New Entra ID Enterprise Application and Configure Custom Attributes for SAML Login for AWS Cognito

How to Create a New Entra ID Enterprise Application and Configure Custom Attributes for SAML Login for AWS Cognito

Why use user.objectid?

Steps to Create a New Enterprise Application in Entra ID

Create a New Application

Configure Single Sign-On (SSO) login

Configure the User Access for SSO login

Configure User Attributes & Claims for SSO Login

Update the AWS Cognito userpool

Retrieve SAML Federation Metadata

Enable the IdP

Configure Attribute Mapping

Enable the External IdP for App Clients

Test the Configuration

Final Steps

Conclusion

Lazy Loading in C# (and not only)

Introduction

Hands-on

Singletons

Instance fields

Improving security in AWS Cognito

A call for debate on how to better secure AWS Cognito

Updating users via the public Cognito API:

The generic solution

The custom solution

Account takeover via unverified email/phone

The solution

Have you faced any of the scenarios above? How did you mitigate the issues? What else caught your attention when it comes to securing Cognito?

Configure Centralised S3 bucket replication from multiple S3 bucket sources

Prerequisites

Create S3 Buckets

Replication is configured via rules.

Test replication

AWS Gameday

A few weeks ago our company hosted an AWS Gameday event throughout our sites. There aren’t too many details online about it so I decided to give you a short heads-up.

Aws Sso From Adconnect To Azure Ad

Secure your website access with Kubernetes NGINX Ingress Controller, OAuth2 and Azure AD

How does it work

Prerequisites

Register an Application

Install Oauth2 Proxy

Installing the NGINX Ingress Controller

Prepare ElasticSearch cluster and Kibana

Deploy Ingress objects

Test your Setup

Recommendation

How to: login into Kibana/AWS OpenSearch using Azure AD

Prerequisites

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

Step 7

Getting full transparency throughout your DevOps process

Getting full transparency throughout your DevOps process

Principal DevOps metrics

One Dashboard for the whole cycle

Development

Build

Operation

Technical implementation

Connecting a custom source

Embedding already rendered widgets

Automatically browse through several dashboards

Securing access

Dockerize the Dashboard

Make technology and culture reinforce each other

Endnotes and Literature

Let's Scale - Part 2!

Part 2: Keep your Infrastructure Code Clean!

Retro

Generate automatically Documentation with Terraform-Doc

Run Security Scans with Terraform-Sec

Keep your Terraform Code Clean with Terraform-Lint

Integrate all the Terraform Helpers in One to your Git Workflow with Pre-Commit

Testing your Terraform Modules as Units

Summary

Why use `user.objectid`?