Kubernetes Secrets management

kube control describe secret


This is the eighth part of the series on building highly scalable multi-container apps using AKS. So far in the series we have covered following topics:

This post is about managing secrets in a Kubernetes cluster. We will be focussing on following topics during this post.

  • Understand the reasons for using secrets in Kubernetes cluster
  • Create secret using Kubernetes manifest
  • Register secret in the AKS cluster
  • Verify secret in AKS cluster
  • Consume secret from the cluster in TechTalks DB deployment while initializing SQL Server 2017 container
  • Consume secret from cluster in the TechTalks API init container to initialize the database
  • Consume secret from cluster in TechTalks API for database access

Understand the reasons for using secrets in Kubernetes cluster

In enterprise solutions it is quite common to have separation of duties applied to different roles. In Production environments developers are not allowed access to sensitive information. Operations teams are responsible for managing the deployments. It is quite common in such scenarios to distinguish which parts of the application are handled by development teams and which part is handled by operations team. Most common example is the database passwords.

These are managed by operations teams and in most cases encrypted before storing in the target environment. Development team can use these passwords using a pre-configured file path or environment variable or some other means. The development team does not need to know the how the password is generated or the exact contents of it. As long as it can source the password by some means, the application would work fine.

The same approach can be used to externalize the passwords or secrets for different environments like Development / QA / Pre-production / Production etc. Instead of hardcoding the environment specific settings we can externalize them using configurations. Lets see in our case how we can use secrets with Kubernetes.

Create secret using Kubernetes manifest

There are different ways in which secrets can be created. As we had been doing in the earlier parts of this series we will use a Kubernetes manifest file to store the secrets information. First and foremost lets encrypt the password that we have been using for the SA account in TechTalks application.

We need to convert the plaintext password into a base64 encoded string. Run the command shown below to generate the string

echo –n ‘June@2018’ | base64

generate secret

Copy the output of the command. We will store this into the Kubernetes Manifest file.

Notice that we set the kind of Kubernetes object as Secret on line 3. In the metadata section we provide the name as sqlsecret. Finally we provide the data. We can provide multiple elements as part of the same secret in the form of key value pairs. In our case we are specifying only one value for sapassword. With this setup we are ready to store our secrets in the Kubernetes cluster.

Register secret in the AKS cluster

Secrets can be registered into the cluster by running the kubectl create command and specifying the manifest filename. This approach is shown in the Kubernetes Secrets documentation. I use a Powershell script to deploy the complete application and all the files in a directory are used as input at the time of deployment. If you wish to deploy just the single manifest file named sa-password.yml use the command

kubectl apply –f sa-password.yml

Verify Secret in AKS cluster

Once the secret is deployed to the cluster, we can verify it in different ways. First of all lets check using the Kubernetes command line.

kubectl get secrets –namespace aks-part4

kubectl secret

We can see the sqlsecret created about 2 hours back. (Took a long time to take the screenshot after creating the secret Smile) Next we can verify the same using the Kubernetes control plane. Brose to the Kubernetes dashboard and look for secrets at the bottom of the page

kube control secrets

We can see the same information in the UI as well. Click on the  name of the secret and we will get to the details of it as shown below

kube control describe secret

The information is the same as what we had provided in the manifest file. lets verify the same in the terminal by using the kubectl describe command

kubectl describe secret

The information matches with what is shown in the UI except for the Annotations part. Now that we know that the secret is available within the Kubernetes cluster, lets turn our focus towards making use of this secret in the services used by our application.

Consume secret from the cluster in TechTalks DB deployment while initializing SQL Server 2017 container

The first place where the secret is used is when we instanciate the SQL Sever 2017 container. This is done as part of the statefulset definition.

Pay close attention to line numbers 29 to 32. Instead of hardcoding the password, we are now reading it from the secrets. We reference the secret by its name sqlsecret and the value using sapassword as the key. In future if the password expires and the operations team replaces the password, the development team does not need to redeploy the container. The new password will be accessible as part of the environment variable to the container. This solves one problem for us with the creation of SQL Server 2017 container. how about the services which uses this container. In our case, the TechTalks API is the one who is dependent on the database and interacts with it.

Consume secret from cluster in the TechTalks API init container to initialize the database

If you remember from the post on init containers, you would recollect that the API container first initializes the database with master data and few initial records. Lets use the secret while calling the initialization script.

Notice line numbers 26 to 30. We use exactly the same approach to extract the secret and store it in an environment variable. This environment variable is then interpolated with the command on line 34. With this step we have removed the hardcoding of sa password from the initialization script in the init container. We still have the connection string inside the TechTalks API container which has the sa password.

Consume secret from cluster in TechTalks API for database access

Look at the yaml file above from line numbers 42 to 46. We extract the secret and then on line 48 we interpolate it with the connection string using $(SA_PASSWORD). With these modifications in place, we removed all the hardcoding of sa passwords in our code.

I did a quick test by adding a new TechTalk using the applications UI. I can verify that the application is running smoothly.

Data accessed via API


Secrets management is quite a powerful concept in software development. During the course of this post we saw that Kubernetes provides built in support for managing secrets. By externalizing the secrets, we also make our applications more scalable. We do not need to hardcode secrets into application. Same code can be deployed to multiple environments by using different configurations.

Another advantage of externalizing secrets is that multiple containers can share the same secret in the cluster. In our case SQL Server container and the API container are sharing the secret. If we did not share the secret, next time there is a change in the sa password, we will need to redeploy both the containers.

Secrets play a very important role in building secure systems. Modern day applications built using DevOps practices rely on managing secrets efficiently. Most cloud providers provide secrets management as a dedicated service like Azure Key vault. For on premise scenario there are products like Hashi corps Key Vault. Hope by now you realize the importance of secrets and the ease with which we can manage them with Kubernetes cluster.

This post is dedicated to my friend Baltazar Chua who has been quite persistent in telling me that I should use secrets instead of plaintext passwords for quite a long time now.

As always the complete source code for the post and the series is available on Github.

Until next time, Code with Passion and Strive for Excellence.



  1. Hi just wanted to give you a brief heads up and let you know a
    few of the pictures aren't loading correctly. I'm not sure
    why but I think its a linking issue. I've tried it in two different web browsers and
    both show the same results.

    1. Thanks for letting me know. I have tried Safari & Chrome browsers on Mac. I did not find any issue loading the images. It might be linked to the browser settings on your machine. Check if you are seeing any warning due to mixed content like http & https. Ignoring the warning might help you to load the images.

  2. Hey There. I found your blog using msn. This is a
    really well written article. I will make sure to bookmark it and come back to
    read more of your useful info. Thanks for
    the post. I'll definitely return.

  3. We stumbled over here from a different web page and thought I may as well check things out.

    I like what I see so now i am following you. Look forward to checking out your web page repeatedly.

  4. Outstanding quest there. What happened after?
    Good luck!

  5. It's actually a great and helpful piece of information. I am satisfied that you simply shared this
    useful info with us. Please stay us up to date like this.
    Thanks for sharing.

  6. Great web site. Lots of helpful info here. I'm sending it to several friends ans additionally sharing in delicious.
    And naturally, thanks on your effort!

  7. May I simply say what a relief to uncover a person that actually
    knows what they are talking about on the net. You definitely know how to bring an issue to light and make it important.

    More people need to look at this and understand this
    side of the story. It's surprising you're not more popular because you certainly possess the gift.

  8. For a small fee, some companies provides instructions on the way to build
    one. Once you have all sufficient manuals and tools with your hand, you may complete it correctly.
    The basics of your magnet is opposites attract and likes repel against each other.

  9. My brother recommended I may like this website. He used to be totally right.
    This publish actually made my day. You cann't consider simply how much time I had spent for this info!

    Thank you!

  10. This is really interesting, You are a very skilled blogger.
    I have joined your rss feed and look forward to seeking more of your great post.
    Also, I have shared your web site in my social networks!

  11. Aw, this was a really good post. Spending some time and actual effort to generate a good
    article… but what can I say… I procrastinate a lot and don't seem to get
    anything done.

  12. Keep on working, great job!

  13. It's hard to come by educated people on this subject, however, you seem like you know what you're talking about!

  14. Appreciating the dedication you put into your site and detailed information you offer.
    It's awesome to come across a blog every once in a while that isn't the same unwanted rehashed information. Excellent read!
    I've saved your site and I'm adding your RSS
    feeds to my Google account.

  15. Hi there! I'm at work surfing around your blog from my new apple iphone!

    Just wanted to say I love reading through your blog
    and look forward to all your posts! Keep up the great work!

  16. Great goods from you, man. I've understand your stuff previous to and you're
    just extremely excellent. I really like what you've acquired here, certainly like what you are stating and the way
    in which you say it. You make it entertaining and you
    still take care of to keep it smart. I can not wait to read far more from you.
    This is really a great web site.

  17. Its like you learn my mind! You seem to understand so much approximately this,
    like you wrote the book in it or something. I believe
    that you just can do with some p.c. to drive the message home a
    bit, but instead of that, this is magnificent blog.
    A great read. I will definitely be back.

  18. I've been browsing online more than 3 hours today, yet I
    never found any interesting article like yours. It's pretty worth enough for me.

    Personally, if all website owners and bloggers made good content as you did,
    the web will be much more useful than ever before.

  19. I used to be able to find good info from your content.

  20. It's amazing to pay a visit this website and reading the views of all mates regarding this
    article, while I am also zealous of getting knowledge.

  21. That is a good tip particularly to those new to the blogosphere.
    Short but very accurate information… Thanks for sharing this one.
    A must read article!

  22. I blog quite often and I truly thank you for your content.
    This great article has truly peaked my interest.
    I'm going to bookmark your website and keep checking for
    new information about once a week. I opted in for your Feed as well.

  23. Excellent way of telling, and pleasant piece of writing to get information about my
    presentation subject matter, which i am going to present in institution of higher education.

  24. hello there and thank you for your info – I have
    certainly picked up anything new from right here. I did
    however expertise a few technical points using this website, as I experienced to reload the
    website a lot of times previous to I could get it to
    load properly. I had been wondering if your web
    hosting is OK? Not that I am complaining, but slow loading instances times will sometimes affect
    your placement in google and can damage your high quality score if advertising and marketing
    with Adwords. Well I'm adding this RSS to my email and can look out for much more of your respective interesting content.
    Make sure you update this again very soon.

  25. Informative article, exactly what I wanted to find.

  26. Thanks for finally writing about >"Kubernetes Secrets management" <Liked it!

  27. Attractive component to content. I simply stumbled upon your site and in accession capital to say that I acquire in fact enjoyed account your weblog posts.
    Any way I will be subscribing on your augment and even I achievement
    you get admission to consistently quickly.

  28. Great info. Lucky me I discovered your website by accident
    (stumbleupon). I have saved it for later!

  29. Hi there, I discovered your site via Google while searching for a similar topic, your site came up, it seems to be good. I have bookmarked it in my google bookmarks.
    Howdy! This blog post couldn’t be written any better! Looking at this article reminds me of
    my previous roommate! He continually kept preaching
    about this. I most certainly will forward this article to
    him. Fairly certain he’s going to have a good read.
    Thank you for sharing