My mantra to clear AZ-300 Azure Architect Technologies certification

Background


Recently I cleared the AZ-300: Microsoft Azure Architect Technologies certification. Many people have asked me how did I go about preparing for the certification exam and what all resources I used to clear the exam. This post is about the approach I took to prepare for this exam. I’ll share my experience with the following topics
  • Skills measured
  • Online courses
  • Hands-on labs
  • Sample test
  • Notes


Skills measured

The first thing to do while preparing for the exam is to understand what skills are measured as part of this test. Head over to the test details at https://www.microsoft.com/en-us/learning/exam-AZ-300.aspx and scroll down to the section which takes us through the skills measured section. It tells us about the % of questions that are likely to appear in the test from different sections. Here is the quick summary of the main topics
  • Deploy and configure infrastructure (25-30%)
  • Implement workloads and security (20-25%)
  • Create and deploy apps (5-10%)
  • Implement authentication and secure data (5-10%)
  • Develop for the cloud and for Azure Storage (20-25%)
In summary 80% of this exam is around the topics related to infrastructure, networking, security and storage.


Online courses

There are multiple options when it comes to online courses. Everybody has their pros and cons. Even before taking the certifications, I have personally used multiple online learning platforms including Pluralsight, Udemy, Edx, Linkedin learning etc. Particularly for this exam, I referred to 3 main online resources
  1. Pluralsight learning path for AZ-300
  2. Udemy course by Scott Duffy
  3. Microsoft Learning with Hands-on Labs

Pluralsight learning path for AZ-300

Pluralsight is one of my favourites when it comes to technical learning. When I started looking  for online resources, I was happy to see Pluralsight had already put together a learning path which is a collection of courses related to AZ-300 exam
.
There are 28 different courses authored by multiple authors. The total content length is more than 60 hours if you wish to cover each and every course. Most of the courses are short and to the point like 1 to 3 hours of length. There are few which are quite lengthy running more than 5 hours. One of the difficulties I found with Pluralsight courses is that there is no predefined structure. You can take them up in any order. It is difficult to understand which course should be taken up first and which ones later.


Udemy course by Scott Duffy

Personally, I always find it easier to refer to multiple resources. I got to know from a couple of other folks who were also preparing for the same exam that Udemy has courses material. A few months back I had subscribed to the 70-353 exam course. Since Microsoft replaced the older exam with AZ-300, the author Scott Duffy had restructured his course to cater to the needs of the new exam.




I found the course structure to be very well laid out and it helped me quite a lot. The content may not be very deep in this course but it covers all the basics required to get through the exam. There are also a lot of references provided for additional learning.


Microsoft learning with Hands-on Labs

Not many people are aware that Microsoft Learning offers free educational content. This is provided on a different axis like based on role or based on technology or product. I selected the Solution Architect role based learning. The content is very specific to the point.

I like the part where you can perform Hands-on Labs without having an Azure subscription. You are given temporary Azure account based on the duration of the topic and you can familiarize yourself with the capabilities of Azure. This is not just limited to the portal, there are cases when predefined VMs are also created with required tools in order to complete the required tasks.




Microsoft learning also has a bit of gamification. Every course you complete, you get some points in the form of experience. Although the points cannot be redeemed its a good incentive to collect more points and indirectly it shows how many features related to Azure you have covered via Microsoft learning resources.


Hands-on Labs

Although the online courses are helpful, I am one of those people who does not feel comfortable unless I try it out myself. In most cases, I used my Azure subscription to get familiar with the concepts. In some cases, it is not possible to do so. Like in case of migrating an on premise VM to cloud. In such cases, I had to rely on the online course. Thing related to Storage accounts, Virtual Networks, VMs, app services, containers is all easily doable using the Azure subscription.

I would highly recommend practising using your own subscription as much as possible. There are multiple reasons for this. First one is that many of the online courses are recorded more than 6 months before. Some of the features discussed in the module may not be available in the current Azure portal. Secondly, there are two sections in the exam where you will be required to perform tasks by accessing live Azure portal. If you have never tried it yourself, I can bet that you will not be able to perform these live tasks during the actual exam.


Sample Test

I think there is an option provided by some test providers to retake the test once in case you do not clear it in the first attempt. In my case, I had used my MVP benefits to get some discount on the online test. There was no such option for me. Which meant that if I did not clear the test in the first attempt, I had to repay the full amount again.

Luckily I found a sample test on Udemy by Scott Duffy and Riaan Lowe named AZ-300 Azure Architecture Technologies Practice Test. It has 2 tests with 50 questions each. Although the format does not match the actual exam, it still gives you an idea of what kind of questions to expect. It was also helpful for me to focus on areas which I was not very comfortable. One point to note is that this exam does not contain the live lab feature and is only restricted to the multiple choice type question.



I had never taken any online certification test before. So this sample test really helped me a lot. If you have already taken other online certifications, maybe you will have a different experience.


Notes

Every individual has different ways of studying. Some learn by reading books, some by watching videos. Others are good at learning by discussing with peers. I personally like to take notes in digital form. One Note is my favourite notes taking app. I made good use of One note’s note taking abilities during the preparation. Apart from using it on multiple devices like iPad pro and Surface Book, I found the handwritten notes along with highlighters and combination of images useful. Here is an example of my handwritten notes using One Note




Conclusion

This is was the first time I was taking any certification exam. In the past, I have never been a fan of certifications. I am thankful to Puneet Ghanshani for changing my perception of certifications. On multiple occasions, I found my discussions with Mayur Tendulkar very helpful. He was the one who suggested that I try Udemy along with other resources.

The certification exam looks at the breadth of the resources around Azure. I was able to complete the exam well before the stipulated time of 3 hours. If I were to go back and prepare for this exam again, I would follow a slightly different path. I would start with Udemy course and use Microsoft Learning in parallel with that. Then I would go through the Pluralsight courses and do the hands-on labs for deep dive kind of sessions.

During the exam, there are different scenario based questions. At first, I found it a bit difficult to understand the structure of the test. You are given a case study and then there are supporting documents like existing infrastructure, technical requirements and target scenario. The multiple choice questions are then related to these sections. The different sections are separated into a multi-tabbed interface. If you do not go through each of the tabs, the multiple choice questions do not make sense. It took me some time to understand this pattern, but once it was understood the rest of the exam went well.

I feel relieved that the monkey is off the back. It is a good starting point and I hope to continue with few more certifications. I hope that the people who are looking to clear this exam find it useful.
Until next time, Code with Passion and Strive for Excellence.
spacer

Migrate Travis CI Build to Azure DevOps

Background

Back in July 2017, I had written about how to setup Mac Book almost at the speed of light using Ansible. I had setup continuous integration using Travis CI. The objective of doing this was to test any changes to the ansible playbook automatically. At that point of time Travis CI provided an option to run the build on a Mac OS. The advantage of doing this was I could test any changes to the playbook without impacting my personal Mac. I had also scheduled to run the build on daily basis. This was really helpful in identifying outdated extensions in Visual Studio Code. I had blogged about this experience in another post how Travis CI saved me from using an unpublished extension in future.

Recently Microsoft announced availability of Azure DevOps which is combination of multiple services helping us to achieve DevOps practices. One of the service available under the umbrella of Azure DevOps is the Azure Pipelines. Azure Pipelines allow us to define build and release pipelines. We can use these pipelines for Windows, Linux and Mac. We can make use of the Hosted Mac agent to run the build pipeline which can run the ansible playbook. This post is about how I migrated an existing Travis CI build to Azure DevOps build pipeline.
We will go through following points over the course of this post
  • Understand how the Travis CI was setup
  • Identify changes required to run the build using Azure DevOps
  • Run Ansible playbook using Hosted Mac Agent 

Understand how Travis CI was setup

I had setup the initial Travis CI automated build using .travis.yml file. This file contains the build definition described in a yaml file. It can be divided into following sections
  • Environment setup : defines sudo access, language as objective-c and osx image as the xcode9.1
  • Before Install: uninstalls existing homebrew and reinstalls the required version
  • Install: installs Ansible and creates folders which are required to run the playbook
  • Script: Runs the actual playbook. Note on line #39, we are using a timeout setting within Travis CI set to 90 minutes with travis_wait. This was set to 90 minutes based on different builds which took approximately 90 minutes to finish. This is the main part of the whole setup. It runs the ansible playbook with main.yml.

Identify changes required to run the build using Azure DevOps


There are slight differences Azure DevOps pipelines and Travis CI. Both support yaml files as a way of declaratively defining the build steps. The greatest advantage of this is that the build definition can be scripted and checked into the source code repository. The ability to use Yaml files for defining build is currently available only for build pipelines in the Azure DevOps. The release pipelines are currently defined using visual components but are expected to have support for Yaml in near future.
Compared to the Travis CI yaml file, Azure pipeline yaml file contains fewer settings. These are
  • Pool: Defines the VM Image to be used which we are setting to macOS 10.13
  • Steps: Defines different steps which form part of the build. We have only a single step which runs the script. As part of the script we install ansible, set up the required variables and run the playbook using the same main.yml file which was used along with the Travis CI build.
The main difference between the two scripts is we do not need to install Homebrew and pip in case of Azure pipeline. The remaining steps related to executing the ansible playbook are exactly same. The macOS hosted agent contains truckloads of build tools preinstalled for our usage.

Run Ansible playbook using Hosted Mac Agent 

Lets trigger the build using a Hosted Mac Agent. After defining the build pipeline in Azure DevOps when we want to trigger the build, we get an option to choose the agent where we would like the build to run. In our case we make sure that the selected agent is Hosted Mac agent.
Before we trigger the build, I would like to check the capabilities hosted Mac agent. We can go into the details of all the software packages installed by default on the agent by selecting Project settings –> Agent Pools –>Hosted macOS –> Details. The agent has almost all the tools required for building most common applications in various programming languages like Java, Node, .Net along with the ones for XCode and Objective C. It also has Homebrew 1.9.2 version preinstalled which our Ansible playbook uses.

We create a new build pipeline as shown in the screenshot below. The process is is pretty straightforward we choose the Github repository as the source and Hosted macOS as the agent. YAML file path points to the azure-pipelines.yaml file inside the Github repo which contains the build definition as declarative steps.


The final step is to Queue the build. Assuming everything is setup correctly, one the build is triggered we can check the real time output log. Below screenshot shows the successful build.


Issues faced during migration from Travis CI to Azure Pipeline

The migration was not a lift and shift operation. I had some minor hiccups along the way. First one was due to the changes to the Homebrew packages. Over a period of time Homebrew had made some changes to the core packages and casks. There were some packages moved from core to cask. One such example is the Chrome Driver. I had to move this from Homebrew package section to the casks section. Another one was Telegram which changed the cask name from Telegram to telegram-desktop. Once these minor issues were fixed, I was able to successfully run the ansible playbook.



The final output shows time taken for each step in the build. As we can see from the screenshot below it took about 16 minutes to complete the whole build.


Conclusion

It was a huge relief to migrate from Travis CI to Azure DevOps build pipeline. One of the biggest benefit I got from this migration is the speed in testing the automation script which runs the ansible playbook. When I was using Travis CI, the build used to take close to 90 minutes to finish. I had to put in a kind of hack to timeout the builds which ran more than 90 minutes. After migration the Azure pipeline for similar set of activities is taking just about 15-20 minutes. That is a huge time saver.

Next time I need to set a Mac Book from scratch with all the software I need, I know that I can test it on an independent machine using Azure pipeline and in 20 minutes or so I would be able to identify if any changes are required to my Ansible playbook before I run in on actual laptop.

Another advantage I get by using a cloud hosted macOS agent is that any new tool or software I want to try to install, I can test it in isolation first and my main laptop will remain untouched. Isn’t that great?


Apart from the benefits to personal users of Mac, the main advantage of using hosted macOS agent is that teams can build their software for iOS using the Azure pipelines. I hope I have give enough pointers for the readers to explore Azure Devops more. You can register for free and use up to 10 build agents in parallel. That is not all, if you have open source projects and public repos you get huge boost in the amount of time the pipelines can be run for free.


Until next time, Code with Passion and Strive for Excellence.


Technorati Tags: ,,,
spacer

Kubernetes Secrets management

kube control describe secret

Background

This is the eighth part of the series on building highly scalable multi-container apps using AKS. So far in the series we have covered following topics:

This post is about managing secrets in a Kubernetes cluster. We will be focussing on following topics during this post.

  • Understand the reasons for using secrets in Kubernetes cluster
  • Create secret using Kubernetes manifest
  • Register secret in the AKS cluster
  • Verify secret in AKS cluster
  • Consume secret from the cluster in TechTalks DB deployment while initializing SQL Server 2017 container
  • Consume secret from cluster in the TechTalks API init container to initialize the database
  • Consume secret from cluster in TechTalks API for database access

Understand the reasons for using secrets in Kubernetes cluster

In enterprise solutions it is quite common to have separation of duties applied to different roles. In Production environments developers are not allowed access to sensitive information. Operations teams are responsible for managing the deployments. It is quite common in such scenarios to distinguish which parts of the application are handled by development teams and which part is handled by operations team. Most common example is the database passwords.

These are managed by operations teams and in most cases encrypted before storing in the target environment. Development team can use these passwords using a pre-configured file path or environment variable or some other means. The development team does not need to know the how the password is generated or the exact contents of it. As long as it can source the password by some means, the application would work fine.

The same approach can be used to externalize the passwords or secrets for different environments like Development / QA / Pre-production / Production etc. Instead of hardcoding the environment specific settings we can externalize them using configurations. Lets see in our case how we can use secrets with Kubernetes.

Create secret using Kubernetes manifest

There are different ways in which secrets can be created. As we had been doing in the earlier parts of this series we will use a Kubernetes manifest file to store the secrets information. First and foremost lets encrypt the password that we have been using for the SA account in TechTalks application.

We need to convert the plaintext password into a base64 encoded string. Run the command shown below to generate the string

echo –n ‘[email protected]’ | base64


generate secret

Copy the output of the command. We will store this into the Kubernetes Manifest file.

Notice that we set the kind of Kubernetes object as Secret on line 3. In the metadata section we provide the name as sqlsecret. Finally we provide the data. We can provide multiple elements as part of the same secret in the form of key value pairs. In our case we are specifying only one value for sapassword. With this setup we are ready to store our secrets in the Kubernetes cluster.

Register secret in the AKS cluster

Secrets can be registered into the cluster by running the kubectl create command and specifying the manifest filename. This approach is shown in the Kubernetes Secrets documentation. I use a Powershell script to deploy the complete application and all the files in a directory are used as input at the time of deployment. If you wish to deploy just the single manifest file named sa-password.yml use the command

kubectl apply –f sa-password.yml

Verify Secret in AKS cluster

Once the secret is deployed to the cluster, we can verify it in different ways. First of all lets check using the Kubernetes command line.

kubectl get secrets –namespace aks-part4

kubectl secret

We can see the sqlsecret created about 2 hours back. (Took a long time to take the screenshot after creating the secret Smile) Next we can verify the same using the Kubernetes control plane. Brose to the Kubernetes dashboard and look for secrets at the bottom of the page

kube control secrets

We can see the same information in the UI as well. Click on the  name of the secret and we will get to the details of it as shown below

kube control describe secret

The information is the same as what we had provided in the manifest file. lets verify the same in the terminal by using the kubectl describe command

kubectl describe secret

The information matches with what is shown in the UI except for the Annotations part. Now that we know that the secret is available within the Kubernetes cluster, lets turn our focus towards making use of this secret in the services used by our application.

Consume secret from the cluster in TechTalks DB deployment while initializing SQL Server 2017 container

The first place where the secret is used is when we instanciate the SQL Sever 2017 container. This is done as part of the statefulset definition.

Pay close attention to line numbers 29 to 32. Instead of hardcoding the password, we are now reading it from the secrets. We reference the secret by its name sqlsecret and the value using sapassword as the key. In future if the password expires and the operations team replaces the password, the development team does not need to redeploy the container. The new password will be accessible as part of the environment variable to the container. This solves one problem for us with the creation of SQL Server 2017 container. how about the services which uses this container. In our case, the TechTalks API is the one who is dependent on the database and interacts with it.

Consume secret from cluster in the TechTalks API init container to initialize the database

If you remember from the post on init containers, you would recollect that the API container first initializes the database with master data and few initial records. Lets use the secret while calling the initialization script.

Notice line numbers 26 to 30. We use exactly the same approach to extract the secret and store it in an environment variable. This environment variable is then interpolated with the command on line 34. With this step we have removed the hardcoding of sa password from the initialization script in the init container. We still have the connection string inside the TechTalks API container which has the sa password.

Consume secret from cluster in TechTalks API for database access

Look at the yaml file above from line numbers 42 to 46. We extract the secret and then on line 48 we interpolate it with the connection string using $(SA_PASSWORD). With these modifications in place, we removed all the hardcoding of sa passwords in our code.

I did a quick test by adding a new TechTalk using the applications UI. I can verify that the application is running smoothly.

Data accessed via API

Conclusion

Secrets management is quite a powerful concept in software development. During the course of this post we saw that Kubernetes provides built in support for managing secrets. By externalizing the secrets, we also make our applications more scalable. We do not need to hardcode secrets into application. Same code can be deployed to multiple environments by using different configurations.

Another advantage of externalizing secrets is that multiple containers can share the same secret in the cluster. In our case SQL Server container and the API container are sharing the secret. If we did not share the secret, next time there is a change in the sa password, we will need to redeploy both the containers.

Secrets play a very important role in building secure systems. Modern day applications built using DevOps practices rely on managing secrets efficiently. Most cloud providers provide secrets management as a dedicated service like Azure Key vault. For on premise scenario there are products like Hashi corps Key Vault. Hope by now you realize the importance of secrets and the ease with which we can manage them with Kubernetes cluster.

This post is dedicated to my friend Baltazar Chua who has been quite persistent in telling me that I should use secrets instead of plaintext passwords for quite a long time now.

As always the complete source code for the post and the series is available on Github.

Until next time, Code with Passion and Strive for Excellence.

spacer

Externalize SQL Server Container State using Persistent Volume Claim

Background

This is the seventh part of the series on building highly scalable multi-container apps using AKS. So far in the series we have covered following topics:

This post is about externalizing the state associated with SQL Server 2017 Linux container to an external volume which we provisioned in the previous post. We will be focussing on following topics during this post:

  • Create a StatefulSet Kubernetes object which will be responsible for creating the SQL Server 2017 Linux container
  • Externalize the state by means of volume mapping to utilize the storage which was provisioned in the previous post
  • Query and update data from SQL Server 2017 Linux container
  • Verify data is persisted if the container restarts
  • Verify the volume mount using Kubernetes cli

Lets get the ball rolling.

Create a StatefulSet Kubernetes object

Kubernetes provides an object of type StatefulSet which is often used when deploying database workloads that store their state on external storage providers. Statefulsets are used when data loss is unacceptable. StatefulSets provide a stable volume mounts across the restarts of pods or even cluster.

Lets see how we can create a StatefulSet. We will use the manifest file shown below to  declaratively set the different properties.

We specify the kind as StatefulSet. Compared to the deployment manifest, there is another difference here. For the StatefulSet, we need to specify the serviceName attribute. The rest of the attributes like metadata and spec are similar to other objects that we have created in the series.

Externalize the state by means of volume mapping

Before we can deploy the statefulset to Kubernetes cluster using kubectl, we need to make couple of other changes to ensure that the volume is mapped correctly. In the same yaml file, scroll towards the bottom and pay attention to the volume mounts and volumes sections.

Lets first look at the volumeMounts section. Here we specify the name for the volume which is mounted as mssqldb. The mountPath specifies which path inside the container is mapped to the volume named mssqldb. The MSSQL Server 2017 Linux container stores the data files inside /var/opt/mssql directory. So we have mapped this path inside the container to an external volume.

Next is the link between the volume and the physical storage which is persisted to the Azure Disk. This association is done in the volumes section. We specify exactly the same name which was assigned to the PVC in previous post, techtalks-data.

The beauty of this is that the container does not know what is the underlying storage provider. In future we can swap the Azure Disk to some other provider and the container need not have to worry about it. As long as the PVC name matches with the claimName, the container will continue to run.

We also expos the service via a load balancer. The other manifest files used during this series can be found in the Github repo. Navigate to the TechTalksDB folder and run the kubectl apply command to create the objects in the Kubernetes cluster. Below are the list of commands I executed to deploy the TechTalksDB, TechTalksAPI and TechTalksWeb containers.

kubectl part1

kubectl commands

Query and update data from SQL Server container

Lets query the data using SQL Operations Studio. The initialization script contains 3 records. I added one more before taking the screenshot. So at this point of time, we have 4 records in the TechTalk table.

database initial state

Lets add a new record and query it back.

database after adding new record

Verify Data is persisted on container restart

This is where the things start to get interesting. Lets delete the pod which is running the SQL Server 2017 container. We can delete the pod in multiple ways. Earlier in the series I had demonstrated how to interact with Kubernetes objects using the kubectl cli. To keep things simple, lets use the kubernetes control plane to delete the pod using GUI option.

Delete statefulpod

Click on the Delete option as shown above. We will be asked for confirmation, go ahead and confirm the deletion.

Delete pod

Verify data is persisted if the container restarts

As soon as the pod is deleted, the replication controller starts another instance of the pod. Any guess why? Go back to the manifest file and check the number of replicas. We have specified the number of replicas as 1. This is the desired state that we requested Kubernetes to maintain. As soon as the replica count became zero, the controller initiates the creation of new pod to make the actual state to what is desired.

For few seconds while the old pod is terminating and the new one is initiating, we will see some errors.

statefulset error

Give it a few seconds to stabilize and within a minute we can see that things are back to normal with the new pod successfully running.

statefulset recreated

The pod is restarted, but does that mean we did not loose the data which was persisted? When I queried the table again using SQL operations studio, I got the results with all the 5 rows intact. To save some space, I will not put the same screenshot again. You can give it a try yourself as a home work.

We can inspect the logs of the newly created pod to see what happened in the background.

SQL Server container logs

Verify the volume mount using Kubernetes cli

We can also look at the different events that took place at the cluster level. Run the command kubectl get events –namespace abc2018sg and we can see the list of events. Check the sequence of events in the screenshot which shows what happened after the pod was killed and new one was recreated.

volume mount events 

I have highlighted 3 main areas above. First one is the deletion of sql2017 pod. Second is the recreation of the stateful container. And third is the volume mount which is the real game changer. If the volume is not there, we would not be able to recover the data.

Conclusion

During the course of this post we saw how stateful sets can mount external volumes using the persistent volume claims. Data is persisted across container restarts. StatefulSets provide a stable endpoint for the containers which need persistent storage like in the case of database container. By externalizing the data from container into persistent volumes we can ensure that even of the cluster is restarted, we can still be able to mount the volumes back to container and achieve scalability.

All that I demonstrated here is also covered as part of the AKS learning series. The video recording of the series is available on Youtube. Thanks to Engineers.SG who are helping to record this series of talks.

AKS Part4 Youtube

As always, the code used during this post is available on Github.

Until next time, Code with Passion and Strive for Excellence.

spacer

Manage Kubernetes Storage using Persistent Volume (PV) and Persistent Volume Claim (PVC)

Background

This is the sixth part of the series on building highly scalable multi-container apps using AKS. So far in this series we have covered following topics.

In this post we will learn about a very important concept of Persistent Volumes (PV) and Persistent Volume Claim (PVC). We will use these concepts to manage data for the SQL Server 2017 Linux container.

Why do we need volume in the first place?

Managing data in a containerized scenario is quite tricky. Containers are ephemeral by design. Which means that they can be created, deleted and rescheduled on any host in the cluster. If the container is recreated, what happens to the data stored inside the container? It is oblivious that the data stored locally inside the container will be lost.

This scenario might be ok while testing. But as we start moving to deploying the applications using containers, we would want to persist the data even if the container restarts or a newer version of the image is used to upgrade the older version of container. In such scenario it makes sense to store the data external to the container. It is in such scenario that external volumes play a very important role.

During the course of this post we will focus on following

  • Provision Persistent Volume (PV)  using Azure Disk
  • Create Persistent Volume Claim (PVC) to the provisioned volume

Provision Persistent Volume (PV)  using Azure Disk

Persistent Volume abstracts from users details of how the storage is provided and how it is consumed. It is a cluster level resource provisioned by administrator and available to the pods to use. Persistent volume supports multiple types of storage including network storage, physical disks or cloud storage like Azure Disk. The storage is provisioned using one of the supported Storage Class. Lets look at an example of creating a storage class based on Azure Disk

We will skip the initial metadata, on line 7 we specify the provisioner as azure-disk. We also specify the parameters storage account type as Standard_LRS and kind as Managed. The parameters will be different for different types of provisioner. You can refer to the storage-class provisioner link below to know more about the supported provisioners with Kubernetes.

Lets go and apply this Kubernetes manifest to our cluster. Use kubectl command line to execute the command

kubectl apply –f 01_storage-class.yml

We can browse to the Kubernetes dashboard and see that the storage class is created successfully.

storage class

The creation of Storage Class is an indication of what kind of storage we are going to provision within the Kubernetes cluster. Lets go ahead and create a Persistent Volume Claim.

Create Persistent Volume Claim to the provisioned storage

Lets look at an example of the Kubernetes manifest file to create a Persistent Volume Claim (PVC).

Above we see the definition of PVC. Line 8 links the storage class named azure-disk. Note that this is the name we gave while creating the storage class earlier. From line 9 onwards we specify the requirements for the storage claim. In our example we have specified ReadWriteOnce as the access mode. For the resource request, we specified 1Gb as the disk space. When we apply this manifest, 1Gb of disk space will be provisioned for us. Run the kubectl command

kubectl apply –f 02_PersistentVolumeClaim.yml

The Persistent Volume has different lifecycle. It takes few minutes to provision the storage and bind it to the Kubernetes cluster. Initially the status of the persistent volume claim will be Pending.

PVC Pending

Once the volume is successfully provisioned it is bound to the claim. We can verify this by looking at the bound state.

PVC Bound

At this point we can go to the Persistent Volumes section on the left of the Kubernetes dashboard and find details about the underlying volume.

PVC

In this view we can find details like the name of the volume, its capacity, access modes, status, associated claim etc. Like we had done in earlier parts of this series, we can use the Kubernetes cli to query the PV object to know more details. Execute the following command

kubectl describe pv <<name of the pvc>>

The output will be as shown in the screenshot  below

kubectl describe PVC

It is same as what we see in the Kubernetes dashboard. We will see how to make use of this Persistent Volume in the next part of this series.

References

[1] Kubernetes Storage class provisioner : https://kubernetes.io/docs/concepts/storage/storage-classes/

[2] Persistent Volumes : https://kubernetes.io/docs/concepts/storage/persistent-volumes/

Conclusion

We saw how to provision a Persistent Volume backed by Azure Disk. With the help of 2 Kubernetes Manifest files we were able to get the disk and associate a claim with it.

Persistent Volumes are quite powerful concept in Kubernetes. It allows developers and cluster administrators to work on different parts of the application. Using Persistent volumes, the administrator can provision storage which can be used by developers without knowing the underlying storage mechanism. This can be used to swap the underlying storage without any need to make changes to application code. This may not be very clear right now, but I am sure it will make sense in the next part where we will see how to utilize these volumes

Until next time, Code with Passion and Strive for Excellence.

spacer

Continuous Kubernetes deployments with Skaffold on Docker for Mac

Background

My previous post was about doing continuous deployment of Kubernetes application using Skaffold to a Minikube cluster. In the recent version of Docker for Desktop (Mac & Windows), there is built in support for Kubernetes. We can decide to deploy multi container applications to a single node local cluster and choose the type of Orchestrator. Currently Docker for Mac supports Docker Swarm and Kubernetes. In this post, I will demonstrate how we can use Skaffold to deploy to a Kubernetes cluster instead of Minikube.
We will perform following steps during the course of this post
  • Enable Kubernetes support for Docker for Mac
  • Set context to work with Docker for Mac
  • Use Skaffold with Docker for Mac

Enable Kubernetes support for Docker for Mac

Lets first enable the support for Kubernetes orchestrator for our installation of Docker for Mac. Navigate to the preferences section of Docker. Select the Kubernetes pane, enable Kubernetes support, select default Orchestrator ad Kubernetes. Finally apply the setting as shown with steps 1 to 4 below. In my case I had these settings already enabled. If you are enabling the settings for the first time it can take a while for the Kubernetes to start the single node cluster.
Kubernetes for Docker
Once everything is setup we should get the Docker is running and Kubernetes is running in green status.

Set context to work with Docker for Mac

We will be using the kubectl command line tool to deploy the artifacts to Kubernetes cluster. kubectl can work with multiple clusters at the same time. We need to ensure that the right context is set for the kubectl. Lets first get the list of all the clusters we have at our disposal. This is done by executing the command
kubectl config get-contexts
kubectl get context
We can see two clusters are currently configured. The minikube cluster is the current context as shown by the * sign in front of the cluster name. We need the context to be set to the docker-for-mac context. This can be done by executing the command
kubectl config use-context docker-for-desktop
kubectl use context
We are now ready to deploy the application to this cluster.

Use Skaffold with Docker for Mac

Once the right context is set, there is no change to be done on the skaffold side. We execute the same command skaffold dev which we did during the earlier post. Skaffold does its magic
skaffold dev - part1
The proper context has been selected by skaffold.
skaffold dev - part2
The build speed is just over 1 second to build the required Docker images. Same way the deploy is done in less than 2 seconds. I have not seen these kind of speeds while using docker compose in the past. With this, now skaffold is continuously monitoring for changes to the source code. Let test it by making some changes in the code.

As you can see from the above screen capture, the updates are really fast. It doesn't even take 1 second to update the deployment. I commented and uncommented the same set of lines that we had used during the previous post.

Conclusion

As can be seen from this demo, we can seamlessly move from minikube cluster to the docker for mac cluster while working with skaffold. It does not require any configuration change to the skaffold.yaml file or to the Kubernetes manifest files. Just  by setting the proper context for kubectl we can have the same productivity gains that we saw during the last post. I hope that developers working with Docker and Kubernetes find this tool useful.
Until next time, Code with Passion and Strive for Excellence.
spacer

Continuous Kubernetes deployments with Skaffold

Background

I have been working with Docker containers and Kubernetes for quite some time now. When working with these technologies the usual workflow involves following steps (bare minimum)

  • containerize you application into a docker image
  • push the changes to the container registry like Docker Hub
  • deploy the containers to a cluster using container orchestration tool like Kubernetes

While developing an application, the developer also tests the application. It is quite time consuming to repeat the whole process every time there a change in the code and you need to rebuild the docker container, push it to registry and do the deployment. What if there was a tool which can do all of this for us and we focus on producing the code?

Introducing Skaffold

My dear friend who is a rockstar when it comes to introducing me to new cool tech, Baltazar Chua mentioned to me about Skaffold. Because Bal has this knack of picking the right tool for the right job, I have a hash tag for him #BalIsARockStar.

Skaffold is a command line tool that facilitates continuous deployment for Kubernetes applications.

In  this post we are going to do the following

  • Install Skaffold on Mac using Homebrew
  • Integrate Skaffold with existing multi container application
  • Test Skaffold using Minikube as target deployment environment

Install Skaffold on Mac using Homebrew

On the Skaffold GitHub repo, in the installation sections you find the steps to install on Linux or Mac OS. For Mac you run the following command

curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-darwin-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin

I am a big fan of using Homebrew for installing packages on Mac. Luckily there is a Homebrew package available for Skaffold. I used this approach to install Skaffold on my Mac. Run the following command to get Skaffold installed.

brew install skaffold

Integrate Skaffold with existing multi container application

I will use the codebase from GitHub repo which I am currently working with the Azure Kubernetes Service (AKS) Learning series. This consists of a Web frontend, a web API project and a database running inside SQL Server 2017 Linux container.

Skaffold works on the basis of very little minimal configuration. First of all we need to define the configuration in a yaml file named skaffold.yaml. This file contains 3 phases.

1. Build

In the build phase we define the build artifacts which are docker images. In this step we specify the location of the Docker files that we need to build whenever there is change in the source code.

2. Push

Once the images are built, they are pushed to the container registry referenced in the image name. I am publishing the images to Docker Hub registry. When we use the Kubernetes cluster locally like minikube or Docker for Mac, the push is skipped because the images are available locally.

3. Deploy

The deploy step ensures that the most recent versions of the artifacts are running inside the cluster. I will be using kubectl to deploy. I can reuse the Manifest files that I had created earlier for the demos.

We can see the contents of the skaffold.yaml file

I have defined the build phase on line 3. On line 4 we list all artifacts to be built as part of this phase. Lines 5 to 10 specify the image names and the location of the Dockerfile.

Next is the deploy phase on line 11. We specify kubectl as a mode of deployment and finally the list of manifests are specified on line 13 & 14. I am using the wildcard option to deploy all the manifests under k8s/Minikube directory. Currently there are 8 different manifests as shown below.

lisf of manifests

With the configuration done, lets put the Skaffold to test.

Test Skaffold using Minikube as target deployment environment

We need t run the Skaffold dev command in the directory where the skaffold.yaml file resides. Check the output of the command

In this case you can see that the build was completed in less than 2 seconds and deployment completed in less than 5 seconds. This is a great beginning. For the first time, it can take a bit long to download the respective images and  build them locally. Once the images are available in the cache it is super fast.

Update source code and experience the magic

Lets test Skaffold by making a code change to the source code. In the screenshot below you see me commenting the block of code which shows the corousel control on the homepage. The moment I comment out the code, skaffold triggers a build and deploy. Its so fast that I could not capture it in full screen mode between the VS Code IDE changes, the iTerm terminal showing the automatic build triggering and then the Safari refreshing the final output. That is the reason, I kept the 3 windows open side by side so you can see in real time how the  changes are impacted. I did these commenting and uncommenting couple of times.

It would take minutes to redeploy the changes if we were to do it without Skaffold. The best part I like about Skaffold is it is completely aware of only the parts which are changed. Notice carefully how it triggers the build for only 1 docker image and subsequently deploys only the impacted service. This is freaking cool.

Another interesting point I found out about Skaffold was its cleanup mechanism. After the testing is completed, I can stop the continuous monitoring and deployment part by stopping the dev command (CONTROL + C). Skaffold does a neat job of cleaning up the resources it created itself.

skaffold cleanup

Gotchas & Limitations

Skaffold currently works only with Linux & Mac. There is no Skaffold available for Windows OS.

Looing at the Github page for installation section, it gives an impression that there is only Linux & Mac support for Skaffold. But as rightly pointed out by Baltazar Chua, there is support for Windows OS as well. You can download the Windows executable from

https://storage.googleapis.com/skaffold/releases/latest/skaffold-windows-amd64.exe

Refer to the detailed documentation for more details.

If you are using Minikube as the target cluster, the Minikube has to be up and running. Skaffold will not start the Minikube instance automatically for you.

Conclusion

As we can see from the quick demonstrations in this post Skaffold is quite a handy utility to add to your toolkit if you are developing containerized application and deploying them to Kubernetes cluster. I can see myself using it during the development phase to improve the inner loop workflow. Based on my past experience I can say that it is a huge productivity boost and can easily save few hours a day. I was using docker compose to build multiple images together. With Skaffold taking care of building, pushing and deploying images continuously, I don't feel the need for docker compose for my workflow. In this post we saw how to use skaffold with minikube Kubernetes cluster.

In the future post I will also demonstrate how to use it with the built in Kubernetes cluster with Docker for Mac.

Until next time, Code with Passion and Strive for Excellence.

spacer