Scaling .Net Core containers with Event Driven Workloads

Background 

Due to the COVID-19 pandemic, many of the developer communities and user groups have been forced to conduct their regular session in a virtual manner. This has provided a great opportunity for organizers and speakers from across the globe to speak at community events and rope in speakers from different parts of the world. This might not have been possible in case of physical events. 

I have been speaking at the local community events in Singapore as well as other parts of Asia for the past 3-4 years. Recently, I got opportunity to speak at the virtual meetup across the globe for the Microsoft Cloud South Florida user group. 

 It started off with a Tweet from Julie Larman that she is getting multiple requests for speaking opportunities but could not fulfil all. She suggested the organizers can extend the opportunities to others who might be interested and available to speak. I thought it was a good opportunity and replied to her tweet. The thread got picked up by Dave Noderer and we managed to set up a virtual meetup in no time. 

Scaling .Net Core Containers with Event Driven Worksloads 

I have presented the topic of autoscaling container using KEDA on multiple occasions in the past for different meetups and events in Asia. I also have a 3 part series on my recently launched YouTube channel about this. The duration of the meetup was 90 minutes and that provided me with an opportunity to do a deep dive on some of the topics which are not possible in a 45 minutes or 1 hour session. 

The application I used in the demo is a dummy events management application called Tech Talks. There is a ASP.Net Core WebAPI which exposes a method to generate random events. These events are pumped into a RabbitMQ queue. We have a .Net Core exe which consumes these messages in a batch. It is the consumer which we use to showcase the autoscaling capabilities using an upcoming project called Kubernetes-based Event Driven Autoscaling (KEDA)




During the session, I demonstrated the following features 
  • Containerize .Net Core Web API and executable using Dockerfile 
  • Build and Publish docker images to a private container registry (Azure Container Registry) 
  • Use Docker-compose to multiple services
  • Use YAML files to describe Kubernetes deployments 
  • Provision AKS cluster using an idempotent Powershell script 
  • Deploy RabbitMQ cluster using Helm charts 
  • Deploy application containers to Kubernetes 
  • Auto scale RabbitMQ consumer using KEDA 
  • Extend the scaling capabilities to serverless Azure Container Instances (ACI) using Virtual Node

By the end of the session, we have expanded the containers to be auto scaled on to serverless Azure Container Instances (ACI) using Virtual Node.


YouTube video recording

The recording of this talk is now available on YouTube


Slides

The slides used during the session are available on 

Source code

The source code is available in GitHub repository.

Conclusion

The session provided me an opportunity to speak for the first time across the globe. I like to attend in-person events as it helps a great deal to network with people. In a virtual event sometimes you feel like you are talking to a screen. It is difficult to gauge the reaction of the audience in virtual event. 

One of the benefits of a virtual event is that we can focus more on the content delivery without getting distracted which could sometimes happen in a in-person event. Depending on which platform or communication tool is used (YouTube live stream / MS Teams/ Zoom etc) the question and answers can be handled separately. Aonther great advantage of virtual event is the ability to record it and share it on platforms like YouTube. People who could not attend due to timezone differences or due to emergencies can find these recording useful.

Until next time, Code with Passion and Strive for Excellence.
spacer

How to improve code quality using GitHub Super Linter

 Background

As developers, we work on multiple programming languages like C#, Java, Python, JavaScript, SQL, etc. Apart from the mainstream programming languages, we also work with different file types like XML, JSON, YAML. Each of these languages and file types has their own styles and conventions. As a language or a format becomes mature, there are standards and best practices which get developed over a period of time. 

Nowadays, many developers are full stack developers or polyglot developers. As part of the Microservices style of development, they might work on the web or Javascript based front end, a Java, C#, Go, Python or some other programming language based middle tier. And then some SQL or NoSQL backend. Each of these components can use a different language. And each language will have its own style. It can be very difficult for new or even experienced developers to follow all the best practices for all the languages at the same time.

Linter to the rescue

A linter is a software or an add on which will help to identify issues in a file with respect to the rules or conventions. It is a static code analysis tool which helps to automate the process of validating common errors, bugs, style-related errors. Some of the Integrated Development Environments have built-in linters for the common programming languages. Visual Studio, for example, can suggest changes to classes and methods. External tools and extensions like Resharper can also help. One of the most popular code editors, Visual Studio Code has many extensions which are specific to a particular language like

I tried doing a search for Linter in the Visual Studio Code Marketplace and as of this writing, there are more than 200 linters available.


All these linters help in making sure an individual developer can follow the rules and styles correctly on their development environment. Things get tricky when we work in teams and multiple developers are working on the same project. Each developer can have their personal opinion. To avoid having multiple styles for the same codebase, it is necessary to standardize the rules across the whole team. 

These rules can then be automatically checked as part of the automated build process. All the modern day Continuous Integration (CI) systems like Azure DevOps, Jenkins, Bamboo, TeamCity, TavisCI etc. allow us to perform static code analysis. 

GitHub Super Linter

Github recently announced what they call the Super Linter. GitHub also allows us to trigger certain actions based on some conditions like source code checkin into a master or main branch or a feature branch. These are called GitHub Actions.

The Super Linter is a collection of more than 30 linters for some of the most commonly used programming languages. With one GitHub action, we can scan the whole codebase and identify any issues in a single go. If the team has its own set of predefined rules for a particular language, we can customize the defaults to use the team or organizational rules.

In the video below, we can see how GitHub Super Linter can be configured for your repository and triggered using GitHub Action.



Conclusion

As demonstrated in the video, GitHub Super Linter can be a great tool for automating the static code analysis. It can easily help to standardize coding practices across multiple languages for teams and organizations. It is very easy to set up and can help a great deal in DevOps practices. Hope you find this useful.

Until next time, Code with Passion and Strive for Excellence.
spacer

DP-200 Implementing an Azure Data Solution Exam preparation guide

 Background

Most of us are working from home due to the ongoing COVID-19 situation. This has saved some time in travelling for me personally. I managed to utilize some of this time to prepare for Azure Role-based certifications. In the past 3 months, I managed to clear AZ-301, AZ-400 and DP-200 certification exams. Many people have asked me about how did I go about preparing for these certifications. This post and the accompanying YouTube video is about my preparation for the DP-200 exam.

DP-200 Implementing an Azure Data Solution Exam

In case you are new to Microsoft Role-based certifications, have a look at this excellent post by Thomas Maurer about selecting the right exam. The DP 200 exam is focused on Data Engineer role and falls under the Data & AI category of tests. It is for the Associate level. I have been part of Data Engineering teams since late 2015. I have worked with Big Data technologies on Hortonworks Data Platform and also on Microsoft Azure. The exam consists of 3 main areas:

  • Implement data storage solutions (40-50%)
  • Manage and develop data processing (25-30%)
  • Monitor and optimize data solutions (30-35%)

Your skills are tested against following core services from Microsoft Azure:
  • Azure Data Lake Storage (ADLS) Gen 2
  • Azure Data Factory
  • Azure Databricks
  • Azure SQL
  • Cosmos DB
  • Azure Stream Analytics
  • Azure Key Vault
  • Azure Monitor
Since the test is focused towards implementing the Data solutions, we need to know the different options available for securing, managing, integrating and monitoring the solutions. I used the Linux Academy course for DP 200 to prepare for this exam. If you are new to Azure I recommend going through the MS Learn paths related to DP 200. Following learning paths are available on MS Learn

We need to understand batch processing, stream processing, structured & unstructured data, different APIs supported by Cosmos DB, different consistency levels. In terms of Data Integrations, it is important to understand the capabilities of Azure Data Factory. For Data Processing, Azure Databricks, Synapse Analytics and Stream Analytics play a very important role. If you are not familiar with the stream processing and real time data integration, focus on understanding the different Windowing mechanisms like sessions window, sliding window, tumbling window, hopping window etc. 

The questions are all multiple choice questions (MCQ) based on case studies. For some questions, there is more than one correct answer and in such cases, we are also required to sequence the steps in the correct order. An example could be the data integration process which requires 5 steps. We are required to sequence all the steps correctly. The practice is very important due to this requirement.

Refer to the below YouTube video for more details about how I prepared for this exam


Conclusion

It is very important to practice as much as possible. The questions are not straightforward and involve selecting the right choices as well as making the right sequence of steps. I would also recommend using more than one reference material. Do not rely only on one source of information. I prefer to combine the eLearning courses from one of the platforms like Linux Acadamy, Pluralsight, Udemy etc. with MS Learn. I also do not recommend using dumps. In my opinion, dumps reduce the actual value of the certification. Please don't take shortcuts when learning new technology. 

Until next time, Code with Passion and Strive for Excellence.

spacer