Learn how to correctly Write, Ship and Analyze Logs using C# .Net, Logstash and Grafana

What you'll learn

How to use Logging infrastructure in C# .Net

How does logging work in Docker Containers

How to use Logstash as a pipeline for digesting logs

Use MySQL to save data

Use Grafana for visibility and alerting

Create a real project for real-life usage

Requirements

Basic knowledge in C# .Net. I'll try to cover the basics, but you should know how to program in C# (or any other high level language)

REST API - This concept should be familiar with you

Basic Docker knowledge. We'll run all required applications using docker containers and docker compose. Everything will be explained, but some basic knowledge will help here.

Some SQL knowledge. Highly recommended to have at least some knowledge on databases, tables and queries. Either way, you won't get stuck if you aren't familiar with this.

Description

Hi there! My name is JavierI've been working as a software engineer for more than ten years.I have a sweet spot for logging and I think it's a very important aspect of software that mostly gets overseen.In the last few years I've been deep diving into logging, and learned how to correctly write, deliver and analyze them.At the end of this course you'll have a working .Net API Server shipping Logs to a Logstash pipeline and being displayed using Grafana Dashboards.Let's break this down.In this course you'll learn:How to write logs in C# .Net Even if C# isn't your daily programming language, everything I'll show will be easily projectable to any other programming language and framework.Handling logs when working with Docker Containers - We'll see how Docker handles the standard output (stdout), how to use docker commands to see the logs and how to configure Docker to ship the logs to wherever we want to.Working with Logstash - A processing pipeline for logs. We'll see what Logstash is, how we can run it and configure it.Finally, we'll learn how to use Grafana to display the data, creating dashboards and panels. The data will come from a SQL database that will be popularized using Logstash.We'll see how easy it is to create alerts on Grafana to notify when something happened according to our logs!

Overview

Section 1: Introduction to the Course

Lecture 1 Introduction to the Course

Section 2: Introduction to logs and Best Practices

Lecture 2 What are logs, when are they useful

Lecture 3 The log levels

Lecture 4 Course Project

Lecture 5 Environment setup

Section 3: Logging in C# and Common Patterns

Lecture 6 Creating an API application - Part 1

Lecture 7 Creating an API application - Part 2

Lecture 8 Using ILogger - In Controller

Lecture 9 Using ILogger - In Program.cs

Lecture 10 Using ILogger - Changing the log level while running

Lecture 11 Using nlog - Install and use nuget

Section 4: Creating a LogMessage Class in C#

Lecture 12 Changing the application to our need

Lecture 13 Create the class itself

Lecture 14 Create a custom ILogger

Lecture 15 Change the log format

Section 5: Dockerizing the application and Logging in Docker

Lecture 16 Create a Dockerfile

Lecture 17 Create a docker-compose file

Lecture 18 How Docker logging mechanism works

Lecture 19 "docker logs" command

Lecture 20 Some changes in the logs

Section 6: Introduction to Logstash and its Role in Log Management

Lecture 21 Introduction to Logstash

Lecture 22 Install using Docker

Lecture 23 Configure logstash to receive our gelf output

Section 7: Building Logstash Pipelines for Advanced Manipulation

Lecture 24 Using Filter

Lecture 25 Using Filter - Continued

Lecture 26 Adding categories for better control

Lecture 27 Using Tags to control the flow

Section 8: Storing and Querying Logs in SQL for Insights

Lecture 28 Install MySQL under Docker

Lecture 29 Create a Database and a Table for the data

Lecture 30 Install MySQL plugin for Logstash

Lecture 31 Create a stored procedure

Lecture 32 Output from logstash statistic data

Section 9: Visualizing Logs with Grafana Dashboards

Lecture 33 Preparing the environment - Install Grafana Docker

Lecture 34 Preparing the environment - Setup data source

Lecture 35 Visualizations - Table visualization to see "raw" data

Lecture 36 Create an alert subscription

Junior developers wanting to broad their knowledge about logging and visibility