Use AWS Step Functions + Sagemaker to Build a Scalable Production Ready Machine Learning Pipeline for Plastic Detection

What you'll learn

Learn how you can use Google's Open Images Dataset V7 to use any custom dataset you want

Create Sagemaker Domains

Upload and Stream data into you Sagemaker Environment

Learn how to set up secure IAM roles on AWS

Build a Production Ready Object detection Algorithm

Use Pandas, Numpy for Feature and Data Engineering

Understanding Object detection annotations

Visualising Images and Bounding Boxes with Matplotlib

Learn how Sagemaker's Elastic File System(EFS) works

Use AWS' built in Object detection detection algorithm with Transfer Learning

How to set up Transfer Learning with both VGG-16 and ResNet-50 in AWS

Learn how to save images to RecordIO format

Learn what RecordIO format is

Learn what .lst files are and why we need them with Object Detection in AWS

Learn how to do Data Augmentation for Object detection

Gain insights into how we can manipulate our input data with data augmentation

Learn AWS Pricing for SageMaker, Step Functions, Batch Transformation Jobs, Sagemaker EFS, and many more

Learn how to choose the ideal compute(Memory, vCPUs, GPUS and kernels) for your Sagemaker tasks

Learn how to install dependencies to a Sagemaker Notebook

Setup Hyperparameter Tuning Jobs in AWS

Set up Training Jobs in AWS

Learn how to Evaluate Object detection models with mAP(mean average precision) score

Set up Hyperparameter tuning jobs with Bayesian Search

Learn how you can configure Batch Size, Epochs, optimisers(Adam, RMSProp), Momentum, Early stopping, Weight decay, overfitting prevention and many more in AWS

Monitor a Training Job in Real time with Metrics

Use Cloudwatch to look at various logs

How to Test your model in a Sagemaker notebook

Learn what Batch Transformation is

Set up Batch Transformation Jobs

How to use Lambda functions

Saving outputs to S3 bucket

Prepare Training and Test Datasets

Data Engineering

How to build Complex Production Ready Machine Learning Pipelines with AWS Step Functions

Use any custom dataset to build an Object detection model

Use AWS Cloudformation with AWS Step Functions to set up a Pipeline

Learn how to use Prebuilt Pipelines to Configure to your own needs

Learn how you can Create any Custom Pipelines with Step Functions(with GUI as well)

Learn how to Integrate Lambda Functions with AWS Step Functions

Learn how to Create and Handle Asynchronous Machine Learning Pipelines

How to use Lambda to read and write from S3

AWS best practices

Using AWS EventBridge to setup CRON jobs to tell you Pipeline when to Run

Learn how to Create End-to-End Machine Learning Pipelines

Learn how to Use Sagemaker Notebooks in Production and Schedule Jobs with them

Learn Machine Learning Pipeline Design

Create a MERN stack web app to interact with our Machine Learning Pipeline

How to set up a production ready Mongodb database for our Web App

Learn how to use React, Nextjs, Mongodb, ExpressJs to build a web application

Create and Interact with JSON files

Put Convolutional Neural Networks into Production

Deep Learning Techniques

How to clean up an AWS account after you are done

Train Machine Learning models on AWS

How to use AWS' GPUs to speed up Machine Learning Training jobs

Learn what AWS Elastic Container Registry(ECS) is and how you can download Machine Learning Algorithms from it

AWS Security Best practices

Requirements

Laptop with Internet Access

AWS account

Knowledge of Python and basic Machine Learning

Spend 20-50 dollars on AWS if you want to follow along with me. Note that you can still follow along without having to pay any money

Description

Welcome to the ultimate course on creating a scalable, secure, complex machine learning pipeline with Sagemaker, Step Functions, and Lambda functions. In this course, we will cover all the necessary steps to create a robust and reliable machine learning pipeline, from data preprocessing to hyperparameter tuning for object detection.We will start by introducing you to the basics of AWS Sagemaker, a fully-managed service that provides developers and data scientists with the ability to build, train, and deploy machine learning models quickly and easily. You will learn how to use Sagemaker to preprocess and prepare your data for machine learning, as well as how to build and train your own machine learning models using Sagemaker's built-in algorithms.Next, we will dive into AWS Step Functions, which allow you to coordinate and manage the different steps of your machine learning pipeline. You will learn how to create a scalable, secure, and robust machine learning pipeline using Step Functions, and how to use Lambda functions to trigger your pipeline's different steps.In addition, we will cover deep learning related topics, including how to use neural networks for object detection, and how to use hyperparameter tuning to optimize your machine learning models for different use cases.Finally, we will walk you through the creation of a web application that will interact with your machine learning pipeline. You will learn how to use React, Next.js, Express, and MongoDB to build a web app that will allow users to submit data to your pipeline, view the results, and track the progress of their jobs.By the end of this course, you will have a deep understanding of how to create a scalable, secure, complex machine learning pipeline using Sagemaker, Step Functions, and Lambda functions. You will also have the skills to build a web app that can interact with your pipeline, opening up new possibilities for how you can use your machine learning models to solve real-world problems.

Overview

Section 1: What we are Building

Lecture 1 Let's look at our End Project

Section 2: Getting Started with AWS and Getting our Dataset

Lecture 2 Source Code for the Course

Lecture 3 Setting up IAM User

Lecture 4 Clarification about AWS S3

Lecture 5 Getting Data for our Project

Lecture 6 Getting dataset Part 1

Lecture 7 Getting dataset Part 2

Lecture 8 Getting dataset Part 3

Lecture 9 Getting dataset Part 4

Section 3: Setting up AWS SageMaker

Lecture 10 Create SageMaker Domain

Lecture 11 Create SageMaker Studio Notebook

Lecture 12 Learning how to Stop and Start SageMaker Notebooks

Lecture 13 Restarting our SageMaker Studio Notebook Kernel

Lecture 14 Upload and Extract Data in SageMaker

Lecture 15 Deleting Unused Files

Section 4: Exploratory Data Analysis

Lecture 16 Loading and Understanding our Data

Lecture 17 Counting total Images and getting Image ids

Lecture 18 Getting Classname Identifier

Lecture 19 Looking at Random Samples from our Dataframe

Lecture 20 Understanding Annotations

Lecture 21 Visualize Random Images Part 1

Lecture 22 Visualise Random Images Part 2

Lecture 23 Matplotlib difference between plt.show() and plt.imshow()

Lecture 24 Visualising Multiples Images at Once

Lecture 25 Correcting our Function

Lecture 26 Visualising Bounding Boxes Part 1

Lecture 27 Visualising Bounding Boxes Part 2 (Theory Lesson)

Lecture 28 Visualising Random Images with Bounding Boxes Part 1

Lecture 29 Wrong Print Statement

Lecture 30 Visualising Random Images with Bounding Boxes Part 2

Lecture 31 Read this Lesson if you have issues with Data Visualization

Section 5: Cleaning and Splitting our Data

Lecture 32 Clean our Train and Validation Dataframes

Lecture 33 Split Dataframe into Test and Train

Lecture 34 Get Images IDs

Lecture 35 Splitting IDs Theory Lesson

Lecture 36 Explanation Regarding Next video

Lecture 37 Moving Images to Appropriate Folders

Lecture 38 Count how many Train and Test Images we have

Lecture 39 Verifying that our Images have been moved Properly Part 1

Lecture 40 Verifying that our Images have been moved Properly Part 2

Section 6: Date Engineering

Lecture 41 Using Mxnet

Lecture 42 Additional Info regarding RecordIO format

Lecture 43 Using Mxnet RecordIO

Lecture 44 Correction Regarding Label width

Lecture 45 Preparing Dataframes to RecordIO format Part 1

Lecture 46 Preparing Dataframes to RecordIO format Part 2

Lecture 47 Moving Images To Correct Directory

Lecture 48 Explanation Regarding the Previous Video

Lecture 49 Verifying that all Images have been Moved Properly

Lecture 50 Read Before Proceeding to the next Lecture

Lecture 51 Creating Production .lst files (Optional)

Section 7: Data Augmentation

Lecture 52 Data Augmentation Theory

Lecture 53 Augmenting a Random Image

Lecture 54 Moving Images to new Folder structure

Lecture 55 Visualising Random Augmented Images Part 1

Lecture 56 Visualising Random Augmented Images Part 2

Lecture 57 Read this Lesson if you have issues visualising your images

Lecture 58 Creating Data Augmentation Function Part 1

Lecture 59 Creating Data Augmentation Function Part 2

Lecture 60 Checking Image Counts Before running the Function

Lecture 61 Correctional Video regarding our Function

Lecture 62 Augmenting Test Dataset and Creating test .lst Files

Lecture 63 Augmenting Train Dataset and Creating .lst File Part 1

Lecture 64 Augmenting Train Dataset and Creating .lst File Part 2

Lecture 65 Verifying that Data Augmentation has Worked

Section 8: Setting up and Creating our Training Job

Lecture 66 Increasing Service Quotas

Lecture 67 Installing dependencies and Packages

Lecture 68 Creating our RecordIO Files

Lecture 69 Uploading our RecordIO data to our S3 bucket

Lecture 70 Downloading Object Detection Algorithm from AWS ECR

Lecture 71 Setting up our Estimator Object

Lecture 72 Setting up Hyperparameters

Lecture 73 Additional Information for Hyperparameter Tuning in AWS

Lecture 74 Setting up Hyperparameter Ranges

Lecture 75 Setting up Hyperparameter Tuner

Lecture 76 Additional Information about mAP( mean average precision)

Lecture 77 Starting the Training Job Part 1

Lecture 78 Starting the Training Job Part 2

Lecture 79 More on mAP Scores

Lecture 80 Monitoring the Training Job

Lecture 81 Looking at our Finished Hyperparameter Tuning Job

Section 9: Analysing Training Job Results

Lecture 82 Deploying our Model in a Notebook

Lecture 83 Creating Visualization Function for Inferences

Lecture 84 Testing our Endpoint Part 1

Lecture 85 Testing out Endpoint Part 2

Lecture 86 Testing our Endpoint from Random Images from the Internet

Section 10: Setting up Batch Transformation

Lecture 87 Setting up Batch Transformation Job locally first

Lecture 88 Starting our Batch Transformation Job

Lecture 89 Analysing our Batch Transformation Job

Lecture 90 Visualising Batch Transformation Results

Lecture 91 Look at this lesson if you have trouble with the Visualisations

Section 11: Setting Up Our Machine Learning Pipeline

Lecture 92 Read this Before Watching the Next Lesson

Lecture 93 Setting up AWS Step Function

Lecture 94 Verify that CloudFormation has worked

Lecture 95 Configure Batch Transform Lambda Part 1

Lecture 96 Configure Batch Transform Lambda Part 2

Lecture 97 Create Check Batch Transform Job Lambda

Lecture 98 Fixing typos and Syntax Erros

Lecture 99 JSON output Format

Lecture 100 Creating Cleaning Batch output Lambda Function Part 1

Lecture 101 Creating Cleaning Batch output Lambda Function Part 2

Lecture 102 Configuring our Step Function Part 1

Lecture 103 Configuring our Step Function Part 2

Lecture 104 Configuring our Step Function Part 3

Lecture 105 Upload Test Data to S3

Lecture 106 Testing our Step Function

Lecture 107 Fixing Errors

Lecture 108 Testing our Step Function with the Corrections

Lecture 109 Verifying that our Step Function Ran Successfully

Lecture 110 Donwloading our JSON file from S3

Lecture 111 Using Event Bridge to set up Cron Job for our Machine Learning Pipeline

Lecture 112 Verify that the Cron Job works

Lecture 113 Verifying that our Pipeline Ran Successfully

Lecture 114 Setting up Production Notebook

Lecture 115 Extending Our Machine Learning Pipeline

Lecture 116 Coding our Process Job Notebook Part 1

Lecture 117 Coding our Process Job Notebook Part 2

Lecture 118 Coding our Process Job Notebook Part 3

Lecture 119 Coding our Process Job Notebook Part 4

Lecture 120 Verifying that the Images have been Saved Properly

Lecture 121 Productionizing our Notebook Part 1

Lecture 122 Productionizing our Notebook Part 2

Lecture 123 Verify that the Entire Machine Learning Pipeline works

Lecture 124 Deleted Unused items from Sagemaker EFS

Section 12: Creating our Web Application

Lecture 125 Clone the Web Application from Github

Lecture 126 Setup MongoDB

Lecture 127 Connect to MongoDB and get AWS Credentials

Lecture 128 Configuring Env file

Lecture 129 Install Node modules

Lecture 130 MERN app Walkthrough Part 1

Lecture 131 MERN app Walkthrough Part 2

Lecture 132 MERN app Walkthrough Part 3

Lecture 133 Output Images Explanation

Lecture 134 MERN app Walkthrough Part 4

Lecture 135 MERN app Walkthrough Part 5

Section 13: Outro

Lecture 136 Clean Up Resources

Lecture 137 Congratulations

For developers who want to take their machine learning skills to the next lever by being able to not only build machine learning models, but also incorporate them in a complex, secure production ready machine learning pipeline