Tuesday, 27 December 2022

Creating Storage Accounts and accessing them via Storage Explorer

 

Convolutional Neural Network (CNN):

Why Azure ML Service?

  • In today’s scenario, many companies struggle to adopt the ML solution because data engineers face slow training and complex deployment. Azure ML service removes those barriers.
  • Azure ML service is a powerful drag and drop interface. where we go from idea to deployment in a few clicks without additional setup required.

Overview Of Convolutional Neural Network (CNN)

  • We mostly use CNN’s in Computer Vision for Image Classificationobject detection, and Neural Style Transfer.

 CV.

  • Problem: Images can be big. with 1000 hidden units we need a lot of space.
  • Solution: Use CNN’s, it’s like scanning over your image with a filter.

What Are Convolutional Neural Networks (CNNs)

  • CNN’s are a specific type of artificial neural network.
  • CNN’s works well with matrix inputs, such as images.
  • There are various kinds of the layer in CNN’s: convolutional layers, pooling layers, Dropout layers, and Dense layers.
  • CNN’s real-world applications: Detecting Handwritten Digits, AI-based robots, virtual assistants, NLP, Electromyography recognition, Drug discovery, Time series forecasting, and self-driving cars.


Image of CNN's Layers

Convolution Layer 

  • In CNN’s we use Convolutional layers which consist of a set of learnable filters and these filters are applied to a subregion of the input image to reduce image dimensions.
  • Ensure the spatial relationship between pixels.


 image of convolution-layer.

Pooling Layer

  • We use a pooling layer in CNN’s to reduce the number of dimensions(width, height) but retains the most important information. A common technique we use is Max Pooling.
  • Max pooling is a type of non-linear sampling. it will divide the input image into set non-overlapping rectangles.
  • Speed up Computation.
  • Makes some of the detected features more robust.

 Image of Pooling Layer.


Dropout Layer

  • We prefer Dropout layers in CNN’s to prevent overfitting by increasing testing accuracy.
  • We generally place the dropout layer with p = 0.5 in between fully connected layers.
  • …….CONV => Relu => Pool => FC => DO => FC => DO => FC……. Image of Dropout layer.


Fully-Connected Layer

  • In CNN’s Fully Connected Layer neurons are connected to all activations in the previous layer to generate class predictions.
  • it’s common to use more than one fully connected layer prior to applying the classifier.

 Image of Fully Connected Layer.

Rules Of Thumb

  1. The input layer should be square. Common size includes 32×32, 64×64, 96×96, 224×224.
  2. The input layer divisible by two multiple times after the first convolution layer is applied.
  3. Convolutional layers size such as 3×3 and 5×5 mostly.
  4. CNN used the POOL layer rather than the Convolutional layer for reducing spatial dimension until you have more exp on Convolutional Neural Networks architectures.


Azure ML Workspace

1)we need to install Azure ML extensions for the Azure CLI. Run the following cmd.

$ az extension add -n azure-cli-ml

2)now we will be able to use extension using az ml cmd.

3)we create a new resource group for the azure ML workspace. Run the following cmd.

$ az group create -n ml -l westus2
$ az ml workspace create -w mldws -g ml

4) we will change to working dir in our shell & export an azure ML workspace config to disk.

$ az ml folder attach -w mlws -g ml -e mls

5) Then we will install python ext to interact with our azure ML from within python. Run this cmd.

python3 -m pip install azure-cli azureml-sdk

  • now we run our authoring environment in the cloud, we can simply navigate to our Machine Learning workspace in Azure and open the Azure Machine Learning interface to use the Notebook viewer provided there. Screenshot of notebooks.
  • To run our code in this env then click on compute and create compute instance. when it will be done then we have an option to start Jupyter notebook. Screenshot of jupyter notebook

 

6) now we can load workspace config from this file without explicitly specifying the workspace and subscription in every experiment. we run the following code to load our workspace.

from azureml.core import Workspace
ws = Workspace.from_config()


Training A Convolutional Neural Networks On Azure 

  • Login to Azure portal with email ID (Azure user credential) which you used during creating Azure account. Screenshot of Azure login page
  • Now see the upper-left corner of the Azure portal, select Create a resource. Screenshot of select create
  • Now search Machine Learning and select Machine Learning then select Create. Screenshot of search ML.
  • Go to Azure Notebooks & then Create a new project.
  • Then download “CNN_Keras_Azure.ipynb” from here & upload it to your project.
  • Ensure “Running on Free Compute” is enabled. Screenshot of azure notebook.


  • Click Download config.json from the Overview section and then Upload config.json in your project folder and then launch option in Azure ML workspace.
  • Now Click on CNN_Keras_Azure.ipynb in your project to open & execute points by points.

This is how we train the convolutional neural network model on Azure with Keras.


Load Balancer And Traffic Manager In Azure

 In Microsoft Azure Architect With Standard Load Balancer, you can scale your applications and create highly available services. The load balancer supports both inbound and outbound scenarios.

In this segment, we will be implementing inbound load balancing and NAT by using the Azure Load Balancer Standard. Then we will configure outbound SNAT traffic and test it by using Azure Load Balancer Standard.

Load Balancer

Configuring VNET Peering, Service Chaining

 Virtual network peering enables you to connect networks in Azure Virtual Network seamlessly.

Azure supports the following types of peering:

  • Virtual network peering: Connect virtual networks within the same Azure region.
  • Global virtual network peering: Connecting virtual networks across Azure regions.

VNET PeeringService chaining enables you to direct traffic from one virtual network to a virtual appliance or gateway in a peered system through user-defined routes.

What Is Azure VNet Peering?

Azure Virtual Network is used for the Virtual Network Peering empowers users to flawlessly communicate with virtual networks in Azure. VNet Peering in Azure allows the traffic of one virtual network to communicate to another virtual network. This is basically used for database failover, disaster recovery, or cross-region data replication.VPN gateways are used in an encrypted connection in the region but VNet Peering provides connection sharing in different regions.

VNet_Peering


Importance Of VNet Peering

  • VNet peering is similar to an inter-VLAN Routing in VLAN of On-premise networks so it works similarly to inter-VLAN connect to one VLAN to another VLAN for communication.
  • In Azure infrastructure, need to connect to virtual networks to each other for sharing traffic which can be applications, backup, replication, recovery, or information sharing.
  • The virtual machines of virtual network connections to other virtual machines of different Virtual network via connection of VNet Peering in the same region or across the region

Types Of VNet Peering

  1. Default VNet Peering: it empowers the connectivity between various VNets within the same Azure region.
  2. Global VNet Peering: it allows Virtual networks to connect across different Azure regions. It provides private peering with low latency and high bandwidth in Azure backbone infrastructure.


Pre-checks Of VNet Peering

  • VNet Peering only establishes between Two virtual networks, it’s not transitive. for example,  VNet P and VNet Q having VNet peering, VNet Q and VNet R having peering than VNet P and VNet R are not having VNet peering.
  • Before Peering at virtual networks check for the non-overlapping IP addresses.
  • The virtual network’s resource cannot set connections with the front-end IP address’s basic load balancer in Global VNet Peering in Azure.
  • Once a virtual network peered with another virtual network then adding or deleting the address range is auto-disabled.


Benefits

  • Network traffic of peered Virtual networks become private.
  • Virtual network peering in Azure allows transferring data across Azure deployment models, subscriptions, and other regions.
  • No downtime issues in global Azure virtual network peering.
  • It configures the connection with high bandwidth Low latency in the VNet region.
  • Global VNet peering has erased the need for VNet to VNet peering Azure configuration. It disabled the use of VPN encryption, public internet, or any gateways.
  • This is a very cost-effective and Time-saving process that controlling the backup, traffic, sharing from different regions


Step-by-Step Configuration

  1. Log in to the Azure portal at  https://portal.azure.com.
  2. Create two Virtual networks in the same or Different regions like Vnet1, Vnet2, etc.
  3. Now go to any one of the two Virtual Networks and select Peerings, under Settings, and then select Add.vnet peering
  4. Configuring the peering for the two virtual networks and select, Add.

    This virtual network:
     means the vnet 1
    Remote virtual network: 
    means the vnet (here vnet2) which you want to peer the vnet1 with.
    create vnet peering
  5. The PEERING STATUS is Connected, as shown in the following picture:vnet peeringIf you don’t see the status, refresh your browser.
    Note: Configuring peering on anyone VNet will configure the peering automatically on the other VNet as well.
  6. Now connect anyone of your VM and then try to ping the Public IP of the second Virtual Machine to test the peering.
    Note: If you are using a Windows Server VM, the ping will fail, because ping uses the Internet Control Message Protocol (ICMP). By default, ICMP isn’t allowed through the Windows firewall.
  7. To allow VM1 to ping VM2 in a later step, enter this command in the VM2 Powershell.
    New-NetFirewallRule –DisplayName “Allow ICMPv4-In” –Protocol ICMPv4windows firewall
    NOTE: You have to enter this command on the other VM. (Here VM2).
  8. Now, close the remote desktop connection to VM2 and connect to the VM1, then again ping the Public IP of the second VM.


Now you will see that your Virtual Machines is connected as it has 100% packages received on pinging to VM2.

ping
You have successfully configured and tested the VNet Peering, I hope now you have a better understanding of VNet Peering in Azure. The virtual network establishes the connection with VPN gateways and VNet Peering but it depends on the condition and requirement of the scenario.

Encrypt A Virtual Machine

In Microsoft Azure Architect, we will learn how to safeguard our VM’s data. We will be using Azure Disk Encryption to do so. Azure Disk encryption can be applied to both Linux and Windows virtual machines, as well as to virtual machine scale sets.

Encrypt VM

Data Science VS Data Analytics VS Data Engineer :

What Is Data?

  • Data is the collection of lots of facts and figures.
  • Whether you understand it or not there is no denying that data is the foundation of any successful company and the business entrepreneurs that are leading the way are aware that looking deeper into data is what will make them tower above the competition.
  • Let’s look at the data science team or big data team. We want to solve a business problem then We’ll do a significant amount of work on data that is available first based on the data analytics and we will provide an insight dashboard after the dashboard is ready.
  • We as a data scientist will use some machine learning and artificial intelligence tools to develop models that could predict future outcomes.


What Is Data Science? (Business Analytics vs Data Science) 

  • It is a discipline relying on data availability, while business analytics does not completely rely on data.
  • Data Science covers part of data analytics, particularly that part which uses programming, complex mathematical, and statistical. it is not completely overlapping Data Analytics but it will reach a point beyond the area of business analytics.
  • It can be used to improve the accuracy of prediction based on data extracted from various activities.
  • Business intelligence fits in data science because it is the preliminary step of predictive analytics because we first analyze past data and extract useful insights and then create appropriate models that could predict the future of ours business accurately.

 Data Science

The Job Roles Of Data Scientist

  • Exploratory Data Analysis (EDA)
  • Processing, Cleaning and Verifying the Integrity of data.
  • Identify trends in data and make unique predictions.
  • Develop an understanding of using Machine Learning Techniques.


Skills And Tools For Data Scientist

  • Understanding of Machine Learning Algorithm and Techniques.
  • Hands-on Data Visualisation tools such as Tableau and Power BI.
  • Experience in Big data tools like Spark and Hadoop.
  • Understanding of Python or R and Expert in SQL.

 Data Scientist Skills.

Data Scientist Use Cases

  • Fraud and Risk Detection.
  • Advertisement optimization.
  • Automation Risk Management.
  • Search Recommenders.
  • Improving E-commerce Experience.
  • Optimizing Rides
  • Music Recommendation

Note: Do Read Our Blog Post on Azure Machine Learning model.

 Use Cases

What Is Data Analytics?

  • Data Analytics is the study of datasets to figure out conclusions from the information using particular systems software. it concentrates on specific areas with specific goals.

 Data Analytics

The Job Role Of Azure Data Analyst

  • Exploratory Data Analysis(EDA)
  • Discover new patterns using Statics Tools.
  • Develop KPI’s and visual dashboards.
  • Clean data

Skills And Tools For Azure Data Analyst

  • Expertise in Stats tools such as R, SAS, Excel, etc.
  • Hands-on BI tools.
  • Proficiency in data mining. Data analyst tools.


Data Analyst Use Cases

  • Social Media Analysis.
  • Customer churn analysis and prevention.
  • market basket analysis.
  • 360-degree view of the customer.

What Is Data Engineering?

  • Data engineering is the form of data science that targets on practical applications of data collection and analysis.

Who Is Data Engineer?

  • Data Engineer makes and amends the systems that data analysts and scientists to perform their work.
  • Data Engineer responsible for storing data, receiving data, transforming data, and made available to the users. Data Engineer.

The Job Role Of A Data Engineer

  • Develop, construct, test, and maintain architecture.
  • Basic understanding of Programming languages and Data structure.
  • Know how to deploy a machine learning model on Azure or other cloud services.
  • Architect pipelines for different ETL operations.

If you have been looking for the best source to learn about the AZ-204 exam preparation, then click here.

Skills And Tools For Data Engineer

  • Architecting a distributed system and creating predictable pipelines.
  • Architecting data stores and Combining data sources.
  • Understanding of python, java, SQL, and C++.
  • Experience in computation software such as Hadoop, Hive, Pig, and Spark. Data Engineer vs Data Scientist


Data Engineer Use Cases

  • Event-Driven Architecture (EDA).
  • Change Data Capture (CDC).
  • Event Stream Processing.

Data Analyst vs Data Scientist vs Data Engineer

  • Data Scientist: Analyze data to identify patterns and trends to predict future outcomes.
  • Data Analyst: Analyze data to summarize the past in visual form.
  • Data Engineer: Preparing the solution that data scientists use for their work.

 data science vs data engineer vs data analytics


ML And AI In Data Science vs Data Analytics vs Data Engineer

  • Machine learning: The ability of machines to predict outcomes without being explicitly programmed to do so is regarded as machine learning. ML is about creating and implementing algorithms that let the machine receive data and used this data to :
    • Make Predictions
    • Analyze Pattern
    • Give recommendations
  • ML can not be implemented without data. Hence it should stay within data analytics completely. Applying ML tools to business intelligence is increased. As data scientists, we are interested in how tools from machine learning can help us improve the accuracy of our estimations.
  • ML software can hold data from the third company and detect new patterns from their data and thus suggest real-time recommendations and insights to managers and other decision-makers. This is a great way to improve the performance of our business.
  • Client Retention: ML helps develop models that predict what a client’s next purchase would be.
  • Fraud Prevention: ML algorithm with prior fraudulent activity data. it will find the patterns which the human brain is incapable of seeing. Image.
  • Artificial Intelligence: It is about simulating human knowledge and decision making with computers. it is quite a general term that can have a rather philosophical interpretation. we have only managed to reach AI through ML.


Why Microsoft Azure?

Computing Power:

  • Big Data & Analytics requires huge computing power because of the huge amounts of data that need to be analyzed.
  • Azure’s compute mostly comes from its Virtual Machines. Both offer scale-on-demand computing capacity, providing the infrastructure needed to run robust Big Data & Analytics solutions.

Storage:

  • Big Data solutions depend on Network and Storage.
  • Azure offers scalable storage features.

Pricing:

  • Azure has a pay-as-you-go model with Microsoft charging its customers by the minute.

Security:

  • Azure both provide the greatest security features to safeguard hacking instances and sensitive data. With 90% of Fortune 500 companies entrusting Azure.

Processing Power:

  • Azure houses ‘Event Hubs,’ displaying enough firepower for data analysis inexpensively and in situations with low latency.

 Microsoft Azure.