A day with .Net

My day to day experince in .net

Archive for the ‘Azure’ Category

Predicting linear regression with Tensorflow and Azure Machine Learning Studio (Comparison ,Gradient descent)

Posted by vivekcek on September 20, 2017

In this post I am trying to evaluate the prediction done by Tensorflow and Azure Machine Learning Studio.
I am using a dataset obtained from Courseera machine learning tutorial. I will provide the dataset at the end of this post.
Here is the predicted value I got from Tensorflow and Azure Machine Learning for the input “8.5172”

Azure

Tensorflow

I used linear regression with Gradient descent optimizer, and below are the values used for epoch and learning rate in Tensorflow and azure ML.
Epoch=1000
Learning rate=0.01

Azure Model and Algorithm settings are given below

Tensorflow code is given below.

import matplotlib.pyplot as plt
import pandas as pd
import tensorflow as tf

# Parameters
display_step = 50
learning_rate = 0.01
training_epochs = 1000

data = pd.read_csv('ex1data1.txt', names=['population', 'profit'])

X_data = data[['population']]
Y_data = data[['profit']]

n_samples = X_data.shape[0]  # Number of rows

# tf Graph Input
X = tf.placeholder('float', shape=X_data.shape)
Y = tf.placeholder('float', shape=Y_data.shape)

# Set model weights
W = tf.Variable(tf.zeros([1, 1]), name='weight')
b = tf.Variable(tf.zeros(1), name='bias')

# Construct a linear model
pred = tf.add(tf.multiply(X, W), b)

# Mean squared error
# cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
cost = tf.reduce_mean(tf.square(pred-Y)) / 2.0

# Gradient descent
# may try other optimizers like AdadeltaOptimizer, AdagradOptimizer, AdamOptimizer, FtrlOptimizer or RMSPropOptimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

# Initializing the variables
init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    cost_value, w_value, b_value = (0.0, 0.0, 0.0)
    for epoch in range(training_epochs):
        # Fit all training data
        _, cost_value, w_value, b_value = sess.run((optimizer, cost, W, b), feed_dict={X: X_data, Y: Y_data})

        # Display logs per epoch step
        if (epoch+1) % display_step == 0:
            print('Epoch:', '%04d' % (epoch+1), 'cost=', '{:.9f}'.format(cost_value), \
                'W=', w_value, 'b=', b_value)

    print ('Optimization Finished!')
    print ('Training cost=', cost_value, 'W=', w_value, 'b=', b_value, '\n')
    print('Evaluation')
    print(w_value*8.5172+b_value)
    # Graphic display
    plt.plot(X_data, Y_data, 'ro', label='Original data')
    plt.plot(X_data, w_value * X_data + b_value, label='Fitted line')
    plt.legend()
    plt.show()

Please use below dataset.

6.1101,17.592
5.5277,9.1302
8.5186,13.662
7.0032,11.854
5.8598,6.8233
8.3829,11.886
7.4764,4.3483
8.5781,12
6.4862,6.5987
5.0546,3.8166
5.7107,3.2522
14.164,15.505
5.734,3.1551
8.4084,7.2258
5.6407,0.71618
5.3794,3.5129
6.3654,5.3048
5.1301,0.56077
6.4296,3.6518
7.0708,5.3893
6.1891,3.1386
20.27,21.767
5.4901,4.263
6.3261,5.1875
5.5649,3.0825
18.945,22.638
12.828,13.501
10.957,7.0467
13.176,14.692
22.203,24.147
5.2524,-1.22
6.5894,5.9966
9.2482,12.134
5.8918,1.8495
8.2111,6.5426
7.9334,4.5623
8.0959,4.1164
5.6063,3.3928
12.836,10.117
6.3534,5.4974
5.4069,0.55657
6.8825,3.9115
11.708,5.3854
5.7737,2.4406
7.8247,6.7318
7.0931,1.0463
5.0702,5.1337
5.8014,1.844
11.7,8.0043
5.5416,1.0179
7.5402,6.7504
5.3077,1.8396
7.4239,4.2885
7.6031,4.9981
6.3328,1.4233
6.3589,-1.4211
6.2742,2.4756
5.6397,4.6042
9.3102,3.9624
9.4536,5.4141
8.8254,5.1694
5.1793,-0.74279
21.279,17.929
14.908,12.054
18.959,17.054
7.2182,4.8852
8.2951,5.7442
10.236,7.7754
5.4994,1.0173
20.341,20.992
10.136,6.6799
7.3345,4.0259
6.0062,1.2784
7.2259,3.3411
5.0269,-2.6807
6.5479,0.29678
7.5386,3.8845
5.0365,5.7014
10.274,6.7526
5.1077,2.0576
5.7292,0.47953
5.1884,0.20421
6.3557,0.67861
9.7687,7.5435
6.5159,5.3436
8.5172,4.2415
9.1802,6.7981
6.002,0.92695
5.5204,0.152
5.0594,2.8214
5.7077,1.8451
7.6366,4.2959
5.8707,7.2029
5.3054,1.9869
8.2934,0.14454
13.394,9.0551
5.4369,0.61705
Advertisements

Posted in Azure, Machine Learning, Tensorflow | Tagged: , | Leave a Comment »

Moving your app to free service plan in microsoft azure.

Posted by vivekcek on November 11, 2016

I was working on a research project with a developer essentials account in azure.
The first problem i faced was to deploy the app from visual studio ,immediately after creating my subscription.
So i just logged in to the portal and created a Web App, unfortunately i forgot to select the free pricing tier.

After someday’s i found that my credit is going down, from the bill it was found that the app service is consuming all money.

Immediately i had to do something. I tried the below steps.

1. Created a new app in the same resource group with a new service plan and free pricing tier.
2. Then i changed the service plan of my existing app to new free service plan.
3. Then tried to delete my old service plan.
4. Haaaa got the error, i can’t delete this plan due to an app is associated with it.
5. But there was no app associated with that plan.
6. After some reserach i found that it is the resource group is causing the issue.
7. I can’t delete my existing default resource group, as my existing app and free service plan is running above it.
8. Now i created a new Resource group and moved my old service plan to this group.
9. Then i was able to delete my old plan and my app is now running freely.

Moving Service plan to new resource group

1
Deleteing the resource group.

2

Posted in Azure | Tagged: | Leave a Comment »

Microsoft Azure – The remote server returned an error: (407) Proxy Authentication Required

Posted by vivekcek on February 23, 2015

I was trying to upload a file to azure blob from an MVC application, which is running in my local emulator.
Then i got a proxy authentication issue.

To solve this, go to your web.config and add this (In windows application add this in app.config)

 <system.net>
    <defaultProxy useDefaultCredentials="true">
      <proxy bypassonlocal="True" usesystemdefault="True"/>
    </defaultProxy>
  </system.net>

Posted in Azure | Tagged: , , | Leave a Comment »

Load balance virtual machines in azure

Posted by vivekcek on December 16, 2012

The new IasS (Infrastructure as a service) facility in windows azure helps us to create virtual machines in azure cloud platform. We can install additional software’s to these virtual machines. We can create a server farm by networking these virtual machines. So we have a lot of control than the PaaS (Platform as a service) feature of azure cloud.

So in this post I am going to explain how to create 2 virtual machine, network them and configure load balancing. My final aim is to host an asp.net application in these 2 machines.

load

If you don’t know how to create virtual machines in azure read my old blog post. Create a virtual machine in azure and host an asp.net web site.

Setup Machine1

1. Select Windows Server 2012 from gallery.

1

2. Provide machine name and password. This password is used to login to your virtual machine through remote desktop.

2

3. Provide a DNS name. This DNS name is used to access the web site hosted in that virtual machine. We create a Standalone virtual machine.

3

4. We don’t want to create an availability set. So select none.

4

5. It will take some time to create a virtual machine. After that you have to add a TCP endpoint to this machine.

8

6. Give name and settings as shown below.

9

7. So all settings for the first machine over now.

Setup Machine2

1. Select Windows Server 2012 from gallery. Use the same VHD used for Machine1.

2. Provide machine name and password.

5

3. This time we are not going to create a standalone machine. We want to connect our Machine2 to Machine1.So select connect to existing virtual machine.

6

4. Select none for availability set.

7

5. After the Machine2 is created we need to configure endpoint for that machine. This time we create a load balance endpoint that connected to endpoint of first machine.

10

6. Give the endpoint settings as given below.

11

Now verify endpoints in Machine1 and Machine2

Machine1

14

Machine2

12

Posted in Azure | Tagged: , , | Leave a Comment »

Create a virtual machine in azure and host an asp.net web site.

Posted by vivekcek on November 18, 2012

From this post you will learn how to create a virtual machine in windows azure.Virtual machine’s are the new IaaS(Infrastructure as a Service) feature in azure.Don’t confuse it with VM Role in PaaS(Platform as a Service).

So what is the advantage over PaaS?.
—————————————————–

PaaS constitute Web Role and Worker Role. When you deploy a Web Role azure will create a virtual machine for you and deploy your ASP.NET web site in that virtual machine.

Consider you have a big ASP.NET application. it will be hard to convert that to a web role project. in that case you can create a virtual machine and host your existing application without any change.

Another advantage deploy a share point farm with domain controller can be achieved using VM’s.

You can install 3rd party software like MySQL.

Simply you will get a virtual server you can do anything with that. Not just hosting asp.net.

STEPS
———————

1. Log in to your azure portal.

2. From Menu select ‘Virtual Machines->Create a virtual machine’.

3. From the next window select ‘COMPUTE->VIRTUAL MACHINE->FROM GALLERY’.

4. From gallery select your OS for virtual machine.

5. Give name and credentials for your virtual machine.

6. Give your DNS name for virtual machine. Through this DNS you can access your IIS hosted application. You should associate a storage account with the virtual machine to store OS installation file. In this example i have a storage account named ‘vivu’. If you don’t have any storage account azure will create one automatically when you create a virtual machine.

7. Next you can create an availability set. If you have only one virtual machine you don’t need to specify an availability set. This feature is used for placing virtual machines in different fault domains.

8. After your virtual machine create you can connect to your machine through remote desktop. Which is very simple don’t need any certificates uploaded.

Click connect and save .rdp file and click on that.

9. Give username and password

10. Enable Web Server Role.

11. Enable ASP.NET

12. Now you need to log back your azure management portal. Select your virtual machine and

select EndPoint.Then click ‘AddEndpoint’.You must configure port 80 TCP endpoint for external communication.

13. Select AddEndpoint.

14. Give details of end point.

16. After endpoint is create access ‘.cloudapp.net’

Posted in Azure | Tagged: , , | Leave a Comment »

Cannot delete storage account – azure Virtual Machine

Posted by vivekcek on November 13, 2012

I create a new VM, which automatically created a Storage account.

I then closed and deleted the VM, but the storage account remained.

When I try and delete it I get:

“Storage account *********** has 1 container(s) which have an active image and/or disk artifacts. Ensure those artifacts are removed from the image repository before deleting this storage account.”

To fix this

Go to Virtual Machines -> “Disks” or “Images” tab -> select the images/disks that you have located in the Storage Account you want to delete -> then hit the “Delete disk” or “delete image” command. Back these up if you need before hitting the delete.

Once the images/disks in the storage account have been deleted, you should be able to delete the storage account. Hope this helps.

Posted in Azure | Leave a Comment »

On Demand Diagnostic Transfer

Posted by vivekcek on April 17, 2012

On Demand transfer helps you to transfer diganostic logs up on your request.

1. Create a simple azure web role project.In the WebRole.cs file add the below code in OnStart() method.This code will start diagnostic monitor with collecting performance counter logs.

 public override bool OnStart()
        {

            var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
            config.PerformanceCounters.DataSources.Add(

                new PerformanceCounterConfiguration()
                {

                    CounterSpecifier = @"\Processor(_Total)\% Processor Time",
                    SampleRate = TimeSpan.FromSeconds(5)

                }

                );

            config.PerformanceCounters.BufferQuotaInMB = 200;
            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
            return base.OnStart();
        }

2. Build the project and it will be deployed to development fabric.Find the deployment-id from compute emulator.In our case it is “deployment16(14)”

3. Now create new windows form project and add reference to Azure SDK DLL’S from ref folder of 1.6 version.

Microsoft.WindowsAzure.ServiceRuntime.dll
Microsoft.WindowsAzure.Diagnostics.dll
Microsoft.WindowsAzure.StorageClient

4. Design a web from with a textbox and button.Enter the deploymentID in text box.

5. The full code of the form application is given below.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.Diagnostics.Management;
namespace OnDeamnd
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            DeploymentDiagnosticManager dianosticManger = new DeploymentDiagnosticManager("UseDevelopmentStorage=true", textBox1.Text);
            var myRoleInstanceDiagnosticManger = dianosticManger.GetRoleInstanceDiagnosticManagersForRole("DiagnosticWeb");

            DataBufferName dataSourceToTransfer = DataBufferName.PerformanceCounters;

            OnDemandTransferOptions transferoptions = new OnDemandTransferOptions();

            //Transfer log from past 1 hour
            transferoptions.From = DateTime.UtcNow - TimeSpan.FromHours(1.0);
            transferoptions.To = DateTime.UtcNow;
            //Message sended to queue when transfer completed
            transferoptions.NotificationQueueName = "transferoptionqueue";

            foreach (var instanceagnet in myRoleInstanceDiagnosticManger)
            {
                Guid requestID = instanceagnet.BeginOnDemandTransfer(dataSourceToTransfer, transferoptions);
            }

        }
    }
}

Posted in Azure | Tagged: | Leave a Comment »

Remotely Change Diagnostic Configuration Azure

Posted by vivekcek on April 13, 2012

In normal scenarios we configure diagnostic in WebRoles OnStart() method.But if we want to change the default configuration remotely,i mean from out side windows azure

using a windows application.We can done this using azure SDK DLL’s.

In this post i will show you how to add a performance counter to a deployed and running web role through a windows application.

1. Create a simple azure web role project.In the WebRole.cs file add the below code in OnStart() method.This code will start diagnostic monitor with default configuration

 public override bool OnStart()
        {
            var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
            return base.OnStart();
                       
        }

2. Build the project and it will be deployed to development fabric.Find the deployment-id from compute emulator.In our case it is “deployment16(14)”

3. Now create new windows form project and add reference to Azure SDK DLL’S from ref folder of 1.6 version.

Microsoft.WindowsAzure.ServiceRuntime.dll
Microsoft.WindowsAzure.Diagnostics.dll
Microsoft.WindowsAzure.StorageClient

4. Design a web from with a textbox and button.Enter the deploymentID in text box.

5. The full code of the form application is given below.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.Diagnostics.Management;
namespace DiagnosticConfig
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void btnConfigure_Click(object sender, EventArgs e)
        {
            DeploymentDiagnosticManager dmanger = new DeploymentDiagnosticManager("UseDevelopmentStorage=true",textBox1.Text);

            //Get the diagnostic mangers of all the instances of the role DignosticWeb
            var DiganosticMangerCollectionOfRoleInstances = dmanger.GetRoleInstanceDiagnosticManagersForRole("DignosticWeb");

            //Create a performance counter configuration that can be applied to all role instances.
            PerformanceCounterConfiguration ProcessorTime = new PerformanceCounterConfiguration() {

                CounterSpecifier = @"\Processor(_Total)\% Processor Time",
                SampleRate=TimeSpan.FromMilliseconds(5.0)
              };
            //Apply performance counter to all role instances

            foreach (var instnceDiagnosticAgent in DiganosticMangerCollectionOfRoleInstances)
            {
                DiagnosticMonitorConfiguration instanceConfig = instnceDiagnosticAgent.GetCurrentConfiguration();
                instanceConfig.PerformanceCounters.DataSources.Add(ProcessorTime);
                instanceConfig.PerformanceCounters.BufferQuotaInMB = 200;
                instanceConfig.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
                instnceDiagnosticAgent.SetCurrentConfiguration(instanceConfig);

            }
        }
    }
}


Posted in Azure | Tagged: , , | 1 Comment »

Custom Error Logs -Azure

Posted by vivekcek on April 11, 2012

You can collect data in a custom log file. The custom log file is created in a local storage resource. You create the local storage resource by using the LocalResource Class and adding the local storage resource to the configuration of the Windows Azure diagnostic monitor. Perform the following steps to initialize the collection of custom log data.

1. Open the service definition file (CSDEF) using your favorite text editor and add the LocalStorage element. The following example shows the element added to the definition of a web role:

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="Diagnostics" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WebRole name="DiagnosticWeb">
    <Sites>
      <Site name="Web">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" />
        </Bindings>
      </Site>
    </Sites>
    <Endpoints>
      <InputEndpoint name="Endpoint1" protocol="http" port="80" />
    </Endpoints>
    <Imports>
      <Import moduleName="Diagnostics" />
    </Imports>
    <LocalResources>
      <LocalStorage name="LocalResource" cleanOnRoleRecycle="false" sizeInMB="200" />
    </LocalResources>
  </WebRole>
</ServiceDefinition>

2. Add the following code in OnStart() of WebRole.cs

 LocalResource loc = RoleEnvironment.GetLocalResource("LocalResource");

            DirectoryConfiguration dirconfig = new DirectoryConfiguration();
            dirconfig.Container = "wad-customlogs-container";
            dirconfig.DirectoryQuotaInMB = loc.MaximumSizeInMegabytes;
            dirconfig.Path = loc.RootPath;

            var diaConfig = DiagnosticMonitor.GetDefaultInitialConfiguration();
            dirconfig.DirectoryQuotaInMB = 200;
            diaConfig.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
            diaConfig.Directories.DataSources.Add(dirconfig);
            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", diaConfig);

            File.Create(loc.RootPath + "vivek.txt");

            return base.OnStart();

The above code will transfer the file vivek.txt to wad-customlogs-container.

Posted in Azure | Tagged: , | Leave a Comment »

Trace Logs

Posted by vivekcek on April 11, 2012

Trace Logs can be configured by the below code.These logs are transferred to azure table named “WADLogsTable”

 public override bool OnStart()
        {
          
            var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
            DiagnosticMonitorTraceListener tmp = new DiagnosticMonitorTraceListener();
            System.Diagnostics.Trace.Listeners.Add(tmp);
            config.Logs.BufferQuotaInMB = 200;
            config.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
            System.Diagnostics.Trace.Write("Test");
            return base.OnStart();
            
        }

Posted in Azure | Tagged: , , | Leave a Comment »