Build a weather forecast system leveraging the power of Windows Azure

In this section, we are going to build a weather forecast system using the power of Windows Azure, Microsoft’s cloud platform. Before we even start with building our application we need to understand the architecture of the application.

1. Understanding the basics of Azure:

Windows Azure is a cloud platform which means we can scale up and scale out our applications built on Azure. Say, we have a website that handles user requests. When we had 100 users per minute, one single IIS server was sufficient to handle those needs. But as the requests increase to 1,00,000 per minute our quota would be limited. So to remove all such restrictions, all good Cloud platforms offer scalability which means you can add to the processing capabilities of your system as and when required. Microsoft Azure offers a greater deal of scalability than any other vendor. It additionally offers you scaling out your apps or Virtual Machines/Servers. Suppose we have 10 servers/VMs to host our web applications each equipped with 2Ghz processor and 2GBs of RAM. Now, we can use this configuration or we can have 20 VMs with 1.4Ghz processing speed and 1GB of RAM; without hindering the capabilities. Instead we increase our fault tolerance and robustness as we increase number of VMs. Also, it reduces the pricing drastically. Here’s an analysis below:

Configuration 1 with 4 Medium Size VMs:
Azure Pricing Model 1
Azure Pricing Model 1
Configuration 2 with 6 small sized VMs:
Azure Pricing Model 2
Azure Pricing Model 2

So the second model helps a lot as far as pricing is concerned.

2. Architecture of the Application

Two core concepts are used in here. We have to understand the topology of the application/service.

  1. User logs into his account and requests for weather data.
  2. User queries are handled by a web service/server called as Web Role in terms of Azure. We shall rename it as ‘Web Role DataServe’.
  3. The Web Role queries or asks for data from a storage service called as Table in Azure.
  4. The weather data as found in the storage table is returned and shown to the user.

Front end Looks like this:

Frontend UI
Frontend UI

Now these are the steps from the front end view. That is, what the user is viewing. What about the fact that how the system can process and retrieve data in real time?
So, The back end view goes like this:

  1. We receive data from a standard open source weather API called openweather API. We run a Python script that periodically stores data for user’s cities. The data obtained is in JSON format.  This periodic engine is called a Worker Role. We rename it as ‘Worker Role DataFetch’.
  2. After data is received it is stored to the Table storage via a standard Python API provided under azure.storage.
  3. So whenever the user asks for data we need not check for it and create unnecessary waiting time.

The architecture is printed below:

architecture and dataflow
architecture and dataflow

3. Code for Worker Role DataFetch:

For the Worker Role we use a Python script that helps us easily parse our JSON data.

from azure.storage import *
import datetime
import re
import urllib2
import math
import time

table_service = TableService(account_name='*********', account_key='**********')

class Temperature(object):
    def toCelcius(self, deg_f):
        return (deg_f-32)*5/9

#minute=minute+1
city='kolkata'
url = "http://api.openweathermap.org/data/2.5/weather?q=kolkata&mode=xml&units=metric"
source=urllib2.urlopen(url)
tomorrow=str(datetime.date.today()+datetime.timedelta(days=1))
htmltext=source.read()
print("")

# search for pattern using regular expressions (.+?)
#if(htmltext.find(adate)!=-1):
direction=''
wind=''
temp=''
humidity=''
condition=''
pattern_direction=re.compile(direction)
pattern_wind=re.compile(wind)
pattern_temp=re.compile(temp)
pattern_humidity=re.compile(humidity)
pattern_cond=re.compile(condition)

# match pattern with htmltext
weather_windspeed=re.findall(pattern_wind,htmltext)
weather_winddirection=re.findall(pattern_direction,htmltext)
weather_temp=re.findall(pattern_temp,htmltext)
weather_cond=re.findall(pattern_cond,htmltext)
weather_humid=re.findall(pattern_humidity,htmltext)

#print "Overall Weather status: ",weather_cond[0][1]
RowKey=datetime.datetime.now().strftime("%d%m%Y%H%M")
temperature = weather_temp[0][0]
#print "Minimum Temperature: ",weather_temp[0][1]
#print "Maximum Temperature: ",weather_temp[0][2]
print "Wind Direction: ",weather_windspeed[0][0]

def checkduplicate():
    tasks = table_service.query_entities('WeatherFetch', "PartitionKey eq 'data'")
    flag=0

    for task in tasks:
        if(task.RowKey==RowKey):
            flag=1
        elif(task.temperature==temperature):
            flag=1

    if(flag==1):
        return True
    else:
        return False

if(not checkduplicate()):
    fetched_data = {'PartitionKey': 'data', 'RowKey': RowKey, 'temperature' : temperature,'min_temperature' : weather_temp[0][1], 'humidity' : weather_humid[0],'windspeed':weather_windspeed[0][0],'winddirection':weather_winddirection[0][1] ,'condition':weather_cond[0][1]}
    table_service.insert_entity('WeatherFetch', fetched_data)
else:
    print 'Foo is empty!'

This basically stores weather forecast data of Kolkata every day. So, this script can be run on a small sized Linux VM via a simple shell script:

#!/bin/bash
#script checks whether time its a full hour like 5:00, 8:00
#and runs a script hourly

now=0
count=0
echo $now
i=12
p=$now
while true
do
now=$(date +"%M")
if [ $(( $now % 100 )) -eq "0" ]; then
count=1
if [ $count -eq "0" ]; then
python /home/abhishek/AzureWorkerRole/AzureTableStore.py
fi
fi
done

This script calls the Python Script hourly and stores weather information in the table.

Now, Table details that means name and primary key can be found at Azure Portal under storage:

authorization for table storage
authorization for table storage

4. Fetching Data from table using Web Role DataServe:

This entire documentation can be found at:

http://www.windowsazure.com/en-us/documentation/articles/storage-dotnet-how-to-use-table-storage-20/#retrieve-all-entities

So, this wraps up another tutorial in Windows Azure.

Entire Code is available on:

https://github.com/abhishekdepro

Happy Coding.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s