A Cloud Guru #CloudGuruChallenge: Improve application performance using Amazon ElastiCache - My Solution

My first attempt at #CloudGuruChallenge. 

I loved the fact that the challenge requires us to us EC2, Redis, RDS (Postgresql) one of the most common scenarios in a typical web application.

Requirement: Improve the performance of a python web app by implementing Redis cache. 

Environment: EC2 (web server), RDS (PostgreSQL), Cache (Elasticache - Redis), Python (flask)

Steps: 

• RDS - Creating PostgreSQL
• Login to AWS console and create stack in cloud formation.
• Upload template to create RDS.
• Open RDS in AWS console
• Note the database URL.
• Elasti cache - Redis
• Login to AWS console and create Elasticache cluster with Redis
• Provide the relavant details and create the Redis cache
• After the cache is created note the Primary Endpoint URL.
• EC2 provisioning
• Login to AWS console and create EC2 instance with defaults
• Note the public IP address
• Allow Redis cache to be only accessible from EC2
• Application installation on EC2 instance
• Login (SSH) to EC2 instance using the pem key and public IP address.
• Execute the commands to install:
• git (to download the source code)
• python3
• psycopg2, flask, configparser and redis modules (required for webapp)
• Run the following command to download the source code from github
• git clone https://github.com/ACloudGuru/elastic-cache-challenge.git
• Change directory to the source code folder and run the following command to create the stored procedure
• psql -h <DBURL> -U username -f install.sql <instanceid>
• Modify the database.ini file in the config folder with RDS database details
• Testing if everything is working file till now
• Run python3 app.py to run the app.
• Open another SSH console in cloud playground
• Type curl http://localhost:5000/ 
• Observe the Elaspsed time. Run it 3-4 times.
• Modifying the code to connect to Redis cache
• Add code in app.py to connect to redis and get/set values
• Before querying the database add code to check cache before querying database
• Run python3 app.py to run the app
• Open another SSH console in cloud playground
• Type curl http://localhost:5000/
• Observe the Elaspsed time. Run it 3-4 times.

Challenges/Learning in the above exercise

1. Using cloud formation to create RDS stack helps to reduce time to create RDS everytime.
2. Redis cache was new to me. Created a t2.micro instance to get started (My cloud playground was suspended because I chose t6 large! which was the default Sorry!)
3. Had challenge for EC2 to access RDS. Modified security group of EC2 to have default security group.
4. Redis cache really reduces the time required to fetch data and improves app performance significantly.
5. Redis returns data in byte data type. Had difficulty in converting to string (UTF8)

Pre-cache load times:

5.02477s

5.04940s

5.04783s

Post-cache load times:

0.00534s

0.00170s

0.00166s

Screenshots:

RDS

Redis cache

EC2 instance

Code for getting cached data

Cloud Formation Template

My feedback was heard!

I am a frequent flyer now :-). I have travelled to US 5 times now. But, I had this one bad experience at Bangalore Airport. Once while travelling, I forgot to put the tag on my bag and I went for security clearing. The security guy (read that idiot #$%&#@) who was checking the bags said that I need to have tag on my bag. He told that in such a manner that would have made anyone feel bad about themselves. I did not show any (or atleast tried) anger there. I continued with the rest of the process and went on my way. But later, I thought about it. I thought that it was the responsibility of the person who issued me the ticket to tell me that I need to put the tag on my bag. So, during my flight I received a feedback form for the airline. In the feedback form, I mentioned this and told that it was the ticket issuer's responsibility to make sure that their passengers are not put to any hardships. Then I forgot about that incident and that feedback.

Recently, while travelling (with the same airline that I had given the feedback) the ticket issuer immediately told me "Sir, please take this tag for your bag.". I did not remember about the feedback I had given before but later on I realised that somebody definitely had read that feedback. I observed that for another guy and they said the same thing to him. Not that its very important, but, its just one of those things which avoids unnecessary hassle and one can have a peaceful journey :-). So my feedback did not go in vain!.

P.S: The airline was Jetairways.

Windows 7

"Shreenivas, you have a packet in the mailbox" shouted my house mate from downstairs in hall. I was sleepy on Saturday afternoon and his words were almost inaudible. But, the word "packet" caught my attention and I immediately woke up from my sleep and started to walk downstairs to look at what "packet" I had received. I was suprised to see that the packet was from Dell and immediately recognized what the packet contained. I had ordered my free copy of Windows 7 OS from Dell from whom I had purchased a laptop few months back.

I started enthusiastically to open the packet to find two DVDs along with the instruction manual. It was a suprise to find exactly what I was thinking because morning I had checked their website for the order status and saw that the status was "shipped". In fact, the order was shipped just 2 days back. The instruction manual was just 1 page and had described exactly 5 steps to upgrade my laptop from Vista to Win 7. These steps involved both the DVDs.

The installation took a good 3 hours!!. The setup had to prepare my laptop for upgrade and install the OS. Everything of course was done by the setup but I had to sit there because I had to switch the DVDs whenever it prompted for them.

My first impression of Windows 7...same as Vista :-(. Although, I have heard reviews of it being a better version than Vista and faster one at that, I havent had any such experience in my 2 days of use. There are of course new features like pinup task bar, shake windows to minimize and auto snap windows etc which I am not sure how much I will be making use of it. The one tool that I like the most that has been added is a snip tool which basically allows you to take snap shot of a portion of screen. This is a huge time saving over using print screen everytime you want to take a snap shot of your screen and use mspaint to snip it.

But now with advent of ubiquitous broadband connections and most of the people preferring to be online I think OS features are just becoming redundant and OS makers need to concentrate on how one can get a user online quickly and the features that one can offer to make online browsing a wonderful experience.

Nilekani Interview on CNN IBN

I just now finished watching this interview on CNN IBN's website. The interview was done with Nandan Nilekani about why India needs a Unique Identification project and does the project itself make any sense. And boy-oh-boy! I was not only impressed by Karan Thapar's interviewing skills but also by the people who have done such extensive research as to probe into so many minute details on Unique ID project and question its viability. Karan talks about many things from the economic benefit of the project to vulnerability of the database to hackers to taking examples of UK, US and other countries who have not gone ahead with similar projects in their country and questions the reasons why it will work in India.

The interview is simply amazing. All Nilekani was able to convey was that the concerns are legitimate and need to be addressed. Watch it! :-)

Virtumonde spyware!

Last week my laptop got infected with Virtumonde spyware. It is the most annoying spyware that I have seen. This spyware gets hidden somewhere in the computer and becomes very very difficult to remove it. I tried to install hijackthis and superantispyware but both were not able to remove this stupid, annoying spyware. More info on this spyware can be found here.

Then after searching for sometime I found a video on youtube (see youtube can be helpful as well :-)) which can be found here. What the video suggested is to restore the system to a point earlier in time and that should take care of the spyware. I had no option but to give it a shot. I restored my system to a date 2 weeks in advance and voila! the spyware was gone. The browser stopped showing stupid ads and I was able to browse peacefully :-).

So if you get infected with this annoying spyware, try doing system restore. But do it at your own risk! :-)

New post

Where to begin - ah! here it is - in my last post I did mention that my next post would be from Houston, TX. That was way back in Dec 2007 when I was travelling to Houston. But, many things have changed now and it is Sept 2009 (yes! you read it right).

I travelled to Houston, stayed there for 15-16 months. Then, went back to India for sometime and now I am back again in US in Washington. Lot has changed and I think I am just there where I was before. I am still trying to learn new things and still trying to understand lot that goes around me. However, now that I am back to US and have 24 hr internet connection, I will try posting something which I notice or would like to share.

Till then, see ya.

Klueless 3 Konquered/Kompleted and Onsite

Finally completed Klueless 3!!!. It was awesome till it lasted. I have played all the "versions" of Klueless and I think all have been good. Although most of levels have sufficient hints in each puzzle but I used to check the Klueless blog for hints which were more helpful. Level 24 was the one level which took long long time to complete because I was going in the wrong direction and never looked in the other direction. Apart from that none of the other levels took that long. I thought level 25 was good because it was something technical which I found it to be my "home ground" :). If you want to play it go ahead and click here. Warning!! Play at your own risk since you will be totally addicted to it and you will have to sacrifice your work/study to complete it. Which I did. :-)

Btw, I am going to Houston, Texas next year i.e. 2008 :-). So next post probably from the "lone star state" i.e. Texas.