SpotWatch & Thesis Update

Amazon recently added even more instance types to the spot pricing market, namely the Cluster Compute and Cluster GPU instances in the US-East region! I updated the SpotWatch application with these two instance types, it is up and running on the website: spotwatch.eu.

Besides from that I've been busy writing my thesis, which seems to be a rather slow process. I just started writing the 4th chapter which presents the design of the proposed broker, this means that a first version of chapter 1 (Introduction), chapter 2 (Environmental Analysis) and chapter 3 (Resource Scheduling) are written. During the following two weeks I will also expand the implementation of my broker prototype and keep in mind that the next (and final) chapter handles about the evaluation of my brokers results.

Lack of Updates

Just a little post to explain the lack of updates the last couple of weeks. Well during this period at Belgian universities students need to study for and take their exams of the courses they took during the first academic semester. So, that is what I've been doing: finishing my project for 'topics in distributed computing' and studying for my exams.
Next week however I'll make some time to start writing a few of the chapters of my thesis and report on what I've been doing on this blog.

UPDATE: started by creating a Latex skeleton project first, at the moment I'm writing an introduction chapter, which handles about cloud computing in general and the Amazon EC2 implementation of it.

GOAL: I'll finish a first version of the first 3 chapters by the beginning of the second semester.

Decision Model Java Implementation

I made some little changes to the java port of the 'Decision Model ...' paper software. And added Javadoc comments to this source code, which can be checked out from the SVN repository. This software needs a lot of memory because it first of all reads a couple of 100000 records to memory from the input CSV file then uses it to do a lot of simulations ... and finally writes all the results from memory back to file. (Actually this process starts all over for every task length). I think to boost the performance it would not be a bad idea to only take the history spot prices of one instance type as input and do the simulation only for one category/instance type at a time. This would almost make all arrays used in the program a dimension smaller. And it's not that hard to then write something around it that runs the program a couple of times with different input for the different instance types. Also the program still takes its own input file, but this can easily be changed by implementing a data input reader that takes the cloudexchange CSV files as input. A backup of the source code of this program can be downloaded here.

Prototype Broker for Resource Allocation

I posted earlier what I was doing during the long weekend

I'm working on a prototype (A Broker for Cost-efficient QoS aware resource allocation in EC2) by combining the tools and knowledge I already acquired during the last couple of weeks"

Except for the fact that I keep feeling sick (for a couple of weeks already), everything went okay. The only issue is the actual scheduling in the time of the different tasks, I wonder if I can use some existing deadline scheduler for this. The prototype (it literaly is: e.g. undocumented, overlapping model and view, ... ) now handles everything for reserved and on-demand instances correctly. The next step (after refactoring and implementing a scheduling algorithm) will be to start adding spot instances. I created two programs:

1. A TaskGenerator that takes some program arguments specifying the task as input and creates a
   CSV file that describes the task and a corresponding CSV file describing the randomly generated workload of the task.
2. The actual EC2 broker prototype which takes the output of the previous program and some CSV files with the EC2 pricing details as input. I chose to create a Swing GUI that creates Gantt charts containing the scheduled tasks. I also created an overview of the corresponding costs. Some screenshots can be found below.

Overview of the scheduled tasks

Every task has a detailed tooltip

Cost overview

The source code of this prototype can be checked out from the SVN repository (EC2Broker project) or an archive can be downloaded right here.

What's next?

I'm working on a prototype (A Broker for Cost-efficient QoS aware resource allocation in EC2) by combining the tools and knowledge I already acquired during the last couple of weeks. An update will follow during the weekend.
I also started reading in 'AMPL: a modeling language for mathematical programming', because we decided that it would be a good idea for me personally to learn a bit about solvers.

What's next?

At the moment I'm making a comparison between spot and on-demand instances: looking for some interesting facts/sources on the web. Afterwards I'll have a look again at the "Decision Model for Cloud Computing under SLA Constraints" paper, to make a decision about how feasible it is to implement the presented model.