mindstorms: Microsoft

Showing posts with label Microsoft. Show all posts

7 Flavors of MapReduce

I am pretty sure that those reading this post already know what MapReduce is (in case you want to refresh your memories here is the PDF). I'm also pretty sure that you've already heard about the open source implementation of MapReduce contributed by Yahoo to Apache Foundation: Hadoop and you have probably heard also about Amazon Elastic MapReduce.

At least that's pretty much all I knew about MapReduce and its implementations. But I have discovered a few other solutions that offer a mapreduce implementation (disclaimer: I haven't tried these projects and I don't know their current status).

Disco

Description: Disco is an open-source implementation of the Map-Reduce framework for distributed computing. As the original framework, Disco supports parallel computations over large data sets on unreliable cluster of computers.

Project: http://discoproject.org/

Skynet

Description: Skynet is an open source Ruby implementation of Google’s MapReduce framework, created at Geni. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers. If you’d like to learn more about MapReduce, see my intro at the bottom of this document.

Project: http://skynet.rubyforge.org/

FileMap

Description: FileMap is a lightweight system for applying Unix-style file processing tools to large amounts of data stored in files. It provides full map-reduce functionality without requiring that you switch your processing to any particular language or runtime environment, install any special software, or have root on your storage and processing nodes.

Project: http://mfisk.github.com/filemap/

GreenPlum

Description: Greenplum Database is a software solution built to support the next generation of data warehousing and large-scale analytics processing. Supporting SQL and MapReduce parallel processing, Greenplum Database offers industry-leading performance at low cost for companies managing terabytes to petabytes of data.

Project: http://www.greenplum.com/

Hadoop

Description: The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing, including Hadoop Core, our flagship sub-project, provides the Hadoop Distributed Filesystem (HDFS) and support for the MapReduce distributed computing framework.

Project: http://hadoop.apache.org/

Amazon Elastic MapReduce

Description: Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Project: http://aws.amazon.com/elasticmapreduce/

There is also a project from Microsoft research that seems to related to mapreduce: Dryad (investigating programming models for writing parallel and distributed programs to scale from a small cluster to a large data-center) and its DryadLINQ module (make large-scale, distributed cluster computing simple, simple enough for ordinary programmers).

Do you know any others? Also, if you have any experience with any of these projects, I'd really appreciate if you can share it with us. Links to posts covering any of the projects are welcome.

Read post...

bookmark | email | 2 comments
Virtualization: Microsoft and VMWare Webinars
Microsoft getVIRTUALnow

Couple of weeks ago I've participated to two webinars about virtualization organized by Microsoft Romania part of their getVIRTUALnow initiative: Overview of Hyper-V Technology; Windows Server 2008 R2; HA and The management of virtual infrastructure.

I must say that at the end of the 2+ hours I have been slightly disappointed. The part I have appreciated the most was the slide decks. They were so great I'd really like to share them with you, but I'm not sure I'm allowed to do so. Unfortunately, the presentations were just vendor pitches and I cannot say I've really learned a lot. I got the feeling that instead of spending the time discussing about Microsoft solutions, their strengths (and weaknesses) and providing tons of details, the talks have most of the time slipped to simple comparisons with competitors' products. I would have also liked to see some more elaborated and consistent demos. All in all, I have to say that even if interesting from time to time, the 2 webinars were not the best use of my time.

VMWare Virtualization Webinars

In order to better understand Microsoft's virtualization and also see how competitors are positioning themselves, I've put together a list of webinars from VMWare, the name that came up quite often during the getVIRTUALnow webinars.

Just to make sure there is some common ground for all the materials I'd suggest to start with:
Intro to Virtualization: [...] who should use virtualization? Why should you use it? How does it work? What does it do? [...] This webcast provides a basic overview to virtualization solutions from VMware and explains the technology that enables virtualizations, as well as the features and benefits of VMware solutions.
View this webcast and learn how virtualization can help you:
- Save time, money and energy while achieving more with the computer resources you already own.
- Transform your IT infrastructure into a competitive advantage.
Desktop Virtualization
ESXi

ESXi is the free VMWare type 1 (native/bare metal) hypervisor.
- VMware ESXi Webstart
- Getting Started with VMware ESXi
Appliances
- An Overview of Virtual Appliances
- Virtual appliances: Strategy, Solutions and Programs
Application Virtualization
- Technical Track: Performance Tuning and Configuration of Resource-Intensive Applications and Databases
- Technical Track: Resource-Intensive Application and Database Performance on VMware Infrastructure
VMWare Advantage
- VMware Infrastructure: The Right Investment for a Tough Economy
- Best Practices for Deploying Your Virtual Infrastructure: VMware’s Own Path to Virtualization
The Cloud and Virtualization

Building the Internal Cloud: Transforming the Datacenter: This webcast will look at what cloud computing is and how you can begin to transform your own IT organization into a self-service, yet centrally controlled business unit.

Terms and Definitions

Hypervisor

also called virtual machine monitor (VMM), is a computer software/hardware platform virtualization software that allows multiple operating systems to run on a host computer concurrently.

VDI

Virtual desktop infrastructure is a technology consisting of a couple components presented in the attached picture:

Last

Who won the getVIRTUALnow prizes?
About getVIRTUALnow
- getVIRTUALnow: Sa trecem la treaba! (2)
- getVIRTUALnow: ce stiti despre virtualizare?
Read post...

bookmark | email | comments
Different ways to react to outages in the cloud: Google vs Microsoft vs Amazon
Launched less than 48 hours ago, the Google App Engine cron support has already seen a long outage spanning around 9 hours:

The scheduled tasks service saw some down time last night from 11:30pm and 2:30am PST (GMT-8), and again from 6am to 12:00pm PST.

I must confess that I don't have a (big) problem with the fact that the service was down. Microsoft Azure has been down for 20+ hours sometimes in March and I've heard of quite a few Amazon AWS outages in the past. But this is not the subject of this post.

What I am actually interested in is to see how these teams are reacting when facing problems, so let's take a look at a couple of examples.

Google

Here is the fragment from the latest report from Google about the issue I've mentioned at the beginning of the post:

This issue has been identified and fixed.

And this is not the first time they are very short on providing any sort of explanation (I have quite a few more examples, but I'll keep it short)

Investigation Complete - Issue Resolved
March 10, 2009 at 1:46 am 2009-03-10T01:45:51Z 2009-03-10T01:45:51Z
Online
We have determined that this spike did not affect the performance or uptime of applications. If you feel we have incorrectly diagnosed this issue please inform us by posting in our developer forum.

Investigation Complete - Issue Resolved
March 9, 2009 at 12:47 am 2009-03-09T00:46:51Z 2009-03-09T00:46:51Z
Online
We have determined that this spike did not affect the performance or uptime of applications. If you feel we have incorrectly diagnosed this issue please inform us by posting in our developer forum.

Microsoft

Let's just compare the above reports with Microsoft's report about their Azure outage. The team has posted on their blog an explanation covering 3 aspects:
- What Happened?
- What Was Affected?
- How Will We Prevent This in the Future?
Leaving aside the fact that as a way to prevent further problems the Azure team has decided to provide by default a second instance that is not counted against the service quota:

For continued availability during upgrades, we recommend that application owners deploy their application with multiple instances of each role. We'll make two the default in our project templates and samples. We will not count the second instance against quota limits, so CTP participants can feel comfortable running two instances of each application role.

the report makes me feel confident in the team capability to deal with problems.

Amazon

Unfortunately I don't have any specific references here, but I clearly remember Amazon been very explicit in their explanations and the plans to address similar issues in the future.

My question now is: who would you trust?

I do feel that failing to provide any details about the problems and their resolution is not the best policy to convince your users about your capability to operate the system and its stability.

What we've got here is a failure to communicate.
Read post...

bookmark | email | comments
Cloud Computing Unification Storm Calming Down
After what seemed to be the start of an interesting debate (with quite a few dramatic declarations), it looks like IBM and Microsoft sat at the same table and came up with a polite common position:

Representatives of CCIF, CloudCamp, Cisco, IBM, Intel, Microsoft and the IEEE-ISTO met while attending the Cloud Computing Expo in New York. Other companies were invited but were unable to attend, generally due to the short notice.

The companies agreed on a shared goal to promote use and awareness of open and interoperable cloud computing. The group brainstormed several ideas including the possibility to build on the momentum created by CloudCamp.

Another topic was the ability to enable participants, from individuals and companies, both large and small, to be able to contribute to and use the results of broad community collaboration. Additionally, the possibility of a trade association or marketing association for cloud computing was discussed but no specific actions were agreed. The final topic was the need to have broader participation from the community in this discussion

But don't let this declaration trick you. The fight for leadership is far for being over! In fact it is just starting and what happened last week was just a rehearsal. Interesting times ahead!

In case you missed the events, here is a series of post summarizing the week:
Resources:
- IBM And Microsoft Bury The Hatchet On 'Open Cloud'... Sort Of
Read post...

bookmark | email | comments
Cloud Computing Unification: Microsoft or Sun?
Looks like this is generating more discussions than I have initially though. You can check the last two links in the resource section for even more detailed coverage.
Even if the Sun cloud computing solution is not yet public, they made available the specifications (Tim Bray is the lead of the spec) under a commons creative license. Moreover they made it clear that Sun would be happy to have their specification becoming the standard for the cloud computing.

On the other hand, Microsoft has built their cloud platform Azure around some open standards too: web addressability, SOAP, XML, and REST and are stating that

Our vision in taking this approach was to ensure that the programming model was extensible and that the individual services could be used in conjunction with applications and infrastructure that ran on both Microsoft and non-Microsoft stacks.

Last but not least there is a cloud manifesto initiative that is taking shape on the Cloud Computing Interoperability Forum:

Our goal is to draft a document that clearly states we (including dozens of supporting companies) believe that like the Internet, the cloud itself should be open. The manifesto does not speak to application code or licensing but instead to the fundamental principles that the Internet was founded upon - an open platform available to all. It is a call to action for the worldwide cloud community to get involved and embrace the principles of the open cloud.

Now if you think about it, who wouldn't want to become the leader in the cloud computing market and coordinate and shape its future? (Forrester, Gardner and almost everybody else is speaking about the bright future of cloud computing). Most probably we will see once again some friction in the industry for the specification leadership role and for the

And the signs of the debate are already here, as Microsoft is already blaming others for not inviting them to the manifesto group (there are tons of links covering the subject in the resource section).

Recently, we've heard about a “Cloud Manifesto,” purportedly describing principles and guidelines for interoperability in cloud computing. We love the concept. We strongly support an open, collaborative discussion with customers, analysts and other vendors regarding the direction and principles of cloud computing. When the center of gravity is standards and interoperability, we are even more enthusiastic because we believe these are the key to the long term success for the industry, as we are demonstrating through a variety of technologies such as Silverlight, Internet Explorer 8, and the Azure Services Platform. We have learned a lot from the tens-of-thousands of developers who are using our cloud platform and their feedback is driving our efforts. We are happy to participate in a dialogue with other providers and collaborate with them on how cloud computing could evolve to provide additional choices and greater value for customers.

We were admittedly disappointed by the lack of openness in the development of the Cloud Manifesto.

I don't believe that either Sun's or Microsoft's current approaches can become the specification. There is indeed a lot to be learnt from their experience, but a specification will have to bring to the table all the companies involved in providing cloud computing solution plus the community. A specification created by providers only has no value with the consumers are not involved.
Resources:
Read post...

bookmark | email | comments
Microsoft to Lead the Cloud Computing Market
Not so long ago, some where saying that Microsoft is loosing the cloud computing bandwagon and that they have no idea what to do about it. Well, I'm pretty sure that now those that said it are biting their tongues. Not only that Microsoft has jumped into the cloud computing bandwagon with their Azure platform, but if they continue this way, I'd say that quite soon they will probably lead the market.

How sure am I about it? Nobody can be sure about anything these days, but I can tell you this: after spending time with both Amazon Web Services and Google App Engine, now I am preparing a virtual environment to start looking into Windows Azure.

Leaving aside my personal opinion, I'd encourage you to take a look at what Azure is offering (and what the plans are). Then do keep in mind that Microsoft has always been into integration and interoperability (in the sense that their products were always created to integrate and interoperate with existing ones). Last, but not least add to the mix the Microsoft BizSpark initiative (a program that between other things is offering startups 3 years free licenses to all Microsoft products) and you'll probably have a new perspective of the impact Microsoft will have on the cloud computing market.

Anyways, here are a couple of the latest announcements and presentations from Mix09 just to wet your taste. (nb: Silverlight required. Some of the videos are also available for download.)
Overview of Windows Azure

See through the Clouds: Introduction to the Azure Services Platform

Building Web Applications with Windows Azure

Windows Azure Storage

What's New in Microsoft SQL Data Services

Cloud Computing: What's in It for Me?

Using the Windows Azure Tools to Build Cloud Services

Simplifying Distributed Access Control with Microsoft .NET Services

Mesh-Enabled Web Applications

Live Framework and Mesh Services: Live Services for Developers

Connecting Applications across Networks with Microsoft .NET Services

A Lap around Microsoft .NET Services

I know this is a lot, but I'd strongly suggest to watch some of them in order to better understand why I think that Microsoft is going to be a leader in the cloud computing market.
Resources:
Read post...

bookmark | email | comments
Idea: Does Google Own the Search?
Tim O'Reilly, Mike Arrington and a lot of other people are discussing the latest Microsoft moves in the search market. Some are believing Google has won this battle, at least for now.

Idea Analytics

Should Microsoft and Yahoo stop fighting for the search market and just leave it in Google's hands?

Microsoft is dropping the Live Book Search, but launches Live Search Cashback. Where does Microsoft want to be positioned in the search market?

Details

This whole discussion started when Tim O'Reilly has published in a blog post the following advice:

my advice to Yahoo!: continue with your plan to outsource search to Google, just like you did before 2002, and plow those increased profits and reduced costs into your own innovation, strengthening the areas where you are #1, exploring new ideas that will make YOUR users insanely happy, and generally focusing on what makes Yahoo! great, rather than on what doesn't. That is, unless Microsoft makes you so good a deal for your search assets that you just can't say no. But either way, let yourself be quit of the destructive competition and focus on adding real value for your users.

My advice to Microsoft: outsource your search to Google too!

Tim O'Reilly

In reply, Mike Arrington is raising quite a few good points
- We are just getting started in search.
  There are so many areas on search that remain to be conquered. Semantic search. Real language/AI search. The deep web. Media search.
- Search monopoly and a healthy internet are mutually exclusive.
- [...] the CPC (cost per click) model is flawed, but in Google’s favor because it puts fraud risk inefficiently on the advertisers, who have no way of controlling it at the search engine level. CPA (cost per action) models work much better, but Google has done little more than test them.
Mike Arrington
Not far ago, I was also expressing my concerns related to video content search, even if there are quite a few contenders on this market.
In the following comments and new Tim's post, both have finally agreed that Tim's original advise was not about leaving Google approach to search unchallenged, but rather something that sounds more like Seth Godin's Purple Cow: instead of playing the catch up role, Microsoft and Yahoo should be investing their money and energy into researching something new, worthy and remarkable.

I also think that Tim O'Reilly opinions were triggered by the Microsoft's attempts to monetize search through the introduction of Microsoft Live Search Cashback, a new service on which the opinions are quite disputed right now.

However, there are signs that Microsoft is starting to look differently at the search market, as it has currently dropped the Live Search Books project (which was a reaction to Google's Book Search).
Related:
Read post...

bookmark | email | comments