Quick Reference to Alternative data storages

Make sure you check myNoSQL a NoSQL blog featuring the best daily NoSQL news, articles and links covering all major NoSQL projects and following closely all things related to NoSQL ecosystem. Everything you need and want to know about NoSQL.

Collaborative effort: Please help me fill in the gaps in the tables below by providing missing data, references to interesting articles, metrics, etc.. Please feel free to suggest new criteria to be included.

This is work in progress..

While it may probably not be exhaustive, my intention is to provide a quick reference to BASE systems (Basically Available, Soft State, Eventually consistent, as opposed to ACID: Atomicity, Consistency, Isolation, Durability) that would offer newcomers an overview of the existing projects in the field.

So far, I've been looking for filling in information about the following characteristics:

• Data model
• Partitioning
• Persistence
• Rebalancing (elasticity)
• Replication (clustering)

I have also included notes about the implementation language and the protocols that can be used with each solution.

If you think I should include other criteria please do let me know.

The projects included so far in the list: Cassandra, CloudBase, CouchDB, Dynomite, HBase, Hypertable, Kai, LightCloud, LucidDB, Memcached, MemcacheDB, MonetDB, MongoDB, Neptune, Redis, Ringo, Scalaris, ThruDB, Tokyo Cabinet + Tyrant, Voldermort.

Alternative Data Storages

Project	Data model	Partitioning	Persistence	Rebalancing	Replication
Cassandra	Column-family (BigTable[5], Dynamo6)	Y[n4]	disk	Y	Y
CloudBase	HDFS/Hadoop[n3]	Y	disk	Y	Y
CouchDB	Doc-oriented	?[n2]	disk	?[n2]	?[n2]
Dynomite	Blob (Dynamo6)	Y	pluggable	Y	Y
HBase	Column-family (BigTable[5])	Y	disk	Y	Y
Hypertable	Column-family (BigTable[5])	Y	DFS (HDFS)	?	Y
Kai	Blob	?	disk	?	?
LightCloud	check Tokyo Tyrant[n5]
LucidDB	Column-based	?	disk	?	N
Memcached[n1]	Blob	Y	RAM	Y	N
MemcacheDB	Blob	?	BerkleyDB	?	Y
MonetDB
MongoDB	Doc-oriented	Y			Y
Neptune
Redis
Ringo	Blob	Y	disk	Y	Y
Scalaris	Blob	Y	RAM		Y
ThruDB	Doc-oriented
Tokyo Cabinet + Tyrant
Voldemort	Structured / Blob / Text	Y	pluggable	N	Y

Notes

• [n1] Memcached: a distributed memory object caching system
• [n2] CouchDB partitioning and replication: according to a 2009 Summer of code proposal: While distributed deployments have been achieved with the help of proxies and smart external scripting, the core of CouchDB itself does not currently support distributing the database across multiple machines. More references about CouchDB cluster:
  • CouchDB Cluster
  • couchdb-lounge: Clustering framework for CouchDB
• [n3] All other criteria for CloudBase have been deduced based on the HDFS/Hadoop capabilities
• [n4] Cassandra: Consistent hashing vs order-preserving partitioning in distributed databases
• [n5] LightCloud seems to be a set of management scripts (Python) for Tokyo Tyrant

Implementation details

Project	Impl.	Client protocol	Refs
Cassandra	Java	Thrift[4]	[1], [2], [3]
CloudBase	Java	JDBC (Java)
CouchDB	Erlang	HTTP + JSON	[1], [2], [3]
Dynomite	Erlang	Thrift[4]	[1], [3]
HBase	Java
Hypertable	C++	C++ API, Thrift[4]
Kai	Erlang
LightCloud	Python + Tokyo Tyrant	Python
LucidDB	Java/C++	JDBC (Java)
Memcached	C	all*
MemcacheDB	C	all* (memcached protocol)
MonetDB	C
MongoDB	C++	API (Python, Java, Ruby, PHP, C++, Perl, Erlang)
Neptune	Java
Redis	C
Ringo	Erlang	HTTP
Scalaris	Erlang
ThruDB	C
Tokyo Cabinet + Tyrant	C	C, Perl, Ruby, Java, Lua
Voldemort	Java	Java

Performance

I usually do not trust micro-benchmarks. I know that performance measuring is an art. But I also know that some are looking for this sort of data and sometimes even the smallest piece of information is more helpful than nothing.

Project	reads/s	writes/s	refs
Cassandra
CloudBase
CouchDB
Dynomite
HBase
Hypertable
Kai
LightCloud	See: Tokyo Tyrant results + this
LucidDB
Memcached	here, 2007, here
MemcacheDB	benchmark data
MonetDB
MongoDB	Performance testing
Neptune
Redis
Ringo
Scalaris
ThruDB
Tokyo Cabinet + Tyrant
Voldemort

Other projects

I have found a couple of other projects, but I couldn't decide if they fit in or not. In case you consider that I should include them please do let me know (a helpful argument is also highly appreciated)

• Neo4j
• Schemafree
• Vertica
• Virtuoso

I'd like to also mention the FriendFeed usage of MySQL, which while not being a new system in itself it was conceived to behave like a BASE .

Resources:

1. Anti-RDBMS: A list of distributed key-value stores
2. Drop ACID and Think about Data (text)
3. Some Notes on Distributed Key Stores
4. Thrift
5. BigTable
6. Amazon's Dynamo (PDF)

Google, Newspapers and Advertising

Update: US newspaper quarter ad sales dropped almost 29% and other numbers in the industry are far for looking encouraging. Meanwhile, Google's ad revenue is growing. The discussions between newspapers and Google seem to intensify.

From a recent interview published on FT.com with Eric Schmidt:

FT: In their quest for revenues newspapers have started talking about trying to persuade you, Google, and specifically Google News, to share a little bit more of the revenue, specifically from their stories that appear on Google News. Wouldn’t that be a good idea?

ES: We’ve decided that the value we provide to the partners is the traffic. So we want to provide incredible numbers of users going to their sites, their content, which is why we urge them to make it deeper, stronger and use better tools and so forth. From our perspective, that’s where the real source would be. In our model, and what we’re doing today, the vast majority of the revenue that comes directly from reading newspapers, in fact, goes to them through all these mechanisms. The real issue here is that when people are reading the news online, we’re not monetising it in aggregate, so if we were to transfer money we would be taking money from something unrelated to newspapers and just paying them, which doesn’t seem like a good sustainable model for anybody.

FT: And don’t you think that there is a risk for you, as well, if you fail to find, perhaps, more of a revenue-sharing model, say, with something like Google News? Your sources of news for that service that you provide will just dry up? You risk killing the goose that’s laying the golden egg?

ES: This is the co-dependence that we were talking about earlier. From our perspective, we depend on the production of very, very high-quality content. If the people who are producing that are getting laid off, it’s really a tragedy for both. So we need the high-quality content. There’s a debate in the industry of exactly how to get it but, ultimately, the problem is not us taking money from some other pocket and subsidising it, ultimately the solution is to build products that really are so good that we make enough money from advertising and subscriptions, to a degree, that they make sense and that there’s enough money to pay for the construction of this high-quality content.

Or simply put:

• Google has no plans to share the revenues generated by their aggregation tools with the newspapers
• Google drives traffic.
• Newspapers should look into finding out better ways to monetize the traffic they get from Google.

I'm wondering if Google is not working on some products to be licensed to newspapers for monetizing the traffic sent back to them.

Some stats about the US newspaper industry (I cannot stop wondering when will our local market start learning something from this):

• Newspaper quarter ad sales shrank 28.9% ($2.6bil) (in a year-to-year comparison)
• Print ad sales declined 29.7% to $5.9 billion
• Online sales down 13.4% to $696.3 million
• Classifieds down 42.3% to $1.5 billion
• Ad sales collapse 16.6% to $37.8 billion in 2008. The worst decline ever.
• 2009 revenues will likely come in lower than $30 billion, less than they did in 1987
• Employment advertising shrank 67.4% to $205.4 million
• Real Estate down 45.6% to $336.9 million
• Auto down 43.4% to $332.8 million
• National campaigns down 25.9% to $1.1 billion
• Retail down 23.7% to $3.3 billion
• 'Other' down 16.5% to $587.7 million

Resources:

• Worst quarter for newspapers: Sales dive $2.6B

Is Online Advertising Overrated?

This was the (debate) topic for the last two WebClub meetings. While I haven't been there, the subject got so much coverage by the blogs that it caught my attention. And just to wet your taste here are a couple of quotes:

Pe de o parte se discută despre "eficiența" mediului online, pe de alta despre susținerea acestuia în lupta cu... aici nu prea înțeleg unde este dușmanul.

Ciprian Stavar

Am fost socat (acesta este cuvantul) de faptul ca oamenii din online sustineau faptul ca online-ul nu este eficient.

Mihai Dragan

At this point I've remembered a topic I've asked my Twitter friends about:

Can anyone explain how TV rating points are measured? I'm wondering why is it considered a more reliable stat than what is used on web

Twitter

I always wondered about that. IMO if they're not recording what runs on ca 70% of all TV sets the stats are unreliable.

FlashDriveDT

using "peoplemeters", meaning electrical devices statistically&relevant delivered or using questionnaires... not reliable at all.

luczian

TV is a traditional media and people/ brands are used to it, internet is new.....still in RO (in 140 characters this sums it up)

[...], but they have faith in this media tool because they are used to buy at CPP and don't understand the CPC's precision

only such a person can say that TV rating points are more "reliable" than web stats

Raluca Goia

Let's also take a look at Google has to say about TV vs web

What if the ads we saw when watching TV were always just what we wanted to see? Well, we believe it is possible to make TV ads more relevant to viewers and to deliver more value to advertisers.
Television is becoming more like the web. Just as users click with their mouse to choose what's most relevant to them on the web, viewers send signals about what they want to see on television with clicks of the remote control.

The Official Google Blog

What about classic media?

Advertisers love data almost as much as Google does. They think it tells them where to spend their money and the return on investment they get. For decades, advertisers accepted dubious measurements of magazine readership (which assume that every allegedly well-worn copy is passed around to large groups) and broadcast audiences. Then along came the most measurable medium in history, the internet, where advertisers can learn about customers than ever before. (nb: my underline)

Rishad Tobaccowala quoted in What Would Google Do?

or quoting Eric Schmidt about Google advertising initiatives in radio and print:

In both cases they didn’t work well enough. We measure our businesses very, very carefully and in both the print and the radio businesses we could not seem to invent or get enough of a signal back to make the network or value really spin – that’s one way to describe it. In our model what happens is as people click on ads and as they use our services we get all sorts of ways to improve our products. And with all that customer feedback we can make it better. But because of the unique structure of radio where it’s a broadcast to a relatively unidentifiable radio, there’s not very much information of what the radio is doing, and similarly for print ads, we could not get that signal. And that’s, ultimately, I think why we have moved on.

FT.com interview with Erich Schmidt

Then, you might think that Ionut is right:

[...] e posibil ca motivul pentru care internetul nu obtine un procent mai mare din bugete sa fie unul mult mai simplu. Nu neaparat eficienta, inteleasa sau nu, satisfacatoare sau nu, nu neaparat raspunsurile directe si performanta, ci ceva mult mai simplu: depasirea sau nu a asteptarilor clientilor.

Ionut Oprea (aka clickio)

But when you see what the romanian online is up to: here and here (the only thing I can think of when seeing these is that they are completely disconsidering their main clients: the readers), you'll probably reach the same conclusion as Vali, while they should probably spend some time thinking about what Dragos Novac is saying or Dragos Manac was writing the other day or Seth Godin's advise.

La noi, lucrurile au ramas in stadiul de portal. Bagam banii doar in alea, ca sigur acolo e publicul nostru. Altfel, de ce ar veni 100.000 de oameni pe portalul X?

Vali Petcu

Resources:

• Ionut Oprea » Cum sa marim cota internetului in bugete: azi, asteptarile clientilor
• Web(?) Club
• Eficienta, my a$$…
• Internetul - intre eficienta si deficienta
• Cum a evoluat internetul de la portal încoace
• De ce este/nu este internetul overrated
• 2009 - anul Internetului in Romania, gluma sau pacaleala
• Tuning in to TV data
• De ce sa-si bata capul agentiile cu interactivitatea campaniilor online?
• Making commercials for the web
• But Who's Counting? (recommended read)

In case you have two hours, thanks to Costin, you can watch both meetings in full:

7 Flavors of MapReduce

I am pretty sure that those reading this post already know what MapReduce is (in case you want to refresh your memories here is the PDF). I'm also pretty sure that you've already heard about the open source implementation of MapReduce contributed by Yahoo to Apache Foundation: Hadoop and you have probably heard also about Amazon Elastic MapReduce.

At least that's pretty much all I knew about MapReduce and its implementations. But I have discovered a few other solutions that offer a mapreduce implementation (disclaimer: I haven't tried these projects and I don't know their current status).

Disco

Description: Disco is an open-source implementation of the Map-Reduce framework for distributed computing. As the original framework, Disco supports parallel computations over large data sets on unreliable cluster of computers.

Project: http://discoproject.org/

Skynet

Description: Skynet is an open source Ruby implementation of Google’s MapReduce framework, created at Geni. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers. If you’d like to learn more about MapReduce, see my intro at the bottom of this document.

Project: http://skynet.rubyforge.org/

FileMap

Description: FileMap is a lightweight system for applying Unix-style file processing tools to large amounts of data stored in files. It provides full map-reduce functionality without requiring that you switch your processing to any particular language or runtime environment, install any special software, or have root on your storage and processing nodes.

Project: http://mfisk.github.com/filemap/

GreenPlum

Description: Greenplum Database is a software solution built to support the next generation of data warehousing and large-scale analytics processing. Supporting SQL and MapReduce parallel processing, Greenplum Database offers industry-leading performance at low cost for companies managing terabytes to petabytes of data.

Project: http://www.greenplum.com/

Hadoop

Description: The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing, including Hadoop Core, our flagship sub-project, provides the Hadoop Distributed Filesystem (HDFS) and support for the MapReduce distributed computing framework.

Project: http://hadoop.apache.org/

Amazon Elastic MapReduce

Description: Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Project: http://aws.amazon.com/elasticmapreduce/

There is also a project from Microsoft research that seems to related to mapreduce: Dryad (investigating programming models for writing parallel and distributed programs to scale from a small cluster to a large data-center) and its DryadLINQ module (make large-scale, distributed cluster computing simple, simple enough for ordinary programmers).

Do you know any others? Also, if you have any experience with any of these projects, I'd really appreciate if you can share it with us. Links to posts covering any of the projects are welcome.

QS: Business Rules for Our Time

After reading Alan Webber's recommendations in this post:

Rule #3: Ask the last question first. The last question is, what's the point of the exercise

Rule #23: Keep two lists, one that holds what gets you up in the morning and one for what keeps you up at night

Rule #4: Don't implement solutions. Prevent problems.

I am strongly considering buying his boook: Rules of Thumb.

Has any of you read it?

Update: Here you can find an interview with the book author.

And if I am at thought provoking reads, I would strongly recommend the KillerInnovations series.

Quick Guide on How to Behave at a Presentation

Today I've spent half a day at a VMWare event organized by IDG in Bucharest (my thanks to the organizers for the invitation). While I'm planning to write another post about how to learn at a vendor event, this post will be a bit different.

I have discovered that quite a few people present at such events do not know how to behave. And I am not talking here about the socializing part, nor about common sense (as in switching off your mobile while attending a presentation, not picking your nose in public, etc), but rather to the way they interact with the presenter and others in the audience during a session. So, here is my short list on how to behave at a presentation:

1. Raise your hand. We all learnt this from the school, but it looks like over years some tend to forget it. It is impolite to speak without permission, so raising your hand will signal that you have something to say.
2. Introduce yourself. You already know the name of the speaker, so considering you are trying to create a conversation, it would be polite to introduce yourself. In case your question is interesting for the rest of the audience, some may consider contacting you afterwards or citing you.
3. Make sure others we'll hear you. The presenter's time is shared between all participants so it would be nice that others will benefit of your question. If it is a private question, you can wait for after the session.
4. Be short and focused. Do keep in mind that there are others in the room and they are there for the presentation (and not for you).
   1. If it is a question: ask your question. In case your question needs some context make sure you are able to formulate that in at most one sentence.
   2. If you have more than one question, make sure you ask permission for asking all of them. Some presenters may prefer to address questions from multiple persons in the room.
   3. If it is a comment: make sure you ask for presenter's permission. The audience is interested in what the presenter has to say and in case you are not an expert in the field others will not care about your comment.
   4. Don't offer free advise to the presenter if he is not asking for it. While you might think that you have a better answer, I'm pretty sure that you don't know the whole context.
   5. Do not rant. And I'll say it again: Do not rant. In case you are not specifically asked for your opinion, I'll most probably not be interested on your comments to comments, comments to questions, etc. And others in the audience may probably feel the same.
5. In case your question was not completely answered, it is better to continue the conversation with the presenter after the session. Others have the right to ask their questions too.

"It doesn't hurt to ask". Actually, it does hurt. It does hurt to ask the wrong way, to ask without preparation, to ask without permission.

Seth Godin

Dedicated to all those that do not know how to behave at a presentation.

Disclaimer: as you can imagine by now, this post was caused by a couple of annoying people at the event. While I am pretty sure they will not read the post and most probably next time they will annoy once again the presenter, me and the rest of the audience, I'll probably have a couple of printed copies in my pocket and I'll make sure to handle them one at that time.

Google: Words vs Meaning

Transparency

Whenever we build out new products and features at Google, we try to ensure that we provide users with two key components: transparency and control.

By looking at the messages the Google App Engine team is posting in reply to reported issues, I have a couple of hypothesis:

1. the Google App Engine team is not aware of this policy
2. the Google App Engine team is not part of Google
3. the above paragraph is just PR

I hope you'll pick the right one(s).

Listening

Another thing I've noticed about the openness of Google is that they consider that opening the communication chains is equivalent to listening to your community. I'll tell you what: having a mailing list or a forum or a blog is not listening to your community and even less building a community. In fact, asking for feedback and not listening to it is much more frustrating than not doing it at all (just check the issue tracker for Google App Engine issue tracker or its corresponding Google Group to learn how to ignore your community).

So, I'll let you decide: is Google really transparent? is Google listening to his community?

DevWorld Retrospective

Monday marked the first edition of the DevWorld.ro an event organized by IDG and addressed to developers and designers.

The good

• I really liked the format of the event with some short sessions at the beginning, followed by specialized sessions for developers and designers and a panel
• I liked most of the sessions. I know it is very difficult to make everyone happy about it, so this is just my feedback
• I liked the fact that attendees have used the breaks to socialize and I've heard a lot of interesting discussions
• The selected projects for the contest were quite good. Their presentations were good too.
• The feedback they have received back looked to be in the right spirit.

The bad

• There was no internet connection and I had to buy it myself from the hotel desk. While I'm aware that getting an internet connection arrangement may be difficult and/or expensive (to somehow confirm this hypothesis, I've noticed that too many such events were no providing an internet connection) I think the organizers should try harder to get one.
• The space allocated for breaks was quite small and it hardly accommodated all the event participants
• Most of the sessions were longer than scheduled and so the agenda was not respected. While I do think that the speakers were the ones responsible for this issue, the organizers should have set up a process to keep it under control
• For the last couple of hours of the event there was no water.

What I'd like

• I'd like to see people in the audience getting more involved during the sessions
• I'd like to see people trying to connect with as many others as possible (instead of spending the breaks with the group they came in)
• and I'd definitely like to see the next edition.

Last, as I couldn't run the whole presentation I'm embedding it here:

Programatori cu capul in nori

View more presentations from Alex Popescu.

PS: Thanks to the fact that the schedule was not respected, I've had the chance to present to the full audience (and I confess I was quite happy with that).

Is there anything I've forgot?

JQuery Popup Dialogs Plugins

A set of jQuery popup, dialog, gallery plugins through two different pairs of eyes: jQuery Popup Dialog and Gallery Plugin Reviews and 10+ promising jQuery Lightbox-Clones Plugin.

The reviewed plugins are:

• FancyBox (both articles)
• FaceBox (both articles)
• Impromptu (1st article only)
• BlockUI (1st article only)
• nyroModal (both articles)
• FancyZoom (both articles)
• Boxy (1st article only)
• Flyout (1st article only)
• Pirobox (2nd article only)
• ThickBox (2nd article only)
• prettyPhoto (2nd article only)
• Interface Imagebox (2nd article only)
• jQuery lightBox plugin (2nd article only)
• Greybox Redux (2nd article only)

I only have used FaceBox as it pretty much covered everything I needed. It's design, look and feel resembles the one of the Facebook dialogs (the name also suggests that). I've also found Coda Popup Bubbles, but haven't used it.

Have you used any of these plugins? Which one would you recommend?

The Bell-Shaped Blog Interactions

In a guest post over Louis Gray's blog, Hutch Carpenter presents the 4 blogging phases and the way they are influencing the level of 7 different types of interactions with the readers:

And what are "interactions"?

• Allowing blog comments
• Responding to blog comments
• Commenting on FriendFeed about your blog
• Tweeting links to your blog posts
• Digging your blog
• Stumbling your blog
• Pimping your blog on others' blogs

Judging on the level of interaction I get, I'd like to conclude that my blog falls in the "industry legend" category, but for doing so I'd have to include the tons of spam I've received. Otherwise, I'll remain on the border between a newbie and the one still trying to figure it out :-).