Michael got a little geetar for Christmas, and I was sitting here working yesterday when I stopped and listened to him singing along.  I couldn't believe it - serious train blues.  I got up and went into the living room and there he was sitting in the middle of his train tracks singing along, "Red Train Going Down the Tracks!".  Listen to him singing about the piston at the end.  What to say, infinite cuteness.

When I was at Sun, I worked on the design for a replicated in-memory data store.  One of the principles was, scale and throughput demands are increasing, and memory is getting cheaper, so why not put it all in memory, and use disk/database only as a backing store.  We provided durability not through writing to disk but through replication, and then writing back to disk (e.g. to a traditional relational database) in the background.  Even if there were a node failure, we would recover from the replica, not from the database.

We couldn't get this project funded for various reasons.  It's been frustrating because I knew we were on to something but couldn't make it happen - and now, as many of us predicted, the industry is moving in that direction - a growing belief in denormalization, caching, and eventual consistency.  But still, many applications write to the database as part of the cycle of a transaction.

But this article at HighScalability hit the nail on the head - you evolve your application from database-centric to cache-centric to memory-centric.  Money quote:

As scaling and performance requirements for complicated operations increase, leaving the entire system in memory starts to make a great deal of sense. Why use cache at all? Why shouldn't your system be all in memory from the start?

Well, that sounds awfully familiar.  And it's true.  If you are replicating anyway, your risk of data loss is pretty minimal, and as Pat Helland, Amazon ex-architect says, computer suck, and you should just plan to apologize sometimes.  If you batch your changes every N minutes, then you have an N minute window where some changes may be lost.  But for many applications, that's OK.  And the wins you get in terms of throughput are significant

http://www.possibility.com/epowiki/Wiki.jsp?page=ItsTheLatencyStupid<br />">Latency was slashed because logic was separated out of the HTTP request/response loop into a separate process and database persistence is done offline.

Note that moving to flash-based disk, with something like Sun's OpenStorage solution, may give you the best of both worlds - durability and speed...

It looks like the way they implemented an all-memory solution was using Gigaspaces, which, by the way, has a solution that deploys and scales automatically on the Amazon EC2 fabric.  And the result: near linear scalability and 1 billion events a day.   Yeah, that's the ticket.

Please ignore...

Arun has a nice set of simple steps for getting MySQL 6.x working with NetBeans 6.5

This keeps happening - the community I need to stay engaged with keeps jumping on new forms of communication, and if you want to stay in tune, you need to jump on.

This first happened with blogging, and then with Facebook.  These I actually don't mind.  I like writing and reading blogs, and Google Reader has made it fairly easy for me to quickly digest what's going on.

Facebook is nice because I keep in touch with friends.  It's not so much about work but about getting a little more connected, particularly to old friends who I don't see much any more.

But this Twitter thing - I really don't like it. Sorry.  I subscribe to someone's Twitter feed because I value their thoughts, but then it's serious drinking from a firehose.  Many of these Thought Leaders are just constantly (I mean constantly, like I don't know how they get any work done) pushing stuff onto Twitter, and my head just spins.  I just un-followed Tim O'Reilly, I just couldn't take it any more.  I tried Guy Kawasaki for less than a day - that was mind numbing.

The problem is, these guys aren't blogging any more.  Tim Bray has gone almost completely radio silent on his blog.  Now Brian Aker's going to do the same thing.  I never hear anything from James Governor's blog any more, but his Twitter feed is really overwhelming, and I had to sign off.   It's a bummer, I really like what these guys have to say, but I just can't take it.

Those of you who follow people on Twitter, how do you do it?  How do you process all this information, get the gold, and get on with your life?  I am at a loss...

A fun poem in the spirit of St. Nick by my half-sister Alicia, who works in indie film. 

I don't understand some of the allusions, but I am encouraged by this part:

Theatrical's dying -- it has been forever,
Filmers are thinking, is self-distrib better?
So Ballast and Crumley and Range Life and Co,
What money is there hoeing that lonesome row?

Not much, but we own it ourselves, which is great
We do this for love, not a studio slate.
On iPhone! On Hulu! On V on Demand!
New platforms that stretch far across all the land!

Death to old models! Minds open for new!
Film wants to be free and cream rises, it's true!
Patiently plotted and pretty to scope,
Niche marketed art films please don't give up hope:

The one-two punch of the Internet and the economic downturn may provide huge opportunities for non-studio-driven independent film to make it to the masses...

Great blog by Closer to the Ideal (can't seem to find the author's real name) about the Depression and how it affected the stock market for years to come.  Something to think about...

There were millions of citizens, like my great-grandmother, who avoided the stock market after 1929. They helped create the low valuations that made the stocks on the stock market a good value, using traditional metrics like P/E ratios. Once upon a time, in a land now far, far away, a P/E ratio of 15 was considered frighteningly high. This month, the Dow Jones is suddenly back under 15, having spent most of the last decade above it. Will this level again come to be seen as normal, or will this eventually be seen as an aberration?

Very interesting editorial by Paul Krugman about the GOP strategy, and in particular this quote by Lee Atwater:

“You’re getting so abstract now you’re talking about cutting taxes, and all these things you’re talking about are totally economic things and a byproduct of them is blacks get hurt worse than whites.” In other words, government is the problem because it takes your money and gives it to Those People.

Wow.  I had never thought of it like that, but there it is in black and white, as it were.  It reminds me of how McCain/Palin were really pushing on the fact that Barack Obama said that it's good to spread the wealth, and how they tried to show that this meant he was a socialist, as if that were an evil, evil word.  The underlying message is the same - I don't want to give my money to Those People...

I'm back from Chama, New Mexico, where I had a wonderful Christmas with my family. For the first six days it snowed constantly. Ben, my half-brother, took the following picture, this gives you an idea of the serious White Christmas we had. It was glorious.

 

Testing...