OK, this post about Ajax Datagrids is for the geeks. I’ve done plenty of posts discussing why HitTail is so amazing as a blogging tool, but only occasionally have I addressed the phenomenal feat of displaying all the world’s logfile data in real-time.
Why do more Web apps not do this? There are several reasons, but one is how massive the data is. I mean, it’s very massive. We’re already letting you step in real-time through hundreds of millions of records, with HitTail less than a year old. Before long, when you click that “next” link, you’ll be navigating database consisting of billions, if not trillions of records.
And it’s all in real-time: next, next, next. Prev, prev, prev. First, last.
And just to flex our muscle, we even go as far as to highlight the keywords for easy visual perusal, and hyperlink them for easy visiting.
This is all done in one simple, elegant Ajax datagrid.
I don’t think we’ll tell you how we’re able to accomplish this, while so few others have. Suffice to say, SQL is not your friend. Web application development tools, particularly the integrated development environments (IDE’s) with record sets, are not your friends. Even agile development frameworks with active records, such as Ruby on Rails, is not really your friend.
To make a box capable of manipulating and letting you navigate such massive amounts of data in real-time with such performance requires thinking outside the box. Early-on, I tried describing how we’re doing this to a few respected colleagues, and SQL has made the database programming community so myopic, that it may be impossible to program incredibly high performance record-stepping applications that don’t require huge record sets and cursors, therefore ruining scaling. The technique is called the indexed sequential access method (ISAM), and it’s broken on the Web. In fact, it may have never hit the Web (as far as I know), and HitTail may be the only known example (someone correct me).
So, what’s behind HitTail? An enterprise database like IBM DB/2? Oracle? Sybase or Informix?
Open source like MySQL, Postgress or Ingress?
Or perhaps a true ISAM database like BTreive (underlying Pervasive)?
The answer may surprise you.
Much is not what it seems in the world of databases, where the ability to construct a high performance Ajax datagrid is more a state of mind than a particular platform.
You have to throw out everything you know, then successfully deal with a whole new set of problems.
If you’re a tech geek with a blog or website, and want to play around with the aforementioned ajax datagrid, then simply register for HitTail. Put the snippet of code on your website. And start surfing the set of records which, among the hundreds of millions of other records, are yours alone.