My Dream Javascript Database

I have a dream

A dream that one day I will be able to use a NoSQL database solution built on NodeJS. A dream that people won’t laugh at my desire to build such a database. A dream that it will work well, and that it will bring together all the best features from my favorite NoSQL databases into one product. A dream that someday I will have the time to spend on something like this, and to learn about Amazon Dynamo/map-reduce/file-journaling/etc hands on in a language I enjoy.

Seriously guys, a Javascript Database would be cool

I’m actually completely serious. For some this dream might be a nightmare (the Java-heads I work with at my day job think I’m insane). But I think it would be completely awesome. And what’s more, I think there will be a lot of other people who think the same.

It seems everything Javascript is all the rage these days, especially NodeJS (node). Some evidences: DotCloud said their most popular stack by far was node. I wrote a VoltDB driver for node which in just a few weeks became the most popular driver. And CouchDB is quite popular. Its interface is only REST and its map/reduce you write in Javascript (though it is implemented in Erlang). Also Riak uses Javascript for its map/reduce.

My Wishlist

I’ve written an extensive list of features I’d like for this so that if I ever get time to work on it I can remember what I wanted. Features come from Riak, CouchDB, MongoDB, and my own imagination. Here are the highlights:

  • It should be written amazon dynamo-style so that adding/removing servers is simple when growing/shrinking the cluster, maintenance is easier
  • Should be written for nodejs. I’m guessing that could bring high-performance since there will be no [insert language here]-to-javascript bridge, and since network and disk IO (which node excels at) are a major part of dynamo. We’ll see
  • Map/reduce using Javascript, allowing multiple phases and links/link-walking like Riak
  • (Don’t roast me) Allowing in-place updates both on an individual document as well as at the end of a map/reduce query. These updates would be Javascript functions that alter the object in-memory to be written back to disk. The object could be updated thousands of times/second while it was in memory and persisted to disk every few hundred milliseconds or so. Speed like MongoDB‘s in-place updates, simplicity by using Javascript to alter the document, and brining the update/code to the document rather than moving the document to the client and back again for the same code to be run. Faster. Especially great for updates over many documents (can be parellelized). Allows only certain fields to be changed instead of the whole document to be overwritten, so safer transformations for the developer (than say Riak or CouchDB). E.g. if I just want to increment a field, I don’t have to worry about whether someone is changing the firstName of the document at the same time and that I might overwrite it with my complete document PUT. [Whew, that was long]
  • RESTful interface, probably bucket/database then key (e.g. /db-name/doc-key)
  • Ability to have map, reduce, and update functions stored as documents in the database and referred to by name (bucket/key) in map/reduce/update queries. Also ability to store complete queries as documents in the database.
  • Pre/post commit hooks on documents allowing validation as well as other-domain side-effects such as sending an email when a document is stored, full nodejs abilities
  • Index creation (like CouchDB views?), but allow sorting
  • Robust URL rewriting so that one might write a self-hosted application
  • User authentication and permissions, pluggable authentication mechanisms
  • Pluggable storage system, in-memory, disk, S3(?)
  • Replicatable like CouchDB so you can have replicates across data-centers, or even partial data on your local machine, maybe even mobile someday
  • Admin GUI for managing the cluster, and email notifications that can be sent out

Reasons for a Javascript Database

Here are my reasons. Node is great at IO, which distributed databases deal a lot with. Node is fast thanks to V8. Many of these databases use Javascript anyway for querying, why not use it for the database? I don’t know Erlang, and I know I wouldn’t be able to sufficiently alter the feature sets of the existing solutions into my ideal database, so why not write my own, everyone else is. :) I like the idea of the database hosting the application, thus replicating it out to multiple nodes along with the data, CouchApp-style. But I want to do more than what CouchDB allows, so I can have real applications completely hosted. And I want links like in Riak. And I want in-place updates like in MongoDB. The only other thing I would want that I wouldn’t actually tackle would be lucene indexing. So maybe I can provide a river for elasticsearch. :)

And the biggest reason is I think it would be so fun to write and so much to learn along the way. Even if nobody ever uses the thing. Actually, that might be safest for everybody.

One Response to “My Dream Javascript Database”

  1. Norbert Fuhs Says:

    i am also working on such project because my server got down and now i am working on an dynamo inspired node.js based solution.
    But i´m new to node.js i just playing with it on my ubuntu box at home with the nginx web server. I also a huge fan of Apache Hadoop my problem is i need to build an app for my home town Osnabrück and want to build it completely in javascript.
    The demo should run on 4 virtual servers.
    Maybe we can work together and discuss things about the idea.
    Best regards,
    Norbert Fuhs