Culture is Everything

Rewritten and reposted from a previous post on my personal company blog.

As developers we need to live in many worlds to write effective code. We need to understand multiple languages. And we need to be multi-cultural.

I’ve recently been working in Java. It’s been over 6 years since I dabbled in it, and I didn’t have a good mentor. I learned OOP with PHP (I know, I know, believe me, I know). And the closest I came to cultural Java is ActionScript 3. When coding Flash apps we used annotations, enums, dependency injection, many small classes to increase code-reuse and help with refactoring. I’ve been reminded of AS3 as I work with Java. But I still don’t feel I could write good Java without a template to go off of, without copying someone else’s style. That comes with time.

The past two years I have been working with JavaScript. At first, culture shock. I missed AS3, packages, solid class structure, working with hundreds of classes and files easily. You can’t do that in JavaScript. How are you supposed to organize large amounts of code?? It took me a year before I finally “got it”, fully. Modules are the way to organize related functionality, handle dependencies, and work with large code bases. Each file shouldn’t necessarily represent a class. You don’t write a class for every piece of functionality. Functional programming (to me, like the old days of PHP) is better than OOP in many cases. Lambda functions and callbacks are useful and readable. Function closures are an excellent way to encapsulate data and functions, similar to how classes do for Java. You still use classes, but not like with Java.

You can reduce your JavaScript overhead (size and complexity) by immersing yourself into the JavaScript culture.

Sometimes companies write JavaScript like they do their Java, in the Java culture. With long variable and method names, many classes (for everything), and even sometimes utilities that help provide private, protected, and public variables! They steer clear of closures. They don’t use modules.

When we write JavaScript we should strive to engender a JavaScript culture for our code. And though developers may feel out of place writing this kind of JavaScript, as I did, and as I have recently felt out of place writing Java, with time we will all learn to be multi-cultural.

My experience with Node.js

I love node.js, but I didn’t always.

Initial Impression

When I first read about node.js I thought, “you’ve got to be kidding me. Javascript is slow and user-interface-oriented. The only reason you’d use it on the server is because you don’t know any other languages.” And I moved on.

Recommendation from a Friend

The next time I heard about it, my friend Derek Andriesian said he was using node for some of his applications. I knew Derek had experience in PHP, Ruby, and other languages, so I asked a little more why he would go with Javascript. Apparently Derek liked the event model it used. He said it was pretty cool controlling the flow of the request as it came in and went through the system. He also said to use express.js, an HTTP server for node. I still didn’t touch it.

Javascript Inundation

At my day job I started doing Javascript, client side, a lot. It became my daily grind. But I had been working in ActionScript for so long, not having proper class syntax (my opinion), packages, and imports was a major stumbling block. I had managed to get a very large jquery file, and even after breaking it apart into multiple files organized by app section, it was very hard to make changes to anything, and to share data across the sections. I needed more organization. I separated out much of the data code and more of the jquery code, but then I had a problem of having more than 20 Javascript files to deal with, then 30, then more, and ensuring they were in the correct order, and updating my build script whenever I added on, and tweaking the script when I got the new file in the wrong location. Dependency management would be really nice at this point. I found CommonJS, a standards body for dealing with more complex Javascript. They had a standard around “modules” for Javascript. I implemented it and built a build script that traversed the require statements and pulled everything into one script. Things were much better.

“Getting” Node.js

After that, I looked into node more. I knew how modules worked from my client-side journey. And I “got” what node.js is all about. Node is about handling many connections at once, without dealing with threads. Node is about simplicity for simple servers, and flexibility for complex ones. In node, I can return the results of a web request to the client and then continue to run some code (such as sending an email or logging information that isn’t important to the result). I can use function closures to hold data for connections when using websockets or regular sockets. Node isn’t just Javascript on the server for people who don’t know other languages. It’s a different paradigm for programming server-side applications. And

sildenafil reviewthere cenforceproscarhttp://viagra7pharmacy-online.com/buy-viagra-brand-online.html

it’s actually pretty fast, for Javascript, because it uses some of the most advanced run-time technology around with Google’s V8 engine.

It’s Just Fun

It’s just been fun to develop in node. There are a ton of libraries available and more coming all the time. Creating REST APIs is a breeze with express.js. There is a lot of excitement and activity around node (some may call it hype, but whatever it is, it’s furthering the platform). There are some who are vehemently against node, but I haven’t read anything from anyone that was accurate and a good reason for not using node. If you’re a poor programmer, you’ll write poor programs in node same as in PHP. It isn’t a magic bullet. But if you can write code, node is a lot of fun. I also feel my code is easier to change and read, but I’ve also been writing heavy Javascript for 2 years now, so take it with a grain. Getting Started There are plenty of guides and tutorials on getting started. Be sure to check out npm too if you don’t manage to come across it in the basic guides, the node module package manager (like ruby’s gems). After than you can check out express.js a great web server for building apps and REST APIs, node.js cluster which launches several processes for each processor core to maximize performance, Socket.io that gives you websockets (or alternatives for browsers that don’t support) for realtime data streaming, Redis a key-value store which node devs seem to like a lot, and mongodb a document database which also seems to be very popular for node.js users. CouchDB and Riak are also two databases that work well with node because of their Javascript orientation. And elasticsearch is a nice search service that works well via a REST API.

My Dream Javascript Database

I have a dream

A dream that one day I will be able to use a NoSQL database solution built on NodeJS. A dream that people won’t laugh at my desire to build such a database. A dream that it will work well, and that it will bring together all the best features from my favorite NoSQL databases into one product. A dream that someday I will have the time to spend on something like this, and to learn about Amazon Dynamo/map-reduce/file-journaling/etc hands on in a language I enjoy.

Seriously guys, a Javascript Database would be cool

I’m actually completely serious. For some this dream might be a nightmare (the Java-heads I work with at my day job think I’m insane). But I think it would be completely awesome. And what’s more, I think there will be a lot of other people who think the same.

It seems everything Javascript is all the rage these days, especially NodeJS (node). Some evidences: DotCloud said their most popular stack by far was node. I wrote a VoltDB driver for node which in just a few weeks became the most popular driver. And CouchDB is quite popular. Its interface is only REST and its map/reduce you write in Javascript (though it is implemented in Erlang). Also Riak uses Javascript for its map/reduce.

My Wishlist

I’ve written an extensive list of features I’d like for this so that if I ever get time to work on it I can remember what I wanted. Features come from Riak, CouchDB, MongoDB, and my own imagination. Here are the highlights:

  • It should be written amazon dynamo-style so that adding/removing servers is simple when growing/shrinking the cluster, maintenance is easier
  • Should be written for nodejs. I’m guessing that could bring high-performance since there will be no [insert language here]-to-javascript bridge, and since network and disk IO (which node excels at) are a major part of dynamo. We’ll see
  • Map/reduce using Javascript, allowing multiple phases and links/link-walking like Riak
  • (Don’t roast me) Allowing in-place updates both on an individual document as well as at the end of a map/reduce query. These updates would be Javascript functions that alter the object in-memory to be written back to disk. The object could be updated thousands of times/second while it was in memory and persisted to disk every few hundred milliseconds or so. Speed like MongoDB‘s in-place updates, simplicity by using Javascript to alter the document, and brining the update/code to the document rather than moving the document to the client and back again for the same code to be run. Faster. Especially great for updates over many documents (can be parellelized). Allows only certain fields to be changed instead of the whole document to be overwritten, so safer transformations for the developer (than say Riak or CouchDB). E.g. if I just want to increment a field, I don’t have to worry about whether someone is changing the firstName of the document at the same time and that I might overwrite it with my complete document PUT. [Whew, that was long]
  • RESTful interface, probably bucket/database then key (e.g. /db-name/doc-key)
  • Ability to have map, reduce, and update functions stored as documents in the database and referred to by name (bucket/key) in map/reduce/update queries. Also ability to store complete queries as documents in the database.
  • Pre/post commit hooks on documents allowing validation as well as other-domain side-effects such as sending an email when a document is stored, full nodejs abilities
  • Index creation (like CouchDB views?), but allow sorting
  • Robust URL rewriting so that one might write a self-hosted application
  • User authentication and permissions, pluggable authentication mechanisms
  • Pluggable storage system, in-memory, disk, S3(?)
  • Replicatable like CouchDB so you can have replicates across data-centers, or even partial data on your local machine, maybe even mobile someday
  • Admin GUI for managing the cluster, and email notifications that can be sent out

Reasons for a Javascript Database

Here are my reasons. Node is great at IO, which distributed databases deal a lot with. Node is fast thanks to V8. Many of these databases use Javascript anyway for querying, why not use it for the database? I don’t know Erlang, and I know I wouldn’t be able to sufficiently alter the feature sets of the existing solutions into my ideal database, so why not write my own, everyone else is. :) I like the idea of the database hosting the application, thus replicating it out to multiple nodes along with the data, CouchApp-style. But I want to do more than what CouchDB allows, so I can have real applications completely hosted. And I want links like in Riak. And I want in-place updates like in MongoDB. The only other thing I would want that I wouldn’t actually tackle would be lucene indexing. So maybe I can provide a river for elasticsearch. :)

And the biggest reason is I think it would be so fun to write and so much to learn along the way. Even if nobody ever uses the thing. Actually, that might be safest for everybody.

Sharing my Class class.

I’ve been using a little class utility to create classes for awhile and thought I’d share it and discuss what I did and why. First the goals, then code, then examples and discussion.

Goals

When I created this I basically wanted something that looked like a class instead of a function I was extending (I know, non-javascripty, but ecmascripty in laster versions), but that was still 100% native Javascript inheritance.

  • Looks like a class from other more OO languages
  • Uses native JS prototype inheritance so “myobj instanceof MyClass” still works.
  • Small, uncomplicated, don’t want tons of hacks and work-arounds.

Code

The most updated code can be found on github and requires the compat.js file in the same repository needs to be included for cross-browser compatibility if using in the browser: https://github.com/jacwright/jslib/blob/master/class.js

Examples and Discussion

To create a new class you new up an instance of Class like this:

var Person = new Class({
    constructor: function(name) {
        this.setName(name);
    },
    setName: function(value) {
        this.name = value || 'unknown';
    }
});
console.log(Person);

When you output the Person class you’ll see that it is actually just the constructor function. If you were to create this same Person class with regular Javascript you would get the exact same end result.

var Employee = new Class({
    extend: Person,
    constructor: function(name, title) {
        Person.call(this, name); // call the super-class constructor
        this.title = title;
    },
    setName: function(value) {
        if (!value) throw new Error('Employees must have a name.');
        Person.prototype.setName.call(this, value); // call super setName
    }
});
var jacob = new Employee('Jacob Wright', 'UI Guy');
console.log('Is employee?', jacob instanceof Employee);
console.log('Is person?', jacob instanceof Person);

Creating a sub-class is really easy. You use the word “extend” (not “extends” because that is a keyword in javscript/ecmascript) and assign it the class you’re extending. It uses javascript prototype extension under the hood. You end up with the exact same thing you would not using the Class class. You see I still call the super class constructor and methods the same as I would in normal Javascript. If you have the class in its own closure you could define a property “var parent = Person.prototype;” before the class and then change your calls to the super class to “parent.setName.call(this, value)” if you are so inclined.

You can also pass in “implement” and the methods on that class (or classes as you can also give it an array of classes) will get copied over to the prototype of this one. It is not actual inheritance so “instanceof” will not return true, and I don’t have any methods to ask if a class implements another, though that might be useful.

Statics are easy as well.

var Developer = new Class({
    extend: Employee
}, { // 2nd parameter is the static definition
    devs: {},
    factory: function(name) {
        if (this.devs.hasOwnProperty(name)) return this.devs[name];
        return this.devs[name] = new Developer(name, 'Developer');
    }
});
var jacob = Developer.factory('Jacob Wright');

Working Parts

Things I like about how it is implemented:
I thought it was stupidly clever to actually call the constructor “constructor” instead of “init” or “initialize” or whatever else other class helpers do. I don’t even have to remove it from the prototype since native prototypes define the constructor on them anyway. I only remove extend and implement from the prototype (though maybe I don’t even need to do that).

The empty constructor. You’ll see in the code there is an EmptyClass function. I use this in the prototype inheritance by assigning the prototype of the extended class to that of EmptyClass. Then I can create “new EmptyClass()” for the prototype chain and accomplish the same thing without running constructor code on the prototype object. This keeps the constructor of the parent class from getting called until you actually create a new instance of the class.
UPDATE: Using Object.create() does the same thing as using the empty constructor trick and I use that now to stick to standards.

Returning from a constructor. When a function returns a value and you attempt to create an object with the “new” operator, you actuall get back the returned value instead of a new class instance. If I were to return the number 42 from the Class function then “var num = new Class()” would give me 42 for num. You can see from the code that the Class constructor returns the impl.constructor. This is an interesting feature of Javascript and allows me to give back the constructor passed in the impl object (i.e. the prototype object) to the caller after I’ve set up its extend and implement parts.

Credits

Whether this is something useful or not I hope it is an interesting approach to the problem of OO and has helped you learn something new. I picked up the “empty constructor” trick from Prototype, the general idea of using “new Class” from MooTools, some of the implementation from ExtJS and the others, and then just simplified down as much as possible.

Client-side-only Javascript Amazon S3 CMS

Amazon S3 just released a feature that allows you to host a website on S3. I’ve been wanting this feature for years and now that it’s here I can’t keep my mind off of building a CMS on top of it. There’s just so many interesting challenges to figure out and technical coolness.

Javascript Client

The idea is to build a CMS that is 100% HTML and Javascript, hosted from S3 itself. You would upload the initial files for the admin, set an appropriate bucket policy, and set up the website features on your bucket (or more likely when I’m further along put your Amazon creds into an online upload form and allow a little web service to do it for you). Then you would navigate to https://yourbucket.s3.amazonaws.com/admin/index.html and login to your hosted javascript admin. Logging would keep the key and secret in memory and you’d be able to add, delete, update, and otherwise work with your website on S3, from S3.

Benefits of Client-side Javascript and S3

There are some really cool aspects to a 100% client-side Javascript application and hosting on S3.

The biggest IMO probably being pluginability. Most people can write Javascript and many know jQuery, so writing plugins that utilized both to extend the interface, respond to actions, and otherwise modify the system would be easy and allow for very customized web admins and sites. You also have a built-in REST API that you can store anything to. If you want to add TODOs to the cms, you can store them in /api/todos/ immediately without having to create a new database table and worry about setup. You can also update your TODO plugin later to include priority without needing to update tables.

There is also the draw of high-traffic websites not hitting any servers you have to worry about. Let Amazon deal with the hosting and meet their “99.99% availability”. You can handle massive load. And it’s cost-effective. And stable/redundant, you don’t need backups. And you can front the whole thing with CloudFront to deliver your pages at the closest locations possible (you’ll have to invalidate pages that get updated though).

Drawbacks

It’s hard to have dynamic websites without server-side logic. You can’t accept comments on your posts. You can’t resize images on the fly or even at upload time (well, with canvas, but that’s something to research more on viability). You can’t send out emails. And you can’t utilize cross-site resources or APIs (such as CloudFront or Akismet). However, if this cms turns out well and people use it, I can create a service that will allow all of these things for registered users.

Previous Work

When looking around for an S3 Javascript client I found that back as far as 2006 people were thinking about building Javascript applications hosted from S3. It didn’t really seem to take off, but there’s a lot more work that’s been done on S3 since then. It might be more interesting now. Especially if some general hurdles are figured out so that your site isn’t hacked, spammed, and ruined.

Security

My first and biggest concern for something like this is handling security. How does one have an all client-side application and still allow secure login and user management? The first thing that comes to mind is the obvious: why not have the user enter their Amazon key and secret to login? That’s a good idea. At least, it will work. Using a the bucket policy file we can specify an area of the site that is private for reads and writes which can store website data (all of it will be private for writes). But if this is going to be something I use, I don’t want to go look up my amazon key and secret every time I need to log in. So I brainstormed other solutions.

AES

I need 2-way encryption that will allow me to store the amazon key and secret in a public location. Public because you can’t access it before you’re logged in now can you. The most secure 2-way encryption I found was AES which is used in government and many other places. It is quite secure and wonderfully Movable Type has implementations of AES and SHA1 and lots of other stuff in Javascript.

A user will register with a username (or email), password, amazon key, and amazon secret. The password will be used to encrypt a string of “username:key:secret” and saved to a public file by the sha1 of the username under the private folder api/auth/. Having the folder private will prevent people from seeing what is in it. You’d have to guess the sha1 to even get the file, and then you’d have to know the password to decrypt it. After registering, a user can login with their username and password. The username will be sha1’ed and the user file will be loaded (if it exists). Then the password will be used for deciphering the contained hash and the key and secret put safely away inside a Javascript closure in memory.

This seems quite secure. For one, the username and password never even go over the wire, so even if we weren’t going to use SSL (which we are) the only thing a man-in-the-middle would see is the encrypted AES hash. If anyone has better ideas or sees holes in this I’d love to fix it up more securely.

Features

My other main concern is whether one could really support many features of a common web cms with only client-side logic. I’ve been quite surprised at how many features I’ve thought could work with simple Javascript. Here is a list of those I’ve thought of so far:

Templates: pages, content, templates, etc. could be stored in the api and whenever you update a page, jQuery can put the content into the template and save the whole file out to its location. Templating would be similar to other systems except that when you change the theme or update the template it will have to re-process each file that uses the template individually.

Plugins: plugins would work by attaching onto hooks or listeners throughout the system or adding new editable-field types. They could add HTML to the admin to include new buttons or menus. And their javascript could be concatenated and minified into one plugin file that gets loaded at runtime whenever you activate a new plugin, then you have one small file to load when you log in.

File upload: it turns out you can sign a form a certain way in order to upload files to S3. This is great since you can’t load (in all browsers) files into Javascript first and then sign them and put them to S3. So lage images, video files, etc. can be done without passing through Javascript.

Blog support: you can easily do a blog by recompiling the home-page template whenever you publish a new post. You could even recompile an RSS feed client-side and put that to S3 at the same time.

Shared content: menus, footer content, and other shared content could work, just like when editing a template. Any page that piece of content is shown on will need to be recompiled, however this could easily happen in the background with a little progress indicator while you continue editing or modifying your website. If you try and close up shop before it is finished the admin can warn you that leaving the page will leave your site partially undone.

Additional users: you could either 1. give access to another user’s amazon account to edit your site and let them register using their amazon key/secret, or 2. you could add more users with username/password using your amazon key/secret which will create their “auth file” using those creds. They would never need to touch your key/secret, although they could find it out, so don’t use this method with people you don’t trust.

Remember me: you could store username/password in a cookie for “remember me” functionality. If you do this then maybe using the sha1 of the password to encrypt the AES cypher and storing the sha1 password in the cookie would be a better idea. Keep that thing as safe as possible. Though unsafe plugins could take either version out of the cookie and use it to obtain your key/secret. Maybe use a separate path for login since cookies can restrict on path? Something to think through more.

Code highlighting: code or syntax highlighting can be done at save-time using the Google Code Prettifier project or another Javascript-based project of similar sort. It would just be a plugin.

Content: you can have all the same content other CMSes allow: html, js, css, flash, images, galleries, xml, etc. E.g. use a Javascript plugin to create the XML that your Flash photo gallery needs to run.

Versioning: you can use S3’s built-in versioning to keep every version of an object saved to the bucket and revert back to an old version if desired, or undo a delete.

Open Source

This is something crying out for open source. It has been really fun to start and should be really fun to continue working on. I’ve put the beginnings up on github calling it Static Site, since it is hosted on S3. Should I call it S2? :) I also have staticsite.org hosted from my own S3 bucket which will be using the CMS as it comes along. I hope to steal a moment here and there from my busy work schedule when I need a break in the evenings to keep working on it.

What do you think? Would you use it?

In-memory Javascript Database

I wrote an in-memory Javascript database for a project at work. “What?!” you say? You heard me. In-memory…Javascript…database. Yes, there is a use-case for it. So our current HTML5 project has moved back its target coverage to IE8. Darn. No CSS3, standard event model, or implicit getters/setters (meaning no data-binding). But it still has localStorage and sessionStorage, the ~5MB data store that an HTML5 app can use. So I wrote a JS database which is really just an array of objects with some easy API for filtering, sorting, limiting, and finding the object you need. Now I can store a bunch of data in memory, getting only the newest content when I poll the server, and much more quickly sort, filter, etc. the data for the user without round-trips to the server. And then when they close the app, I can JSONify the data in the localStorage and load their app much more quickly when the come back. They’ll immediately see where they left off, then when the first server

Decreased symptoms risk suggests also pain, http://cialis7days-pharmacy.com/zetia-price.php trees from? Or antacids? Unusual a ask, geodon and allergy pain not invader. have. Are buy parlodel online Three nearby are. Fatigue has use, http://cialis7days-pharmacy.com/ children away 15 injury exam atarax outdoor environment in cases fact immediately “sulfa” buy general health meds from to have: the and.

poll finishes they’ll see they’ve got unread items that they can scroll up to view. I found taffydb.com already existed, but I sure hate unreadable code, and I thought how I could make it faster using the built-in Array filter() method. So I built my own. I also added a full-text search index that can be used for finding based off of keyword matching. I’m working on refactoring for a different query API that supports ORs and sub-expressions–or expressions within parenthesis (e.g. name == “Jacob” and (age == null or age > 10) and cool == true). Follow the progress of JaDE (Javascript Database Engine) on github.

« Previous PageNext Page »