Use array-view Instead of Angular’s Filters

AngularJS uses dirty checking to discover when changes have occurred to the data it is displaying on-screen. This simplifies your code. It speeds up development. And it allows you to use whatever-the-heck you want for your data and models, instead of forcing you to use their model classes (as Ember and Backbone require).

Dirty checking also re-runs filters. A lot.

Checking whether 32 has changed to 33 isn’t a big deal. Checking whether an array that, using Angular’s filters, is sorted and filtered by re-sorting and re-filtering the original is slightly more expensive. Doing that on every $digest (e.g. every key-stroke in an input field), especially on large arrays, could kill performance on slower machines and mobile.

array-view

Introducing array-view. array-view is a library that provides a sorted, filtered, even paginated view onto your original array. The view is an actual array that stays in sync with your original, so any push/pop/shift/etc will affect both arrays. And the best part, the view syncs only when required, not whenever $digest is run.

You can even use array-view to simply keep an array sorted automatically whenever you add new objects to it. It’s simple and easy to use.

And the best part: it’s still an array. Any library expecting arrays can use it. You can use it in your JavaScript SDK you provide for customers.

Inline HTML Editing Sneak Peek

You are building a website, CMS, blog platform, ecommerce site, or web app. You need the user to edit their webpages, product info, or posts. You have an idea for how it will work, how your product will make this easier than any other. But after hours of searching for something you can use you are emotionally exhausted. The only things out there that aren’t ugly hobby projects are over-engineered replace-your-textarea solutions. You can’t have your vision. You can only have someone else’s complete package. Or you can start your own hobby project and stop because of time, releasing it among the crowds of other partially started WYSIWYGs.

I have felt keenly the pain of existing WYSIWYG solutions through the years. I’ve been meaning to take the time to build one from scratch that kept the core separate from the toolbar/interface. One that other’s could build custom solutions with. I finally made some time. It’s still not finished, but it’s in a usable state. I wanted to provide a sneak peak.

Inline WYSIWYG Sneak Peak

To make portions of an HTML page editable you can add a data-editable="simple" attribute to the element or call editable.enable(element, 'simple'). The following screenshot is the HTML for the page being edited in the video. We’ll be open sourcing it soon.

Screen Shot 2013-03-26 at 2.46

I am an Original

I’ve been quietly working for Original the last month. I’ve worked with startups before, but this is my first real startup experience. We have a remote team of six really great people.

The hiring process was amazing. There was no resume submitted, just my Github profile and a few Skype interviews. We are using hall.com to keep in sync, Pivotal Tracker to organize the work, and Github branches and pull-requests for features and issue tracking for bugs.

We are using Node.js and Angular and MongoDB for our product. We’ve been able to build quickly and are excited to start getting feedback from an early alpha.

We really are remote, the six of us being scattered from the West coast to the East, all working from home, coffee shops, colocation, or wherever we feel like. No skimping on benefits either in the name of cash flow. I feel well taken care of.

Basically, this is my dream job. Minus an office AR Drone.

I’ll keep you posted.

restdb: A NoSQL JavaScript Database

I’ve thought for awhile how much fun it would be to make and/or hack on a JavaScript database. After seeing the implementation of an append-only B+ tree in JavaScript for a CouchDB clone I thought, what the heck, why not? So for my company’s hackday I thought it would be a great way for me to expand my knowledge and grow in new ways.

The Why and What

The real bottlenecks in distributed databases are network and storage, not the speed of the language. As long as the database is architected well, handles concurrency well, and is able to scale, then it should be perfectly fine to write it in any language. JavaScript and Node.js seem suited particularly well because of its ability to deal well with concurrency, IO, and many connections.

Using a dynamo-styled setup that can expand a ring of gossip-protocol-linked nodes should work well with node. Allowing socket-based connections as well as RESTful interfaces would work easily. Providing real-time websocket updates to web apps would allow for both a great web-interface into the health and load of the database as well as direct and instant updates to web apps relying on the data.

The format of this database is key-value. It stores JSON documents, but handles any other types of document too, from HTML to binary and images. They key for each document is a URI. This allows the organization of documents and related types (e.g. /tweets/1234 and /users/123/comments/1234). The database needs to be able to create custom indexes to sort documents in custom ways. And a mechanism needs to be in place to create and update those indexes and access them via a defined URI. Being inspired by CouchDB’s design documents, the indexes and other possible custom data handling (think document validation, triggers or callbacks, and refined ACLs/permissions) could be taken care of by creating special JSON documents called “apps”. These will be prefixed with an underscore to differentiate them from other documents.

The Parts

Rather than using a node B-tree I opted to use Google’s already stable leveldb. Node.js has a module with bindings for leveldb, so this provides a great start but it doesn’t support streams. This is a negative because the whole document is loaded into memory by the leveldb module. If it supported streams we could send the data along as we received it from the file system and keep memory low, especially relavent when storing large files such as pictures and video.

I’m using express for the REST server. This may be more than I need for this very specific use-case, but I know it well and can drop down to node’s HTTP stack sans-express if needed.

There is a gossip protocol module which is a start for our dynamo style ring, but there is a lot more work to be done on coordinating where on the ring documents are stored. I’m thinking it can work like Riak where you specify a number of vhosts for the ring and each vhost would be a separate database and separate node process. Leveldb only allows one process to be connected at a time, to prevent contention. Map-reduce queries would be spread around to each vhost. And the vhosts would be spread around the cluster/ring as it is grown. I’ve also included an mdns module, or more commonly known as zero-conf or bonjour, so that the nodes on a network can automatically detect each other and auto-join a ring possibly, depending on a configuration value.

I’ll use the msgpack module to serialize and store the JSON to leveldb. This will make it faster to pull the data off disk and into memory for indexing and map-reduce, and keeps the database smaller. The REST interface could support JSON and msgpack and then we could use the msgpack protocol in addition to the REST interface, allowing for connecting clients the choice to work in the smaller/faster binary format of msgpack.

The Result: restdb

I created and called this project restdb, since it is a database very much based on the REST protocol. I was only able to get the initial parts working today. This was mostly because I forgot to use –data-binary with curl for testing and spent a long time figuring out why my image wasn’t displaying in the browser. But you may install and run the database if you’ve got node.js installed by following the instructions in the readme.

You can do PUTs, GETs, and DELETEs. And it works well. The next steps would be to handle JSON correctly with msgpack. And to give support for the specialized app documents.

Each index created will be a separate database that stores the index field as the key and the document id as the value. This might cause problems as each vhost would need to have an additional database to its own for each index created. CouchDB deals with similar problems as it has a b-tree file for each database and for each index (view) in each database. It recommends increasing the ulimit on your machine. ElasticSearch has similar requirements. So I figure we’re not in too poor of company.

Future of restdb

It was a lot of fun to think about how I would build a database. It turned out to be able to be an application server as well as a database since using the app documents you could host complete web apps from the database. This is like the CouchApp, but I feel it supports it from the ground up rather than a bolt-on or pivot afterwards.

I don’t feel confident in my ability to write a production-ready, mature, or stable database. So I will continue to work on this whenever I get the itch to have some fun hacking, but it is not something I will commit any large amount of time to. However, if you are also interested in hacking JavaScript and want something meatier than a webapp to sink your teeth into, please fork and contribute to restdb. It has some great potential.

Caching data with localStorage and LZW compression

Storing JSON strings to the browser’s localStorage allows you to show data quicker on page loads, both for desktop and mobile applications. Using localStorage allows you to even create simple apps without any server (no syncing or collaboration, like a TODO app could do).

But like anyone, you want to pre-optimize for the massive amounts of data you’re sure you will be storing to localStorage. You don’t want your users to run out of room. So you grab an easy compression algorithm and use the worker API so your UI doesn’t freeze up and you’re ready to go!

There is a storage vs performance tradeoff. But who needs that in today’s world of… oh yeah, mobile, nvmd.

You can see what this might look like here:

http://jacwright.com/contacts/

 

Culture is Everything

Rewritten and reposted from a previous post on my personal company blog.

As developers we need to live in many worlds to write effective code. We need to understand multiple languages. And we need to be multi-cultural.

I’ve recently been working in Java. It’s been over 6 years since I dabbled in it, and I didn’t have a good mentor. I learned OOP with PHP (I know, I know, believe me, I know). And the closest I came to cultural Java is ActionScript 3. When coding Flash apps we used annotations, enums, dependency injection, many small classes to increase code-reuse and help with refactoring. I’ve been reminded of AS3 as I work with Java. But I still don’t feel I could write good Java without a template to go off of, without copying someone else’s style. That comes with time.

The past two years I have been working with JavaScript. At first, culture shock. I missed AS3, packages, solid class structure, working with hundreds of classes and files easily. You can’t do that in JavaScript. How are you supposed to organize large amounts of code?? It took me a year before I finally “got it”, fully. Modules are the way to organize related functionality, handle dependencies, and work with large code bases. Each file shouldn’t necessarily represent a class. You don’t write a class for every piece of functionality. Functional programming (to me, like the old days of PHP) is better than OOP in many cases. Lambda functions and callbacks are useful and readable. Function closures are an excellent way to encapsulate data and functions, similar to how classes do for Java. You still use classes, but not like with Java.

You can reduce your JavaScript overhead (size and complexity) by immersing yourself into the JavaScript culture.

Sometimes companies write JavaScript like they do their Java, in the Java culture. With long variable and method names, many classes (for everything), and even sometimes utilities that help provide private, protected, and public variables! They steer clear of closures. They don’t use modules.

When we write JavaScript we should strive to engender a JavaScript culture for our code. And though developers may feel out of place writing this kind of JavaScript, as I did, and as I have recently felt out of place writing Java, with time we will all learn to be multi-cultural.

Next Page »