Sammy.js, CouchDB, and the new web architecture
UPDATE: My slides and a protected instance (will open up for play soon) or swinger, available here
I’ll be honest. When I created Sammy.js, It was more of an academic exercise then a means to an end. I had ideas for how it could be useful, and had some immediate uses for it, but the initial thought was ‘recreating Sinatra’s API in JavaScript’. Little did I know – with a little introduction and the work of some clever individuals, its real value would be revealed.
We’re at the dawn of a new age on the web, or maybe or horizon, or a precipice; the metaphor doesn’t matter, the point is, things are changing. Specifically, as browsers become more and more powerful, as developers we’re given new opportunities to rethink how we architect web applications. Whether or not some of the predictions of ‘the death of the traditional desktop’ are correct – whether or not the near future desktop is just the browser – we now have the power, without proprietary technologies (flash, etc) to create truly desktop like experiences on the web. Thats nothing mind-blowing, we’ve seen it coming for a little while now.
Ok, then: let me blow your mind.
Because of the great power that modern browsers provide us as Javascript developers a new paradigm of web development is arising. Lets take a look at the current/traditional model:
Here we have a traditional stack. The ‘power’ of the application lies on the server: our development platform of choice (Rails, PHP, etc) sits on top of our database of choice (probably relational, probably SQL). This server platform renders finished HTML and maybe decorates those pages with the use of our JS library and our application specific code (Your $(hit)). In this model we’re using probably at least 3 different programming languages and almost fully on the speed and reliability of our servers for getting data to the user. There’s probably also at least 3 steps in either direction between data and the user:
- User clicks a link on a page or follows a URL
- The request is sent to the server
- App receives request passes it on to Model/ORM layer
- ORM requests the data from the Database
- Database returns the data to the ORM
- ORM returns the data in a parse-able form to the App
- App converts it into user-readable format and displays it to user
Obviously, Rails and most other languages frameworks abstract a lot of this away, so we don’t really have to think in all these steps. They’re there, though.
Welcome to the new world. HTTP Databases and JSON Storage. The simple act of making the database and the browser more powerful on either end has destroyed the need for the middle tier. In the new architecture, Our database (JSON/HTTP based: CouchDB, Cloudkit) serves data as JSON directly to the browser. On the browser side we create a much smaller/tighter ‘controller’ layer with JavaScript. This handles the directing of the user to the right place, the displaying of the data to the user, and the conversion of user interaction into state + data. This middle piece is jQuery + Sammy.js.
In my presentation at jQuery Conf this past weekend (slides available here) I demonstrated this concept in a rather meta-tastic way. The presentation tool I used was called Swinger and it was written using CouchDB, CouchApp, and Sammy.js. The application logic was contained in a single JS file that used a Sammy application to do all the routing, getting data in and out of CouchDB and displaying it in a particularly pretty way. The coolest part about this whole thing is that even beyond taking out the middle layer, this implementation removes the traditional static file serving model, too. With the power of CouchApp and _attachments, the javascript and HTML files needed to serve the application are served directly out of the database. Its like CouchDB and Sammy.js were made for each other. Think about the new steps for this architecture:
- User clicks a link on the page or follows a URL
- Data is requested via AJAX from the server
- The server proxies to the database or processes the request directly
- The data is returned to Sammy.js as JSON
- Sammy.js uses the data it needs and displays it to the user
There are a number really amazing and exciting things to note here. First, notice how the application is written in JavaScript AND the data is JavaScript. The JSON store might actually be written in another language (CouchDB is Erlang, Cloudkit is Ruby) but we don’t really ever have to touch those. Personally, I really like JavaScript. Sure, I miss some of Ruby’s syntax and niceties, but I’ll give that up pretty quickly for something that still shares some of the similar good features (closures, loose typing) and has a ubiquitous deployment environment (the browser). Second, the end result is probably similar if not better to the end user, as we eliminate the need to reload the page at every request, resulting in a better perceived speed.
One of the provisions I gave for my talk is that “I’m less concerned with speed than awesomeness”. That rings pretty true here. I don’t have an architecture like this running in production and getting hit like twitter. I can’t really speak yet to how well it ‘scales’ or how it preforms under load. I CAN tell you about how awesome it is and how easy and fun it was to develop swinger with Sammy. That’s really all I need to know right now.
Subscribe via RSS
Thanks for sharing this Aaron — I’m sending it over to my developer colleagues who weren’t at jqcon right now!
[…] http://www.quirkey.com/blog/2009/09/15/sammy-js-couchdb-and-the-new-web-architecture/ a few seconds ago from Gwibber […]
This is really cool, thank’s for this contribution. I will definitely look into this.
Interesting stuff Aaron! Looking forward to the slides.
Really think you’re onto to something here. Enjoyed you’re talk at JQuery Conf. Seems like an excellent solution for, at least, a class of applications.
Head’s spinning with possibilities. Wondering how Sammy.js has been used by others? thx
Hi,
thanks for this wonderfully writtern article. But, i have a question. (I haven’t used couchDB yet,so it may be obvious) What about authentication of the request to couchDB ? in normal case, server side logic takes care of that but if all data is availble through HTTP API from couchDB isn’t that security threat.
thanks
Gaurav verma
@Guarav: CouchDB’s has built-in authentication that works but is still pretty basic. It would also be pretty easy to set up a pretty thin middle tier with nginx or apache that handles HTTP basic authentication.
Not a bad post, informing other developers of new developments is a good thing!
I played for a short time with the JS/CouchDB stack for web development and noticed a number of issues that arise:
1) CouchDB just isn’t secure enough and exposing it so openly could lead to a number of issues if you get hacked.
2) JavaScript is far from benign, it runs locally on the machine and I know there is a large population of malware JavaScript (hence why I use NoScript).
3) Because JavaScript runs locally the user would essentially have every ability to modify the application logic (another problem).
Note that many of these shortcomings could also be interesting highlights, for example, users being able to change the JS application logic to modify the application to their own desires.
Some of these issues could also be solved with some complicated signing/certificate systems, but that would also require the data store (CouchDB in this example) to have those capabilities.
In the end, it is an excellent development, and may serve well for simple applications or applications that have enough transparency (Wikis?) to not care…
I look forward to a redevelopment of the entire stack, what if one were to create a truly distributed operating system (instead of a web OS, which is clunky)? No doubt there would be many challenges, but, in a society that is constantly on; it would seem that the web browser and ‘web’ stack in general are growing too complex and large for their own good.
I like that idea: a distributed OS… Quite a few ideas pop up from that one.
But there is more to security than just authentication.
If the power of your app is all in JS, what stops a savvy user from changing the JS on their client to muck with your system? Somewhere, server-side control is necessary to validate what comes in from the browser.
@dave @parnell: I think you’re right, there are some issues around security for sure – however, two points:
1 – CouchDB authentication is getting there. Theres work being done on per-db/user auth right now. Cloudkit already has OAuth baked in. Really you could also create a SUPER thin middle teir that just handles authentication and passes through everything else.
2 – In terms of users messing around with the front end code – sure! they could tottaly do that – but you can do that with any site now, right? Theres just MORE logic on the front here. Authentication/Authorization and the protection of your business logic should be handled on the backend, but as I said that should be pretty thin. What are you protecting besides data? If you’re front-end logic is you’re business’s/app’s most valuable property then obviously this architecture isnt for you.
That said, this is very early days – whether or not this proves to be a long term solution, i’m sure it will spur some new + very cool ideas.
It’s all cool but how google can crawl it?
@scotty definitely another acknowledged issue – there are ways around it (having the site/tempaltes work with the full URLs and the # tags) but this is less important for applications anyway.
I think you nailed it 🙂 We have had the same thoughts about new web architecture, where the limit between client and server and server is moving down to the server (http://blog.superfeedr.com/dev/gospel/web-apps/the-future-of-web-apps/)
The only comment that I would add though is that I don’t think that the backend will be reduce to the storage, I would just add a small “API” brick, but I guess the fact that you put REST in there means that you somehow had that in mind 🙂
Well done. It seems you have rediscovered client/server, except for that part where neither the frontend or backend perform as well and the marshalling is done by JSON (which is WAY better than the binary protocols the old dogs USED to use. And if they are really hard up for that, I say they can whip up a jquery backend for Thrift.)
Powerbuilder in Javascript should rock. Many congratulations on the awesomeness.
(Disclaimer: CouchDB dev here 🙂
—
Great article.
Re Crawling: CouchDB has built in transformation functions that can turn any JSON into static HTML for google to crawl.
Re Security: Even if CouchDB’s auth model will go all the way eventually, the main use case for it is having an instance on each personal device (laptop, desktop, set top, mobile, whatever) with all a user’s local data (and maybe a a copy of her peer’s data). Security is less an issue here with a centralised server system. CouchDB replication is used to sync all these devices and auth & replication filters already can ensure that a user can only replicate the date she is allowed to. With that data, and the application itself, she can do what she want.
Cheers
Jan
—
@tom: I’ll post a couple links soon for other projects using it here, soon. Thanks
[…] Sammy.js, CouchDB, and the new web architecture (tags: web internet javascript webdev framework database http json js jquery sql architecture couchdb sammy nosql) […]
I would certainly agree with you, I’ve said it before:
http://www.slideshare.net/secret/EcxO2A5MZia1q4
— MV
[…] they guy from QuickleyBlog created a new way to run web apps. It’s based on sammy.js. Sammy.js, in turn, is based on Sinatra. Sinatra is a very light […]
[…] “a tiny javascript framework built on top of jQuery inspired by Ruby’s Sinatra.” In Sammy.js, CouchDB, and the new web architecture, Aaron talks about the new age that we on the web are entering into. He writes that the current […]
I host a few applications like the new one you’ve described for high volume web sites. Making this scale well (into the GigE/sec+ range) is not as easy as it sounds.
One key to the scalability is to get good caching for hot data that’s continuously re-read in order to keep hotspots in your object store from crippling performance in the application. If your app has hot spots of data, then you’ll want to work right at the object store (in the REST part of your NEW diagram) to allow efficient insertion of a cache (CDN, etc.) between the client and the application for reads, or integration with a distributed memory cache like memcached. Ideally your object store (CouchDB in your setup) has sort of functionality built in, rather than relying on disk reads (or even page cached reads) per request on a storage given node.
I prefer to have a CDN between the client application and the data store in order to reduce the RTT (latency) of fetching frequently accessed data that’s common between multiple end-users of the application. This is particularly important if you have a globally distributed user base. If you use a good CDN that works well for small files, then you can achieve a cache that’s closer to your end-users than your object store, driving better performance, and a higher degree of efficiency and interactivity.
Making this work right means that you need full RFC2616 support in your REST layer so that If-Match (section 14.24) capability is fully supported, you set Cache-Control headers properly to suit your application (needs vary here, no single solution fits all).
Now you might argue that data should be de-normalized in the object store so that you don’t have hotspots. Perhaps that you should pepper replicas of frequently accessed data all over the object store so that all nodes get equal shares of work. That’s fine, but that model is not as efficient as having a high performance distributed cache. It just spreads out the work. You get horizontal scale, but no real efficiency boost.
wondered if you’ve looked at http://www.persvr.org/ to give you javscript on both sides of the wire.
Beautifully, simple model. RESTful interface with CouchDB; I love it. But sa others have mentions there are concerns with user authorization. (Not authentication; this is not the problem).
CuouchDB will probably get there as far as simple data access control is concerned. (Who’s allowed to read/create/update/delete a document).
However, for all but the simplest of apps, more granular user authorization will be required.
Example: User “A” cannot create an order document where the total cost exceeds $5000; User “B” is limited to $10000.
These types of edits must be enforced in a locked room behind a firewall running in a secured user account. They cannot be done on the end-users browser.
I suspect the good people of the CouchDB community will try to add these types of business predicates, just as SQL-98 did with the ASSERT declaration (and I suspect they will be no better received).
@ben: Not at persvr, but definitely at other server side JS frameworks. I like node.js in particular.
@David: Document validation actually already exists in CouchDB. You can ensure that documents have the correct author or structure very easily. What’s currently in progress is better authorization/authentication rules per-database and per-document.
[…] web architecture: The Near Future of Web Apps Tags: application designer, application generators, Domain Specific Language, DSL, […]
[…] Shared QuirkeyBlog » Blog Archive » Sammy.js, CouchDB, and the new web architecture […]
[…] Source:QuirkeyBlog » Blog Archive » Sammy.js, CouchDB, and the new web architecture […]
[…] web and internet growth to a level where it needs another, a better, a native technological infrastructure, different from classic client/server and desktop […]
[…] I’m more interested in playing with Sammy.js. The two application stack figures on the blog page (and in the Swinger slides) are interesting. Take away the couchdb bit, add Sakai’s K2, and […]
have you thought about using sammy.js w/ narwhal and jack? http://www.narwhaljs.org
They seem like a natural fit
Check out this blog I wrote a while back on JS3: http://labs.mudynamics.com/2009/01/14/js3/ One difference was that I ended up using Helma as the web server. So it’s still 3-tier, but JavaScript the whole way. You can see this in action on pcapr: http://www.pcapr.net (nosql here).
[…] Sammy.js, CouchDB, and the new web architecture Welcome to the new world. HTTP Databases and JSON Storage. The simple act of making the database and the browser more powerful on either end has destroyed the need for the middle tier. […]
[…] web storage needs. We had a seminar @Lab and document oriented dbs seem to have a definite niche. CouchDB is another product in the […]
[…] QuirkeyBlog » Blog Archive » Sammy.js, CouchDB, and the new web architecture (tags: jquery couchdb) […]
I think we’re missing some kind of templating capability with the suggested stack. The HTML/Javascript get’s pretty ugly.
Well Aaron your throwaway reference to twitter was quite prophetic! The new Twitter site seems to have jumped on the “new web architecture” bandwagon, proving that it is also scalable.
I wonder how much they were influenced by sammy.js?
Hi Aaron,
The link to sammy.js is broken.
http://code.quirkey.com/sammy/
I would love to have a look at the code, can you please have a look at it?
Cheers,
Chris
Sorry, it was a problem with the index code and github pages. Fixed now!
[…] Sammy.js, CouchDB, and the new web architecture (tags: javascript jquery couchdb programming) […]
Is it straight forward to extend Sammy to coffeescript?
I know coffee is really a transliterator and therefore should just work with just about any javascript but I wad curious about any pitfalls I’m not seeing for a big switch.
It reads so smooth and coffee is classy.