The Pageforest Blog

The Pageforest Blog http://blog.pageforest.com Building a web application service for JavaScript developers. posterous.com Fri, 15 Apr 2011 15:34:00 -0700 The "Ambiguous Operator" in JavaScript http://blog.pageforest.com/the-ambiguous-operator-in-javascript http://blog.pageforest.com/the-ambiguous-operator-in-javascript

I found this style of non-deterministic programming to be "mind blowing".

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Mon, 14 Mar 2011 18:05:00 -0700 Fixing Recursive onResize Calls on the iPad (iOS 4.2 and 4.3) http://blog.pageforest.com/fixing-recursive-onresize-calls-on-the-ipad-i http://blog.pageforest.com/fixing-recursive-onresize-calls-on-the-ipad-i

We're writing Pageforest applications for the iPad (e.g., Quoridor). One really confusing issue popped up that seems to be a bug only when running in the full-screen mode of the iPad.

When we turn the display from portrait to landscape we would get a window.resize event. But then, as a side-effect of measuring a DOM element (like reading elem.offsetHeight), a RECURSIVE CALL to our window.resize event handler is called. I couldn't believe it - but it's true. This is the one case I've seen where the browser makes a re-entrant call to an event handling function.

To fix it, we created a simple function to serialize all calls to our real event handler:

$(window).resize( function () { setTimeout(onResize, 0); } );

Now all calls to our onResize function will be serialized, instead of occuring re-entrantly while we are in the middle of handling the first resize event.

We're not sure why our app is getting two calls to resize during a portrait to landscape flip - it looks like it might be giving us a smaller intermediate size, for the purposes of displaying a rotation animation of the page to the new orientation.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Tue, 01 Mar 2011 19:44:00 -0800 Demo of Application Manifest http://blog.pageforest.com/demo-of-application-manifest http://blog.pageforest.com/demo-of-application-manifest

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Mon, 28 Feb 2011 09:29:00 -0800 Offline Applications with Pageforest http://blog.pageforest.com/offline-applications-with-pageforest http://blog.pageforest.com/offline-applications-with-pageforest

HTML5 applications support an offline mode - all the files needed for an application can be saved in the client, so it can run without and internet connect. For a great explanation of offline applications see:

http://diveintohtml5.org/offline.html

As you can see from the article, the concept is simple - but it can be a hassle to use:

Create a Manifest file, and list all of the application files that you wanto be available offline. Be sure your server serves this file using mime type text/cache-manifest. You can see an example of a manifest file here: http://scratch.pageforest.com/app.manifest
Add a reference to the manifest in the <html> tag of your main application page (all the html pages for your application, if you have more than one).

Seems simple enough. The problem comes in when you change your application files.

If your files change, but your don't modify your manifest, then the browser will not know to re-download the new files, and will instead use the out-dated version stored in its cache. So, you have to be sure to modify the manifest file in some way (usually by updating a version number in the file).
If you add a file, you have to, of course, remember to add it to the manifest, or it won't be available when your application is offline.

pf.py to the rescue - autogenerated app.manifest

In order to simplify the process for Pageforest applications, I've introduced a new command into the pf.py utility:

$ pf.py manifest

When you issue this command, an app.manifest file will be automatically created for you, listing all the files in your application. More than that, when an app.manifest file exists in the root directory of your application, it will be automatically be updated whenever you do a put command to the server. So if any files change or are added, you don't have to manually update the manifest file yourself.

Customizing Manifest Files

You may want to add additional resources that are not part of your private application files to your manifest. For example, the scratch application adds:

http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js
/lib/beta/css/client.css
/lib/beta/js/json2.min.js
/lib/beta/js/utils.js
/static/images/appbar/green-left.png
/static/images/appbar/green-center.png
/static/images/appbar/green-right.png
/static/images/appbar/down.png
/static/images/appbar/logo.png

When placed above the AUTOGENERATED line in your manifest file, pf.py will preserve these lines and not modify them.

Excluding Files From a Manifest

If you have a large number of big files, you may not want to include them in the manifest (browsers will typically not download more than a few megabtes of data to the application cache). To exclude them from being added to your manifest, you can put an EXCLUDE directive in the auto generated section of your app.manifest:

#!EXCLUDE: big-images

Any files in the big-images directory will now be omitted from the manifest.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Fri, 14 Jan 2011 11:30:00 -0800 Using PF.py to deploy files to the Pageforest.com server. http://blog.pageforest.com/using-pfpy-to-deploy-files-to-the-pageforestc http://blog.pageforest.com/using-pfpy-to-deploy-files-to-the-pageforestc

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Thu, 13 Jan 2011 23:21:00 -0800 Intro Video for Pageforest http://blog.pageforest.com/intro-video-for-pageforest http://blog.pageforest.com/intro-video-for-pageforest

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Fri, 10 Dec 2010 14:08:00 -0800 Pageforest at Seattle JS Meetup http://blog.pageforest.com/35970196 http://blog.pageforest.com/35970196

I presented Pageforest to the Seattle JavaScript user's group last night.

It was great to have the forcing function of a presentation to wrap up some loose ends, and get the site into shape to work well.

The other speaker, Jordan Dobson showed of his amazingly pristine designs for iPhone apps using web-kit CSS craziness. He's able to to some "no-images" layouts that look beautiful, perform great, and load really quickly on the phone.

I also met Thomas Yip, of BeeDesk at a Startup Weekend event this week. He's looking at using Pageforest for his new product - could be the "killer app" to show off our platform?

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Mon, 18 Oct 2010 10:13:00 -0700 The iPad platform http://blog.pageforest.com/the-ipad-platform http://blog.pageforest.com/the-ipad-platform

I just got back from a 3 week trip where I left my laptop behind, and only took my iPad (and my mobile phone). Spending more time with the iPad has convinced me that it represents a sea-change just as important as Windows was in the early PC evolution. What makes the iPad such an interesting new platform?

Form factor - This is the most obvious difference between the PC platform (Mac and Windows) and the iPad. The fact that you can hold the screen in your lap and interact with a touch-screen interface. Microsoft has proved that this is NOT sufficient, as we've seen a lot of failures in the tablet form factor for Windows-based machines. But Apple's implementation of a touch-screen computer is particularly simple and elegant. It is a pleasure to use unlike any other PC experience I've had.
Application Security - I've been hearing more and more that iPad users are installing and using MANY more applications than Windows or Mac users. When I reflect on my own behavior, I realize how hesitant I've been to install Windows applications for fear of corrupting my system (crashes, slow-downs, security holes). The way iOS manages applications it effectively creates a sandbox so your apps can't really damage your system as a whole. And the single-application running at a time, ensures that your system resources cannot be consumed by background applications running without your knowledge.
App Store - The most important aspect of the iPad platform is the iTunes App Store. The iPad would not have been successful without the previous success of the iPhone. Not only did it solve the chicken and egg problem of having a large number of apps available on launch-day, but it also had several years of building up a developer community that was familiar with building apps for the iPhone/iPad platform. Because Apple manages the distribution and payment of applications, App developers have a frictionless means of selling their applications to users. It boggles the mind that Microsoft did not long ago develop a similar system for Windows. Perhaps they didn't think it was necessary, since anyone could distribute Windows apps on their own. But they did not count on the perceived complexity and trust issues that would prevent most users from being promiscuous purchasers of Windows applications.

But there is a dark-side to the Apple-owned iPad/iPhone platform; the fact that it is controlled by a single company which exercises strict control on entrance into the App market and protects it's own competitive interests by rejecting applications that they themselves want to monetize (they have already had the US Justice department get involved in anti-trust complaints, such as when Google tried to deliver it's Google Voice service on the iPhone).

I'm looking forward to an Android-based tablet, and the introduction of another App marketplace being run by Amazon. That will give developers a great choice with more than one option for application distribution.

I also think there's an opportunity for the HTML5 platform to evolve into a competing API/platform for application distribution. Web apps will evolve to have all the capabilities of the iOS platform (offline support, local storage, rich graphics (SVG + Canvas), camera and microphone support, etc.). But we still lack an App Store equivalent for web apps. Google announced one around Chrome at Google IO in 2010 - perhaps we will see them ship it with the introduction of Chrome-based tablets.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Wed, 07 Jul 2010 08:35:00 -0700 DRY (Don't Repeat Yourself) Design Patterns http://blog.pageforest.com/dry-dont-repeat-yourself-design-patterns http://blog.pageforest.com/dry-dont-repeat-yourself-design-patterns

One of the Pageforest applications I've been working on has prompted me to write a template generation language (modeled after Django's Template Language - DTL). DTL has a number of features that allow you to re-purpose templates and combine them in interesting ways. Yet, as a template language, it does not use the familiar concepts we, as programmers, are used to in our primary programming languages.

This got me thinking about the generalized patterns we use to help us design complex systems, by decomposing them into simpler parts. I'll compare DTL with JavaScript to show how the facilities in one comapare to those in the other.

Includes - DTL has a {% include "file.html" %} tag that can be used for basic composition of one template inside another. This is much like functional decomposition in a programming language; you can call a function from within another function:
   function foo() {
   bar();
   }
But note that DTL does not support parameter substitution - or function arguments. This makes {% include %} a much less powerful and useful feature (and, in fact, is not much used in most Django applications).
Inheritance - DTL allows one template to extend another via the {% extends "base.html" %} tag. This turns out to be a very practical way to concisely design templates for different pages in an application, and yet enforce some regularity in the top-level design of the web site. Developers would typically create a master template that describes the top level layout and navigation for their site, and then individual site pages can extend that template, and replace sections with page-specific content.
This is accomplished with {% block name %} tags. A child template can re-define any named block that appears in the template it extends. This is similar to inheritance-based composition in class-based languages. You can think of each block as a method in a base class, which can be selectively over-ridden by derived classes.

Yet, DTL is missing some important techniques for composition.

Parameterized templates - As mentioned above, there is not method of invoking a template, but replacing formal parameters from a calling template. Variables in templates are basically dynamically scoped, so a calling template can define a variable that it knows will be used in an included template.
Template as datatype - Just as we frequently create functions dynamically in JavaScript, and pass them around as data, it would be similarly useful to allow templates to be stored as data, passed in variables, and then invoked dynamically. There is not "eval" or "apply" function that would enable this type if template programming.

DTL is a very nice, and constrained, template definition language. And I like that fact that it does not expose the full complexity of a programming language to its users (which was it's creator's design philosophy). Yet it would be very useful to build language extensions via functional-like programming practices directly in the template language, without having to resort to escape functions that require writing custom tags and filters in Python.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Fri, 04 Jun 2010 02:13:00 -0700 A Mandelbrot Set Viewer - Pushing the Envelope with Blob Storage http://blog.pageforest.com/a-mandelbrot-set-viewer-pushing-the-envelope http://blog.pageforest.com/a-mandelbrot-set-viewer-pushing-the-envelope

As we've been developing the Pageforest service, we've been developing sample applications in parallel. We think the best way to motivate features to support in our service, is to actually build applications that utilize those services. In fact, we explicitly stated that we won't implement any feature that we don't have an imminent need from a JavaScript application developer (including ourselves!).

If you've seen our simple Scratch application sample, you'll see that a simple Pageforest application can be written in just a few lines of code. And your application can save and load data from a cloud data-store on behalf of your users.

What you may not have realized, is that documents can contain much more than a single JSON blob storage. In fact, each "document" in our system, can also have associated with it, any number of child "Blobs". The permissions for reading and writing these blobs are controlled by their parent document. Blobs can contain any Internet data-type, including images, sounds, pdf's, html documents, javascript files, or JSON persistence.

With this feature in mind, we decided to push the envelope with a Mandelbrot set viewer application with the following goals:

Use the Canvas element to draw the Mandelbrot set using JavaScript only (no flash, no plug-ins).
Cache images rendered in the client, into Blobs in the data store - so that once any user has viewed a region of the set, it would be available to any other user without having to recompute it.
Use HTML 5's Web Workers to compute image tiles in the background - keeping the UI responsive even when the CPU is busy with intensive image processing.
Use Google's Map (v3) API to provide a famiilar navigation interface to the Mandelbrot set, making it as easy to pan and zoom over the Mandelbrot set as it is to view maps and satellite imagery of the Earth.
Use the spare CPU cycles of concurrent connected browsers, to create a peer-to-peer compute cloud to further speed up calculation of desired image tiles.

We started the project over the Memorial Day weekend, and today we have a working prototype that meets all but the last goal.

Mandelbrot-blog

If you would like to play with the Mandelbrot Set Viewer, be aware that you must be signed in to Pageforest in order to generate tiles (you should be able to view existing image tiles without signing in).

The way the Mandelbrot Viewer works, is that whenever the map UI generates a request for an image tile (all of the tiles at all of the magnifications have been assigned names according to their position - even if the tile hasn't been generated yet), we simultaneously query the Blob store to see if the tile exists. If it doesn't, we queue up a tile creation task and send it to a Worker. Because workers don't have direct access to Canvas elements for drawing, we compute the data for the bitmap in the Worker, and send that back to the parent window when the Worker is done. It can then be quickly saved into a Canvas element, converted to a PNG file, and then uploaded to the server.

We had some difficulty getting compatability across browsers to support raw binary upload's via AJAX, so we instead just send a base64 (text) encoded version of the file, and decode the data on the server before storing it.

Once the tile is generated, we update the url in the map image, so the browser attempts to download the tile again.

As is all of Pageforest's code, this example has been made open source. You're welcome to make a copy to make your own variations.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Wed, 26 May 2010 12:51:00 -0700 Pageforest Version 0.6.0 http://blog.pageforest.com/pageforest-version-060 http://blog.pageforest.com/pageforest-version-060

Today we have released version 0.6 of our client library for Pageforest. We had to make some major changes to the low-level REST api to fix a security flaw we discovered in our first version. As a result any of the code built using the 0.5 versions of our library no longer work (sorry!).

On the plus side, I think our "Scratch" sample application is pretty simple and very functional for those that would like to experiment with Pageforest. See our QuickStart page to get the steps to create your own application starting from our sample.

We're also aware that we're pretty light on documentation now. If you do take a stab at building a Pageforest app, please let us know what your experiences are (support@pageforest.com), and we'll be happy to help you with any questions you have.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/364296/mike-profile.png http://posterous.com/users/3sO6rrFt5VXH Mike Koss mckoss Mike Koss Tue, 27 Apr 2010 10:31:00 -0700 Key-Value storage API demo http://blog.pageforest.com/key-value-storage-api-demo-0 http://blog.pageforest.com/key-value-storage-api-demo-0

Here's a snapshot of our first official demo application. It's basically one JavaScript file with few dependencies (jQuery and HMAC-SHA1) that uses the Pageforest API to authenticate an existing user, then store and retrieve data in the Key-Value store. The source code of this version is available here: http://code.google.com/p/pageforest/source/browse/?r=eeeeee#hg/examples/keyvalue (JavaScript application and Django project for the server side on Google App Engine).

Next, we are going to change the auth mechanism and require that username and password are entered only on a trusted page like https://auth.pageforest.com/ in a separate browser tab. Then the JavaScript app doesn't have to deal with the details of cryptographic authentication, and malicious applications don't have access to user credentials.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/391320/square.png http://posterous.com/users/3sycdpwQbidb Johann C. Rocholl nxdom Johann C. Rocholl Fri, 09 Apr 2010 16:42:00 -0700 Memcache mixin for datastore models http://blog.pageforest.com/memcache-mixin-for-datastore-models http://blog.pageforest.com/memcache-mixin-for-datastore-models

So we made a datastore model mixin that will transparently use memcache. It is available on Google Code under the MIT license, just like the rest of pageforest.com: http://code.google.com/p/pageforest/source/browse/appengine/utils/cacheable.py

To use it, just inherit your model from it:

from google.appengine.ext import db from utils.mixins import Cacheable class App(Cacheable): name = db.StringProperty()

The interesting part is that it automatically reduces datastore write contention by skipping datastore put if the write rate is consistently high. The App Engine datastore only supports 5 updates per second. So if one entity gets 10 updates per second, the Cacheable mixin makes sure that it's always saved to memcache but the datastore is updated only once every 2 seconds.

Permalink | Leave a comment »

]]> http://files.posterous.com/user_profile_pics/391320/square.png http://posterous.com/users/3sycdpwQbidb Johann C. Rocholl nxdom Johann C. Rocholl Fri, 06 Nov 2009 23:00:00 -0800 Google's new Closure JavaScript optimizer http://blog.pageforest.com/googles-new-closure-javascript-optimizer http://blog.pageforest.com/googles-new-closure-javascript-optimizer

I'm pretty excited to see the release, today, of Google's Closure JavaScript compiler.

Closure goes way beyond a simple JavaScript minifier. It can do things like unwind function calls, and replace them with the body of the function (inlining). It also changes local variable names to single characters.

You can either download the compiler locally, or use their web service (though the UI or via a REST API). Here a sample of how aggressively Closure can reduce your code size:

function Foo(string)
{
 alert(string);
}

Foo("hello");

In Simple optimization mode this yields:

function Foo(a) {
 alert(a)
}
Foo("hello");

In Advanced mode this compresses to:

alert("hello");

I'm still learning how to use Closure optimally for some of my code. For example, in Advanced mode, my JavaScript Namespace code is pretty severely compressed. First, Simple optimization yields:

Media_http2bpblogspot_nwhod

While Advanced Optimization saves a few hundred more bytes, but mangles some variable names that should be left alone as external method names:

Media_http2bpblogspot_jiuev

There is a tutorial on how to annotate your code to make sure that Advanced optimization does not break your code by applying variable renaming too aggressively.

To fix my Namespace code I add these lines:

// Export names
var p = Namespace.prototype;
p['Extend'] = p.Extend;
p['Define'] = p.Define;
p['Import'] = p.Import;
p['SGlobalName'] = p.SGlobalName;

which then add the following lines to my function in optimized form to restore the "exports" from my class library:

d = f.prototype;
d.Extend = d.d;
d.Define = d.c;
d.Import = d.f;
d.SGlobalName = d.g

With all these fixes, I'm able to get a clean compile of the Namespace library that compresses down to:

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad Wed, 05 Aug 2009 00:34:00 -0700 How Does Google Count Absolute Unique Visitors? http://blog.pageforest.com/how-does-google-count-absolute-unique-visitor http://blog.pageforest.com/how-does-google-count-absolute-unique-visitor

As a test of the Answer service, Mahalo, I posed the following question:

How does Google Analytics calculate Absolute Unique Visitors?

I know that Google claims that they can report on the number of Absolute Unique Visitors over any time period. What I can't figure out is how they can be calculating this without doing very expensive database queries. I feel they must be making an approximation of some sort.

Otherwise, they would have to query the unique set of users who visited the site across a large time span, and remove duplicates in real time. They could not afford to do this for a site with millions of unique users.

I will reward the tip to the person who best answers this question by providing a feasible solution to the technical problem or explaining how the reported value is approximated. Even better, if it is backed by an authoritative explanation from Google developers.

Note that the crux of the problem is to avoid double-counting Returning Visitors that are duplicately counted across the time span of a report.

Unfortunately, even the best answerer did not understand the question. Perhaps there were not enough users on the site, nor did they have people with the needed expertise to figure out what I was asking.

I rescinded my $5 "tip", and actually got some Mahalo users mad at me for doing so. After given the problem some more thought, this is what I cam up with:

Here's how I would calculate Absolute Unique Visitors:

Data Collection

On the first visit of a user, for each day, I record how many days since their last visit (for the "returning" visitors - as opposed to the "new" visitors).

Data Aggregation

When Analytics is processing the raw data, they can collect buckets of counters for the total number of visitors that:

New (never visited before)
Visited 1 day ago or more (aka all "returning visitors")
Visited 2 days ago or more
Visited 3 days ago or more
etc. (they may choose to cut off the number of buckets at some reasonable maximum - which would set a max on the reported ranges they could accurately display).

Note that these are cumulative numbers - each bucket has strictly fewer users than the previous one.

Reporting

When the site owner asks for the Absolute Unique Users across a date range, the reporting engine can scan all the dates in the period and accumulate a sum as follows (pseudo-code):

Assumes Data[DAY] containing values:
  NEW - Number of new users who arrived that day
  RETURNING[N] - Number of users who arrived that day with a haitus of N 

days or more.

UNIQUE = 0
for DAY from 1 to N:
  UNIQUE += Data[DAY].NEW
  UNIQUE += Data[DAY].RETURNING[DAY]

UNIQUE is thus, the sum of all NEW users reported on each day (who are always unique), and then only those returning users who were not counted in a prior day (since they were last on the site before the beginning of the reporting period).

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad Fri, 10 Jul 2009 23:53:00 -0700 JSComposer - A JavaScript Composition Utility for Google AppEngine (Python) http://blog.pageforest.com/jscomposer-a-javascript-composition-utility-f http://blog.pageforest.com/jscomposer-a-javascript-composition-utility-f

After developing a JavaScript namespace facility, I needed a simple way of merging and/or minifying my javascript source files from my AppEngine (Django) application. So, I developed a simple python module that can be used to:

Merge multiple JavaScript files into one (for faster download).
Minifies your javascript on the server (and stores it in memcache for fast retrieval)
Allows you to include javascript files individually on your test server for easy debugging.

Both the namespace library and jscomposer have been placed in the public domain, so feel free to use as you see fit:

namespace.js
jscomposer.py

I would love to hear from you if you use either of these libraries.

-- Mike

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad Fri, 19 Jun 2009 18:04:00 -0700 JavaScript Namespaces http://blog.pageforest.com/javascript-namespaces http://blog.pageforest.com/javascript-namespaces

One thing that JavaScript programmers have to deal with is corruption of the global namespace. Every time you define a simple function, or other variable at the top level of a web page, the names you've chosen could potentially come in conflict with names used by other developers or libraries that you are using. In the browser, all global variables become properties of the window object.

I've been dealing with this in an ad-hoc manner until recently. I would create a single global variable for all my code, and then define all my functions and variables within it, like this:

var PF = {
  global: value,
  ...,

MyFunc: function(args)
    {
    ...
    },

...
};

I tend to want to migrate code from one project to another quite frequently, so putting all the code in one namespace was becoming quite tedious as I was editing the code to move it into different namespaces for different projects. Inspired by Python, I've developed a more general method of defining and importing namespaces across different modules/namespaces of javascript code.

Here is a typical way to define a new namespace, and import another namespace into it so you can reference code from other libraries succinctly.

global_namespace.Define('startpad.base', function(ns) {
    var Other = ns.Import('startpad.other');

    ns.Extend(ns, {
        var1: value1,
        var2: value2,
        MyFunc: function(args)
            {
            ....Other.AFunction(args)...
            }
    });
       
    ns.ClassName = function(args)
    {
    };
       
    ns.ClassName.prototype = {
        constructor: ns.ClassName,
        var1: value1,
           
    Method1: function(args)
        {
        }
    };
});

The benefits of this approach are:

Isolation of code without polluting the global (window) namespace with multiple names. A single global name ('global_namespace') is added to the window object.
Easy to import code from another namespace, and assign it a short local name (e.g., 'Other', above).
Allow javascript code to be loaded into the browser without regard for execution order. Forward references (to a namespace that hasn't been loaded yet), work fine as the Import function will pre-create a namespace object when it is referenced, and then fill it in when the namespace is defined.
Long names can be assigned that are unique using a heirarchy similar to DNS names. E.g., since I own startpad.org, I claim the "startpad" name as a top level global namespace, and can use names like "startpad.base", or "startpad.timer", for libraries that I am building
Namespaces can be versioned simply by naming convention. For example, I could load in the same browser, namespaces for "startpad.timer.v1" and "startpad.timer.v2".

There still remains a problem of javascript composition. I don't like to include lots of different script files in the same web page. So you still have to combine the source code from multiple different independent script files into one file. This can be done as part of a build process (along with javascript minification), or through a composition service running on your web server (I hope to write one of these in Python for my AppEngine projects).

I am placing namespace.js into the public domain. Let me know if you end up using it, or have suggestions for improvements.

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad Wed, 15 Oct 2008 17:18:00 -0700 Beware Mutable default value initializers in Python http://blog.pageforest.com/beware-mutable-default-value-initializers-in http://blog.pageforest.com/beware-mutable-default-value-initializers-in

As a new Python programmer I was surprised by the behavior of default parameter expressions. I knew they were only executed once when the function is first defined, but I hadn't realized the ramifications for using a static dictionary object as a default expression. This little gotcha hit me yesterday and took quite a while for me to figure out what was happening. Here's some sample code:

def Bad(dict={}):
    print "Bad: %s" % repr(dict)
    dict['p'] = 1

Bad()
Bad()

Which results in:

Bad: {}
Bad: {'p': 1}

Since the value of dict is mutable, it can be changed as a side effect of the function. So all subsequent calls will use a default value that has been modified by previous calls to the function! This can be (and was) a nightmare to track down in a large program.

I fixed this by changing to the following:

def Good(dict=None):
    if dict == None:
        dict = {}
    print "Good: %s" % repr(dict)
    dict['p'] = 1

Good()
Good()

Which results in:

Good: {}
Good: {}

The fix de-couples the effects of one call on subsequent calls (as was the intention of the original code).

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad Wed, 03 Sep 2008 23:50:00 -0700 Securing REST API's against "Drive By" Requests http://blog.pageforest.com/securing-rest-apis-against-drive-by-requests http://blog.pageforest.com/securing-rest-apis-against-drive-by-requests

REST protocols are increasingly popular to enable API calls to be sent to web services. These requests can be generated from one server to another, or they can be embeded in client-side web pages. But client-side only requests are inherently unsafe; anyone that can navigate to that page, can have REST calls made against a 3rd party service without their knowledge.

I couldn't find a brief summary of common practices for securing REST API's, so I did some investigation. I reviewed the del.ico.us api to see how they deal with these problems. They have a REST api that, among other things can add and remove bookmarks for a given user. So there needs to be some security so that a 3rd party does not execute this API on behalf of a given user without their permission.

For example, executing this "API" call:

https://api.del.icio.us/v1/posts/add?url=http://mckoss.com&description=security+test

will cause a bookmark to be created in your del.icio.us account. This could even be hidden in a web page as an emdeded <script> tag, so that you would not even be aware that it is happening. Note that POST's are easy to create in the client as well as GET's, so there is not really any added security to using POST over GET.

In order to prevent this, del.icio.us implements two countermeasures:

All calls to their API that modify data require Basic Authentication.
If there is a Referer value in the request header, they return a 403 Forbidden error.

Note that, even if I am logged in to del.icio.us, I will have a authentication cookie on my machine. But this is not sufficient to authenticate to the API - since it uses a different form or authentication. It is possible, however, that a user has previously logged into the del.icio.us API and cached their credentials in the browser. So this protection is a speed-bump, but not fail safe protection.

The second requirement protects users from "drive-by" API calls - the fact that I am visiting a web page should not allow that page to issue requests on my behalf. All mainstream browsers will send a Referer header whenever they make a request from another web page.

One work-around is to download an html page to your local machine, and execute it from your file system. In this case, there is no Referer and the API request is executed. So, a malicious site would have to trick a user into downloading a page and opening the local copy - which is much harder to do. There is a more robust alternative to these measures, that more modern REST servers implement. That is, to require that some user-specific secret information be transmitted along with every request. That way, a random malicious web page will have no way of generating an API call for an unknown potential victim.

The FriendFeed API accomplishes this simply by generating a random RemoteKey for each user. Any service that wishes to make API calls on behalf of the user must include the current value of that user's RemoteKey. In case of a security breach, the user can reset the key (and give the new key to all the services he wishes to continue to use).

It's also not uncommon for web application frameworks to generate hidden session keys within a form that is unique to the session. That way they eliminate the ability for pages to cross-post forms from malicious web sites.

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad Mon, 07 Jul 2008 20:33:00 -0700 Back from PageForest Haitus http://blog.pageforest.com/back-from-pageforest-haitus http://blog.pageforest.com/back-from-pageforest-haitus

That was a long time between posts. But StartPad.org is now up and running, and in the capable hands of Zach who's dealing with the day-to-day organization, so I'm getting to spend more time on PageForest.

I've also switched my application platform to Google AppEngine - I'm having a grand time learning Python and doing a bunch more JavaScript development.

I've spent quite a bit of time building on my JavaScript unit testing framework. I'm open-sourcing that code, and I've made some significant improvements during PageForest development, so I'll update the core project soon (hosted on Google Code).

I've been working so much in Firefox, I hadn't noticed that IE testing was getting stale. In general, the different browsers are very compatible when it comes to JavaScript - most of the platform differences are in HTML, CSS, and eventing models. The core language is very standardized. But I have hit one major gotcha when it comes to IE JavaScript (aka JScript): handling of empty elements at the end of an array definition.

As a (now) Python programmer, it's very common to define an (open ended) array as:

a = [1,2,3, ];

Note the dangling comma with no trailing element. In Firefox (and Python), this creates a 3 element array - the final comma is ignored. But, in IE, this creates a 4 element array with undefined as the 4th element.

I had used this paradigm extensively, w/o really realizing it. So when I switched to testing on IE, it took me quite a while to track down all the subtle bugs based on extra undefined elements added to my arrays.

JavaScript SHOULD be well enough defined that one of Firefox or IE is INCORRECT in their implementation of arrays (and I hope it's IE - since it's nice to be able to leave placeholders for future array elements in array definitions). Others have experience this problem too; the mozilla JavaScript documentation explicitly allows for it:

If you include a trailing comma at the end of the list of elements, the comma is ignored.

According to my reading of the ECMA-262 spec IE is non-conforming. Section 11.1.4 states:

Whenever a comma in the element list is not preceded by an AssignmentExpression (i.e., a comma at the beginning or after another comma), the missing array element contributes to the length of the Array and increases the index of subsequent elements. Elided array elements are not defined.

so, this definition:

a = [1,,2,];

should be equivalent to:

a = [1,undefined,2]; // Firefox - correct

but NOT:

a = [1,undefined,2,undefined]; // IE - incorrect

Even worse, a trailing comma at the end of an Object initializer is fatal in IE. This statement:

o = {a:1, b:2,};

produces a Runtime Error ("Error: Expected identifier, string or number") and your code will not execute! Here, it may be that IE is actually behaving correctly (according to spec), but it's still rather annoying.

You can see how your browser performs by visiting dangle.htm.

Permalink | Leave a comment »

]]> http://posterous.com/images/profile/missing-user-75.png http://posterous.com/users/5fiRcDMM7lKx StartPad startpad StartPad