After the latest round of scala is teh complex, I started thinking a bit more about the similar roles implicit conversions in scala play to extension methods in C#. I can't objectively comment on whether Extension Methods are simpler (they are still a discoverability issue), but I can comment that the idea of implicit conversion to attach new functionality to existing code really intrigued me. It does seem less obvious but also a lot more powerful.
Then i thought, wait, C# has implicit.. Did they just invent extension methods without needing to? I've never used implicit conversion in C# for anything but automatic casting, but maybe I was just not imaginative enough. So here goes nothing:
public class RichInt {
public readonly int Value;
public static implicit operator RichInt(int i) {
return new RichInt(i);
}
public static implicit operator int(RichInt i) {
return i.Value;
}
private RichInt(int i) {
Value = i;
}
public IEnumerable<int> To(int upper) {
for(int i = Value;i<=upper;i++) {
yield return i;
}
}
}
Given the above I was hoping I could mimic scala's RichInt to() method and write code like this:
foreach(var i in 10.To(20)) {
Console.WriteLine(i);
}
Alas that won't compile. Implicit conversion only works on assignment, so i had to write
foreach(var i in ((RichInt)10).To(20)) {
Console.WriteLine(i);
}
So I do have to use an extension method to create To() in C#.
And, yes, i'm grossly simplifying what scala's implicts can accomplish. Also, I wish I could have block scope using statements to import extension methods for just a block the way scala's import allows you to handle local scope implicits.
Tried to upgrade node from 0.4.5 to 0.6.x and my my micro kept falling over dead. I know it's an edge case, but it's an annoying set of symptoms that I figured I should post in case someone else runs into the same issue.
tl;dr => It's not a node problem its an AWS kernel issue with old AWS AMIs and Micro instances
So I have a micro that's about a year old, i.e. beta AWS-AMI, but i gather the same problem happens with pretty much every AMI prior to 2011.09. I was running node 0.4.5, but had started using 0.6.4 on my dev and some modules were now dependent on it. Since micro instances go into throttle mode when building anything substantial, i hope to use the build from my dev server. The dev machine is centos, so i crossed my fingers, copied the build over and ran make install. No problem. Then i tried npm install -g supervisor and it locked up. Load shot up, the process wouldn't let itself be killed and i got a syslogd barf all over my console:
Message from syslogd@ at Wed Dec 28 00:58:19 2011 ... ip-**-**-**-** klogd: [ 440.293407] ------------[ cut here ]------------ ip-**-**-**-** klogd: [ 440.293418] invalid opcode: 0000 [#1] SMP ip-**-**-**-** klogd: [ 440.293424] last sysfs file: /sys/kernel/uevent_seqnum ip-**-**-**-** klogd: [ 440.293501] Process node (pid: 1352, ti=e599c000 task=e60371a0 task.ti=e599c000) ip-**-**-**-** klogd: [ 440.293508] Stack: ip-**-**-**-** klogd: [ 440.293545] Call Trace: ip-**-**-**-** klogd: [ 440.293589] Code: ff ff 8b 45 f0 89 .... ip-**-**-**-** klogd: [ 440.293644] EIP: [] exit_mmap+0xd5/0xe1 SS:ESP 0069:e599cf08
So i killed the instance. Figuring it was config diffs between centos and the AMI, i cloned my live server and fired it up as a small to get decent build perf. Tested 0.6.4, all worked, brought it back up as a micro and, blamo, same death spiral. Back to small instance, tried 0.6.6 and and once again as a small instance it worked, but back as a micro it still had the same problem.
Next up was a brand new AMI, build node 0.6.6 and run as micro. Everything was happy. So it must be something that's gotten fixed along the way. Back to the clone and yum upgrade. Build node, try to run, death spiral. Argh! So finally i thought i'd file a ticket with node.js, but first looked through existing issues and found this:
Node v0.6.3 crashes EC2 instance
which pointed me at the relevant Amazon release notes which had this bit in it:
After using
yumto upgrade to Amazon Linux AMI 2011.09, t1.micro 32-bit instances fail to reboot.
There is a bug in PV-Grub that affects the handling of memory pages from Xen on 32bit t1.micro instances. A new release of PV-Grub has been released to fix this problem. Some manual steps need to be performed to have your instance launch with the new PV-Grub.
As of 2011-11-01, the latest version of the PV-Grub Amazon Kernel Images (AKIs) is 1.02. Find the PV-Grub AKI's for your given region by running:
ec2-describe-images -o amazon --filter "manifest-location=*pv-grub-hd0_1.02-i386*" --region REGION.Currently running instances need to be stopped before replacing the AKI. The following commands point an instance to the new AKI:
ec2-stop-instance --region us-east-1 i-#####
ec2-modify-instance-attribute --kernel aki-805ea7e9 --region us-east-1 i-#####
ec2-start-instance --region us-east-1 i-#####.If launching a custom AMI, add a
--kernelparameter to theec2-run-instancescommand or choose the AKI in the kernel drop-down of the console launch widget.
Following these instructions finally did the trick and 0.6.6 is happily running on my old micro instance. Hope this helps someone else get this resolved more smoothly.
I consider the start of my programming carreer to be when I learned Genera LISP on Symbolics LISP machines. Sure I had coded in Basic, Pascal and C, and unfortunately Fortran, before this, but it had always just been a hobby. With LISP, I got serious about languages, algorithms, etc.
Genera LISP had its own object system called Flavors, much of which eventually made it into CLOS, the Common Lisp Object System. Flavors had capabilities called Wrappers and Whoppers, which provided aspect oriented capabilities before that term was even coined. Both achieved fundamentally the same goals, to wrap a function call with pre and post conditions, including preventing the underlying function call from occuring. Wrappers achieved this via LISP macros, i.e. the calls they wrapped were compiled into new calls, each call using the same wrapper sharing zero code. Whoppers did the same thing except dynamically, allowing the sharing of whopper code, but also requiring at least two additional function calls at runtime for every whopper.
So what's all this got to do with javascript? Well, yesterday I got tired of repeating myself in some CPS node coding and just turn my continuation into a new continuation wrapped with my common post condition, and so I wrote the Whopper capability for javascript. But first a detour through CPS land and how it can force you to violate DRY.
So in a normal synchronous workflow you might have some code like this:
function getManifest(refresh) {
if(!refresh && _manifest) {
return _manifest;
}
var manifest = fetchManifest();
if(!_manifest) {
var pages = getPages();
_manifest = buildManifest(pages);
} else {
_manifest = manifest;
if(refresh) {
var pages = getPages();
updateManifest(pages);
}
}
saveManifest(manifest);
return _manifest;
};
But with CPS style asynchrony you end up with this instead:
function getManifest(refresh, continuation, err) {
if(!refresh && _manifest) {
continuation(_manifest);
return;
}
fetchManifest(function(manifest) {
if(!_manifest) {
getPages(function(pages) {
_manifest = buildManifest(pages);
saveManifest(_manifest,function() {
continuation(_manifest);
});
}, err);
return;
}
_manifest = manifest;
if(refresh) {
getPages(function(pages) {
updateManifest(pages);
saveManifest(_manifest,function() {
continuation(_manifest);
});
}, err);
} else {
saveManifest(_manifest,function() {
continuation(_manifest);
});
}
}, err);
};
Because the linear flow is interrupted by asynchronous calls with callbacks, our branches no longer converge, so the common exit condition, saveManifest & return the manifest, is repeated 3 times.
While I can't stop the repetition entirely, I could at least reduce it by capturing the common code into a new function. But even better, how about I wrap the original continuation with the additional code so that I can just call the continuation and it runs the save as a precondition:
function getManifest(refresh, continuation, err) {
if(!refresh && _manifest) {
continuation(_manifest);
return;
}
continuation = continuation.wrap(function(c, manifest) { saveManifest(manifest, c); });
fetchManifest(function(manifest) {
if(!_manifest) {
getPages(function(pages) {
_manifest = buildManifest(pages);
continuation(_manifest);
}, err);
return;
} else {
_manifest = manifest;
if(refresh) {
getPages(function(pages) {
updateManifest(pages);
continuation(_manifest);
}, err);
} else {
continuation(_manifest);
}
}, err);
};
What makes this capture possible is this extension to the Function prototype:
Object.defineProperty(Function.prototype, "wrap", {
enumerable: false,
value: function(wrapper) {
var func = this;
return function() {
var that = this;
var args = arguments;
var argsArray = [].slice.apply(args);
var funcCurry = function() {
func.apply(that, args);
};
argsArray.unshift(funcCurry);
wrapper.apply(that, argsArray);
};
}
});
It rewrites the function as a new function that when called will call the passed wrapper function with a curried version of the original function and the arguments passed to the function call. This allows us to wrap any pre or post conditions, including pre-conditions that initiate asynchronous calls themselves, and even lets the wrapper function inspect the arguments that the original function will be passsed (assuming the wrapper decides to call it via the curried version.
continuation = continuation.wrap(function(c, manifest) {
saveManifest(manifest, c);
});
The above overwrite the original continuation with a wrapper version of itself. The wrapper is passed c, the curried version of the original function and the argument that continuation is called with, which we know will be the manifest. The wrapper in turn calls the async function saveManifest with the passed manifest and passes the curried continuation as its continuation. So when we call continuation(_manifest), first saveManifest is called which then calls the original continuation with the _manifest argument as well.
So there I was trying to get jade-mode running and it kept dropping into fundamental mode. The error it gave in *Messages* was:
File mode specification error: (void-function whitespace-mode)
Problem was i was running emacs-22.2.3, the latest in centos and thereby the AWS Linux AMI. And whitespace-mode, the major mode that jade-mode and stylus-mode rely on requires emacs-23. Fine, tell me what repo to add and let me get on with my life. What, no rpm's?
That's where i usually draw the line and say to myself that it's something that i probably don't need. Over the years, i've developed an aversion to building from source, if only because then i have software on my machine that other rpm's can't count on as pre-requisites. But this time, that wasn't gonn fly. I wanted jade-mode!
As my usual build recipies go, this is what i had to do on an existing AWS Linux AMI, so some of the yum deps are missing. Don't worry, when you run ./configure it'll bitch and it'll usually be some *-devel package. So here goes building from source:
yum -y install ncurses-devel cd /tmp wget http://ftp.gnu.org/pub/gnu/emacs/emacs-23.3a.tar.gz tar zxf emacs-23.3a.tar.gz cd emacs-23.3 ./configure --prefix=/usr/local --with-xpm=no make make install
The other day i was musing about improving readability by liberating verbs from the containing objects that own them. Similar things are accomplished with mix-ins or traits in other languages, but I wanted to go another step further and allow for very context specific mapping, rather than just mixing ambigiously named methods from other objects. Since the exports construct looked a lot like a map of expressions, I decided to see what replicating the behavior with existing C# would look like.
To review, this is what I want (in C#-like pseudo code + exports):
class PageWorkflow {
UserService _userService exports { FindById => FindUserById };
PageService _pageService exports {
FindById => FindPageById,
UpdatePage(p,c) => Update(this p,c)
};
AuthService _authService exports {
AuthorizeUserForPage(u,p,Permissions.Write) => UserCanUpdatePage(u,p)
};
UpdatePage(userid, pageid, content) {
var user = FindUserById(userid);
var page = FindPageById(pageid);
if(UserCanUpdatePage(user,page)) {
page.Update(content);
} else {
throw;
}
}
}
And this is C# implementation of the above:
public class PageWorkflow {
private readonly Func<int, User> FindUserById;
private readonly Func<int, Page> FindPageById;
private readonly Action<Page, string> Update;
private readonly Func<User, Page, bool> UserCanUpdatePage;
public PageWorkflow(IUserService userService, IPageService pageService, IAuthService authService) {
FindUserById = (id) => userService.FindById(id);
FindPageById = (id) => pageService.FindById(id);
Update = (page, content) => pageService.UpdatePage(page, content);
UserCanUpdatePage = (user, page) => authService.AuthorizeUserForPage(user, page, Permissions.Write);
}
public void UpdatePage(int userid, int pageid, string content) {
var user = FindUserById(userid);
var page = FindPageById(pageid);
if(UserCanUpdatePage(user, page)) {
Update(page, content);
} else {
throw new Exception();
}
}
}
As I mentioned, it's all possible, short of the context sensitive extension method on Page. Lacking extension methods, I was going to name the imported method UpdatePage, but since it would be a field, it conflicts with the UpdatePage workflow method despite functionally having different signatures.
All in all, the public workflow UpdatePage is pretty close to what I had wanted, but the explicit type declaration of the each exports makes it boilerplate that is likely not worth the trouble of writing, and, no, i won't even consider code generation.
This exercise along with every other language feature I've dreamt up for Promise does illustrate one thing quite clearly to me: My ideal language should provide programatic access to its parser so that the syntax of the language can be extended by the libraries. Internal DSLs are a nice start with most languages, but often they fall short of being able to reduce default boilerplate and just create new boilerplate. Sure, designing the language to be more succinct is desirable, but if anything, that only covers what the language creators could imagine. Being able to tweak a language into a DSL for the task at hand, a la MPS, seems a lot more flexible.
Is this added flexibility worth the loss of a common set of constructs that is shared by all programmers knowing language X? It certainly could be abused to become incomprehensible, but I would suggest that even knowing language X joining any Team working on a project of sufficient complexity adds its own wealth of implicit patterns and constructs and worse than a language whose compiler is extended, these constructs are communicated via comments and documentation that are not part of language X, i.e. the compiler and IDE lack the ability to aid someone learning these constructs.
Considering this, I really need to do a survey of languages that already offer this capability as well as take a closer look at MPS and Nemerle to see if the language I want is just a few parser rules a way from an existing meta programming language.
This is yet another in my series of posts musing about what my ideal language would look like. This one is about readability.
Most code I write these days seems to utilize three types of classes: Data, Services and Workflow.
These are generally POCO object hierarchies with fields/accessors but minimal logic for manipulating that data. They should not have any dependencies. If an operation on a data object has a dependency, it's really a service for that data. Data objects don't get mocked/stubbed/faked, since we can just create and populate them.
These are really containers for verbs. The verbs could have been methods on the calling object, but by pulling them into these containers we enable a number of desirable capabilities:
These end up being classes, primarily because in most OO languages everything's a class, but really workflow classes are organizational constructs used to collected related workflows as procedural execution environments. They can be set up with pre-requisites and promote code re-use via shared private members for common sub-tasks of the workflow. They are also responsible for condition and branching logic to do the actual work.
Actions (requests from users, triggered tasks, etc.) start at some entry point method on a workflow object, such as a REST endpoint, manipulate data via Data objects using services, the results of which trigger paths defined by the workflow.
Let's look how this works out for a fictional content management scenario. I'm using a C#-like pseudo syntax to avoid unecessary noise (ironic, since this post is all about readibility):
class PageWorkflow {
...
UpdatePage(userid, pageid, content) {
var user = _userService.FindById(userid);
var page = _pageService.FindById(pageid);
if(_authService.AuthorizeUserForPage(user,page,Permissions.Write)) {
_pageService.UpdatePage(page,content);
} else {
throw;
}
}
}
UpdatePage is part of PageWorkflow, i.e a workflow in our workflow class. It is configured with _userService, _pageService and _authService as our service classes. Finally user and page are instances of our data classes. Nice for maintainability and separation of concerns, but awkward from a readibility perspective. It would be much more readable with syntax like this:
class PageWorkflow {
...
UpdatePage(userid, pageid, content) {
var user = FindUserById(userid);
var page = FindPageById(pageid);
if(UserCanUpdatePage(user,page)) {
page.Update(content);
} else {
throw;
}
}
}
Much more like we think of the flow. Of course this could easily be done by creating those methods on PageWorkflow, but that's the beginning of the end of building a god object, and don't even get me started on putting Update on the Page data object.
So let's assume that this separation of purposes is desirable — i'm sure there'll be plenty of people who will disagree with that premise, but the premise isn't the topic here. What we really want to do here is alias or import the functionality into our execution context. Something like this:
class PageWorkflow {
UserService _userService exports { FindById => FindUserById };
PageService _pageService exports {
FindById => FindPageById,
UpdatePage(p,c) => Update(this p,c)
};
AuthService _authService exports {
AuthorizeUserForPage(u,p,Permissions.Write) => UserCanUpdatePage(u,p)
};
...
}
Do not confuse this with VB's or javascript's with keywords. Both import the entirety of the referenced object into the current scope. The much maligned javascript version does this by importing it into the global namespace, which, given the dynamic nature of those objects, makes the variable use completely ambiguous. While VB kept scope ambiguity in check by forcing a . (dot) preceeding the imported object's members, it is a shorthand that is only questionably more readable.
The above construct is closer to the @EXPORT syntax of the perl Exporter module. Except instead of exporting functions, exports exports methods on an instance as methods on the current context. It also extends the export concept in three ways:
Instead of just blindly importing a method from a service class, the exports syntax allows for aliasing. This is useful because imported method likely defered some of its functionality context to the owning class and could collide with other imported methods, e.g. FindById on PageService and UserService.
As the methodname is rewritten, the argument order may no longer be appropriate, or we may want to change the argument modifiers, such as turn a method into an extension method.
UpdatePage(p,c) => Update(this p,c)
The above syntax captures arguments into p and c and then aliases the method into the current class' context and turns it into a method attached to p, i.e. the page, so that we can call page.Update(content)
But why stop at just changing the argument order and modifiers. We're basically defining expressions that translate the calls from one to the other, so why shouldn't we be able to make every argument an expression itself?
AuthorizeUserForPage(u,p,Permissions.Write) => UserCanUpdatePage(u,p)
This syntax curries the Permissions.Write argument so that we can define our aliases entrypoint without the last argument and instead name it to convey the write permissions implcitly.
Great, some new syntactic sugar. Why bother? Well, most language constructs are some level of syntactic sugar over the raw capabilities of the machine to let us express our intend more clearly. Generally syntactic sugar ought to meet two tests: make code easier to read and more compact to write.
The whole of the import mechanism could easily be accomplished (except maybe for the extension method rewrite) by creating those methods on PageWorkflow and calling the appropriate service members from there. The downside to this approach is that the methods are not differentiated from other methods in the body of PageWorkflow therefore not easily recognizable as aliasing constructs. In addition the setup as wrapper methods is syntactically a lot heavier.
The exports mechanism allows for code to be crafted more closely to how we would talk about accomplishing the task without compromising on the design of the individual pieces or tying their naming and syntax to one particular workflow. It is localized to the definition of the service classes and provides a more concise syntax. In this way it aids the readibility as well as theauthoring of a common task.
I just had a lovely object lesson in lazy evaluation of Iterators. I wanted to have method that would return an enumerator over an encapsulated set after doing some sanity checking:
public IEnumerable<Subscription> Filter(Func<Subscription, bool> filter) {
if(filter == null) {
throw new ArgumentNullException("filter","cannot execute with a null filter");
}
foreach(var subInfo in _subscriptions.ToArray()) {
Subscription sub;
try {
var subDoc = XDocFactory.LoadFrom(subInfo.Path, MimeType.TEXT_XML);
sub = new Subscription(subDoc );
if(filter(sub) {
continue;
}
} catch(Exception e) {
_log.Warn(string.Format("unable to retrieve subscription for path '{0}'", subInfo.Path), e);
continue;
}
yield return sub;
}
}
I was testing registering a subscription in the repository with this code:
IEnumerable<Subscription> query;
try {
query = _repository.Filter(handler);
} catch(ArgumentException e) {
return;
}
foreach(var sub in query) {
...
}
And the test would throw a ArgumentNullException because handler was null. What? But, but i clearly had a try/catch around it! Well, here's where clever bit me. By using yield, the method had turned into an enumerator instead of a method call that returned an enumerable. That means that the method body would get squirreled away into an enumerator closure that would not get executed until the first MoveNext(). And that in turn meant that my sanity check on handler didn't happen at Filter() but at the first iteration of the foreach.
Instead of doing "return an Iterator for subscriptions", I needed to do "check the arguments" and then "return an Iterators for subscriptions" as a separate action. This can be accomplished by factoring the yield into a method called by Filter() instead of being in Filter() itself:
public IEnumerable<Subscription> Filter(Func<Subscription, bool> filter) {
if(filter == null) {
throw new ArgumentException("cannot execute with a null filter");
}
return BuildSubscriptionEnumerator(Func<Subscription, bool> filter);
}
public IEnumerable<Subscription> BuildSubscriptionEnumerator(Func<Subscription, bool> filter) {
foreach(var subInfo in _subscriptions.ToArray()) {
Subscription sub;
try {
var subDoc = XDocFactory.LoadFrom(subInfo.Path, MimeType.TEXT_XML);
sub = new Subscription(subDoc );
if(filter(sub) {
continue;
}
} catch(Exception e) {
_log.Warn(string.Format("unable to retrieve subscription for path '{0}'", subInfo.Path), e);
continue;
}
yield return sub;
}
}
Now the sanity check happens at Filter() call time, while the enumeration of subscription still only occurs as its being iterated over, allowing for additional filtering and Skip/Take additions without having to traverse the entire possible set.
This isn't a review of the conferences as much as my impression of the different forces acting upon javascript, the language. Before I start, i should get my bias out of the way, as it likely colors my observations: Like many I came to javascript out of nessessity and seeing a C-like syntax tried to make it fit into a mold it was ill-suited for and much frustration ensued. I've taken the language at face value, and being a fan of expressions and lambdas, have found it to be fun and flexible. That said, it does have some well documented warts and in many ways these warts are what are behind the different forces pulling at the language.
jsconf and nodeconf had significantly different vibes, but where I had expected the difference to be due to server vs. client people, it seemed that the difference was more closely aligned to the relationship the attendees had to javascript. My impression is that jsconf is a community brought together by the common goal of creating amazing experiences in the browser. Some embrace the language as is, others rely on frameworks (or this year's hottness, micro-frameworks) to make them productive, while yet others try to bend the language to their will by using javascript as a compilation target.
Of those using javascript as a compilation target, coffeescript was the clear star, with enough talks using it as their defacto language that got the impression that it was a natively supported language. The next to last #jsconf talk featuring @jashkenas even nullified the B Track entirely and was joined by @brendaneich to talk about JS.Next. The talk covered proposed and accepted changes to javascript, and coffeescript was held up as testbed for fast prototyping and experimentation with possible syntax changes
The final jsconf talk was clearly meant to come off as a Jobsian lead-in to a big reveal. This reveal was traceur, google's transpiler for trying out what google wants JS.Next to look like. I don't know whether it was the relatively stilted presentation style or the fact that it re-hashed a lot of Brendan's presentation, but the crowd lacked enthusiam for both the presentation and the reveal. I personally liked what they were proposing, but I can't say I disagree with one attendee later describing it as having a condescending tone, something like "we're here to rescue you from javascript". Brendan seemed to have read the talk this way as well.
All in all, jsconf clearly seemed to be celebrating the possibilities ahead and the power of the language to be mutated into virtually any form. More than once I overhead someone say that they were sold on coffeescript and would try it for their next project.
The following night was the the nodeconf pre-party. I had the pleasure of talking extensively with @izs (of npm fame) and @mikeal about various topics javascript and node. Being the language geek that I am, I brought up traceur and coffeescript and was quick to realize that this was a different crowd than jsconf: Nodeconf is a community that chose javascript as their primary language, finding it preferable to whatever language they had worked with before. Clearly the node community does not need language changes to enable their productivity.
This impression of a community happy with the state of their chosen tool was re-enforced throughout the next day at nodeconf. One talk on Track A was "Mozilla Person, Secret Talk". When I suggested that it would likely be about Mozilla's efforts to create node on top of spidermonkey one of the guys at our table said that if that was the case, he would have to go and check out Track B. As the Mozilla person turned out to be Brendan, our tablemate did leave. The talk itself was briefly about V8Monkey and SpiderNode, the two abstraction layers Mozilla is building to create a node clone, and largely a re-hash of Mozilla's JS.Next talk. The post talk questions seemed generally uninterested in JS.Next and were mostly different forms of "what do we have to gain from SpiderNode."
Clearly the node community is not beholden to any browser vendor. They've created this new development model out of nothing and are incredibly productive in that environment. The velocity of node and the growth of the npm ecosystem is simply unmatched. What node has already proven is that they don't need rescuing from javascript as it stands. Javascript is working just fine for them, thank you.
I do believe that Javascript is at a cross-roads, and being the only choice available for client-side web development, it is being pulled into a lot of directions at once by everyone wanting to influence it with bits from their favorite language. It is clear that JS.Next is actually going to happen and bring some of the most significant changes the language has seen in an age. I can't say I'm not excited about the proposals in harmonizr and traceur, but I certainly can understand why this looming change is seen as a distraction by those who have mastered the current language. Being more of a server-side guy nodeconf was clearly my favorite of the two conferences and while I had started the week in Portland with the intention of writing my future node projects in coffeescript, I've now decided to stick with plain old javascript. I fear not doing so would only lead me back into my original trap of trying to make the language something it wasn't which in the end would only hurt my own productivity.
I started this year with a surprise blogging momentum and it was going really great until i started this post at the beginning of March. I made the mistake of writing a novel on the subject, which just ended up in a meandering draft that completely killed all other writing. Lessons learned: If it takes more than two sessions to write a post, scrap it. So here's a single session redux:
REST is wonderful as a query pattern. It easily handles resources and collections of resources and let's you represent hierarchical data models. But for anything than a pure data store, that same pattern is horrible for writes. Posting/putting whole documents at a location comes with complications like what's readonly vs. writable, how are business rules applied, how do you handle partial updates, etc. Yes, it's all possible, but it just imposes lots of ad-hoc and opaque rules on the API.
Before you tell me that you've solved all that, let's just get this clear: Most REST APIs out there either are HTTP-RPC or at least use some RPC in them, but call themselves RESTful, cause it's, like, cool. I'm no REST purist, but I'm willing to bet that your solution to these problems almost always involves a couple of RPC style calls in your RESTful APIs, which only proves my point.
Consider a public user API and how to deal with the user's password:
-- POST:/users --
<user>
<name>bob</name>
<email>bob@bar.com</email>
<password>foo</password>
</user>
-- GET:/users/{id} --
<user id="123">
<name>bob</name>
<email>bob@bar.com</email>
<password-hash>a2e2f5</password-hash>
</user>
-- PUT:/users/{id} --
???
On the POST, we really want the password twice, otherwise we're just a data store pushing the responsiblity for business logic off on the client. On the GET we certainly don't want to return the password. And finally, how do we even update the password? We'd want it in the document twice, plus the old password. So much for a symmetric resource model.
This same problem occurs with Entity models in ORMs: The query and write data models are treated as symmetric, when in reality what we query for and what we manipulate seldomly follows the same model. Query models usually end up getting simplified (flattened, normalized) to favor updates, and update models contain mutable data inappropriate for a specific action.
On the data manipulation side, the CQRS (Command-query Resposibility Separation) pattern has been gaining favor. In it, data is retrieved via queries that match view models, while commands take only the data affected by the command and the command explicitly reflects the user story it advertises.
Commands are procedural, taking as input only the data they require. That certaintly matches HTTP-RPC: It's not a modified resource being stored, although the contract may imply the manipulation of a resource. This pattern gives far greater freedom to manipulate subsets and supersets of resources than a REST PUT can offer and is a more natural match for how data is manipulated in user stories.
On the query side, we've freed REST from representing models that need to be modifiable via PUT, allowing more complex and denormalized data. Yes, this breaks the REST mantra of canonical location of a resource, but that mantra is largely a reflection of having to have a canonical location for manipulating the data. Once models are query only, denormalization isn't a problem anymore, since the command responsible for modification takes on the responsibility of making sure the denormalized changes are appropriately propagated.
Together the use of HTTP-RPC for write and REST for query, we get HTTP-CQRS. Applying this pattern to that public user API from before, we might deal with the password like this:
-- POST:/commands/users/create --
<user>
<name>bob</name>
<email>bob@bar.com</email>
<password1>foo</password1>
<password2>foo</password2>
</user>
-- GET:/query/users/{id} --
<user id="123">
<name>bob</name>
<email>bob@bar.com</email>
<password-hash>a2e2f5</password-hash>
</user>
-- POST:/commands/users/{id}/changepassword --
<command>
<old-password>foo</old-password>
<new-password1>bar</new-password1>
<new-password2>bar</new-password2>
</command>
While you could go all SOAP-like and just have a /commands endpoint and require the action in the body, using descriptive URIs greatly simplifies API comprehension, imho. By separating query and command reponsibility for web services the API actually becomes more descriptive and opens up a lot of operational patterns that aren feasible or at least not sensible with pure RESTful APIs.
If there is one language feature of .NET that I've become increasingly apprehensive of it is events. On the surface they seem incredibly useful, letting you observe behavior without the observed object having to know anything about the observer. But the way they are implemented has a number of problems that makes me avoid them whenever possible.
The biggest pitfall with events is that they are a common source of "memory leaks". Yes, a managed language can leak memory — it happens anytime you create an object that is still referenced by an active object and cannot be garbage collected. The nasty bit that usually goes unmentioned is that an event subscription represents an object holding a reference to the observed instance. Not only does this go unmentioned, but Microsoft spent years showing off code samples and doing drag and drop demos of subscribing to events without stressing that you need to also unsubscribe from them again.
Every "memory leak" I've ever dealt with in .NET traced back to some subscription that wasn't released. And tracking this down in a large project is nasty work –taking and comparing memory shapshots to see what objects are sticking around, who subscribes to them and whether they should really still be subscribed. All because the observer affects the ability of the observed to go out of scope, which seems like a violation of the Observer pattern.
A pattern I've implemented from scratch several times (the side-effect of implementing core features in proprietary code) is the Weak Event pattern, i.e. an event that uses a weak reference as the subscription, so that the observed class isn't pinned in memory by a subscriber.
.NET 4 Microsoft has even formalized this with the WeakEventManager to implement the Weak Event Pattern, although I prefer just overriding the add and remove on an event and using weak references under the hood. While this changes the expected behavior of events and is unexpected in public facing APIs, I consider it the way events should have been implemented in the first place, and use it as default in my non-public facing code.
A better way of implementing the Observer pattern is IObservable from the Reactive Framework (Rx). Getting a stream of events pushed at you is a lot more natural for observation and allows for following a number of different behaviors in one observer. It also provides a mechanism for terminating the subscription from the observed end, as well a way deal with exceptions occuring in event generation. For new APIs this is definitely my prefered method of pushing state changes at listeners.
A pattern I encounter frequently are one time events that simply signal a change in state, such as a connection being estatblished or closed. What I really want for these is a callback. I've added methods in the vein of AddConnectedCallback(Action callback), but always feel like their unintuitive constructs born out of my dislike of events, so generally I just end up creating events for these after all.
I could just use a lambda to subscribe to an event an capture the current scope much like the .WhenDone handler of Result, the lambda is anonymous making it impossible to unsubscribe:
xmpp.OnLogin += (sender,args) => {
xmpp.Send("Hello");
// but how do I unsubscribe now?
};
The mere fact that lambdas are being shown as convenient ways to subscribe to events without any mention about the reference leaks this introduces just further illustrates how broken both events and their guidance are. Using this closure, simplifies attaching behavior at invocation time and makes sure that unsubscribe is handled cleanly.
Doing a lot of asynchronous programming work with MindTouch DReAM's Result continuation handle (think TPL's Task, but available since .NET 2.0), I decided that being able to subscribe to an event with a result would be ideal. Inspired by Rx's Observable.FromEvent, I created EventClosure, which can be used like this:
EventClosure.Subscribe(h => xmpp.OnLogin += h, h => xmpp.OnLogin -= h)
.WhenDone(r => xmpp.Send("Hello"));
Unfortunately, like Observable.FromEvent, you have to set up the subscribe and unsubscribe using an Action provided handler, since there isn't a way to pass xmpp.OnLogin as an argument and do it programatically. But at least now the subscribe and unsubscribe are handled in one place and I can concentrate on the logic I want executed at event invocation.
I could have implemented this same pattern using Task, but until async/await ships, Result still has the advantage, aside from continuation via .WhenDone or Blocking via .Block or .Wait, Result also gives me the ability to use a coroutine:
public IEnumerator<IYield> ConnectAndWelcome(Result<Xmpp> result) {
var xmpp = CreateClient();
var loginContinuation = EventClosure.Subscribe(h => xmpp.OnLogin += h, h => xmpp.OnLogin -= h);
xmpp.Connect();
yield return loginContinuation;
xmpp.Send("hello");
result.Return(xmpp);
}
This creates the client, starts the connection and suspends itself until connected, so it can then send a welcome message and return the connected client to its invokee. All this happens asynchronously! The implementation of EventClosure looks like this (and could easily be adapted to use Task instead of Result):
public static class EventClosure {
public static Result Subscribe(
Action<EventHandler> subscribe,
Action<EventHandler> unsubscribe
) {
return Subscribe(subscribe, unsubscribe, new Result());
}
public static Result Subscribe(
Action<EventHandler> subscribe,
Action<EventHandler> unsubscribe,
Result result
) {
var closure = new Closure(unsubscribe, result);
subscribe(closure.Handler);
return result;
}
public static Result<TEventArgs> Subscribe<TEventArgs>(
Action<EventHandler<TEventArgs>> subscribe,
Action<EventHandler<TEventArgs>> unsubscribe
) where TEventArgs : EventArgs {
return Subscribe(subscribe, unsubscribe, new Result<TEventArgs>());
}
public static Result<TEventArgs> Subscribe<TEventArgs>(
Action<EventHandler<TEventArgs>> subscribe,
Action<EventHandler<TEventArgs>> unsubscribe,
Result<TEventArgs> result
) where TEventArgs : EventArgs {
var closure = new Closure<TEventArgs>(unsubscribe, result);
subscribe(closure.Handler);
return result;
}
private class Closure {
private readonly Action<EventHandler> _unsubscribe;
private readonly Result _result;
public Closure(Action<EventHandler> unsubscribe, Result result) {
_unsubscribe = unsubscribe;
_result = result;
}
public void Handler(object sender, EventArgs eventArgs) {
_unsubscribe(Handler);
_result.Return();
}
}
private class Closure<TEventArgs> where TEventArgs : EventArgs {
private readonly Action<EventHandler<TEventArgs>> _unsubscribe;
private readonly Result<TEventArgs> _result;
public Closure(Action<EventHandler<TEventArgs>> unsubscribe, Result<TEventArgs> result) {
_unsubscribe = unsubscribe;
_result = result;
}
public void Handler(object sender, TEventArgs eventArgs) {
_unsubscribe(Handler);
_result.Return(eventArgs);
}
}
}
While this pattern is limited to single fire events, since Result can only be triggered once, it is a common enough pattern of event usage and one of the cleanest ways to receive that notification asynchronously.