After the latest round of scala is teh complex, I started thinking a bit more about the similar roles implicit conversions in scala play to extension methods in C#. I can't objectively comment on whether Extension Methods are simpler (they are still a discoverability issue), but I can comment that the idea of implicit conversion to attach new functionality to existing code really intrigued me. It does seem less obvious but also a lot more powerful.

Then i thought, wait, C# has implicit.. Did they just invent extension methods without needing to? I've never used implicit conversion in C# for anything but automatic casting, but maybe I was just not imaginative enough. So here goes nothing:

public class RichInt {
  public readonly int Value;

  public static implicit operator RichInt(int i) {
    return new RichInt(i);
  }

  public static implicit operator int(RichInt i) {
    return i.Value;
  }

  private RichInt(int i) {
    Value = i;
  }

  public IEnumerable<int> To(int upper) {
    for(int i = Value;i<=upper;i++) {
      yield return i;
    }
  }
}

Given the above I was hoping I could mimic scala's RichInt to() method and write code like this:

foreach(var i in 10.To(20)) {
  Console.WriteLine(i);
}

Alas that won't compile. Implicit conversion only works on assignment, so i had to write

foreach(var i in ((RichInt)10).To(20)) {
  Console.WriteLine(i);
}

So I do have to use an extension method to create To() in C#.

And, yes, i'm grossly simplifying what scala's implicts can accomplish. Also, I wish I could have block scope using  statements to import extension methods for just a block the way scala's import allows you to handle local scope implicits.

Logging, to me, is a basic requirement of any code I write. There's nothing more useful to track down an odd bug than being able to turn on debug logging in production, potentially filter it down to the area of interest and get some real live data. When that code is causing trouble at a customer location, this capability becomes worth its weight in gold.

And while I am a big Interface abstraction and IoC nut, logging is so fundamental to me that I do use a static logger instance in every class and just have a hard requirement for log4net. I just want this to work and not have people worry about having to inject an ILog into my classes, forcing them to think about a concern of mine. It's far less obtrusive this way, imho.

But it does mean that no matter how small and compact my code is, i suddenly tack on a ~240KB dll requirement, often several times larger than my own code. And there goes my "i don't want them to worry about a concern of mine" out the window.

Depending on log4net only when it's there

That got me thinking about how I can avoid this dependency. I knew that if my code path doesn't hit a code in a referenced DLL, the absense of the DLL does not cause a problem. Of course that means i can't even reference their interfaces.

Instead I had to create a duplicate of the log4net ILog interface and proxy my calls to it. The result is a single file, Logger.cs with the following wire-up:

internal static class Logger {

  private static readonly bool _loggingIsOff = true;

  static Logger() {
    try {
        Assembly.Load("log4net");
        _loggingIsOff = false;
    } catch {}
  }

  public static ILog CreateLog() {
    var frame = new System.Diagnostics.StackFrame(1, false);
    var type = frame.GetMethod().DeclaringType;
    return _loggingIsOff ? (ILog)new NoLog() : new Log4NetLogger(type);
  }
  ...
}

(Full code is here)

Because it's not supposed to create a dependency, it's marked internal. Yes, that means that every assembly has a copy of it, but that's really not significant in the grand scheme of things. In returns for a couple of redundant bytes, I can now do add a logger to every class simply with:

private static readonly Logger.ILog _log = Logger.CreateLog();

And if log4net is present and configured, commence the flow of debug statements. Meanwhile I can ship my code without the log4net DLL.

I'm currently at #monospace and over lunch was discussing message passing with @ackenpacken and @briandonahue. I was lamenting that to create an immutable object in C# was just a lot harder than a mutable one. You either have to create a builder of sorts or create very verbose constructors and never mind trying to derive a new immutable instance with just one field changed. I suggested that if only private setters could be used with object initializers, this would be solved and realized that I could just use anonymous objects as my initializers and so after finishing with talks for the day I coded up a proof of concept.

// create a new data class
public class Person {
  public int Id { get; private set; }
  public string Name { get; private set; }
}

So this is a perfectly good immutable object. So good, we can't even initialize it. Enter Droog.ImMutie.Immutable:

// create and instance
var person = Immutable.Create<Person>(new {Id = 43, Name = "Bob"});

The argument to Immutable.Create is simply object, so we can pass in an anonymous object with whatever properties we want. Create then provides and instance and copies over values on property name and type match. And if we want to change the Name?

// derive new instance with different name (using extension method)
person = person.With(new {Name = "Robert"});

With let's you provide a new anonymous object, will clone the previous instance and then overwrite each newly provided field by name/type match.

Very simple, no error checking, not the most efficient code, but it works and you can get it from github

One dirty little secret about Visual Studio 2008 and even Visual Studio 2010 is that while MSBuild governs the solution build process, the .sln file is not an MSBuild file. The .*proj files are, but solution isn't. So trying to customize the build on the solution level seemed really annoying.

As I dug around trying to find the Solution level equivalent of the Build Events dialog from Visual Studio, Sayed Ibrahim pointed out that in Visual Studio 2010 there is now a hook to let you inject some before and after tasks, but unfortunately the problem I was trying to solve was the build process for MindTouch DReAM, which is still in Visual Studio 2008.

Approach 1: Generating the solution msbuild .proj

Digging around further, I found out that you could get the MSBuild file that the solution was turned into. By setting the environment variable MSBuildEmitSolution=1 and running MSBuild will write out the generated .proj file.

While this enables you to edit it and add new tasks, it means that your build script will drift out of sync with the solution as it is modified. I initially went down this path, since the build i wanted was very specialized to the distribution build. That let me eliminate 90% of the .proj file and I felt confident that the smaller the .proj, the simpler it would be to keep it in sync with the solution.

Approach 2: Calling the solution from a different build script

But wait, all the solution .proj did was call MSBuild on each of its projects. So if one MSBuild script can call another, why do i even need to use a generated version of the solution? Turns out you don't. You can write a very simple MSbuild script, that in turn calls the .sln, letting MSBuild perform the conversion magic, and you still get your pre and post conditions.

<Project DefaultTargets="Build" ToolsVersion="3.5" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Target Name="Build">
        <CallTarget Targets="PreBuild"/>
        <CallTarget Targets="Dream"/>
        <CallTarget Targets="PostBuild"/>
    </Target>
    <Target Name="PreBuild">
        <Message Text="Pre Build" />
        ...
    </Target>
    <Target Name="PostBuild">
        <Message Text="Post Build" />
        ...
    </Target>
    <Target Name="Dream" Outputs="@(DreamBuildOutput)">
        <Message Text="Building DReAM" />
        <MSBuild Targets="Rebuild"
                 Projects="src\MindTouchDream.sln"
                 Properties="Configuration=Signed Release; Platform=Any CPU; BuildingSolutionFile=true;">
            <Output TaskParameter="TargetOutputs" ItemName="DreamBuildOutput" />
        </MSBuild>
        <Message Text="Done building DReAM" />
    </Target>
</Project>

Now that I've implemented this, I am surprised that when I looked for a solution, this didn't come up in google and I hope that this post helps the next person that runs into this issue. The only drawback (which it shares with the first approach) is that this script is only for manual execution. Building from within Visual Studio can't take advantage of it.

Yesterday I needed to create a bit of code that should retry an http request on failure and i wanted this to happen in a non-blocking fashion, so a plain loop was out. Using DReAM's Plug, I can easily set up a .WhenDone handler, but I need something to call back recursively. I could have created a new method that calls itself recursively, but inspired by my recent scala reading, i wanted to define the function to execute to exist only in the scope of the current function. The resulting code was this:

public void Restart(string wikiId) {
    Action restarter = null;
    var retry = 0;
    restarter = () => {
        retry++;

        // call the stop endpoint
        _self.At("host", "stop", wikiId)
            .Post(new Result<DreamMessage>()).WhenDone(
                m => {

                    // if the endpoint was unavailable and we've got retries left ...
                    if(m.Status == DreamStatus.ServiceUnavailable && retry <= 3) {

                        // ... sleep for 2 seconds ...
                        Async.Sleep(2.Seconds()).WhenDone(r => {

                            // ... then try again
                            restarter();
                        });
                    }
                },
                e => {
                    // plug's don't throw exceptions, they communicate failure in the resulting message
                }
            );
    };

    // kick the restart attempt off
    restarter();
}

Completely non-blocking, yay. Of course, like most continuation based async code, it has a tendency to walk off the right side of the screen, but that's a different topic.

Discussing this pattern, Steve suggested that instead of explictly calling myself to execute the next retry, the Action should take another action as its argument. By passing continuation in, any kind of continuation could be called, even a no-op that terminates the chain.

Ok, cool, let's just change the signature of restarter to Action<Action>… no wait, it need to be an action that takes and action that takes an action, that…. hrm, you see the problem…

Action<Action<Action<Action<ad infinitum>>>>

I needed a recursive delegate definition and that is simply not possible with Func<> and Action<>. I needed something like:

delegate void Recursor(Recursor continuation)

So, unlike my previous statement, there are still scenarios where you have to write a delegate

My inability to cleanly insert an intercepting actor to spawn new actors on demand discussed in my previous post about Calculon illustrated tunnelvision in my expression based messages. While these are a great way to express the message contract, they are finally just one way of defining message payloads. The pipeline should be ignorant of message payloads and be able to route any payload based on meta-data alone.

Routing revisited

I had to stop hiding the IMessage format under hood and just accept that ExpressionTransport really represents a convenience extension for generating message payloads. This change not only makes it possible for any actor to accept any kind of message, but also to makes the routing a lot simpler and flexible. Now there is only a single, much simpler ITransport:

public interface ITransport {
    ActorAddress Sender { get; }
    void Send(IMessage message);
}

I've also attached ther sender onto the transport, which means there's one less dependency to inject. Given this, ExpressionTransport and MessageTransport simply becomes extension methods on ITransport:

public static class TransportEx {
    public static void Send<TRecipient>(
       this ITransport transport,
       Expression<Action<TRecipient>> message
    ) {
       transport.Send(new ExpressionActionMessage<TRecipient>(
          transport.Sender,
          ActorAddress.Create<TRecipient>(), message
       ));
    }
    // etc.
}

This provides a lot more flexibility for messaging, since new payloads can be created by implementing IMessage. It does mean that dynamic dispatch would be nice for dispatching the different payloads against their respective receivers. Oh well, we can work around that.

Determining what messages to accept

Now it's possible for an actor not of type TRecipient to receive an ExpressionMessage<TRecipient>, perform some action and re-dispatch it via Send(). By default routing is done by Id, but missing a direct receiver we can now insert additional pattern matching to determine a suitable recipient. For this I've created an interface to let actors expose their accept criteria:

public interface IPatternMatchingActor {
    Expression<Func<MessageMeta, bool>> AcceptCriteria { get; }
}

While this could have been done by convention, I couldn't think of a reason other than current fashion to not use an interface contract to expose this actor capability, so an interface it was. The reason it's an Expression and operates on the MessageMeta rather than the expression is to facilitate serialization of the criteria for delivery across the process boundary so messages can be qualified for acceptance before attempting to cross that boundary.

Better plumbing

I now have everything I need to switch the notify.me bots to Calculon. Once that is done, and working reliably, it'll be time to improve the dispatch plumbing to improve speed and reduce concurrency bottlenecks in dispatch.

I'm currently extending functionality in the notify.me bots and in order to make this easier, I'm refactoring the adhoc actor-like message processing system I built into one a bit more flexible for adding features quickly. Right now message senders and receivers are hard-coupled and use blocking dictionary lookups for dispatch. They also act on instances of each other, which allowed some insidious calls to sneak in during moments of weakness.

As I embarked on my refactor, I wanted to make sure the replacing infrastructure removed assumptions about the entities communicating among each other but also wanted to avoid the pitfall of designing something overly generic. For that I had to first define what it was I needed to be able to do, so I'd only build what I need. At the same time, I decided to pull the replacement into its own Assembly, so that implementation specific coupling wouldn't leak back into the plumbing again. The resulting system has been named in honor of the greatest of all acting robots, Calculon, and is available in its present work-in-progress form on github.

The current actors

XmppBot

The bot is responsible for dispatching messages to users and receiving user messages and presence status. The bot passes messages for a user on to the user's UserAgent actor and receives messages to send to the user from the UserAgent. For distribution and maintenance simplicity, each bot and its related actors was implemented as a separate process.

UserAgents

UserAgents keep the state of the user, such as presence, including all resources (different clients) connected and queues up messages coming from the message queue until the user is in a state to receive messages. It has its own persistence layer, allowing idle users to expire and be recreated as incoming traffic from either the bot or the message queue requires it.

MessageQueue

The message queue is a client to our store-and-forward queueing system. Messages from users are pushed into the this actor via long-polling and user data/actions that affect other notify.me systems (such as analytics) are pushed into the appropriate queues as they are handed to the MessageQueue by other actors (generally UserAgents).

What capabilities are needed?

Register actors

At the root of the system, exists the Stage, which exposes the ActorBuilder:

_stage.AddActor<IXmppAgent>().WithId("bob@foo.com").Build();

The assumption is that actors may require a transport and their own address at construction time and that they are completely isolated, i.e. no reference is ever exposed. The builder will inject these framework owned dependencies if detected in a constructors signature. In order to allow for more flexible construction and the ability to have some kind of IoC container act as a factory, the builder exposes hooks like this:

_stage.AddActor<IXmppAgent>().WithId("bod@foo.com").BuildWithExpressionTransport(
    (transport,address) => container.Resolve<IXmppAgent>(transport,address)
);

The above assumes a container such as Autofac which can resolve a type and be provided typed parameters to optionally inject.

Send messages without knowing that a receiver exists

This is the root of the dispatch system. I need to be able to send the message without a reference to receiver and let the transport worry about immediate delivery, queueing for later or routing it to some controller that will bring the recipient into existence. None of those concerns should be visible to the sender. Using semantics introduced in "Type-safe actor messaging approaches", and slightly tweaked by implementation, provides me with a way of asynchronously calling methods on unknown recipients:

public interface IExpressionTransport {
    void Send<TRecipient>(Expression<Action<TRecipient>> message);
    void Send<TRecipient>(Expression<Action<TRecipient, MessageMeta>> message);
    Result SendAndReceive<TRecipient>(Expression<Action<TRecipient>> message);
    Result SendAndReceive<TRecipient>(Expression<Action<TRecipient, MessageMeta>> message);
    Result<TResponse> SendAndReceive<TRecipient, TResponse>(
        Expression<Func<TRecipient, TResponse>> message
    );
    Result<TResponse> SendAndReceive<TRecipient, TResponse>(
        Expression<Func<TRecipient, MessageMeta, TResponse>> message
    );
    IAddressedExpressionTransport<TRecipient> For<TRecipient>(string id);
}

The main addition is the ability to inject MessageMeta, a class containing meta information such as Sender and Recipient into the receiver without the Sender having to specify this data.

Send/Receive by Id (UserAgent target)

For UserAgent messages, there are thousands of actors each with a unique Id. While currently that Id is a Jid I don't want to tie the internals to Xmpp specific details, so Id should be an plain string and let the transport worry about the meaning and routing implications of that string.

The ability to send by Id is provided by IExpressionTransport.For<TRecipient>(string id). The returned interface IAddressedExpressionTransport<TRecipient> mirrors IExpressionTransport, representing a intermediate storage of the receiver id, thus providing a fluent interface that permits the following calling convention:

_transport.For<Recipient>(id).SendAndReceive(x => x.Notify("hey", "how'd you like that?"));

Send/Receive by Type (XmppBot/MessageQueue targets)

If I stay with the process-per-bot for the bot and messagequeue actors, there would be a single instance for these actors and I can address them directly by Type. The semantics for these message are already expressed by IExpressionTransport.

Spawn UserAgent on demand

Of course, dealing with unkown recipients begs the question where do these recipients come from? I need to be able intercept messages for Id's that are not yet in the system and spawn those recipients on the fly. Wanting to stay with actors for anything but the base plumbing, this facility should be handled by actors that can receive these messages and tell the plumbing to instantiate a new actor.

The same interface to access the ActorBuilder exposed by the stage is encapsulated by the IDirector:

public interface IDirector {
    ActorBuilder<TActor, IDirector> AddActor<TActor>();
    void R(ActorAddress address);
}

The director being a framework owned actor can of course be called via messaging, allowing a new actor to be registered with:

_transport.Send<IDirector>(
    x => x.AddActor<IXmppAgent>().WithId("bob@foo.com").Build()
);

That leaves the ability to intercept messages that don't have a recipient, and redispatch those messages once the interceptor was able to spawn the actor. Both of those are not compatible with Expression based messages since they are coupled to a pretty specific contract. This is the one piece I don't have in Calculon at the time of this writing and the problem is discussed below.

Retire UserAgent on Idle

When a UserAgent sits idle for a while, it should be possible to remove it from the actor pool. Since the actor instance doesn't know that anything about the framework that owns it, there needs to be a message that can be sent to the actors mailbox that shuts it down, ideally disposing and IDisposable actors.

The interface IDirector introduced above includes a method for just that:

_transport.Send<IDirector>(
    (x,m) => x.RetireActor(_address, m);
);

This could be send by an actor itself, or by a governing actor that is responsible for a number of actors in a pool. Under the hood, this is where un-typed messages come into play, since they can be sent without a matching method on the recipient, and therefore could have special meaning to the mailbox that manages the recipient. I.e. sending the retire message to the director, simply causes it to send an untyped retire message to the actors mailbox, which will then shut itself down and dispose the actor. The interface for untyped messages (providing a more traditional Receive(msg) actor messaging model) is provided by this interface:

public interface IMessageTransport {
    void Send<TMessageData>(TMessageData messageData);
    Result<TOut> SendAndReceive<TIn, TOut>(TIn messageData);
    IAddressedMessageTransport For(string id);
}

Rather than force an interface on the receiving actor, messages not simply swallowed by the mailbox are delivered to the actor by convention, looking for a Receive method with appropriate TMessageData.

What capabilities are desirable?

The above is the basic requirements to provide the same functionality already present in the notify.me bot daemons, but using generalized plumbing. It's certainly sufficient to get the code underway, but in itself doesn't provide a lot more than the status-quo other than simplifying the extensibility and maintainability of UserAgents.

To expand on the present feature set and move other parts of the notify.me system to this actor infrastructure I have the following additional design goals for Calculon:

POCO Actors

One of my lead design goals was not to force any interface or baseclass requirements on actors, i.e. it should be possible to author actors as plain old C# objects (POCO). Actors should exist as their own isolated execution environment and their functionality testable without any part of Calculon in play. Dependencies such as transport and address are completely optional and injected by signature.

Actor monitoring and restart

Another aspect I would like to see is the erlang-style let it crash philosophy. It should be possible for an actor to subscribe to another actor to monitor its health. I'm not sure what "crash" should mean at this time, since using Result as the completion handle already captures exceptions and marshalls them to the caller.

My plan is to let these semantics emerge from use cases, as I put Calculon into production.

Remote Actors

One of primary benefits of actors for concurrency is that it cleanly decouples the pieces from each other and lets you move these pieces around for scalability. Being able to serialize the messages would allow dispatchers to send messages across the wire to other nodes in the actor network. For this, I need to determine a format for serializing the LINQ expressions used in ExpressionMessage. That means that any value captured by the expression needs to be serializable itself. Unfortunately checking whether an expression can be serialzied will be a runtime rather than compile time check.

Serializable messages are desirable even for local operation to enforce the share nothing philosophy. As it stands right now, shared object references could be used as message arguments, which defeats the purpose of this system. However, for performance reasons, I will likely employ Subzero to avoid unnecessary copying.

Dynamically load code and replace Actor implementation during runtime

Once there exists remote actor capability, it is possible to traverse AppDomain boundaries easily. That means that we could launch actors in different AppDomain. Conceivably, we should be able to drop a new implementation dll into a directory, load it up and have a control actor shutdown existing actors and and subsume their capabilties with its own implementation. Since we're serializing messages, changes to an actor's implemtation or even interface do not matter, as long as the method signatures previously published still exist.

Current Status

The "needed" capabilities, except for message intercept and re-dispatch, are currently implemented, although the infrastructure is a very simple implementation with lock contention issues under load. However those limitations are no more severe than my current setup, so it's good enough to migrate to, letting me improve and expand the plumbing against a working system.

The main stumbling block is dealing with interception. Right now delivery is done by Id and Type, and for expression based messages Type is fairly binding, at least to use the message. Of course the if the primary reason to intercept messages is to create the missing recipient, the interceptor would not need to be able to unwrap the message, just re-dispatch.

The simple way to implement this is to make interceptors hook into the dispatch framework, rather than actors in their own right. They could then be tied to internals and simply be part of the mailbox matching code and spawn and insert a new mailbox when triggered. However, I would rather stick with actors for everything and make the framework as invisible as possible, which also means that capture and re-dispatch should also be possible without exposing the internals of the framework. I.e. right now nobody outside of the plumbing ever even sees an expression message instance and I'd like to keep it that way.

Since I already know that I want to have actors that can choose to accept messages based on the sender, rather than a recipient id, it's clear that I need better pattern matching capabilities for actors to expose that let them indicate their interest in accepting a message and that I need some neutral payload format that can be re-addressed and re-dispatched. So that's still one part of the puzzle I have to solve.

After porting my mod_mono ASP.NET MVC application to Ruby and Rails and setting up Phusion Passenger up to run the application under mono, I finally figured out how to fix the leaking semaphore issue. The real title of this post should probably be "PEBKAC or Don't assume errors are unrelated, you idiot".

Recap of the problem

The problem manifests itself as a build up of semaphore arrays by the apache process, which is visible via ipcs. When the site is first started the output looks like this:

[root@host ~]# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x01014009 1671168    root       600        52828      48
0x0101400a 1703937    root       600        52828      25
0x0101400c 1736706    root       600        52828      35

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 10616832   apache     600        1
0x00000000 10649601   apache     600        1
0x00000000 10682370   apache     600        1
0x00000000 10715139   apache     600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Eventually it'll look like this:

[root@host ~]# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x01014009 1671168    root       600        52828      48
0x0101400a 1703937    root       600        52828      25
0x0101400c 1736706    root       600        52828      35

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 10616832   apache     600        1
0x00000000 10649601   apache     600        1
0x00000000 10682370   apache     600        1
...
lots more
...
0x00000000 11141158   apache     600        1
0x00000000 11173927   apache     600        1
0x00000000 11206696   apache     600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

At some point all ASP.NET pages will return blank pages. No errors, no nothing, .NET logging reports normal behavior, but no content is sent. And you can restart the mono processes and apache all you want, it won't come back. Sorry.

What's really wrong

Since day one i'd been receiving warnings at apache startup up, but since i didn't understand what they meant and things seemed to be working, i had been ignoring the,. Of course, that was a lie on its face. Things were clearly not working, with the leaking semaphores, but I conveniently filed the two issues as unrelated in my head and ignored them at my peril. The warning was this:

[Mon Jan 24 00:12:50 2011] [crit] The unix daemon module not initialized yet.
Please make sure that your mod_mono module is loaded after the User/Group
directives have been parsed. Not initializing the dashboard.

This warning was repeated for as many times as I had ASP.NET vhosts defined. I looked at my vhost configurations and saw nothing about users and groups and thought it was some weird mono issue and left it at that. But the actual problem was not in the vhost configuration but in httpd.conf. The problem was this default section:

#
# Load config files from the config directory "/etc/httpd/conf.d".
#
Include conf.d/*.conf

#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.
#
# User/Group: The name (or #number) of the user/group to run httpd as.
#  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
#  . On HPUX you may not be able to use shared memory as nobody, and the
#    suggested workaround is to create a user www and use that user.
#  NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
#  when the value of (unsigned)Group is above 60000;
#  don't use Group #-1 on these systems!
#
User apache
Group apache

Obviously User and Group are set after all vhost configs are loaded. Pretty much exactly what the warning was saying (doh). I simply moved the Include below User/Group and since then I have not seen more than 9 semaphores and I've restarted mono, rebuilt the application and hit the app with ApacheBench, the combination of which used to drive semaphores up.

Since things are working now and ASP.NET MVC under mod_mono is significantly faster than the Rails port, I'm sticking with ASP.NET MVC for production right now, monitoring semaphores to make sure this really did fix the problem.

For notify.me I hand-rolled a simple actor system to handle all Xmpp traffic. Every user in the system has its own actor that maintains their xmpp state, tracking online status, resources, resource capability, notification queues and command capabilities. When a message comes in either via our internal notification queues or from the user, a simple dispatcher sends the message on to the actor which handles the message and responds via a message that the dispatcher either hands off to the Xmpp bot for formatting and delivery to the client or sends it to our internal queues for propagation to other parts of the notify.me system.

This has worked flawlessly for over 2 years now, but its ad-hoc nature means it's a fairly high touch system in terms of extensibility. This has led me down building a more general actor system. Originally Xmpp was our backbone transport among actors in the notify.me system, but at this point, I would like to use Xmpp only as an edge transport, and otherwise use in-process mailboxes and serialize via protobuf for remote actors. I still love the Xmpp model for distributing work, since nodes can just come up anywhere, sign into a chatroom and report for work. You get broadcast, online monitoring, point-to-point messaging, etc. all for free. But it means all messages go across the xmpp backbone, which has a bit of overhead and with thousands of actors, i'd rather stay in process when possible. No point going out to the xmpp server and back just to talk to the actor next to you. I will likely still use Xmpp for Actor Host nodes to discover each other, but the actual inter-node communication will be direct Http-RPC (no, it's not RESTful, if it's just messaging).

Definining the messaging contract as an Interface

One design approach I'm currently playing with is using actors that expose their contract via an interface. Keeping the share-nothing philosophy of traditional actors, you still won't have a reference to an actor, but since you know its type, you know exactly what capabilities it has. That means rather than having a single receive point on the actor and making it responsible for routing the message internally based on message type (a capability that lends itself better to composition), messages can arrive directly at their endpoints by signature. Another benefit is that testing the actor behavior is separate from its routing rules.

public interface IXmppAgent {
    void Notify(string subject, string body);
    OnlineStatus QueryStatus();
}

Given this contract we could just proxy the calls. So our mailbox could have a proxy factory like this:

public interface IMailbox {
    TRecipient For<TRecipient>(string id);
}

allowing us to send messages like this:

var proxy = _mailbox.For<IXmppAgent>("foo@bar.com");
proxy.Notify("hey", "how'd you like that?");
var status = proxy.QueryStatus();

But messaging is supposed to be asynchronous

While this is simple and decoupled, it is implictly synchronous. Sure .Notify could be considered a fire-and-forget message, .QueryStatus definitely blocks. And if we wanted to communicate an error condition like not finding the recipient, we'd have to do it as an exception, moving errors into the synchronous pipeline as well. In order to retain the flexibility of a pure message architecture, we need a result handle that let's us handle results and/or errors via continuation.

My first pass at an API for this resulted in this calling convention:

public interface IMailbox {
    void Send<TRecipient>(string id, Expression<Action<TRecipient>> message);
    Result SendAndReceive<TRecipient>(string id, Expression<Action<TRecipient>>  message);
    Result<TResponse> SendAndReceive<TRecipient, TResponse>(
        string id,
        Expression<Func<TRecipient, TResponse>>  message
    );
}

transforming the messaging code to this:

_mailbox.Send<IXmppAgent>("foo@bar.com",a => a.Notify("hey", "how'd you like that?"));
var result = _mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
    "foo@bar.com",
    a => a.QueryStatus()
);

I'm using MindTouch Dream's Result<T> class here, instead of Task<T>, primarily because it's battle tested and I have not properly tested Task under mono yet, which is where this code has to run. In this API, .Send is meant for fire-and-forget style messaging while .SendAndReceive provides a result handle — and if Void were an actual Type, we could have dispensed with the overload. The result handle has the benefit of letting us choose how we want to deal with the asynchronous response. We could simply block:

var status = _mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
        "foo@bar.com",
        a => a.QueryStatus())
    .Wait();
Console.WriteLine("foo@bar.com status:", status);

or we could attach a continuation to handle it out of band of the current execution flow:

_mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
        "foo@bar.com",
        a => a.QueryStatus()
    )
    .WhenDone(r => {
        var status = r.Value;
        Console.WriteLine("foo@bar.com status:", status);
    });

or we could simply suspend our current execution flow, by invoking it from a coroutine:

var status = OnlineStatus.Offline;
yield return _mailbox.SendAndReceive<IXmppAgent, OnlineStatus>(
        "foo@bar.com",
        a => a.QueryStatus()
    )
    .Set(x => status = x);
Console.WriteLine("foo@bar.com status:", status);

Regardless of completion strategy, we have decoupled the handling of the result and error conditions from the message recipient's behavior, which is the true goal of the message passing decoupling of the actor system.

Improving usability

Looking at the signatures there are two things we can still improve:

1. If we send a lot of messages to the same recipient, the syntax is a bit repetitive and verbose
2. Because we need to specify the recipient type, we also have to specify the return value type, even though it should be inferable

We can address both of these, by providing a factory method for a typed mailbox:

public interface IMailbox {
    IMailbox<TRecipient> To<TRecipient>(string id);
}

public interface IMailbox<TRecipient> {
    void Send(Expression<Action<TRecipient>> message);
    Result SendAndReceive<TResponse>(Expression<Action<TRecipient>>  message);
    Result<TResponse> SendAndReceive<TResponse>(
        Expression<Func<TRecipient, TResponse>>  message
    );
}

which let's us change our messaging to:

var actorMailbox = _mailbox.To<IXmppAgent>("foo@bar.com");
actorMailbox.Send(a => a.Notify("hey", "how'd you like that?"));
var result2 = actorMailbox.SendAndReceive(a => a.QueryStatus());

// or inline
_mailbox.To<IXmppAgent>("foo@bar.com")
    .Send(a => a.Notify("hey", "how'd you like that?"));
var result3 = _mailbox.To<IXmppAgent>("foo@bar.com")
    .SendAndReceive(a => a.QueryStatus());

I've included the inline version because it is still more compact than the explicit version, since it can infer the result type.

Supporting Remote Actors

The reason the mailbox uses Expression instead of raw Action and Func is that at any point an actor we're sending a message to could be remote. The moment we cross process boundaries, we need to serialize the message. That means we need to be able to programatically inspect the inspection, and build a serializable AST as well as serialize the captured data members used in the expression.

Since we're talking serializing, inspecting the expression also allows us to verify that all members are immutable. For value types, this is easy enough, but DTOs would need be prevented from changing so that local vs. remote invocation won't end up with different result just because the sender changed it's copy. We could handle this via serialization at message send time, although this looks like a perfect place to see how well the Freezable pattern works.

This isn't yet another .NET developer defecting to Ruby. I have very little interest in making Ruby my primary language. I've done a couple of RoR projects over the years, nothing serious I admit, but I just don't seem to enjoy it in the way that so many of my peers do. That said, RoR does hit a sweetspot for websites. The site I'm porting has very little in terms of business logic — it's primarily HTML templating with navigation — so this was an exercise to circumvent my mod_mono issues.

I'm a huge C# fanboy, but having worked with ASP.NET MVC for a while I have to admit that the amount of cruft one has to assemble to stay DRY in ASP.NET templating is just not worthwhile. While views can be strongly typed, it's an exercise in frustration trying to write templates generically. Maybe this becomes easier with dynamic usage in MVC3, but i haven't checked it out. What certainly doesn't help is that the MVC team decided to make TemplateHelper internal, turning the addition of helpers in the vein of .DisplayFor or .EditorFor into a major task that still ends up being a pile of hacks. Now I'm not an ASP.NET MVC expert and there's probably a lot of extension points I just don't know about. But the articles on extending it that I have found are usually pages of code. I shouldn't have to become a framework internals expert just to add some generic templating extensibility.

Ok, enough ranting. ASP.NET MVC is still a huge improvement over webforms, but right now I'm watching Manos de Mono and OWIN to see what develops in .NET land for websites there. The ASP.NET stack, in my opinion, is just too heavy for something that should be simple.

So, why RoR instead of node.js, since I claimed that I was going to get serious about javascript this year? Mostly because this port has a deadline, so use what you know applies, and it's a production site, so use known stable tech applies. Another benefit was that RoR uses the same <% %> syntax as webforms views and MVC was clearly heavily inspired by RoR.

I ported the site over 3 nights, maybe 10 hours of cumulative seat time which feels like time well spent. Strategic search and replace got me 80% there, faking Html. for my custom extension in RoR got me another 10%, leaving only 10% for actual new business logic written in ruby. Once I get to more complex business logic for the site I may stick to Ruby, although I know I'll be sorely tempted to write it as REST services in C# on top of Dream.