I've been using Apache with mod_mono for some ASP.NET MVC2 projects and kept having problems with semaphore arrays being leaked. Under 2.6.7 this even broke xbuild after a while. I then went to 2.8 and 2.8.1, but it didn't stop the leaks. I posted on the mono-devel list and after lack of response simply asked if anyone was actually running ASP.NET under mod_mono, which also elicited no replies. Finally, I posted the problem on stackoverflow, again without any resolution.

The mod_mono problem

The problem manifests itself as a build up of semaphore arrays by the apache process, which is visible via ipcs. When the site is first started the output looks like this:

[root@host ~]# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x01014009 1671168    root       600        52828      48
0x0101400a 1703937    root       600        52828      25
0x0101400c 1736706    root       600        52828      35

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 10616832   apache     600        1
0x00000000 10649601   apache     600        1
0x00000000 10682370   apache     600        1
0x00000000 10715139   apache     600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Eventually it'll look like this:

[root@host ~]# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x01014009 1671168    root       600        52828      48
0x0101400a 1703937    root       600        52828      25
0x0101400c 1736706    root       600        52828      35

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 10616832   apache     600        1
0x00000000 10649601   apache     600        1
0x00000000 10682370   apache     600        1
...
lots more
...
0x00000000 11141158   apache     600        1
0x00000000 11173927   apache     600        1
0x00000000 11206696   apache     600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

At some point all ASP.NET pages will return blank. No errors, no nothing, .NET logging reports normal behavior, but no content is sent. And you can restart the mono processes and apache all you want, it won't come back. Sorry.

What does work is to remove all semaphore arrays via ipcrm and restart apache. For the time being i've had a script in cron that did this:

/usr/bin/ipcrm sem $(/usr/bin/ipcs -s | grep apache | awk '{print$2}');
/etc/init.d/httpd restart;

Unfortunately, the leaking semaphores are somehow related to traffic, so eventually i'd either have to increase the frequency of the restart script or make it more intelligent. I opted for neither and decided to try out nginx+fastcgi+mono.

Installing and Configuring nginx+fastcgi+mono

Like my mono 2.8.1 install, I'm doing this on an Amazon Linux AMI 1.0. And like that article, this isn't so much a recipe than a log of my actions. Note that this was done after the 2.8.1 install from source so there might be dependencies i'm not mentioning since they'd already been addressed.

First, the simple part, the yum install:

yum install nginx

Append the below to /etc/nginx/fastcig_params:

# mono
fastcgi_param PATH_INFO "";
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

Now, let's assume there's an apache vhost config in /etc/httpd/conf.d/foobar.conf that looks like this:

Include conf.d/mod_mono.conf

MonoSetEnv MONO_DISABLE_SHM=1

<VirtualHost *:80>
  ServerName www.foobar.com
  ServerAdmin admin@foobar.com
  DocumentRoot /foobar/http/www

  ErrorLog      /foobar/log/www/error_log
  CustomLog     /foobar/log/www/access_log common

  MonoServerPath www.foobar.com "/opt/mono-2.8.1/bin/mod-mono-server2"
  MonoDebug www.foobar.com true
  MonoApplications www.foobar.com "/://foobar/http/www"
  MonoAutoApplication disabled
  AddHandler mono .aspx .ascx .asax .ashx .config .cs .asmx .axd

  <Location "/">
    Allow from all
    Order allow,deny
    MonoSetServerAlias www.foobar.com
    SetHandler mono
  </Location>
</VirtualHost>

The equivalent nginx config in /etc/nginx/conf.d/foobar.conf would look like this:

server {
  server_name  www.foobar.com;
  access_log   /foobar/log/www/nginx.access.log;

  location / {
    root /foobar/http/www;
    index index.html index.htm default.aspx Default.aspx;
    fastcgi_index /;
    fastcgi_pass 127.0.0.1:9000;
    include /etc/nginx/fastcgi_params;
  }
}

Now we need to set up the fastcgi server:

fastcgi-mono-server4 /applications=/:/foobar/http/www/ /socket=tcp:127.0.0.1:9000

and finally we can start nginx:

/etc/init.d/nginx start

Voila, ASP.NET MVC2 under nginx. This may have other issues, but i have not yet observed them, so this seems to be a way to get around the mod_mono issues.

Of course that's a bit cumbersome. What we really need is an init script so we can start and stop teh fastcgi server like other services:

#!/bin/sh

# chkconfig:   - 85 15
# description:  Fast CGI mono server
# processname: fastcgi-mono-server2.exe

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/mono-2.8.1/bin
DESC=fastcgi-mono-server2

WEBAPPS="/:/foobar/http/www/"
LISTENER="tcp:127.0.0.1:9000"

MONOSERVER=/opt/mono-2.8.1/bin/fastcgi-mono-server2
MONOSERVER_PID=$(ps auxf | grep "${LISTENER}" | grep -v grep | awk '{print $2}')

case "$1" in
        start)
                if [ -z "${MONOSERVER_PID}" ]; then
                        echo "starting mono server"
                        ${MONOSERVER} /applications=${WEBAPPS} /socket=${LISTENER} &
                        echo "mono server started"
                else
                        echo ${WEBAPPS}
                        echo "mono server is running"
                fi
        ;;
        stop)
                if [ -n "${MONOSERVER_PID}" ]; then
                        kill ${MONOSERVER_PID}
                        echo "mono server stopped"
                else
                        echo "mono server is not running"
                fi
        ;;
esac

exit 0

And now we can start and stop fastcgi properly.

After all that, i'll likely not use it

While this takes care of my ASP.NET troubles, it now means that I'd have to migrate the various php packages over as well. WordPress is no problem, but OpenCart would be a bit of hacking, which is really the last thing I want to do when it comes to ecom.I thought about running both nginx and apache and using one to proxy the sites on the other (since EC2 won't let me attach multiple IPs to a single host), but decided against that as well, since it would just be a hack of a different color. There's also the option of running fastcgi against apache, but I've not found any docs on how to set up ASP.NET MVC that way, all the existing examples map ASP.NET file extensions to fastcgi, which isn't an option.

Apache is still the most supported solution, so when integrating a number of sites on a single host, it ends up being the best option. It's just that mod_mono doesn't seem to be playing along for me So, I hatched a scheme to rid myself of ASP.NET for this site, since it really only has trivial business logic and I have a holiday coming up. More on that later.

My current dev machine is running XP 64, which is a first for me. In the default setup IIS was not installed, so I went through Add/Remove Programs and installed it, which gave me IIS 6. This in turn has several tabs for ASP.NET, but try as you might none of these are what actually turns on ASP.NET and you just end up with mysterious 404s on a application enabled directory that’s configured just like the working ASP.NET on your other machine.

Well, it turns out that ASP.NET (even though it shows up in the Properties tabs) is not installed by default and if you go to Web Service Extensions you won’t see it there. So next, track down aspnet_regiis which is in the Framework directory and run

aspnet_regiis -i

Then go back to IIS Manager -> Web Service Extensions where ASP.NET should now be an available extension. Enable it and finally ASP.NET works.

As I mentioned in my update to my last post, the custom URLs break down on postback because the form doesn’t realize where it’s supposed to post back to. To get around this, we need two new classes, a new Page base class and a custom HtmlTextWriter:

public class RewriteBaseClass: System.Web.UI.Page
{
 protected override void Render(HtmlTextWriter writer)
 {
   CustomActionHtmlTextWriter customWriter = new CustomActionHtmlTextWriter(writer);
   base.Render(customWriter);
 }
}

and

public class CustomActionHtmlTextWriter : HtmlTextWriter
{
 public CustomActionHtmlTextWriter(HtmlTextWriter baseWriter) : base(baseWriter)
 {
 }

 public override void WriteAttribute(string name, string value, bool fEncode)
 {
   if( name == "action")
   {
     value = HttpContext.Current.Request.RawUrl;
   }
   base.WriteAttribute (name, value, fEncode);
 }
}

This seems to work, although I’m not yet convinced it’s the best way or without side-effects. Need to play with it a bit more.

One thing I used to do in mod_perl under apache is use the <Location> directive to feed my PerlHandlers instead of using extensions. Not only did that mean that my apps had nice URLs like

foo.com/myapp

but I would usually use paths as arguments (as is now common in RESTful applications) such that

foo.com/myapp/user/add

meant that i was calling the PerlHandler myapp and the PathInfo /user/add could be interpreted as arguments. Much nicer than

foo.com/myapp.pl?mode=user&action=add

or

foo.com/myapp/useradd.pl

On the ASP.NET side, everything always seemed very file system based, giving it an almost CGI feel, even though under the hood it couldn’t have been further from a CGI. Sure you could register your own extensions, but again, extensions and directories — so filesystem.

I figured it must be possible to process request by hand and it turns out to be rather simple: In IIS just map * (not .*) to ASP.NET for your webapp and you can catch every request. And you don’t have to give up on the existing ASPX, ASMX or ASHX infrastructure. Just use HttpContext.RewritePath(string path) to process the incoming requests and send them off to your regular pages or webservices.

By default you loose the path info that you’d receive in the equivalent Apache <Location> request. You can fix this with the following Application_BeginRequest code:

protected void Application_BeginRequest(Object sender, EventArgs e)
{
      int start = Request.ApplicationPath.Length+1;
      string path = Request.Path.Substring(start);
      string[] info = path.Split(new char[] {'/'},2);
      string handler = info[0];
      string pathinfo = "/"+info[1];
      string rewritePath = "~/"+handler+".aspx"+pathinfo;
      this.Context.RewritePath(rewritePath);
}

This will take a URL such as foo.com/myapp/user/add and call myapp.aspx with PathInfo of /user/add.

Update: Ok. So this doesn’t work quite as well as I’d hoped, since roundtrips will change the URL around. While it doesn’t error out, the URL gets ugly again and you have to do some clean-up of your path info. Basically on postback foo.com/myapp/user/add becomes foo.com/myapp/user/add/myapp.aspx.

So somehow the action of the default form needs to be changed (not legal by default). Overriding the renderer or using Javascript seem like options. I’ll post again when I have a working, non-hideous solution.

I’ve been wondering for a while how you could reliably tell if you are currently running under ASP.NET. This is really only of interest to be because of the ThreadSingleton vs. Static Singleton issue. The best way I’ve found so far is:

bool isASP_NET = ( System.Web.HttpContext.Current == null )?false:true;

Kind of annoying, because you have to reference System.Web in your Project, which you wouldn’t otherwise.

I looked through most of classes that come out of the System and mscorlib assemblies but couldn’t find anything good and reliable (i.e. didn’t want to use AppDomain and see if the config file was called web.config.. Sounds like an accident just waiting to happen.) Still, there’s got to be a better way.

So a while back I read an an article about the Client Callback Feature in ASP.NET 2.0, which allows ASP.NET 2.0 to do updates to a page without a full page rountrip to the server. But wouldn’t you know it, it was specific to Internet Explorer. The article ended on a hopeful note with:

Please note that not every browser supports client callbacks, so two new boolean properties named SupportsCallback and SupportsXmlHttp were added to the Request.Browser object. Currently, both properties will return the same value, but the two properties were created because client callbacks might be implemented using a different technique other than the XMLHTTP ActiveX object in the future.

2.0 is still not out, but cross-browser support using XMLHttpRequest is certainly possible to anyone wanting to roll their own code. A very nice example of this can be found here.

ASP.NET 2.0 isn’t out yet. So i figured, i might as well see if the stance for Client Callbacks and cross-browser support has changed since that first article. Info was only vague, but i think the answer is indeed maybe? — at least if this blog post is favorably interpreted… Although this blog did not interpret it as such…

If it’s not in there, I’m sure going to find out how hard it would be to subclass the appropriate classes to create a W3C XMLHttpRequest capable version