nginx+mono vs. apache+mod_mono
I've been using Apache with mod_mono for some ASP.NET MVC2 projects and kept having problems with semaphore arrays being leaked. Under 2.6.7 this even broke xbuild after a while. I then went to 2.8 and 2.8.1, but it didn't stop the leaks. I posted on the mono-devel list and after lack of response simply asked if anyone was actually running ASP.NET under mod_mono, which also elicited no replies. Finally, I posted the problem on stackoverflow, again without any resolution.
The mod_mono problem
The problem manifests itself as a build up of semaphore arrays by the apache process, which is visible via ipcs. When the site is first started the output looks like this:
[root@host ~]# ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x01014009 1671168 root 600 52828 48
0x0101400a 1703937 root 600 52828 25
0x0101400c 1736706 root 600 52828 35
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 10616832 apache 600 1
0x00000000 10649601 apache 600 1
0x00000000 10682370 apache 600 1
0x00000000 10715139 apache 600 1
------ Message Queues --------
key msqid owner perms used-bytes messages
Eventually it'll look like this:
[root@host ~]# ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x01014009 1671168 root 600 52828 48
0x0101400a 1703937 root 600 52828 25
0x0101400c 1736706 root 600 52828 35
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 10616832 apache 600 1
0x00000000 10649601 apache 600 1
0x00000000 10682370 apache 600 1
_...
lots more
..._
0x00000000 11141158 apache 600 1
0x00000000 11173927 apache 600 1
0x00000000 11206696 apache 600 1
------ Message Queues --------
key msqid owner perms used-bytes messages
At some point all ASP.NET pages will return blank. No errors, no nothing, .NET logging reports normal behavior, but no content is sent. And you can restart the mono processes and apache all you want, it won't come back. Sorry.
What does work is to remove all semaphore arrays via ipcrm and restart apache. For the time being i've had a script in cron that did this:
Unfortunately, the leaking semaphores are somehow related to traffic, so eventually i'd either have to increase the frequency of the restart script or make it more intelligent. I opted for neither and decided to try out nginx+fastcgi+mono.
Installing and Configuring nginx+fastcgi+mono
Like my mono 2.8.1 install, I'm doing this on an Amazon Linux AMI 1.0. And like that article, this isn't so much a recipe than a log of my actions. Note that this was done after the 2.8.1 install from source so there might be dependencies i'm not mentioning since they'd already been addressed.
First, the simple part, the yum install:
Append the below to /etc/nginx/fastcig_params:
# mono
fastcgi_param PATH_INFO "";
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
Now, let's assume there's an apache vhost config in /etc/httpd/conf.d/foobar.conf that looks like this:
Include conf.d/mod_mono.conf
MonoSetEnv MONO_DISABLE_SHM=1
<VirtualHost \*:80>
ServerName www.foobar.com
ServerAdmin admin@foobar.com
DocumentRoot /foobar/http/www
ErrorLog /foobar/log/www/error_log
CustomLog /foobar/log/www/access_log common
MonoServerPath www.foobar.com "/opt/mono-2.8.1/bin/mod-mono-server2"
MonoDebug www.foobar.com true
MonoApplications www.foobar.com "/://foobar/http/www"
MonoAutoApplication disabled
AddHandler mono .aspx .ascx .asax .ashx .config .cs .asmx .axd
<Location "/">
Allow from all
Order allow,deny
MonoSetServerAlias www.foobar.com
SetHandler mono
</Location>
</VirtualHost>
The equivalent nginx config in /etc/nginx/conf.d/foobar.conf would look like this:
server {
server_name www.foobar.com;
access_log /foobar/log/www/nginx.access.log;
location / {
root /foobar/http/www;
index index.html index.htm default.aspx Default.aspx;
fastcgi_index /;
fastcgi_pass 127.0.0.1:9000;
include /etc/nginx/fastcgi_params;
}
}
Now we need to set up the fastcgi server:
and finally we can start nginx:
Voila, ASP.NET MVC2 under nginx. This may have other issues, but i have not yet observed them, so this seems to be a way to get around the mod_mono issues.
Of course that's a bit cumbersome. What we really need is an init script so we can start and stop teh fastcgi server like other services:
#!/bin/sh
# chkconfig: - 85 15
# description: Fast CGI mono server
# processname: fastcgi-mono-server2.exe
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/mono-2.8.1/bin
DESC=fastcgi-mono-server2
WEBAPPS="/:/foobar/http/www/"
LISTENER="tcp:127.0.0.1:9000"
MONOSERVER=/opt/mono-2.8.1/bin/fastcgi-mono-server2
MONOSERVER_PID=$(ps auxf | grep "${LISTENER}" | grep -v grep | awk '{print $2}')
case "$1" in
start)
if [ -z "${MONOSERVER_PID}" ]; then
echo "starting mono server"
${MONOSERVER} /applications=${WEBAPPS} /socket=${LISTENER} &
echo "mono server started"
else
echo ${WEBAPPS}
echo "mono server is running"
fi
;;
stop)
if [ -n "${MONOSERVER_PID}" ]; then
kill ${MONOSERVER_PID}
echo "mono server stopped"
else
echo "mono server is not running"
fi
;;
esac
exit 0
And now we can start and stop fastcgi properly.
After all that, i'll likely not use it
While this takes care of my ASP.NET troubles, it now means that I'd have to migrate the various php packages over as well. Wordpress is no problem, but OpenCart would be a bit of hacking, which is really the last thing I want to do when it comes to ecom.I thought about running both nginx and apache and using one to proxy the sites on the other (since EC2 won't let me attach multiple IPs to a single host), but decided against that as well, since it would just be a hack of a different color. There's also the option of running fastcgi against apache, but I've not found any docs on how to set up ASP.NET MVC that way, all the existing examples map ASP.NET file extensions to fastcgi, which isn't an option.
Apache is still the most supported solution, so when integrating a number of sites on a single host, it ends up being the best option. It's just that mod_mono doesn't seem to be playing along for me :( So, I hatched a scheme to rid myself of ASP.NET for this site, since it really only has trivial business logic and I have a holiday coming up. More on that later.