ILoggable

A place to keep my thoughts on programming

January 16, 2011 geek, mono , , , , ,

nginx+mono vs. apache+mod_mono

I've been using Apache with mod_mono for some ASP.NET MVC2 projects and kept having problems with semaphore arrays being leaked. Under 2.6.7 this even broke xbuild after a while. I then went to 2.8 and 2.8.1, but it didn't stop the leaks. I posted on the mono-devel list and after lack of response simply asked if anyone was actually running ASP.NET under mod_mono, which also elicited no replies. Finally, I posted the problem on stackoverflow, again without any resolution.

The mod_mono problem

The problem manifests itself as a build up of semaphore arrays by the apache process, which is visible via ipcs. When the site is first started the output looks like this:

[root@host ~]# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x01014009 1671168    root       600        52828      48
0x0101400a 1703937    root       600        52828      25
0x0101400c 1736706    root       600        52828      35

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 10616832   apache     600        1
0x00000000 10649601   apache     600        1
0x00000000 10682370   apache     600        1
0x00000000 10715139   apache     600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Eventually it'll look like this:

[root@host ~]# ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x01014009 1671168    root       600        52828      48
0x0101400a 1703937    root       600        52828      25
0x0101400c 1736706    root       600        52828      35

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 10616832   apache     600        1
0x00000000 10649601   apache     600        1
0x00000000 10682370   apache     600        1
...
lots more
...
0x00000000 11141158   apache     600        1
0x00000000 11173927   apache     600        1
0x00000000 11206696   apache     600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

At some point all ASP.NET pages will return blank. No errors, no nothing, .NET logging reports normal behavior, but no content is sent. And you can restart the mono processes and apache all you want, it won't come back. Sorry.

What does work is to remove all semaphore arrays via ipcrm and restart apache. For the time being i've had a script in cron that did this:

/usr/bin/ipcrm sem $(/usr/bin/ipcs -s | grep apache | awk '{print$2}');
/etc/init.d/httpd restart;

Unfortunately, the leaking semaphores are somehow related to traffic, so eventually i'd either have to increase the frequency of the restart script or make it more intelligent. I opted for neither and decided to try out nginx+fastcgi+mono.

Installing and Configuring nginx+fastcgi+mono

Like my mono 2.8.1 install, I'm doing this on an Amazon Linux AMI 1.0. And like that article, this isn't so much a recipe than a log of my actions. Note that this was done after the 2.8.1 install from source so there might be dependencies i'm not mentioning since they'd already been addressed.

First, the simple part, the yum install:

yum install nginx

Append the below to /etc/nginx/fastcig_params:

# mono
fastcgi_param PATH_INFO "";
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

Now, let's assume there's an apache vhost config in /etc/httpd/conf.d/foobar.conf that looks like this:

Include conf.d/mod_mono.conf

MonoSetEnv MONO_DISABLE_SHM=1

<VirtualHost *:80>
  ServerName www.foobar.com
  ServerAdmin admin@foobar.com
  DocumentRoot /foobar/http/www

  ErrorLog      /foobar/log/www/error_log
  CustomLog     /foobar/log/www/access_log common

  MonoServerPath www.foobar.com "/opt/mono-2.8.1/bin/mod-mono-server2"
  MonoDebug www.foobar.com true
  MonoApplications www.foobar.com "/://foobar/http/www"
  MonoAutoApplication disabled
  AddHandler mono .aspx .ascx .asax .ashx .config .cs .asmx .axd

  <Location "/">
    Allow from all
    Order allow,deny
    MonoSetServerAlias www.foobar.com
    SetHandler mono
  </Location>
</VirtualHost>

The equivalent nginx config in /etc/nginx/conf.d/foobar.conf would look like this:

server {
  server_name  www.foobar.com;
  access_log   /foobar/log/www/nginx.access.log;

  location / {
    root /foobar/http/www;
    index index.html index.htm default.aspx Default.aspx;
    fastcgi_index /;
    fastcgi_pass 127.0.0.1:9000;
    include /etc/nginx/fastcgi_params;
  }
}

Now we need to set up the fastcgi server:

fastcgi-mono-server4 /applications=/:/foobar/http/www/ /socket=tcp:127.0.0.1:9000

and finally we can start nginx:

/etc/init.d/nginx start

Voila, ASP.NET MVC2 under nginx. This may have other issues, but i have not yet observed them, so this seems to be a way to get around the mod_mono issues.

Of course that's a bit cumbersome. What we really need is an init script so we can start and stop teh fastcgi server like other services:

#!/bin/sh

# chkconfig:   - 85 15
# description:  Fast CGI mono server
# processname: fastcgi-mono-server2.exe

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/mono-2.8.1/bin
DESC=fastcgi-mono-server2

WEBAPPS="/:/foobar/http/www/"
LISTENER="tcp:127.0.0.1:9000"

MONOSERVER=/opt/mono-2.8.1/bin/fastcgi-mono-server2
MONOSERVER_PID=$(ps auxf | grep "${LISTENER}" | grep -v grep | awk '{print $2}')

case "$1" in
        start)
                if [ -z "${MONOSERVER_PID}" ]; then
                        echo "starting mono server"
                        ${MONOSERVER} /applications=${WEBAPPS} /socket=${LISTENER} &
                        echo "mono server started"
                else
                        echo ${WEBAPPS}
                        echo "mono server is running"
                fi
        ;;
        stop)
                if [ -n "${MONOSERVER_PID}" ]; then
                        kill ${MONOSERVER_PID}
                        echo "mono server stopped"
                else
                        echo "mono server is not running"
                fi
        ;;
esac

exit 0

And now we can start and stop fastcgi properly.

After all that, i'll likely not use it

While this takes care of my ASP.NET troubles, it now means that I'd have to migrate the various php packages over as well. WordPress is no problem, but OpenCart would be a bit of hacking, which is really the last thing I want to do when it comes to ecom.I thought about running both nginx and apache and using one to proxy the sites on the other (since EC2 won't let me attach multiple IPs to a single host), but decided against that as well, since it would just be a hack of a different color. There's also the option of running fastcgi against apache, but I've not found any docs on how to set up ASP.NET MVC that way, all the existing examples map ASP.NET file extensions to fastcgi, which isn't an option.

Apache is still the most supported solution, so when integrating a number of sites on a single host, it ends up being the best option. It's just that mod_mono doesn't seem to be playing along for me 🙁 So, I hatched a scheme to rid myself of ASP.NET for this site, since it really only has trivial business logic and I have a holiday coming up. More on that later.

 

 

6 to “nginx+mono vs. apache+mod_mono”

Trackbacks/Pingbacks

  1. […] nginx+mono vs. apache+mod_mono […]

  1. Porting ASP.NET MVC to Ruby on Rails says...

    […] nginx+mono vs. apache+mod_mono […]

  2. Nicklas Overgaard says...

    Hi,
    My company has launched an Apache + mod_mono site for a customer about a month ago. We have been forced to switch back to a Windows server with .net as Mono was memory leaking HEAVILY. The site is currently handling ~40.000 requests a day, but we have to restart apache every 18 to 24 hours – every time we restart it has eaten all ram on the server… It's dualcore with 6 Gb physical memory and 2 gb swap.
    I have also contacted the mono-dev-list in vain to find a solution to the problem.
    I also performed a benchmark of the server with ApacheBench, and already at 15 concurrent users the response time is 4 seconds on average (compared to 200 ms on a comparable IIS server). At 20 concurrent request, the time lies somewhere between 40 to 100 seconds(!) – again the IIS server does not care at all.
    Just wanted to support the fact that Apache + mod_mono is not production ready yet.
    How is your nginx + mono fast CGI site holding up? Is it a viable solution, or should we just stick with Windows.

  3. arne says...

    I actually fixed my apache issues as detailed here: http://www.claassen.net/geek/blog/2011/01/fixing-leaking-semaphores-with-mod_mono.html

    I've been running apache+mod_mono since January on mono 2.8.1 on an AWS micro instance and have had no memory issues. Admittedly it's a low traffic site (hence the micro instance).

    But i just hit it apache bench -c 50 -n 1000 and got this:

    Total transferred:      9325812 bytes
    HTML transferred:       8951756 bytes
    Requests per second:    15.89 [#/sec] (mean)
    Time per request:       3147.318 [ms] (mean)
    Time per request:       62.946 [ms] (mean, across all concurrent requests)
    Transfer rate:          144.68 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        1   11 113.8      1    1502
    Processing:    24 2818 7185.2    499   36397
    Waiting:       20 2799 7143.7    495   36378
    Total:         25 2829 7215.9    500   37355
     

    Load shot up, but memory didn't budge. When I ran it with -c 20 the avg was 9ms, so clearly the -c 50 is stretching the micro instance.

    Of course that doesn't prove anything, since what your site is doing vs. what mine is doing (basically MVC for templating more than anything else), so you are likely hitting some subsystem that's got some memory issues. But overall the plumbing seems to be sound with mono.

     

     

  4. Nicklas Overgaard says...

    Nice that you actually managed to fix your issues. Unfortunately our problem is not with the semaphores….
     
    It's a webshop, which is using the InProc session store extensively. We have had random runtime errors where the customers order is suddenly lost and stuff that's suddenly null references which worked the minute before. Maybe the entire issue resides in the session handler, who knows.
     
    Also we are running with mono 2.10.1, maybe there is a regression somewhere inbetween 2.8 and 2.10. It would be very interesting to see how your site behaves under mono 2.10, if you have some time to spare 🙂
     
    Do you have any tips and tricks on how to make a memory profile of our running production system, so we can get some hints on what is going on?

  5. Rui Marques says...

    Currenlty using mono 2.10.5 on top of nginx. No memory issues detected yet.

  6. kripper says...

    Nicklas, InProc sessions are buggy. I fixed the StateSessionHandler some months ago and my patch is now in the main mono trunk. Concurrent access was killing sessions randomly due to race conditions (mutex salat). This was happening because Apache allows  concurrent access to a same session (at least, for images), thus one thread was destroying things while other was reading them.

Leave a comment