RE: mmap errors in libvirt guest

From: Matt Keys 
------------------------------------------------------
I've totally disabled SELinux on the guest. On the host side, I did see an
apparmor profile for the libvirt guests. I put that particular one into
disabled for testing but there was no difference. I've also asked in the
#virt oftc channel, #kvm freenode channel (full of redhat engineers), and
#cacti freenode channel. No responses.

 

 

From: chugalug-bounces@chugalug.org [mailto:chugalug-bounces@chugalug.org]
On Behalf Of Lynn Dixon
Sent: Sunday, June 24, 2012 1:29 AM
To: CHUGALUG
Subject: Re: [Chugalug] mmap errors in libvirt guest

 

Just a suggestion Matt, but have you put SELinux in permissive for
troubleshooting?  It could be some oddball SELinux context giving you
problems.  

On Sun, Jun 24, 2012 at 1:24 AM, Matt Keys  wrote:

So I've got a problem I could use a wizard's help with. I've got a ubuntu
10.04 amd64 kvm host I recently upgraded to 12.04. I took the original 7200
sata drive out and replaced it with a ssd and fresh installed 12.04 on it. I
updated everything to the latest, shut it down, installed the 7200 sata in
sdb and powered it back on. Everything good so far. I copied the kvm guests
and config over to 12.04 and try to fire one up. I ran into a problem there,
the guest xml definitions have a slight change between versions where you
have to search/replace the word "serial" with "console". Once I did that
they fired up just fine. 

 

One of the guests is a centos 6.2 32bit running nagios/cacti. Cacti (version
0.8.7g, spine 0.8.7 from built from source, and rrdtool 1.3.8 from centos
default repos) was graphing the cpu/mem/net/disks of the guest and other
devices. Since the drive geometry changed, I just blew away the old
device/graphs and recreated them. The rrd files weren't being created for
some reason, so I touch them and chown the appropriate user/group. It
eventually creates 4 out of 14 graphs on two different SNMP polled devices.
Localhost "Local Linux" graphs all get created fine. For the other 10/14
graphs, it throws this error :

 

ERROR: mmaping file '/var/www/html/cacti/rra/4/27.rrd': Cannot allocate
memory

 

I enable debugging for the graphs, copy/paste the rrdtool command into a
shell script and execute it as root. Same thing. Copy/paste one that works,
check with diff, and they're identical in syntax other than the destination
filename. Permissions and ownership are identical.

 

So yeah, it's a memory allocation error but it's got plenty of free memory
to use. At the time it was around 512MB free, 2GB swap allocated but not
using any of it. I tried throwing an extra gig of ram at it, still no
difference.

 

I began to wonder if it was something with that guest, so I stood up a brand
new centos 6.2 32bit guest and started from scratch. This time with the
latest and greatest cacti, spine, rrdtool all compiled from source. rrdtool
has a configure option of using mmap or not. With mmap it gave me the same
error. Without it, I'm getting :

 

ERROR: short read while reading header rrd->stat
===============================================================
From: Matt Keys 
------------------------------------------------------
I forgot to mention ksm. Sometime in 10.04 ksm was introduced. I was running
it up until about 2 weeks ago and the box was always patched current, so I'm
pretty sure it was running before the upgrade. I'm certain it is running now
on 12.04 :

 

root        61  1.0  0.0      0     0 ?        SN   Jun21  34:38 [ksmd]

 

. but there's not much ubuntu documentation on the subject. but RH has some
here :