Using vMA As Your ESXi Syslog Server

This is something I did a while ago, but it came to my attention that people didn't; a) Know that it's recommended to use a syslog server with ESXi b) You could use an application built in to vMA called vilogger.

Although it is stated in The Architecture of VMware ESXi…..

Because the in-memory file system does not persist when the power is shut down, log files do not survive a reboot. ESXi has the ability to configure a remote syslog server, enabling you to save all log information on an external system. 

…..it is not a well known fact. So that is partly the reason for writing the post. The other reason is to introduce you to vilogger, which is part of the vMA. Of course you can use which ever syslog server you wish,  if you plan to use your own, be sure to checkout Managing VMware ESXi page #68 to view the configuration steps.

I'm not going to take you through the steps of installing vMA, nor am I going to tell you all about what the vMA (vSphere Manage Assistant) does. If you want to read more about that please find the relevant links in the Sources section at the bottom of the page. But I am going to take you through the steps I took to use vMA as my ESXi syslog server.

First of all, download the vMA from here. Import and configure it using the steps in the vSphere Management Assistant Guide

Changing vMA localtime settings to avoid incorrect log timestamps

When vMA collects the logs from your ESXi Host, sometimes the logs have the ESXi Host timestamp and sometimes they will have the vMA Localtime timestamp. I'm not exactly sure why this happens, but it does. (You may or may not know that ESXi uses UTC as its timezone when it timestamps the logs. You can read more about that here. VMware have told me this cannot be changed.) 

So to get around the issue of differing timestamps in your collected logs we need to change the localtime on the vMA to UTC. This can be done using the following steps:

  • sudo rm /etc/localtime
  • sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime

If you use NTP to sync your environments time, it might be worth adding in your ntp servers onto the vMA.

  • sudo nano /etc/ntp.conf
  • Add in your ntp servers under the heading: # Use public servers from the pool.ntp.org project.
  • Configure ntpd to start on reboot: sudo /sbin/chkconfig ntpd on
  • Restart ntpd: sudo /sbin/service ntpd restart
  • Make sure your ntp servers are reachable: sudo ntpq -p 

Whilst your at it you could change your keyboard for those of you outside the US.

  • sudo vi /etc/sysconfig/keyboard
    •  change KEYTABLES=”us” to use the keyboard you have, for example: KEYTABLES="en"

Adding a second Hard Disk to store your logs

Because of the amount of ESXi Hosts I have to manage I decided that I would add a second Virtual Disk to house my logs.  You may not need to do this if you only have 1 or two Hosts.

  • Add a second Hard Disk to the vMA VM
  • PowerOn vMA
  • Login using vi-admin
  • Format the new Disk with the following command: fdisk /dev/sdb
    • Use the n command to create a new partition
    • Use the p command to make the new partition a primary partition
    • Press 1 to make it partition #1
    • Use the default for the First Cylinder
    • Use the default for the Last Cylinder
    • Use the p command to verify the partition table
    • Use the w command to write the partition table to the hard disk
  • Press Enter a couple of times to confirm it's finished and return to the command prompt
  • Now we need to format the partition using the follow command: sudo mkfs -t ext3 /dev/sdb1

Now we have our new disk ready, we now need to mount it.

  • Edit /etc/fstab using the following command: nano /etc/fstab
  • Enter in the following line: /dev/sdb1               /var/log/syslog            ext3    defaults,auto        1 2
  • Use Cntrl+X then Y to save the file
  • Next: cd /var/log/ then mkdir syslog 
  • Change the owner of the /syslog dir: sudo chown vi-admin:root /var/log/syslog
  • finally mount /var/log/syslog to mount the disk

Ok, now our new Disk is mounted, we need to tell the vilogger application to store your logfiles there. The default location is /var/log/vmware so we need to change it.

  • Edit the vilogger config file using: nano /etc/vmware/viconfig/vilogdefaults.xml (or /etc/vmware/vMA/vMA.conf in v4.1)
  • Change the file to match this: <location>/var/log/syslog</location> (This appears twice, change both)
  • Use Cntrl+X then Y to save the file
  • Restart vilogger by: service vmware-vilogd restart

Now vilogger is set to store your ESXi logfiles onto your new disk. 

Configure vMA to collect your logs

Next we configure vMA and vilogger to collect your ESXi logfiles.

  • Connect your ESXi Host to vMA using the following command: sudo vifp addserver <FQDN of ESXi Host> 
  • Enter the Root Password for the Host
  • Once the Host has been added you can double check by running the following command: vifp listservers
  • Now set vilogger to start collecting your logs: vilogger enable –server <FQDN of ESXi Host>  –numrotation 20 –maxfilesize 10 –collectionperiod 10

vilogger

You will see the above displayed, showing that vilogger is now collecting logs from your  Host.

Values –numrotation 20 –maxfilesize 10 –collectionperiod 10 can be changed to suit your needs. If you just run the vilogger –help command you will see what options and values are available.

Viewing your logs

Now your vMA is collecting your logs you can view them by: dir /var/log/syslog/<FQDN of Host> You can view your logs in real time if you wish, read more about there here.

  • An example: tail -f /var/log/syslog/<FQDN of Host>/vpxa.log

Sources:

  • Thanks for the info, I did have to change the owner of /var/log/syslog in order for vilogd to stay running: sudo chown vi-admin:root /var/log/syslog. The vilogd log was not very helpful though (/var/log/vmware/vima/vilogd.log).

    And an easy way to have it enable logging for all servers that have been added to the vMA is to use a for loop:
    for SERVER in $(vifp listservers | awk '{print($1)}'); do vilogger enable –server $SERVER –numrotation 20 –maxfilesize 10 –collectionperiod 10; done

  • Craig

    I get the following error when trying to enable logging for one of my ESXi hosts

    Error: Unable to communicate with vmware-vilogd daemon.

    have ran through the steps again and pretty sure I haven't missed anything.

    Anyone help ?

  • Hi Craig, email me via the contact page and I'll try and help.

    Simon

  • Hi Chris, cheers for the feedback. As soon as I get a moment to test, I'll see if I needed to chmod that directory.

    Simon

  • Cragdoo

    email sent as requested. Many thanks

  • To get around the error Craig has been experiencing, try:

    sudo /etc/init.d/vmware-vilogd stop
    sudo /etc/init.d/vmware-vilogd start

    Simon

  • Matt Liebowitz

    Chris is right – I had to run the chown command to make this work as well.

  • Thanks Matt, I've added the owner change into the steps above.

    Simon

  • Larry

    For osme reason, my esxi log are getting dump to the var/log/vmware dir instead of the syslog dir. The folders get created in the vmware folder.

    I have the settings for the vMA.conf file pointing to the syslog folder. Any help would be kindly appreciated.

    Larry

  • JD

    The vMA.conf file is a little different than previously … there's multiple <location> stanzas. The one you want to modify is for vMALogCollector:

    <vMALogCollector>
    <collectionPeriod>10</collectionPeriod>
    <numRotation>5</numRotation>
    <maxFileSize>5</maxFileSize>
    <location>/logs</location>
    </vMALogCollector>

    JD

  • Larry

    Thank you for your reply. I figured it out. in adding the “/var/log/syslog” path I accidentally copied and pasted the string at the top of the XML file. Therefore, the system ignored the file and went with defaults. When Irealized that my XML file was not structured correctly and removed the bad data, it started to work as expected.

    Thanks again!

    Larry

  • Great article and very useful. I've just used it to setup vMA on my company's VMware cluster.

  • Thanks for sharing this instructional guide.

  • Great guide.

  • mickeyO

    Hi All,

    Anyone see the following message in vilogd.log? This only happens when I enable logging for our vcentres, the esxi hosts are logging with no issues.

    [root@cmswg-vma001v vma]# tail -f vilogd.log
    [2011-01-26 15:09:42.657 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:42.710 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:42.764 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:42.817 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:42.870 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:42.923 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:42.977 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.030 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.083 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.136 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.189 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.242 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.295 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.348 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.401 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs
    [2011-01-26 15:09:43.455 43CB0940 error ‘App’] Unable to genrate key to collect vpxd logs

  • Yann Bizeul

    There is something I don’t get.
    This setup looks like “polling” to me, and I have the feeling that ESX is not redirecting its logs to the vMA using syslog.
    Basically, I’d like to redirect synchronously all logs to the vMA, so I get green light in VMware HealthAnalyzer about “Use remote syslog logging to improve manageability”

  • Oscar

    as soon as I enabled the log file my VMA crashed with memory issue and wont even let me to disable them back
    just be careful and backup before you test this command

  • Yann Bizeul

    This is weird. Kernel panic ?
    You should be able to revert, at least y disconnecting the vMA from the network so that it do no longer fetch logs.

  • logiboy123

    How do you size your syslog drive for holding the data? Is there a sizing estimator or calculator or a per host recommendation?

  • Hi didn’t make any calculations, I just put on a 500GB HDD. That was more than plenty for the 40-50 Hosts we had at the time.

  • Jamie

    I have enabled logging for my two ESXi 4.1 hosts but when I run the ‘vilogger list’ command it shows scheduled and then failed for each log on each host.

    I have tried to remove and re-add the hosts into vifp but this hasn’t made any difference.

    How do I troubleshoot this error?

  • Nitin Singh

    excellent approach for setting vMA .. this is the first time that I have seen all minor things to be kept in mind while installing this appliance. many many thanks!

  • Anonymous

    Help! I’m getting some problems with this approach generating wayyy too many vim.event.UserLoginSessionEvent
    vim.event.UserLogoutSessionEvent entries in my vCenter databases. Is there a way to prevent these sessions from being logged in the vCenter DB? It’s causing us to go over the 4GB limit for SQL Express and causing headaches to the point where I’m about to disable this and look for a cleaner solution.

  • Alecia

    Great directions however I am having a problem. For the host I set up in ‘vilogger list’ I see: ‘Connection Failure’ for hostd, messages, and vpxa. The location is set properly as /var/log/syslog/. Any ideas?

  • Did you manage to add your host via “vifp addserver” ok?

  • Alecia

    Yep!

  • Teamfrown

    Actually, an easier way to enable for all added vifp servers is to type; “vilogger enable” without specifying any hosts, it will enable them all automatically….

  • I didn’t know that, is that new with 4.1?

  • Tom

    Actually in vMA.conf the location setting must be changed THREE times — the third one is especially important since it’s what moves the host logs into the new /syslog location.  HTH Tom

  • gustano

    “”Because the in-memory file system does not persist when the power is
    shut down, log files do not survive a reboot. ESXi has the ability to
    configure a remote syslog server, enabling you to save all log
    information on an external system. “”

    Is this also true for ESXi 5.0???

  • Rory

    vilogger is deprecated in vMA 5, instead you can configure your vCenter
    Server (or other Windows machine) as syslog server using the installer
    disc http://www.virtuallyghetto.com/2011/07/free-linux-windows-syslog-alternatives.html

  • Radu

    The location should be /var/log/syslog/ only. The fqdn is added automatically.

  • Radu

    Did you set the ESXi hosts to send the logs remotely? Config, adv settings, syslog, remote.

 
Get Adobe Flash player