Categories: Gestalt ITVMware

Using vMA As Your ESXi Syslog Server

This is something I did a while ago, but it came to my attention that people didn't; a) Know that it's recommended to use a syslog server with ESXi b) You could use an application built in to vMA called vilogger.

Although it is stated in The Architecture of VMware ESXi…..

Because the in-memory file system does not persist when the power is shut down, log files do not survive a reboot. ESXi has the ability to configure a remote syslog server, enabling you to save all log information on an external system. 

…..it is not a well known fact. So that is partly the reason for writing the post. The other reason is to introduce you to vilogger, which is part of the vMA. Of course you can use which ever syslog server you wish,  if you plan to use your own, be sure to checkout Managing VMware ESXi page #68 to view the configuration steps.

I'm not going to take you through the steps of installing vMA, nor am I going to tell you all about what the vMA (vSphere Manage Assistant) does. If you want to read more about that please find the relevant links in the Sources section at the bottom of the page. But I am going to take you through the steps I took to use vMA as my ESXi syslog server.

First of all, download the vMA from here. Import and configure it using the steps in the vSphere Management Assistant Guide

Changing vMA localtime settings to avoid incorrect log timestamps

When vMA collects the logs from your ESXi Host, sometimes the logs have the ESXi Host timestamp and sometimes they will have the vMA Localtime timestamp. I'm not exactly sure why this happens, but it does. (You may or may not know that ESXi uses UTC as its timezone when it timestamps the logs. You can read more about that here. VMware have told me this cannot be changed.) 

So to get around the issue of differing timestamps in your collected logs we need to change the localtime on the vMA to UTC. This can be done using the following steps:

  • sudo rm /etc/localtime
  • sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime

If you use NTP to sync your environments time, it might be worth adding in your ntp servers onto the vMA.

  • sudo nano /etc/ntp.conf
  • Add in your ntp servers under the heading: # Use public servers from the pool.ntp.org project.
  • Configure ntpd to start on reboot: sudo /sbin/chkconfig ntpd on
  • Restart ntpd: sudo /sbin/service ntpd restart
  • Make sure your ntp servers are reachable: sudo ntpq -p 

Whilst your at it you could change your keyboard for those of you outside the US.

  • sudo vi /etc/sysconfig/keyboard
    •  change KEYTABLES=”us” to use the keyboard you have, for example: KEYTABLES="en"

Adding a second Hard Disk to store your logs

Because of the amount of ESXi Hosts I have to manage I decided that I would add a second Virtual Disk to house my logs.  You may not need to do this if you only have 1 or two Hosts.

  • Add a second Hard Disk to the vMA VM
  • PowerOn vMA
  • Login using vi-admin
  • Format the new Disk with the following command: fdisk /dev/sdb
    • Use the n command to create a new partition
    • Use the p command to make the new partition a primary partition
    • Press 1 to make it partition #1
    • Use the default for the First Cylinder
    • Use the default for the Last Cylinder
    • Use the p command to verify the partition table
    • Use the w command to write the partition table to the hard disk
  • Press Enter a couple of times to confirm it's finished and return to the command prompt
  • Now we need to format the partition using the follow command: sudo mkfs -t ext3 /dev/sdb1

Now we have our new disk ready, we now need to mount it.

  • Edit /etc/fstab using the following command: nano /etc/fstab
  • Enter in the following line: /dev/sdb1               /var/log/syslog            ext3    defaults,auto        1 2
  • Use Cntrl+X then Y to save the file
  • Next: cd /var/log/ then mkdir syslog 
  • Change the owner of the /syslog dir: sudo chown vi-admin:root /var/log/syslog
  • finally mount /var/log/syslog to mount the disk

Ok, now our new Disk is mounted, we need to tell the vilogger application to store your logfiles there. The default location is /var/log/vmware so we need to change it.

  • Edit the vilogger config file using: nano /etc/vmware/viconfig/vilogdefaults.xml (or /etc/vmware/vMA/vMA.conf in v4.1)
  • Change the file to match this: <location>/var/log/syslog</location> (This appears twice, change both)
  • Use Cntrl+X then Y to save the file
  • Restart vilogger by: service vmware-vilogd restart

Now vilogger is set to store your ESXi logfiles onto your new disk. 

Configure vMA to collect your logs

Next we configure vMA and vilogger to collect your ESXi logfiles.

  • Connect your ESXi Host to vMA using the following command: sudo vifp addserver <FQDN of ESXi Host> 
  • Enter the Root Password for the Host
  • Once the Host has been added you can double check by running the following command: vifp listservers
  • Now set vilogger to start collecting your logs: vilogger enable –server <FQDN of ESXi Host>  –numrotation 20 –maxfilesize 10 –collectionperiod 10

You will see the above displayed, showing that vilogger is now collecting logs from your  Host.

Values –numrotation 20 –maxfilesize 10 –collectionperiod 10 can be changed to suit your needs. If you just run the vilogger –help command you will see what options and values are available.

Viewing your logs

Now your vMA is collecting your logs you can view them by: dir /var/log/syslog/<FQDN of Host> You can view your logs in real time if you wish, read more about there here.

  • An example: tail -f /var/log/syslog/<FQDN of Host>/vpxa.log

Sources:

Simon Long

View Comments

  • Thanks for the info, I did have to change the owner of /var/log/syslog in order for vilogd to stay running: sudo chown vi-admin:root /var/log/syslog. The vilogd log was not very helpful though (/var/log/vmware/vima/vilogd.log).

    And an easy way to have it enable logging for all servers that have been added to the vMA is to use a for loop:
    for SERVER in $(vifp listservers | awk '{print($1)}'); do vilogger enable --server $SERVER --numrotation 20 --maxfilesize 10 --collectionperiod 10; done

    • Hi Chris, cheers for the feedback. As soon as I get a moment to test, I'll see if I needed to chmod that directory.

      Simon

      • Hi All,

        Anyone see the following message in vilogd.log? This only happens when I enable logging for our vcentres, the esxi hosts are logging with no issues.

        [root@cmswg-vma001v vma]# tail -f vilogd.log
        [2011-01-26 15:09:42.657 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:42.710 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:42.764 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:42.817 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:42.870 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:42.923 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:42.977 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.030 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.083 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.136 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.189 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.242 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.295 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.348 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.401 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs
        [2011-01-26 15:09:43.455 43CB0940 error 'App'] Unable to genrate key to collect vpxd logs

    • Actually, an easier way to enable for all added vifp servers is to type; "vilogger enable" without specifying any hosts, it will enable them all automatically....

  • I get the following error when trying to enable logging for one of my ESXi hosts

    Error: Unable to communicate with vmware-vilogd daemon.

    have ran through the steps again and pretty sure I haven't missed anything.

    Anyone help ?

  • For osme reason, my esxi log are getting dump to the var/log/vmware dir instead of the syslog dir. The folders get created in the vmware folder.

    I have the settings for the vMA.conf file pointing to the syslog folder. Any help would be kindly appreciated.

    Larry

  • The vMA.conf file is a little different than previously ... there's multiple <location> stanzas. The one you want to modify is for vMALogCollector:

    <vMALogCollector>
    <collectionPeriod>10</collectionPeriod>
    <numRotation>5</numRotation>
    <maxFileSize>5</maxFileSize>
    <location>/logs</location>
    </vMALogCollector>

    JD

    • Thank you for your reply. I figured it out. in adding the "/var/log/syslog" path I accidentally copied and pasted the string at the top of the XML file. Therefore, the system ignored the file and went with defaults. When Irealized that my XML file was not structured correctly and removed the bad data, it started to work as expected.

      Thanks again!

      Larry

  • There is something I don't get.
    This setup looks like "polling" to me, and I have the feeling that ESX is not redirecting its logs to the vMA using syslog.
    Basically, I'd like to redirect synchronously all logs to the vMA, so I get green light in VMware HealthAnalyzer about "Use remote syslog logging to improve manageability"

  • as soon as I enabled the log file my VMA crashed with memory issue and wont even let me to disable them back
    just be careful and backup before you test this command

    • This is weird. Kernel panic ?
      You should be able to revert, at least y disconnecting the vMA from the network so that it do no longer fetch logs.

  • How do you size your syslog drive for holding the data? Is there a sizing estimator or calculator or a per host recommendation?

    • Hi didn't make any calculations, I just put on a 500GB HDD. That was more than plenty for the 40-50 Hosts we had at the time.

Leave a Comment
Share
Published by
Simon Long
Tags: ESXisyslogvma
14 years ago

Recent Posts

Google Cloud VMware Engine @ VMworld 2021

Another VMworld is upon us!!! Sadly, it's only virtual again this year. However, that does…

3 years ago

Google Cloud VMware Engine – Learning Resources

As part of my recent move to Google, I'm working on quickly getting up to…

3 years ago

Hey Google!

I am delighted to announce the next chapter in my career. Today is my first…

3 years ago

EP13 – Defending Remotely

In episode thirteen of The VCDX Podcast, I am joined by two special guests who…

3 years ago

Getting Started With Oracle Cloud VMware Solution (OCVS) – Migrating Workloads Using VMware HCX

In my recent ‘Getting started with Oracle Cloud VMware Solution (OVCS)’ post; Getting Started With…

3 years ago

Getting Started With Oracle Cloud VMware Solution (OCVS) – Connecting To An On-Premises Environment

In my recent ‘Getting started with Oracle Cloud VMware Solution (OVCS)’ post; Getting Started With…

3 years ago