Not often I face issues with VMware products that were previously working with flaws and I do consider VMware hypervisor and it’s components a very reliable and robust solution.
When I got into the office this morning vCenter Server service was not running and a manual restart or server reboot did not fix the issue. The event viewer showed the following errors:
The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Log directory: C:\Documents and Settings\svc_vmware_vcenter\Local Settings\Application Data\VMware\vpx.
and VPX log showed:
–2009-10-08 08:51:29.222 05276 error ‘App’– Error inserting events: "ODBC error: () – " is returned when executing SQL statement "INSERT INTO VPX_EVENT WITH (ROWLOCK) (EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, CHANGE_TAG_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
The Troubleshooting Process
Following internet feeds I have seen different solutions for the same issue, ranging from user permissioning and database redeployment to full reinstallation of vCenter.
That is the beauty of troubleshooting and being a good savvy engineer (LOL). If I had followed the recommendations from the web I would still be rebuilding the environment.
Just looking at the log files and Windows Event Viewer not much other than a ODCB error and vpdx.exe faulty application could be seen.
At the end of the day SQL2005 ran out of disk space due to database backups being hosted at the server and consuming the whole lot of available disk space. This by itself is a major issue and I have raised that with the database team.
- Don’t trust everything you read on the internet.
- Not all solutions apply to your problem.
- Be careful about choosing the solution as it could potentially make the problem worse.
- Stop and think about your environment, look at diagrams.
- Analyze the problem with a segmented approach.
- Read log files and often enable debug mode, when possible.
- Involve other IT members to get to the solution or at least to the troubleshooting process.