What the Oracle / VMware support statement really means...and why

One issue I am asked about frequently is the Oracle support statement on metalink concerning VMware virtualization of Oracle database servers. I would like to clarify that issue, as much as humanly possible, because the truth, as usual, is far different from the prevailing perception.

First of all, I have to ask your forgiveness. As many of you know, I am an attorney. This is a flaw in my character which I have struggled for years to overcome. As I like to say, I am a recovering lawyer. It's a day to day thing. There are days when I have frequent moments of pomposity, pedantry, arrogance, inflexibility, stubbornness, and all the other negative personality traits we associate with lawyers. Forgive me if I revert into lawyer mode for the purpose of parsing and clarifying the Oracle metalink support statement on VMware. It seems called for somehow. So bear with me.

Note: If you would like to read the Oracle metalink support statement on VMware, the easiest way to do so is here. Or you can go to metalink directly here. The metalink URL obviously requires a metalink account.

The first statement in the metalink support note (249212.1) is the following:

Oracle has not certified any of its products on VMware virtualized environments.

This sentence, while literally true, is very misleading. The reason why the statement is so misleading is that Oracle does not certify things like this. Let me explain what I mean.

VMware virtualization simply provides virtualized hardware upon which an operating system runs. Oracle certifies operating systems, no question about that. But the operating systems running under VMware are typically Oracle supported versions. (Certainly running an Oracle supported OS version under VMware is possible.) For example, I routinely run Oracle Enterprise Linux within a VMware VM.

What Oracle does not certify is the underlying hardware. At least not for the normal database product (typically called Oracle Database 11g). RAC is a different story. I will cover RAC later. Let's stick with the normal single-instance database software for now.

For Oracle Database 11g, there is no requirement for hardware compatibility whatsoever. Search away on the Oracle website for a hardware compatibility list for the database product. You will find none.

Take VMware again. VMware ESX 3.5 provides a virtualized SCSI environment based upon either the BusLogic or the LSILogic SCSI card. These are incredibly common SCSI cards, millions of them having been sold and deployed for many years. From what I can tell, Oracle has never certified this set of SCSI cards, or any other SCSI card for that matter. Ditto for the Intel x86 processor, AMD Opteron, Broadcom NIC, Emulex HBA, or any other of the common hardware components used to run Oracle.

Do you run Oracle on a Dell PowerEdge server? Not certified. HP Proliant? Nope. IBM? You guessed it. Sun? Same.

I think you get the idea.

You might also ask: Does Oracle have a hardware compatibility list for the Oracle Enterprise Linux distribution? Surely that is an OS and would require an HCL of its own?

You would be wrong. OEL is simply a repackaging of Red Hat Enterprise Linux, and thus Oracle simply adopts the Red Hat HCL. This support statement for OEL can be found here. Interestingly Red Hat apparently supports VMware virtualization of its Linux distribution.

What all this means, again, is that the initial statement in the metalink note on VMware support is essentially misleading. The rest of the statement is even more interesting. It reads:

Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

Again, this statement is literally accurate, but also misleading. It sounds very scary. You would think that you will have to duplicate your entire virtualized environment on normal hardware to get Oracle support for any issue. The way this works in practice is very different from what the support statement seems to say.

Assume you call Oracle support with an issue relating to the performance of a query. You have executed a query which contains a CONNECT BY clause. You see very poor performance. You have pulled an explain plan on the query and it looks odd to you. You think the execution of the query is very suboptimal, and you suspect that it could be dramatically improved. You call Oracle's support center to discuss the issue.

Consider: Will Oracle support ask you if your OS is virtualized? I would submit it is very unlikely. Why? OS virtualization is irrelevant to the issue at hand. This is an entirely internal issue within the Oracle kernel, specifically the query execution engine. Parsing a query has to do with issues of table statistics, query syntax, hints, and the like. The OS version simply is not a consideration.

Now, assume you call Oracle support with an issue in which you have executed the service oracleasm listdisks command, and you are seeing no disks in the output. Your Oracle instance has also crashed. The ASM instance is up, but shows no diskgroups available.

Will Oracle support ask you if you are virtualized? You bet. Absolutely. And they should. You have an issue which is intimately associated with the I/O layer of the operating system. This is actually not a database issue at all. It is an I/O subsystem issue. Virtualization may very well be your problem. You should probably call VMware support right now. In fact, I would suspect that if you ran the fdisk -l command from your Linux terminal session, you would see something very similar.

Resolving this issue will require you to interact with VMware as well as Oracle. The point is that in the range of issues, there are those which Oracle can resolve without any interaction with the OS layer, and others where the OS layer is very much an part of the resolution. In this range, the Oracle support statement really says that everyone will do the right thing. Certainly, that is the practical reality.

Now, let's deal with the issue of RAC. Oracle's final statement in the support statement covers this:

NOTE: Oracle has not certified any of its products on VMWare, and use of Oracle products in the RAC environment is also not supported.

The first part of the statement is simply a rehashing of the initial, misleading statement. I suspect that the second part of the statement is a typo and should actually read:

...use of VMware products in the RAC environment is also not supported.

Use of Oracle products in the RAC environment is obviously supported. In any event, in a RAC context, Oracle does have a certified hardware compatibility program, and VMware is not on it. Since VMware software is effectively kryptonite around Oracle (for reasons which I will cover shortly), this is inevitable at this point. I will say that there is no technical reason why VMware virtualization of RAC database servers could not be supported. Certainly, it works.

Now, as to the why. That has to do with Oracle's agenda around the issues of virtualization, licensing and HA.

Taking the virtualization piece first, Oracle wants to own the entire stack. They have made that abundantly clear in many other areas, including the following:

• Elimination of Veritas by destroying Veritas Cluster as a viable product (replaced with CRS), and the same for Veritas File System (replaced by ASM)
• Elimination of Sun by adopting Linux as a low-cost alternative to Solaris. (Ironically, Sun has now been acquired by the company primarily responsible for its downfall, a subject of another future blog post.)
• Attempted elimination of Red Hat by subsuming Red Hat Linux under Oracle Enterprise Linux. (Admittedly, this one has been of dubious success so far. But stay tuned.)
• Challenge to EMC and NetApp in the area of storage using the Oracle Exadata Storage array (also the subject of a future blog post).

I think you get the idea. Virtualization is a critical control point in the stack. Ceding leadership to VMware in this area allows another giant software company to challenge Oracle's domination in the data center. That cannot be allowed.

Licensing is another thorny issue. VMware will tend to improve CPU utilization and efficiency on Oracle database servers. All of that costs Oracle money in license revenue. Which is to say, saves the customer money on Oracle license costs. This is obviously beneficial to the customer, and detrimental to Oracle.

Finally, by allowing VMware HA cluster to protect Oracle database servers, Oracle would be allowing another HA solution as an alternative to RAC. RAC is Oracle's crown jewel in terms of licensing revenues. RAC is one of the two major reasons why customers choose to run Enterprise Edition at 4X the cost of Standard Edition (the other being Data Guard). Plus, you pay the RAC upcharge. Don't get me wrong. I run RAC in the lab, and EMC will continue to be a strong supporter of it. For many applications, RAC is absolutely required. For many others, it's total overkill, the software equivalent of swatting a fly with a nuclear bomb. VMware HA cluster could provide an acceptable level of HA to those customers at a fraction of the cost of RAC. But, again, that costs Oracle license revenue. Beneficial to the customer but detrimental to Oracle.

The bottom line is that the reasons for Oracle's statement concerning VMware support are political and not technical. And definitely not altruistic. VMware virtualization of Oracle database servers provides many significant advantages to the customer in many environments. Certainly, VMware is far more mature and robust than OVM. (I will cover OVM in a future post on this blog.) VMware is the vast market leader in the hypervisor space. Significant cost and manageability savings can be achieved by virtualizing Oracle database servers with VMware. Despite this, Oracle has chosen to issue an essentially misleading support statement on VMware virtualization, and has taken the position in numerous sales calls that VMware virtualization of Oracle database servers is unsupported and dangerous.

This does nothing to serve the customer. It serves Oracle corporation, and its political agenda. In my view, customers who wish to pursue virtualization of Oracle database servers should do so. I will certainly continue to post on this subject.

Come on Oracle: Get with it. Embrace VMware. EMC and VMware embrace Oracle. Are you an Oracle customer using VMware? What are your thoughts?

Posted by the Oracle Storage Guy April 23, 2009.

Comments

[RussellH]

I'd like to offer a clarification.

"Parsing a query has to do with issues of table statistics, query syntax, hints, and the like."

With Oracle 8i and earlier, that was true. Starting with Oracle 9i, Oracle can use system statistics, such as I/O and CPU performance and utilization as input to the query optimizer. These things could be affected by being deployed in virtualized hardware.

http://download-uk.oracle.com/docs/cd/A97630_01/server.920/a96533/stats.htm

DBMS_STATS.GATHER_SYSTEM_STATS

----------------

Response:

Your comment is technically true but somewhat pedantic. I did not point out the painfully obvious: Everything eventually has to do with the operating system. And I did know that the cost based optimizer was adopted in version 8, and the rule based optimizer was used previously.

On the issue of whether virtualization has to do with a particular support issue, it is a matter of degree. If a query is doing a full table scan instead of using an index, it is theoretically possible (albeit astronomically unlikely) that it is at the very edge of the decision point, and the OS statistics returned by the virtualization layer has shoved it in one direction or another. But that would be a case in which the use of the index was a close call, and the query performance would be almost the same either way.

It is far, far more likely that a gross event like using a full table scan which causes the query to take far longer than it should is due to faulty statistics (yes these are still very much used after 8i) rather than virtualization. Or some other issue with the query execution engine may be a factor.

If I simplified this unduly, then I apologize for the mis-communication. I assumed that my reader would have the ability to extrapolate.

Regards,
Jeff

[Iain Mobberley]

I wondered if you can now re-consider the future of RAC clusters on VMware now that vSphere is available in the market, specifically with the regard of VMwareFT?

I assume that this further weakens Oracle's position with regards their own "HA" functionality versus what is now available from VMware?

--------------

Response by TOSG:

We have completed testing of FT in the context of our virtualized solutions. FT does work with Oracle database servers. However, it is presently limited to only one vCPU, which limits scalability. For our testing, we typically use 4 vCPUs per VM.

I think the single vCPU restriction will eventually be expanded. At that point, I think FT will be more useful. In the meantime, I recommend it for small database server workloads which can fit onto a single vCPU.

[Arun]

Thank you for parsing the language in the metalink article. I am an Oracle EBS consultant and many customers were scared to migrate to vmware because of the Oracle support policy even though as you very clearly explained there is no technical reason for non-RAC environments.

Arun