Oracle Storage Guy

Upcoming Webcast for Oracle Database Virtualization using VMware

Please join me in my upcoming EMC Live! event. I will be presenting on Efficient and High Performance Virtualization of Oracle Database Environments using vSphere, similar to my presentation at VMworld in September. However, I have added significant material, so those who saw the VMworld presentation should still join this webcast. Anyone who is interested in virtualization of Oracle database servers should find this presentation interesting as well.

This event will be presented live on October 29 at 11:00 a.m. EDT. Details can be found here.

Charles Phillips' response to customer input

For information on the response to Charles Phillips request for customer input on VMware, see Chad Sakac's blog. I heartily agree with Chad's comments. In addition, I will add the following point, which I think is key to the discussion.

Charles Phillips makes effectively the same statement that is contained in the metalink statement on support, i.e.:

What we've said on other VMs is that we will respond to support issues as they arise but so far we've elected not to formally certify any third party VMs....

The metalink support statement is very similar. It says:

Oracle has not certified any of its products on VMware virtualized environments.

In my opinion, both statements are slightly disingenuous. Certainly, they are misleading.

Other than RAC, Oracle simply does not certify things like this. There is no Oracle certification, so far as I can tell, for any hardware platform, other than RAC certified configurations. And we are not talking about RAC here.

As Chad points out, Oracle does not maintain a Hardware Compatibility Program for its software. For this reason, telling customers not to run VMware because it is not "certified" to run with Oracle software is simply meaningless. The same can be said of Dell servers, Intel CPUs, Emulex HBAs, Broadcom NICs and almost all of the rest of the stack on which Oracle software commonly runs.

Ask yourself this simple question: What is it that VMware provides? Folks think of VMware as a software company. In many respects this is wrong headed thinking. VMware is a platform vendor. What VMware provides is virtualized hardware. Thus, VMware provides a virtualized CPU, network connection, SAN connectivity, etc.

In that respect, VMware is far more similar to, say, Dell than it is to, say, Red Hat. Certainly, VMware does not provide an OS environment upon which the Oracle software runs directly. Rather, VMware provides a hardware (albeit virtualized hardware) stack upon which an OS environment like Red Hat Enterprise Linux (RHEL) runs.

Oracle does provide certification of things like RHEL. But, as I have pointed out before, RHEL runs just fine on VMware, and is actually certified to do so by Red Hat.

Perhaps it would help if VMware behaved more like a platform vendor and less like a software developer. That is another conversation. Personally, I would welcome that. Such a change would require VMware to be more involved in solutions development for applications vendors like Oracle, which I think would be a very welcome development.

In the meantime, I respectfully disagree with the assertion by Charles Phillips that Oracle software should not be run on VMware because it is not "certified" by Oracle. That assertion is simply inaccurate and misleading in my view.

Your comments are welcome as usual.

Charles Phillips Asks for Customer Input on VMware

I just received the following email:

Re: Need a favor - Oracle

You are not going to believe this.  Some VMware folks met with Charles Phillips, the president or CEO of Oracle and he said no customers had ever mentioned to him that they wanted Oracle to support their products on VMware.  Or modify the licensing scheme.  He offered if anyone knew of customers who did want better or more support for Oracle on VMware, or virtualization friendly licensing, to email him directly.  His email is Charles.phillips@oracle.com and he really needs to hear that customers run Oracle on VMware, and better support / licensing would be nice!

For any customers who are interested in Oracle on VMware, please send email to Mr. Phillips directly.

VMworld Aftermath and Oracle OpenWorld Premath

Realizing that it has been a long time since I blogged (the past few months have been crazy), much of interest has happened. EMC continues to make significant inroads into the NAS / Oracle database storage market, in competition with NetApp, which delights me. I was a speaker at VMworld 2009 earlier this month, speaking on VMware virtualization of Oracle database servers. And, of course, OOW looms, for which I am a demo captain and responsible for a major EMC high-end NAS demo as well as a speaker in the VMware booth.

In this regard, I have a couple of questions for you:

1. How many of you are doing actual virtualization of Oracle database servers with VMware? I really need to hear from you. EMC is doing quite a bit of work in the area of VMware virtualization, and some of that will be present at OOW.
2. How many of you are using EMC NAS with Oracle? I know about some of you, but not all. If you are using this combination, please comment to this blog. Let us know of your experiences, please.

I promise to be more vocal, and less remote, in the next few weeks as we approach OOW. I will be at the show, obviously, and very much look forward to talking to many of you live. Please come by the booth and say hello.

Paul Maritz Keynote at EMCWorld Mentions Oracle Performance on VMWare

As you all know, EMCWorld was this week, and I attended. One of the most interesting aspects of the event was the Paul Maritz keynote. Paul Maritz is the new President and CEO of VMware as most of you undoubtedly know as well.

During the keynote, Mr. Maritz said the "o" word, i.e. Oracle. In fact, Oracle was a very prominent feature of his discussion. He presented the results of an OLTP performance benchmark that VMware has done recently. The basic results are as follows:

Metric	Native	VM
Throughput in business transactions per minute (1,000s)	293	250
Disk IOPS (1,000s)	71	60
Disk bandwidth (MB/s)	305	258
Network packet rate receive (1,000s)	12	10
Network packet rate send  (1,000s)	19	17
Network bandwidth receive (MB/s)	25	21
Network bandwidth send (MB/s)	66	56

Overall, this indicates that VMware virtualization has no more than a 15% performance penalty over physically booted, which I consider to be actually very good. Certainly, given the other benefits of virtualization with VMware, this would be acceptable for many customers for many of their Oracle database environments.

Now, to deal with the obvious question which I expect to be raised: Why did VMware not see the performance advantage of virtualization that I did in my previous post on comparing physically booted performance to virtualized performance?

The answer will becomes apparent when you review the VMware performance study, linked above. My peformance comparison was between the following:

1. Oracle RAC 10g (i.e. clustered)
2. VMware ESX 3.5 with HA Cluster (i.e. clustered)

Their study is between:

1. Oracle Database 11g (i.e. not clustered)
2. VMware ESX 4.0 (also not clustered)

The most important consideration is the issue of clustering. Their study shows a single 8-way Intel white box being used as a server for both physically booted and virtualized. (Similar to our study, they used identical hardware for both configurations.) Our study shows a cluster of 4 8-way Dell PE2900 servers being used for both configurations.

Closely reading the VMware study, they used a single database server with identical memory and CPU settings for both physically booted and virtualized database servers. Thus, in the VMware study, everything was done within a single instance database server with a single database image.

Another issue is that we federated the databases on the virtualized side. Thus we had 4 clustered instances and 1 database on the physically booted side, and 8 independent database servers (consisting of a non-clustered instance and a database image) on the virtualized side. As I said in my previous post (linked above), creating a larger number of database images had a performance advantage, especially compared to RAC, where cache fusion and lock management add significant overhead.

I have received many comments that this makes the results from the EMC study non-comparable. I somewhat agree, and I will deal with that issue at length in a future post. The VMware performance result is certainly more apples-to-apples than mine.

However, in my defense, I will say (as I will explain at length in my up-coming post) that the configuration I used is identical to that used by Microsoft SQL Server in their clustered product. No one seriously contends that Microsoft SQL Server with MSCS is not competitive to Oracle RAC. Certainly, Oracle considers it to be competitive, as the abundant technical literature on their own website demonstrates.

Thus, the study performed by the Oracle CSV group at EMC is a natural and obvious step: We need to understand the performance differences (as well as cost and manageability differences) between Oracle RAC and Oracle Database with VMware HA Cluster. And when we do that study, we need to configure the databases in the most natural and logical way, taking full advantage of the features of each environment. This necessarily leads you down the path of a federated environment on the virtualized side, and a single image on the RAC / physically booted side.

The VMware study is undoubtedly very valuable and important: It demonstrates that VMware can provide world-class performance for Oracle production OLTP database environments. It shows that the performance penalty of VMware in this context is modest and manageable.

More on this later.

Oracle Wants To Win The Virtualization Wars – But Will Customers Lose?

In my previous post on Oracle's support of VMware virtualization, I pointed out that Oracle is behaving like a monopolist: Attempting to establish control over the entire IT stack, including application, OS, storage and virtualization. Oracle's recent moves in the area of virtualization further demonstrate this trend, potentially benefiting Oracle to the detriment of their customers and the industry.

Oracle recently revised one of its metalink support statements regarding virtualization of Oracle software products. This is the statement concerning support for virtualization with Oracle E-Business Suite. The revised version can be found here. (Note: This URL requires a metalink account.)

The changes in this statement are very interesting, in that they speak volumes concerning Oracle's intention regarding the use of VMware virtualization with their products, especially Oracle Database. Prior to May 8, the title of this support statement was:

Platform Vendor Virtualization Technologies and Oracle E-Business Suite

With the revision on May 8, the title was changed to:

Hardware Vendor Virtualization Technologies on non x86/x86-64 Architectures and Oracle E-Business Suite

Note the use of "Hardware Vendor" rather than "Platform Vendor" and the addition of "on non x86/x86-64 Architectures". Further, prior to May 8, the text of the support statement included the following:

The use of platform (emphasis mine) vendors' virtualization technologies (both software and hardware based) (emphasis mine) to host Oracle E-Business Suite 11i and R12 is covered by Oracle's policy with regards to 3rd-party products - that is, they are 'not explicitly certified, but supported' (emphasis theirs).

On May 8, this was revised to read:

The use of hardware (emphasis mine) vendors' virtualization technologies to host Oracle E-Business Suite 11i and R12 follows the same policy as Oracle's policy with regards to customizations - that is, they are 'not explicitly certified, but supported' (emphasis theirs).

Note the replacement of the term "platform vendor" with "hardware vendor", and the deletion of the phrase "both software and hardware based". Clearly, Oracle intends to carefully limit the support statement to only hardware vendors' virtualization on non-x86/x86-64 platforms at this point. Or to state it in the reverse, to exclude software vendors' virtualization products on the x86/x86-64 platform. And who is the dominant market leader for software virtualization on that platform? VMware, of course.

Further, the earlier version of the support statement included VMware explicitly, and this was deleted from the revised version.

I am speculating, but my guess is that this move on Oracle's part was prompted by the post to Chris Wolf's blog (which he issued on May 6), stating:

The bottom line – Oracle now offers best effort support for all of its E-Business Suite applications on any x86 hypervisor.

This post was based upon the earlier version of the support statement, obviously, which granted best efforts support to all virtualization products, both software and hardware, on all platforms, and included VMware explicitly. Others posted as well, basically claiming victory on VMware virtualization of Oracle software products. The timing is extremely suspicious, and it seems pretty clear that Oracle closed the loophole which might be perceived to grant official support to virtualization of Oracle Database using VMware.

The other move is, of course, Oracle's acquisition of Virtual Iron, a startup which has been nipping at VMware's heels for some time. The intent of that acquisition is obvious: To shore up the Oracle VM offering which has big gaps when compared to VMware.

Combined together, these moves present a fairly stark and obvious message: It appears that Oracle is carefully and intentionally excluding VMware from its support statement. It also seems very obvious that this is without any technical justification whatsoever, given that Oracle has granted explicit support for virtualization products from other vendors which are completely homologous and comparable to VMware. The apparent reason is to benefit OVM, Oracle's competitive offering against VMware, by forcing Oracle Database customers to run that virtualization product instead.

If VMware were a niche player in the virtualization market, that would be one thing. Excluding such a player might be seen as mean spirited, but the impact would be minimal. In the case of VMware, though, Oracle is excluding the vast, vast market leader in the virtualization market.

This is exactly like Oracle attempting to exclude Microsoft Windows from its list of supported OS platforms and promoting Linux as the exclusive OS platform for running the Oracle Database software product. Permitting Linux as an alternative to Windows as your OS platform for running the Oracle Database is fine. Even promoting Linux is the preferred alternative is imminently justifiable. But attempting to exclude an OS platform with the market dominance of Windows would be met with howls of protest by numerous customers. I submit that this is exactly what Oracle is attempting to do by excluding VMware from its best efforts support policy.

I have several questions for Oracle:

1. Do you have any conceivable technical justification for excluding VMware from best-efforts support, given your support for less mature, less popular software virtualization products sold by other vendors, yourself included?
2. How are your customers served by forcing them onto a virtualization platform with less maturity and less market penetration than VMware?
3. As Chuck Hollis pointed out in his blog, competing on your own merits to own the datacenter is one thing. Using your dominance in one area (databases) to force your customers to accept dominance in another area (virtualization) smacks of anti-competitiveness. Can't you tolerate a free and fair competitive environment? What are you afraid of?
4. Given that there is abundant evidence from extensive testing (including at EMC) that VMware virtualization of Oracle databases is both viable and compelling, what is your real issue here?
5. If you have no reasonable response to the above questions, then when will you extend official best-efforts support to VMware which you have already done to other equivalent virtualization products?

Response to comments from Alessandro Perilli blog

I have noticed that a large number of comments have been submitted to the recent post on Alessandro Perilli’s blog which links to my recent post on Oracle’s support policy for VMware virtualization. Many, if not most, of these comments are actually comments directed at my post, not Alessandro’s post. Therefore, I will respond to them here. I am also attempting to post this as a comment on Alessandro’s blog, but that has not appeared on his site as of yet, and I wanted my response to be available.

To respond to several of the points raised by these comments:

Why would anyone want to run Oracle on a single vCPU under VMware?

Not sure how to parse this one. VMware ESX 3.5 allows 4 vCPUs per VM. Our performance testing was done with 4 vCPU VMs (2 VMs per server on a 8 core box, consisting of 2 Quad-core processors). 4 Servers were in the HA cluster, so a total of 32 physical CPUs were in the configuration, all of which were allocated to vCPUs on 8 VMs. This is laid out pretty thoroughly in the reference architecture published on Powerlink and EMC.com. You can find the EMC.com version here.

In terms of why you would want to run Oracle in a virtualized environment using VMware, that is pretty well laid out in my blog. But to summarize:

1. We found better performance in a virtualized environment using VMware HA cluster vs. Oracle RAC. I will deal with the fairness of that comparison on my next point.
2. The cost of VMware HA cluster is less than Oracle RAC, and VMware is appropriate for many customers. Not all, but many. For those customers, RAC would also work, but is, again, vastly more expensive. I lay out the usage cases where VMware HA cluster works as an alternative to RAC in another item below.
3. Manageability is higher with VMware HA cluster than RAC. Believe me, I should know. I run both routinely in our testing environment.

Why should I believe that performance of VMware HA cluster is higher than RAC when VMware will not allow third parties to benchmark their product?

Well, I am a third party, and the program I manage publishes performance benchmark results of both virtualized and physically booted Oracle production environments. These are available on Powerlink. The actual performance results are pretty confidential and proprietary, but I tend to open the kimono on this blog, as you have seen. We are using a TPC-C-like workload run under Quest Benchmark Factory for Databases. We do not claim that this is a published and audited TPC-C result, obviously. However, we have lots of experience running this.

I can tell you no one was more surprised by the VMware performance result than I was. We are still continuing to actively profile performance of both physically booted and virtualized Oracle environments. (In fact, we are beginning to profile OVM as well.) I had no particular axe to grind on this either way. The results were simply what they were.

RAC and VMware HA cluster are not comparable, are they?

Of course they are. Both Oracle Cluster Ready Services (the underlying technology behind Oracle RAC) and VMware HA cluster are cluster software products. They do basically the same sort of thing: Provide high availability for applications. They simply do so in a different sort of way. RAC provides one single database image across multiple physically booted servers. CRS provides transparent failover of VMs using VMware VMotion technology. Both are intended to protect application uptime and client access by providing a high availability solution.

Granted, they have different levels of HA. I would consider RAC to be a fault tolerant technology, in that the physical loss of a node will not result in database downtime (but may result in loss of client access). VMware HA cluster is a high availability product. A brief downtime is inevitable when a VM is being rebooted on a surviving node of the cluster after node failure. In our experience, VMware HA cluster works pretty well at getting the VMs back up and running, though.

In the end, it depends on what the customer needs, the level of HA being one of the issues. I cover that more in a later section.

Isn't RAC free with SE?

This is true, but very misleading. RAC is free with SE when the total CPU core count in the entire cluster is 2. This is a trivial cluster. Beyond that, EE is required.

And that is where the cost savings come in. EE is required for basically two products in most customer configurations:

1. RAC
2. Data Guard

RAC also carries the RAC upcharge above the cost of EE, making it the most expensive Oracle database software product, by far.

Assuming Data Guard is not required (and storage vendors have been competing with Data Guard using products like MirrorView and RecoverPoint for many years), then it comes down to RAC.

Assuming you can provide HA in another manner, then RAC is not required, and therefore SE can be used instead of EE, at 25% of the cost, plus savings on the RAC upcharge. I think you see the point.

The bottom line is that VMware HA cluster provides costs savings assuming the customer scenario allows this product to be used. Which is what I cover next.

Not all customers can use VMware for an HA solution, and need RAC instead. Right?

Not actually in the comments, but important. Again, EMC is a strong supporter of RAC. We run it in our lab, and we will continue to do so. No one would say that RAC does not work as well as VMware HA cluster or that it is not a great product. It is. But it is expensive and complex.

What RAC provides is two things:

1. A single database image
2. True fault tolerance

These are both great advantages, but not all customers need this. For example, I personally visited a very large Fortune 100 company. For confidentiality reasons, I will not use the name here, but believe me, they are a household name. This customer had many single-instance database servers running on 1U and 2U unclustered machines throughout their datacenter. The cost to manage these servers was immense, in the 9 figures per year. Yeah, that's a lot of jack.

The customer wished to consolidate all of these servers into a database cloud. I postulate that the way to do that could consist either of RAC or VMware HA cluster. But consider:

• What is the level of HA provided by the current configuration? Very low. Each physical server is presently a single point of failure. VMware HA cluster would be a big improvement.
• Does the customer need a single database image? Not at all. These servers are already islands of data.

The simplest and quickest way for this customer to consolidate the servers in this scenario, given the choice between RAC and VMware HA cluster, is VMware HA cluster. It is also the least expensive of those two alternatives.

I suspect that many, many customers are in this situation. For many applications VMware HA cluster is a viable high availability and consolidation solution for production Oracle databases. I would certainly not put a large ERP system on VMware. Nor the billing application for a large telco. But many, many applications could be put up on VMware just fine.

This is in many ways comparable to NAS vs. SAN. I am employed by EMC, so it may surprise you to hear me say this, but remember I came to EMC from NetApp. In my experience (not a scientific sample, but still) approximately 90% of the Oracle databases running in the world could be stored on an NFS server with absolutely no change in the client experience or uptime of the database. The same it probably true with VMware. Perhaps the percentage is higher or lower. I certainly have less experience with VMware at this point than NAS. But time will tell.

BTW, it would help me if folks could post comments to content written by me on my own blog, rather than having to run around and find it elsewhere. Just a thought...