Oracle Storage Guy

Oracle Throws in the Towel on VMware Licensing - Reprise

Given the incredible number of comments I received on my last blog post, and the content of those comments, it is very obvious that folks are extremely confused by what I meant by that post, as well as the comments by Mr. Garsthagen (Oracle Director Level Employee), referenced in that post.

The confusion is typified by the following comment, the most recent I have received:

Oracle does not recognize either (Vmware/DRS Affinity) as a hard partition

First, to be completely clear, I have never stated, nor do I believe, that VMware (or any other software hypervisor for that matter) constitutes hard partitioning as that is defined in Oracle's own documentation. Hard partition has to do with licensing a subset of the cores on a database server. I have always maintained that all cores on a  physical server where Oracle is installed and/or running must be licensed to run Oracle.

The issue has to do with licensing of a subset of servers in a VMware DRS / HA cluster. Prior to VMware's white paper on Oracle licensing and support, I maintained that the most conservative position was to create a dedicated Oracle DRS / HA cluster, and thus limit VMs running Oracle to only that set of servers. This illustrated as follows:

Once the VMware Oracle licensing and support white paper was published, and particularly once I became aware of the work of Dave Welch with House of Brick, Daniel Hasselink with License Consulting, and Jason Keough with iQuate, my thinking changed radically.

First of all, the issue with the above approach is simple: Not all customers have the necessary scale of Oracle servers in order to dedicate an entire VMware DRS / HA cluster to just running Oracle. For these customers, it is important to have Oracle VMs limited to a subset of the servers in the cluster. The following would illustrate this:

The essence of this approach is that the configuration must limit Oracle VMs to a subset of the physical servers in the cluster. This complies with Oracle's License and Service Agreement (OLSA), the only relevant document covering Oracle licensing, which states that Oracle software licensing attaches to the CPU cores on physical servers on which Oracle is "installed and/or running". Thus, if you provide a mechanism to limit the physical servers where Oracle VMs are allowed to run, and also provide a methodology to track this over time, then that method satisfies the OLSA.

Fortunately, VMware vSphere provides such a mechanism: It is called Mandatory Host Affinity, and means that a VM is only allowed to either boot or migrate (via vMotion either manually or through a DRS Cluster load balancing process) onto a defined set of servers only. VMware vCenter also provides VM logging which provides a detailed record of the physical servers on which each VM has run. Finally, license compliance software vendors such as iQuate provide tools to track each Oracle VM's location on physical servers within the cluster, and this tracking data has been certified and accepted by Oracle for licensing purposes.

The net-net is that the issue of Oracle licensing of a sub-cluster within VMware vSphere has been resolved, as indicated by Mr. Garsthagen in his online video. He simply restates the obvious. VMware vSphere provides tools which allow you, the Oracle customer, to license a subset of a VMware DRS / HA cluster, provided that you:

• Enforce strict rules regarding where the Oracle VMs are allowed to run
• Track compliance with those rules over time
• Are prepared to provide this tracking data to Oracle LMS on demand

At this point, numerous Oracle customers have provided this data to Oracle LMS and have been allowed to run Oracle on a VMware sub-cluster with no problems in terms of licensing.

Again, to reiterate my clarification, it was never my intent to imply that VMware vSphere (or any other software hypervisor) provides hard partitioning. Hopefully, this is now cleared up. Please let me know your thoughts via comments to this post.

Oracle Throws in the Towel on VMware Licensing

As all of my readers know, I have been a strong proponent of VMware virtualization of Oracle Database servers for license cost savings purposes. Predictably, Oracle has pushed back on this issue in the past. Well, they have now thrown in the towel.

In an online video, Richard Garsthagen, Director of Cloud Business Development EMEA for Oracle, has stated publicly that VMware host affinity rules (when combined with vMotion logging) work just fine, thank you very much, for purposes of establishing where Oracle software is "installed and/or running" for purposes of the Oracle Software License Agreement (OSLA).

Previously, Oracle has maintained that all servers in a cluster must be licensed if even a single VM on any server in the cluster is running Oracle software. VMware has officially disagreed with this statement in their white paper on Oracle Licensing and Support. But the opposing view by Oracle has been a serious deterrent to adoption of VMware virtualization of Oracle servers. No longer.

I think we can consider this issue closed. Customers can now virtualize Oracle database servers without any concern about Oracle hitting them with unreasonable license cost demands. This is undoubtedly great news, and a great watershed event in this area.

Oracle licensing on VMware: Exceptional Michael Webster blog post

As my comment on Michael Webster's recent blog post entitled Fight the FUD – Oracle Licensing and Support on VMware vSphere states, I am in violent agreement with everything he says regarding Oracle licensing and support for VMware virtualization of Oracle database servers. I heartily recommend this blog post to everyone concerned with Oracle's recent behavior regarding VMware virtualization of their products.

Comments by Dave Welch of House of Brick on Oracle on VMware Licensing

I thought it worth republishing Dave Welch's (of House of Brick) comments to my discussion on the Everything Oracle at EMC online community on the subject of Oracle licensing costs on VMware vSphere configurations. Here are Dave's comments:

All,

Let me start by bringing all of you into the courtroom. There are three issues that you will observe me as counsel provide to the jury as part of my allowed instruction (I am not an attorney in real life).

I offer the definitions in this paragraph only to make this post as self-sufficient as possible and not with intent to talk down to anyone.  Case law refers to precedents established in other court cases. Case law can supplement statutory law enacted by legislative bodies and can stand in lieu of statutory law when applicable statutory law does not exist.

Back to the courtroom: 

1. You cannot prove a negative.
2. Just about every contract that any of our organizations enters into or sees contains a clause that essentially states that the only terms that are binding have been reduced to the fully-executed instrument itself as well as other explicitly-referenced written material, and all verbal agreements and/or representations are null and void. The language usually allows for the obvious possibility of written, fully executed amendment.
3. As often as original documents are unavailable, courts in all fifty states recognize case law called "best available image.” This refers to copies which become recognized as legally binding as if they were the original. “Best available image” case law has been universally recognized since 1989 when as the Medical Records Systems Analyst for 23 hospitals and 7 clinics in 3 states, I had responsibility to begin architecture for imaging of medical records.

I offer the following as the exclusive binding source documentation for this post:

• The Oracle License and Services Agreement (OLSA)
• Oracle’s published Software Investment Guide
• Oracle’s published Licensing Data Recovery Environments – an extract of the Software Investment Guide

The OLSA is bi-laterally executed between Oracle and the customer. I have seen many of them as you can imagine in my role as the lead of House of Brick’s Oracle Enterprise License Assessment business line. The OLSA always contains:

• A reference to other unnamed published documentation, which given the nature of the reference, refers to the Licensing Data Recovery Environments extract of the Software Investment Guide as well as the current Oracle Support policy. Because of the nature of these external OLSA references, I make it my business to occasionally archive current versions of the Software Investment Guide and Licensing Data Recovery Environments document such that any given OLSA can be clearly tied to the external terms in effect as of the OLSA’s execution.
• A Non-Disclosure Agreement as to the OLSA’s terms. As Oracle publishes template OLSA copies, a reasonable and customary interpretation of the NDA’s coverage is pricing (discounts) as well as any other non-standard qualitative terms that the customer may have negotiated with Oracle. We have initiated for customers and seen Oracle approve non-standard qualitative OLSA terms. However, we have never bothered to initiate sub-cluster licensing terms as there is no need to do so.

The sum and detail of the stock published OLSA and its external Software Investment Guide for at least the last decade and to-date with respect to processor-based licensing is two-fold:

• Customers must license all physical cores or sockets on a host where Oracle executables are “installed and/or running” (with physical cores factored per Oracle’s published Core Factor Table, and potentially subject to the so-called 10-day rule [whose terms became more restrictive sometime during 2007]). Notice the tense. Oracle customers are contractually obligated for licensing the physical servers where the Oracle executables are and have been, not where they might go. To imply otherwise without explicit contractual inducement would not be unlike concluding that I am legally obligated to purchase transportation to or obtain a visa for destinations that I clearly have the capability of visiting but where I have neither ever been nor yet made a determination to even visit.
• Furthermore, customers must pay a license to cover the use of remote mirroring at the storage unit or shared disk array layer to transmit Oracle executables to a SAN whether or not that set of Oracle executables is “installed and/or running” on any physical host connected to the SAN.

When discussing Oracle licensing, there are two common areas of confusion.

1. Oracle itself defines both of the terms “Hard Partitioning” and “Soft Partitioning” in its Partitioning document, published since 2002. The document makes clear that both “Hard Partitioning” and “Soft Partitioning” have to do with carving up physical cores/CPUs within a single physical server. It is Oracle’s right to define the terms as they relate to the OLSA. I frequently see Oracle reps (knowingly?), customers and others confuse these terms over into a discussion of licensing physical servers within a sub-cluster. As often as unintentional term confusion occurs, I believe it important for authors to make corrective edits after the confusion has been pointed out, especially on a topic with such scaled financial ramifications.
2. To say that Oracle has not made a statement supporting vSphere DRS Host Affinity Rules (as has been asserted in this thread) is beside the point. That is clearly in the realm of proving a negative. That DRS Host Affinity Rules may or may not be used as a mechanism to assure that Oracle executables are not “installed and/or running” on unlicensed physical hosts is customer-discretionary and irrelevant to the OLSA’s binding terms. I might add that methods existed to restrict VM movement to a sub-cluster of physical hosts long before the vSphere 4.1 advent of DRS Host Affinity Rules.

I tried to force both clarifications into Gartner Group’s Chris Wolf’s November 10, 2010 blog thread via comment posting.  I feel that I failed to get Mr. Wolf to directly address my point before he elected to close the thread.  I feel that I failed both verbally and in writing during the pre-publication interview process to preventively insert the clarifications into TechTarget’s Beth Pariseau’s May 4, 2011 article titled “Oracle licensing for vSphere 4.1 irks VMware pros.” Furthermore, the TechTarget article’s title unfortunately implies that Oracle may have licensing terms specific to vSphere version 4.1, which it does not, or licensing terms specific to VMware technologies in any way, which it does not. Thankfully the November 2011 VMware Corp. whitepaper (to which I contributed) “Understanding Oracle Certification, Support and Licensing for VMware Environments” got it right. I can confirm that the entire white paper received all the multi-disciplinary pre-publication review that you would imagine appropriate for a piece of its importance and ramifications.

House of Brick is occasionally invited by customers and/or partners to get on the phone with Oracle to deal with Oracle licensing concerns that Oracle has verbally conveyed to the customer.  We always welcome these opportunities.  The most recent call was a couple months ago. An Oracle Corp. licensing specialists was on the call. The impetus for the call was that Oracle had previously told the customer that all nodes in a physical cluster must be licensed, not just those where Oracle executables are “installed and/or running.” We asked the Oracle licensing specialist about the statement.  He informed us that it was Oracle’s policy.  We asked where the policy was written.  He replied that it is an unwritten policy. At this point, we reminded him of the binding OLSA language whereby all verbal conveyances and understandings were voided at the execution of the document leaving only the printed OLSA terms in effect. That was the end of the discussion. The customer left the call entirely and appropriately satisfied as to their full legal compliance with their proposed sub-cluster licensing scenario. This vignette is by no means an isolated incident. The outcome is always predictable and consistent.

A previous post on this thread states ‘I've heard Dave at HOB contend the same thing you've heard, which is essentially: "DRS/Host Affinity SHOULD be fine"...’  The quote could be interpreted inappropriately out of context of my consistent, persistent statements. What I have always said is that any method (manual or automatic) to restrict movement of Oracle executables to a licensed sub-cluster of physical hosts is fully OLSA-compliant and legally defensible. At the release of vSphere 4.1, I added the statement that DRS Host Affinity Rules happens to be one, the newest, and the easiest of multiple available methods of restricting Oracle executables’ movement.

There are comments on this thread that I believe fall under the umbrella of attempting to prove negatives. Oracle has no legal obligation to go on paper with sub-cluster licensing clarifications specific to vSphere. In my view, Oracle has no financial incentive to do so.  Stop looking for that to happen.

Occasionally people actually insist that I produce an Oracle-authored document that says Oracle will not ding them for not licensing all nodes in a physical cluster. That strikes me like asking me to provide an Oracle-authored document that says Oracle will not make a claim against me for choosing to drive a Ford as opposed to a GM. Not only is it asking me to prove a negative which I cannot do, but it delves into an area that is none of Oracle’s business since it is not listed in my OLSA obligations.

I see no need for Oracle to comment on their recognition of the validity of vSphere logs to prove vSphere VM movement. vSphere logs may well be the best available mechanism to document Oracle executables’ travels.  Although vSphere logs may not be identical to “best available image” case law, “best available image” is instructive to the audit log/verification concept under discussion. As of November, 2010, the majority of the world’s servers are virtual. ~85% of those virtual machines are VMware. That is tenured technology and no one is questioning the de facto authority of those logs. Oracle has the OLSA-granted right to present themselves at any time without notice for the compliance inspection of their customers’ physical hardware. Oracle has options to audit the current and historical location of where their executables are “installed and/or running.” That those options may rely on mechanisms not provided by Oracle is beside the point.

As for the question on the thread as to whether vSphere logs could be maliciously manipulated, security industry experts have said that eighty percent of security breaches come from within the firewall. I see a corollary to that statistic. Oracle has plenty of financially incented prospective friends inside its customer organizations. It is sad that human integrity is such that posting of large rewards has become necessary to incent individuals to turn in their employers for knowing theft of software. To me, OLSA obligations are like the U.S. tax code. I believe in paying every penny I owe. However, beyond that, it is my discretion to who or what I donate and in what amount. I have no patience with individuals or entities that premeditate the creation of OLSA compliance issues.  I similarly have no patience with the knowing spreading of FUD by some professionals in what could be construed as extortion of funds beyond customers’ executed contractual obligations. I will continue to vigorously promote and defend the legal rights of both software vendors and their customers even if that means I induce accelerated hair loss through rapid, frequent hat swapping.

What Oracle verbally represents on this or any other contractual topic for that matter is of absolutely no relevance whatsoever. Look to the OLSA and extant external referenced documentation as of the OLSA’s execution date. Should you still be dissatisfied, look to court history of judgments against customers attempting to exercise their sub-cluster licensing rights. Let me save you some time. Relevant court history does not exist. There is a simple reason: the terms of your executed OLSA are clear, sufficient, binding. The terms explicitly invalidate any and all verbal representations.

Given what I have presented here, I see no legal, risk management, or moral need for and am not inclined to encourage and/or endorse air gap clusters strictly for Oracle licensing purposes. I am concerned that such talk may scare parties away from investigating, understanding, and asserting their legal rights. Furthermore, in my observation, such talk gets quickly quoted out of context of the original motive. I believe such talk has a similar long-term effect on the market as paying ransom for hostages has on future hostage taking. The SMB market has every contractual right to leverage sub-cluster licensing to facilitate crossing the chasm over to Business Critical Application virtualization. That larger enterprises may find it convenient to use air-gap cluster separation of their Oracle workloads is great, but to do so has absolutely no bearing whatsoever on their legally binding contractual obligations to Oracle Corporation.

Some on the thread have suggested this or that course of action to minimize risk with Oracle. Occasionally people express concern to me that Oracle has far deeper legal pockets than they do. I believe those that make these points do not understand the power of their position. I cannot think of one reason that Oracle could possibly want this thing to go to court and indeed I believe Oracle has everything to lose minimally moving forward if and when it does. Once such a judgment was out, it would cause an immediate slow-down on non-contractual cash donations that some of Oracle’s customers have felt induced to make.

VMware's Official Support Statement Regarding Oracle Certification and Licensing

Stake in the Ground

VMware has put a stake in the ground with respect to Oracle licensing of VMware VMs running Oracle. The gist of this statement regarding certification, support, and licensing is that DRS host affinity rules, combined with vCenter audit trails showing where VMs have actually run, are sufficient for Oracle licensing purposes. The summary of the document states:

DRS Host Affinity rules can be used to run Oracle on a subset of the hosts within a cluster. In many cases, customers can use vSphere to achieve substantial licensing savings.

vCenter VMotion Logging

Concerning vCenter VMotion logging, the document states:

With VMware vMotion and DRS technologies you can migrate a live virtual machine running Oracle software from Host A to Host B for server maintenance or load-balancing purposes. In such instances you should ensure that the migration occurs between fully licensed hosts by using vSphere capabilities such as DRS Host Affinity—that is, both Host A and Host B must be fully licensed hosts from an Oracle licensing perspective (as described in section 2.1). VMware vCenter™ Server generates several migration log files maintained at /var/log/vmware/hostd.log and /vmfs/volumes/datastore/vm/vmware.log that can be leveraged to track and record such virtual machine movements across hosts for compliance purposes. Additionally, VMware provides an extensive open API that allows compliance tools to generate user-friendly reports using this data. In particular, VMware vCenter Configuration Manager provides host-level change-tracking mechanisms that enable you to record virtual machine movements across hosts. Since this hostlevel change tracking leverages an open API, third-party configuration-management solutions may also provide some of this functionality for VMware environments.

CPU Affinity

VMware's document also contains a discussion concerning CPU affinity. In that statement, VMware calls Oracle's bluff with respect to the hard partitioning statements concerning OVM on Oracle's website. Again, VMware's document states:

VMware enables you to pin a virtual machine to certain CPUs inside the host (using CPU pinning or CPU affinity). We believe this technology is every bit as robust and reliable as the “hard partitioned” technologies to which Oracle accords preferential subsystem pricing, and should enable customers to license only a subset of the host capacity. Unfortunately Oracle does not recognize this approach as a valid hard partitioning for its licensing mechanism. So today customers must license all the CPUs in the host and follow the “fully licensed host” approach for VMware environments.

Net-Net

The net-net is that VMware is endorsing the view that fully-licensed servers are required, but that Oracle servers can be part of a larger DRS / HA cluster environment, so long as VMs running Oracle are strictly limited to licensed servers, and this can be proven using audit trail documentation.

I cannot overstate the importance of this in terms of Oracle licensing. This is a sea change, folks.

Oracle Licensing on VMware - no magic

Acknowledgements to Bart Sjerps for this content.

There seems to be a lot of confusion on licensing when customers consider running Oracle databases on VMware. Part of the confusion is caused by Oracle on purpose (classic FUD) by suggesting licensing is more expensive on VMware than on physical servers. The reality couldn't be more different - I strongly believe that many customers can actually *save* on database licenses by going virtual. But to understand how to achieve this, you need to know a few things - I hope I can clear this up in a short explanation. I will keep the discussion to Oracle database licenses and ignore application/middleware etc. for now.

License models

Customers typically license their basic database by one out of three options:

1. License by CPU (core) - the more CPU cores, the more licenses are needed. There is a processor core factor depending on the type of CPU and can be 0.25, 0.5, 0.75 or 1.0.
2. License by named user - the more named users, the more licenses are needed. The amount of CPU's is not important, neither the amount of total databases. Typically one license pack per 25 users.
3. Enterprise License - the customer negotiates a contract for the whole company and afterwards can deploy as many databases on as many servers/cpus as he wants.

If a customer uses 2 or 3, then it does not matter if they run virtual or physical. But there are also no license savings possible without re-negotiating their contracts. I don't want to go as far as suggesting to customers to change their license models so we leave this as-is for now.

In my experience, most enterprise customers use either cpu licensing or enterprise contracts. Some have different licensing methods for different business units. Oracle can be very creative in customer-specific contracts so expect to find a different situation for each individual customer.

But let's assume CPU licensing for the sake of this discussion.

Maintenance & support

Users typically buy the CPU licenses  but then have to pay maintenance for the time they use the licenses. Yearly maintenance cost is about 25% of license (list price). I have no information on typical discounts. I expect customers to get at least 50% discount off the price list (but only on licenses, not on maintenance AFAIK).

Database Edition and options

The plain database license comes in 3 versions (for servers):

1. Standard Edition One - Maximum 2 processors, no options allowed. Only used for testing and very small deployments
2. Standard Edition (SE) - Maximum 4 processors, no options allowed. Only used for smaller sizes and workloads (but stay tuned)
3. Enterprise Edition (EE) - No limitations and on top of EE, you can have many licensed features. Most customers will use this, at least for production databases

On top of the basic Database license, most customers use a set of options, each requiring additional licenses per CPU. The most common options are:

• Real Application Clusters (RAC) - allows many servers running the same database (active-active clustering) to allow scale out performance and high availability.
• Real Application Clusters One Node - same but one database can only run actively on one node. For high availability only.
• Active Data Guard - remote replication using log shipping. Note that standard Data Guard is free, but Active Data Guard allows the standby database to be opened for read-only purposes and offers some extra features.
• Partitioning - allows tables to be split up in smaller chunks. Absolutely required when running large databases and no downtime can be tolerated. Eases administration work and offers some performance benefits.
• Real Application Testing - allows workloads to be recorded and re-played on another database to do performance and functionality testing
• Advanced Compression - allows database blocks to be compressed - requiring less storage and boosting performance (in most cases).
• Diagnostics Pack / Tuning pack - provides automated reports. Oracle AWR (Advanced Workload Reports - a performance reporting tool) is part of Tuning Pack.

In my experience, nearly all customers have partitioning. Most customers have tuning/diagnostics pack. Some customers have RAC. Some customers have the other options. There are more options available but these are the most common.

Many customers have 3 or more options - sometimes the options cost more than the base database license - especially if they use RAC they will have most of the other options, too.

Running on a cluster

If a database runs on a cluster, then Oracle assumes the database can make use of any processor in the cluster. This is independent on what kind of cluster is used (so can be MSCS, HP MCSG, Vmware, Oracle RAC, etc).

This is basically the foundation for all FUD and confusion. For example, if you deploy a VMware farm (cluster) of 16 servers, and all virtual machines run all kinds of stuff (file/print, exchange, apps, etc etc) and only one tiny virtual machine in the corner, with only one virtual CPU runs a small Oracle database, you would expect only to pay for one CPU core - but Oracle's reasoning is that this tiny VM can be dynamically moved (VMotion) to all nodes in the cluster and on any processor. Therefore, all CPU's have to be fully licensed by Oracle. So in this case, running the single database on a (small) physical server would be cheaper than running on a VM in the farm.

Total cost of the stack

In a typical database server deployment, the cost of the database licensing is far greater than the cost of the hardware + OS licenses combined. I have no hard numbers but I assume the average DB license cost (plus options) is 10 times larger than the cost of the server + OS.

So a $5,000 server would typically require $50,000 on licenses. Then because maintenance is 25% yearly, the total cost of licenses over a 3 to 5 year period is even higher - so for a 5 year TCO the total license cost might be $75,000 (assumption - could also be closer to $100,000 - and no, I didn't make a mistake with an extra zero, Oracle *really* is this expensive).

Utilization

It is very hard to size a typical Oracle database based application. There are no good methods or calculations to figure out how much CPU power, disk I/O and memory is needed to run a given app. So historically, project teams size their database servers for peak loads, and because they cannot predict how big the peak load is, they double the resources "just in case". The end result is that most database servers are way oversized in terms of CPU and memory.

Most physical deployed database servers will average on about 10-15% CPU load (or less). However, they will peak to higher loads at certain times, such as monday morning when many users log in, or when month/quarter/year-end batch processing is started, etc.

Then, the utilization numbers can be influenced by other tasks of the processors. Some common causes of "artificially high" CPU loads on database servers:

• CPU is involved in storage mirroring (i.e. Host Level Mirroring - using Oracel ASM or a Unix volume manager)
• CPU is involved in file transfers over the IP network
• Backup (non-serverless, using CPU, Network and I/O bandwidth)
• Customers run the application server on the same machine driving up CPU load - This can drive up CPU load from 10% to 90% or more !!
• Same for Middleware and Enterprise Service Buses (Think Oracle BEA, IBM Websphere, SAP Netweaver, etc)
• A  bunch of monitoring/management agents burn CPU cycles (Tivoli, BMC, HP Openview, CA, etc). Each agent maybe consuming 1% but add it up and you have another 5-10% overhead.
• Administrators generate database dumps / exports and run their own reports, scripts and tools. They run ad-hoc queries as well that should not be on production.
• Poorly tuned database servers cause paging and other CPU overhead - hard to diagnose but driving up CPU and I/O significantly.
• Database admin tasks (table reorganizations, (re)building indexes, converting tablespaces, ...)
• And so on...

All of these cause the processors, expensively licensed for database processing, to do other stuff.

So if a server is running at 15% utilization, then the utilization caused by the database workload itself might only be 10% and the rest caused by other stuff (whether needed or not).

Needless to say that Oracle likes customers to use their expensive licensed CPU's for other tasks because it forces them to buy additional CPUs sooner and therefore drive their license revenues.

Isn't life great for an Oracle rep? ;-)

Number of databases

Most customers run many databases. For the average enterprise customer that I visit, 100+ databases is a normal number. A big global that I visited runs 3000+ Oracle databases worldwide (and this is only the scope of this specific project team). Imagine the cost of licensing all these databases on all individual servers...

Why so many? Well, customers do not like to share multiple applications on one database (and often this is not even supported). So if you run SAP ERP, Oracle JD Edwards, your own banking app and a few others, they all require their own production database.

For each production database, you might find an acceptance environment, test system, development server, maybe a staging area to load data into the data warehouse, maybe a firefighting environment, a standby for D/R, a training system and so on. Customers will rarely share production environments on the same server (unless virtualized or at least with workload management segregation). Sometimes they share a few databases for non-prod on a server. So for, say, 100 databases, the average customer runs between 30 and 50 (physical) servers.

Power of big numbers

It does not require rocket science to understand that many of these databases do not require peak performance at the same time. A development system typically drives workload during daytime (when developers are coding new application features). A data warehouse runs queries during the day and loads in the evening. For a production system it depends on the business process. An acceptance system might sit idle for weeks and then suddenly peak for a few days preparing for a new version deployment into the live production system. And so on.

So what if you could share resources across databases - without influencing code levels, security, stability and so on?

If that would be possible - you would not size for "peak load times two" anymore. You would size for what you expect and assume an average utilization of, say, 70% over the whole landscape. If one database needs extra horsepower, there is enough available in the landscape.

How much license cost would you save by bringing down the number of CPU's so that utilization goes up from 10% to 70%?

What would be the effect on power, cooling, floor space, hardware investments, time-to-market?

What would be the business advantage of not limiting production performance of a single server, by whatever was sized during initial deployment? Risk avoidance?

What would be the business advantage of solving future performance issues by just adding the latest and greatest Intel server in the cluster and Vmotion the troubled database over?

Wasn't this exactly why we started server virtualization in the first place about 8 years ago? And why EMC aquired VMware?

Wouldn't you think the average Oracle sales rep is scared to death when his customer starts considering to run his databases on a virtual (cloud) platform? Would it make sense for him to drive his customers mad with FUD around licensing, support issues and whatever he can think of to prevent his customers going this way? Even threatening to drop all support if they continue to go in that direction?

If Oracle is scared of losing license revenue, wouldn't you think there is a huge potential for savings for our customers here?

The journey to the private database cloud

So how should we deal with this?

A few starting points:

• Oracle supports VMware. Period. Any other claim of Oracle reps can be taken with a grain of salt (to be more specific: it's nonsense).
• Oracle does NOT certify VMware. Then again, Oracle does not certify anything except their own hard- and software. But IMO, support is all you need and the discussion around certification leads nowhere.
• Oracle might ask the customer to recreate issues on a physical server if they suspect problems with the hypervisor. Isn't it great that we can do this easily with Replication Manager? ;-)
• Oracle only supports Oracle RAC on VMware for one specific version (11.2.0.2). Any other version with RAC is not recommended on VMware because of support issues. Expected to change in the future.
• Both EMC and VMware offer additional support guarantees for customers deploying Oracle on Vmware. So where Oracle pulls back, EMC and VMware will fix any issue anyway.
• Performance is no longer an issue. With Vsphere 5, a single virtual machine can have 32 virtual processors, 1 TB ram and drive 1 million iops. Only the most demanding workloads would not fit in this footprint. But with customers running hundreds of databases, maybe we should start with the 95% + that DO fit and make significant savings there. By the time we're done, VMware will have Vsphere 6 and who knows what happens then.

How to get around the licensing issue

As I said, Oracle requires licenses for all servers in a cluster. So how do you limit the number of licenses? By deploying an Oracle-only VMware cluster. Only run Oracle databases here. No apps, no middleware, no fileservers, and try to move everything off that does not relate to database processing. No host replication, no storage mirroring, etc.

Say you have a legacy environment with 10 servers, each with 16 cores, so you have 160 cores licensed with oracle EE and a bunch of options. Average CPU load is 15% but let's assume 20% to be conservative.

I claim that a single VMware cluster with 3 servers each with 32 cores will easily do the job. Now we have 3 * 32 = 96 cores to be licensed. 96/160 = 0.6 = 60% so we saved 40% on licensing right away. Probably the average CPU load on the whole cluster will still be much less than 70% so we can gradually add a bunch more databases until we average out on 70%.

If the old system was not running Intel x86 but SPARC, PA-RISC or POWER cpu's then the processor factor was probably 1.0 or 0.75. Intel has 0.5. So for 96 cores (Intel) you would need to pay 48 full licenses. Another 33% savings.

The savings of 40% on licensing will easily justify an investment of a nice new EMC storage infrastructure with EFDs, FAST-VP and all other goodies. Do you think the customer will push us hard for a $0.01 lower GB price competing HDS or Netapp if we just saved them millions in Oracle licenses?

But the story does not end here.

Additional savings

Let's assume the customer needed high availability and scale-out performance and was running Oracle RAC. RAC is the most expensive licensed option and you need at least two for a two-node cluster. But VMware allows for HA (High Availaiblity clustering) as well. Using VMware HA instead of RAC, you would have to fail-over and recover the database in case of an outage - if the customer cannot tolerate this then he needs to stick with RAC (only for mission critical databases!). But most customers can live with 5 minutes of downtime in case a server CPU fails and in that case, replacing RAC with VMware HA can save them another big bunch of dollars.

Let's assume that with virtualization you justified the investment in a nice EMC infrastructure with Flash drives to replace the competitive gear. Now the Oracle cluster is no longer limited by storage I/O's and can drive more workload out of the same 3 VMware servers in the cluster. But you can also replace host mirroring (where applicable). You can implement snapshot backups to get the I/O load away from the production servers. You removed the middleware and apps stuff from the database servers - reducing CPU utilization and allowing even more headroom for DB consolidation - all without buying extra licenses from Oracle.

You have a customer who wants even more?

What if they create TWO database clusters for VMware? One for production (running Oracle Enterprise Edition (EE) with all the options they need) and one for Non-prod (running Oracle Standard Edition (SE) without options - good enough for test/dev and smaller, non-mission critical workloads). I bet the number of non-prod databases will be much more than prod. By removing the expensive options AND moving from Enterprise to Standard Edition, you saved another ton of money on Oracle licensing as SE is much cheaper than EE. But be aware - the devil is in the details and using Standard Edition is not for the faint-of-heart (for example, you could no longer clone a partitioned database to a SE enabled server because of the missing license and functionality). Still if the customer is keen on saving as much as possible, then this might be the final silver bullet...

Do they run a huge Enterprise Data warehouse? Carefully find out if they have troubles with it and see if you can position Greenplum - saving another bag of money and speeding up their BI queries. But be careful, in an Oracle-religious shop it might backfire on you...

Reality Check

I had this discussion already with a few enterprise customers. And found that although the story is easy in theory, the reality is different. If a customer already has the 160 CPU licenses purchased from Oracle, then the Oracle rep will not happily give a money-back in return of the shelfware licenses. So in that case the customer can only save on maintenance and support. But having enough licenses on the shelf, he would not have to purchase any more for the next 5 to 10 years. So talk cost avoidance instead of immediate savings. And again, if they are licensed by user or have a site license, then saving on licenses will be a tough discussion. Still, the savings on power/cooling/hardware/floorspace would still be significant enough to proceed anyway.

And don't forget the other benefits of private cloud of which we all know how to position: they are no different for Oracle than for other business applications.

Final thought

For this to work you need a customer that is willing to work with you and be open on how they negociated with Oracle, and a team of DB engineers to work with you to make it happen. If internal politics cause significant roadblocks then you will get nowhere.

It's not an easy sell but the rewards can be massive. We're only just starting to figure out how to convince customers and drive this approach. Feedback welcome and let me know if you need support.

Resources

The online Everything Oracle at EMC community has lots of information on this subject. See in particular this presentation which I co-presented (with Sam Lucido) at this year's VMworld.

Landing page with Oracle 's price list: http://www.oracle.com/us/corporate/pricing/price-lists/index.html

Download "US Oracle Technology Commercial Price List" for the database license document. Read the fine print because it's not always as simple as it seems.

Processor Core Factor information:http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf