Oracle Storage Guy

Oracle Licensing on VMware - no magic

Acknowledgements to Bart Sjerps for this content.

There seems to be a lot of confusion on licensing when customers consider running Oracle databases on VMware. Part of the confusion is caused by Oracle on purpose (classic FUD) by suggesting licensing is more expensive on VMware than on physical servers. The reality couldn't be more different - I strongly believe that many customers can actually *save* on database licenses by going virtual. But to understand how to achieve this, you need to know a few things - I hope I can clear this up in a short explanation. I will keep the discussion to Oracle database licenses and ignore application/middleware etc. for now.

License models

Customers typically license their basic database by one out of three options:

1. License by CPU (core) - the more CPU cores, the more licenses are needed. There is a processor core factor depending on the type of CPU and can be 0.25, 0.5, 0.75 or 1.0.
2. License by named user - the more named users, the more licenses are needed. The amount of CPU's is not important, neither the amount of total databases. Typically one license pack per 25 users.
3. Enterprise License - the customer negotiates a contract for the whole company and afterwards can deploy as many databases on as many servers/cpus as he wants.

If a customer uses 2 or 3, then it does not matter if they run virtual or physical. But there are also no license savings possible without re-negotiating their contracts. I don't want to go as far as suggesting to customers to change their license models so we leave this as-is for now.

In my experience, most enterprise customers use either cpu licensing or enterprise contracts. Some have different licensing methods for different business units. Oracle can be very creative in customer-specific contracts so expect to find a different situation for each individual customer.

But let's assume CPU licensing for the sake of this discussion.

Maintenance & support

Users typically buy the CPU licenses  but then have to pay maintenance for the time they use the licenses. Yearly maintenance cost is about 25% of license (list price). I have no information on typical discounts. I expect customers to get at least 50% discount off the price list (but only on licenses, not on maintenance AFAIK).

Database Edition and options

The plain database license comes in 3 versions (for servers):

1. Standard Edition One - Maximum 2 processors, no options allowed. Only used for testing and very small deployments
2. Standard Edition (SE) - Maximum 4 processors, no options allowed. Only used for smaller sizes and workloads (but stay tuned)
3. Enterprise Edition (EE) - No limitations and on top of EE, you can have many licensed features. Most customers will use this, at least for production databases

On top of the basic Database license, most customers use a set of options, each requiring additional licenses per CPU. The most common options are:

• Real Application Clusters (RAC) - allows many servers running the same database (active-active clustering) to allow scale out performance and high availability.
• Real Application Clusters One Node - same but one database can only run actively on one node. For high availability only.
• Active Data Guard - remote replication using log shipping. Note that standard Data Guard is free, but Active Data Guard allows the standby database to be opened for read-only purposes and offers some extra features.
• Partitioning - allows tables to be split up in smaller chunks. Absolutely required when running large databases and no downtime can be tolerated. Eases administration work and offers some performance benefits.
• Real Application Testing - allows workloads to be recorded and re-played on another database to do performance and functionality testing
• Advanced Compression - allows database blocks to be compressed - requiring less storage and boosting performance (in most cases).
• Diagnostics Pack / Tuning pack - provides automated reports. Oracle AWR (Advanced Workload Reports - a performance reporting tool) is part of Tuning Pack.

In my experience, nearly all customers have partitioning. Most customers have tuning/diagnostics pack. Some customers have RAC. Some customers have the other options. There are more options available but these are the most common.

Many customers have 3 or more options - sometimes the options cost more than the base database license - especially if they use RAC they will have most of the other options, too.

Running on a cluster

If a database runs on a cluster, then Oracle assumes the database can make use of any processor in the cluster. This is independent on what kind of cluster is used (so can be MSCS, HP MCSG, Vmware, Oracle RAC, etc).

This is basically the foundation for all FUD and confusion. For example, if you deploy a VMware farm (cluster) of 16 servers, and all virtual machines run all kinds of stuff (file/print, exchange, apps, etc etc) and only one tiny virtual machine in the corner, with only one virtual CPU runs a small Oracle database, you would expect only to pay for one CPU core - but Oracle's reasoning is that this tiny VM can be dynamically moved (VMotion) to all nodes in the cluster and on any processor. Therefore, all CPU's have to be fully licensed by Oracle. So in this case, running the single database on a (small) physical server would be cheaper than running on a VM in the farm.

Total cost of the stack

In a typical database server deployment, the cost of the database licensing is far greater than the cost of the hardware + OS licenses combined. I have no hard numbers but I assume the average DB license cost (plus options) is 10 times larger than the cost of the server + OS.

So a $5,000 server would typically require $50,000 on licenses. Then because maintenance is 25% yearly, the total cost of licenses over a 3 to 5 year period is even higher - so for a 5 year TCO the total license cost might be $75,000 (assumption - could also be closer to $100,000 - and no, I didn't make a mistake with an extra zero, Oracle *really* is this expensive).

Utilization

It is very hard to size a typical Oracle database based application. There are no good methods or calculations to figure out how much CPU power, disk I/O and memory is needed to run a given app. So historically, project teams size their database servers for peak loads, and because they cannot predict how big the peak load is, they double the resources "just in case". The end result is that most database servers are way oversized in terms of CPU and memory.

Most physical deployed database servers will average on about 10-15% CPU load (or less). However, they will peak to higher loads at certain times, such as monday morning when many users log in, or when month/quarter/year-end batch processing is started, etc.

Then, the utilization numbers can be influenced by other tasks of the processors. Some common causes of "artificially high" CPU loads on database servers:

• CPU is involved in storage mirroring (i.e. Host Level Mirroring - using Oracel ASM or a Unix volume manager)
• CPU is involved in file transfers over the IP network
• Backup (non-serverless, using CPU, Network and I/O bandwidth)
• Customers run the application server on the same machine driving up CPU load - This can drive up CPU load from 10% to 90% or more !!
• Same for Middleware and Enterprise Service Buses (Think Oracle BEA, IBM Websphere, SAP Netweaver, etc)
• A  bunch of monitoring/management agents burn CPU cycles (Tivoli, BMC, HP Openview, CA, etc). Each agent maybe consuming 1% but add it up and you have another 5-10% overhead.
• Administrators generate database dumps / exports and run their own reports, scripts and tools. They run ad-hoc queries as well that should not be on production.
• Poorly tuned database servers cause paging and other CPU overhead - hard to diagnose but driving up CPU and I/O significantly.
• Database admin tasks (table reorganizations, (re)building indexes, converting tablespaces, ...)
• And so on...

All of these cause the processors, expensively licensed for database processing, to do other stuff.

So if a server is running at 15% utilization, then the utilization caused by the database workload itself might only be 10% and the rest caused by other stuff (whether needed or not).

Needless to say that Oracle likes customers to use their expensive licensed CPU's for other tasks because it forces them to buy additional CPUs sooner and therefore drive their license revenues.

Isn't life great for an Oracle rep? ;-)

Number of databases

Most customers run many databases. For the average enterprise customer that I visit, 100+ databases is a normal number. A big global that I visited runs 3000+ Oracle databases worldwide (and this is only the scope of this specific project team). Imagine the cost of licensing all these databases on all individual servers...

Why so many? Well, customers do not like to share multiple applications on one database (and often this is not even supported). So if you run SAP ERP, Oracle JD Edwards, your own banking app and a few others, they all require their own production database.

For each production database, you might find an acceptance environment, test system, development server, maybe a staging area to load data into the data warehouse, maybe a firefighting environment, a standby for D/R, a training system and so on. Customers will rarely share production environments on the same server (unless virtualized or at least with workload management segregation). Sometimes they share a few databases for non-prod on a server. So for, say, 100 databases, the average customer runs between 30 and 50 (physical) servers.

Power of big numbers

It does not require rocket science to understand that many of these databases do not require peak performance at the same time. A development system typically drives workload during daytime (when developers are coding new application features). A data warehouse runs queries during the day and loads in the evening. For a production system it depends on the business process. An acceptance system might sit idle for weeks and then suddenly peak for a few days preparing for a new version deployment into the live production system. And so on.

So what if you could share resources across databases - without influencing code levels, security, stability and so on?

If that would be possible - you would not size for "peak load times two" anymore. You would size for what you expect and assume an average utilization of, say, 70% over the whole landscape. If one database needs extra horsepower, there is enough available in the landscape.

How much license cost would you save by bringing down the number of CPU's so that utilization goes up from 10% to 70%?

What would be the effect on power, cooling, floor space, hardware investments, time-to-market?

What would be the business advantage of not limiting production performance of a single server, by whatever was sized during initial deployment? Risk avoidance?

What would be the business advantage of solving future performance issues by just adding the latest and greatest Intel server in the cluster and Vmotion the troubled database over?

Wasn't this exactly why we started server virtualization in the first place about 8 years ago? And why EMC aquired VMware?

Wouldn't you think the average Oracle sales rep is scared to death when his customer starts considering to run his databases on a virtual (cloud) platform? Would it make sense for him to drive his customers mad with FUD around licensing, support issues and whatever he can think of to prevent his customers going this way? Even threatening to drop all support if they continue to go in that direction?

If Oracle is scared of losing license revenue, wouldn't you think there is a huge potential for savings for our customers here?

The journey to the private database cloud

So how should we deal with this?

A few starting points:

• Oracle supports VMware. Period. Any other claim of Oracle reps can be taken with a grain of salt (to be more specific: it's nonsense).
• Oracle does NOT certify VMware. Then again, Oracle does not certify anything except their own hard- and software. But IMO, support is all you need and the discussion around certification leads nowhere.
• Oracle might ask the customer to recreate issues on a physical server if they suspect problems with the hypervisor. Isn't it great that we can do this easily with Replication Manager? ;-)
• Oracle only supports Oracle RAC on VMware for one specific version (11.2.0.2). Any other version with RAC is not recommended on VMware because of support issues. Expected to change in the future.
• Both EMC and VMware offer additional support guarantees for customers deploying Oracle on Vmware. So where Oracle pulls back, EMC and VMware will fix any issue anyway.
• Performance is no longer an issue. With Vsphere 5, a single virtual machine can have 32 virtual processors, 1 TB ram and drive 1 million iops. Only the most demanding workloads would not fit in this footprint. But with customers running hundreds of databases, maybe we should start with the 95% + that DO fit and make significant savings there. By the time we're done, VMware will have Vsphere 6 and who knows what happens then.

How to get around the licensing issue

As I said, Oracle requires licenses for all servers in a cluster. So how do you limit the number of licenses? By deploying an Oracle-only VMware cluster. Only run Oracle databases here. No apps, no middleware, no fileservers, and try to move everything off that does not relate to database processing. No host replication, no storage mirroring, etc.

Say you have a legacy environment with 10 servers, each with 16 cores, so you have 160 cores licensed with oracle EE and a bunch of options. Average CPU load is 15% but let's assume 20% to be conservative.

I claim that a single VMware cluster with 3 servers each with 32 cores will easily do the job. Now we have 3 * 32 = 96 cores to be licensed. 96/160 = 0.6 = 60% so we saved 40% on licensing right away. Probably the average CPU load on the whole cluster will still be much less than 70% so we can gradually add a bunch more databases until we average out on 70%.

If the old system was not running Intel x86 but SPARC, PA-RISC or POWER cpu's then the processor factor was probably 1.0 or 0.75. Intel has 0.5. So for 96 cores (Intel) you would need to pay 48 full licenses. Another 33% savings.

The savings of 40% on licensing will easily justify an investment of a nice new EMC storage infrastructure with EFDs, FAST-VP and all other goodies. Do you think the customer will push us hard for a $0.01 lower GB price competing HDS or Netapp if we just saved them millions in Oracle licenses?

But the story does not end here.

Additional savings

Let's assume the customer needed high availability and scale-out performance and was running Oracle RAC. RAC is the most expensive licensed option and you need at least two for a two-node cluster. But VMware allows for HA (High Availaiblity clustering) as well. Using VMware HA instead of RAC, you would have to fail-over and recover the database in case of an outage - if the customer cannot tolerate this then he needs to stick with RAC (only for mission critical databases!). But most customers can live with 5 minutes of downtime in case a server CPU fails and in that case, replacing RAC with VMware HA can save them another big bunch of dollars.

Let's assume that with virtualization you justified the investment in a nice EMC infrastructure with Flash drives to replace the competitive gear. Now the Oracle cluster is no longer limited by storage I/O's and can drive more workload out of the same 3 VMware servers in the cluster. But you can also replace host mirroring (where applicable). You can implement snapshot backups to get the I/O load away from the production servers. You removed the middleware and apps stuff from the database servers - reducing CPU utilization and allowing even more headroom for DB consolidation - all without buying extra licenses from Oracle.

You have a customer who wants even more?

What if they create TWO database clusters for VMware? One for production (running Oracle Enterprise Edition (EE) with all the options they need) and one for Non-prod (running Oracle Standard Edition (SE) without options - good enough for test/dev and smaller, non-mission critical workloads). I bet the number of non-prod databases will be much more than prod. By removing the expensive options AND moving from Enterprise to Standard Edition, you saved another ton of money on Oracle licensing as SE is much cheaper than EE. But be aware - the devil is in the details and using Standard Edition is not for the faint-of-heart (for example, you could no longer clone a partitioned database to a SE enabled server because of the missing license and functionality). Still if the customer is keen on saving as much as possible, then this might be the final silver bullet...

Do they run a huge Enterprise Data warehouse? Carefully find out if they have troubles with it and see if you can position Greenplum - saving another bag of money and speeding up their BI queries. But be careful, in an Oracle-religious shop it might backfire on you...

Reality Check

I had this discussion already with a few enterprise customers. And found that although the story is easy in theory, the reality is different. If a customer already has the 160 CPU licenses purchased from Oracle, then the Oracle rep will not happily give a money-back in return of the shelfware licenses. So in that case the customer can only save on maintenance and support. But having enough licenses on the shelf, he would not have to purchase any more for the next 5 to 10 years. So talk cost avoidance instead of immediate savings. And again, if they are licensed by user or have a site license, then saving on licenses will be a tough discussion. Still, the savings on power/cooling/hardware/floorspace would still be significant enough to proceed anyway.

And don't forget the other benefits of private cloud of which we all know how to position: they are no different for Oracle than for other business applications.

Final thought

For this to work you need a customer that is willing to work with you and be open on how they negociated with Oracle, and a team of DB engineers to work with you to make it happen. If internal politics cause significant roadblocks then you will get nowhere.

It's not an easy sell but the rewards can be massive. We're only just starting to figure out how to convince customers and drive this approach. Feedback welcome and let me know if you need support.

Resources

The online Everything Oracle at EMC community has lots of information on this subject. See in particular this presentation which I co-presented (with Sam Lucido) at this year's VMworld.

Landing page with Oracle 's price list: http://www.oracle.com/us/corporate/pricing/price-lists/index.html

Download "US Oracle Technology Commercial Price List" for the database license document. Read the fine print because it's not always as simple as it seems.

Processor Core Factor information:http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf

 

Darryl Smith's Oracle RAC Recent Blog Post

EMC IT's Darryl Smith recently issued an outstanding blog post on our ECN community, the subject of which was: Is Oracle RAC Becoming More of a Corner Case? I thought I would add a few thoughts to Darry's blog post.

I have been saying for a while that most Oracle databases do not need the level of uptime and fault tolerance that RAC provides, so Darryl and I are certainly thinking along the same lines.

I think that the value proposition for vSphere and NFS are very similar. Given that I have spent the bulk of my career pushing NFS and NAS for Oracle database storage, the synergy is obvious at least to me.

Both technologies are about having a "good enough" infrastructure for an Oracle database which, while certainly important to the business, is not the back-end for an online catalog, or an online securities trading app.

For databases like that, I would recommend neither vSphere nor NFS. But the vast, vast majority of databases running Oracle do not fall into this category.

In the case of NFS, for years I made the statement that I believe that 90% of all Oracle databases running in datacenters all of the world could be run over an NFS mount with absolutely no change in performance, reliability, or user experience. (There would, on average, be a big reduction in cost and improvement in manageability, though.)

vSphere is exactly the same. For anything other than the most barn-burning performance, with absolutely the highest standards of fault tolerance, vSphere works just fine. It provides a very high level of reliability, a few minutes a year of downtime, at a vast reduction in cost and complexity compared to RAC.

Because the value propositions are so similar, I believe that the combination of NFS and vSphere is going to become increasingly popular. We'll certainly see how it turns out.

 

Oracle OpenWorld 2011 Technical Sessions: Part 1

This begins a series of blog posts on the subject of technical sessions that EMC presented at OOW2011. To begin with, we will examine the session that I co-presented with Kevin Jernigan entitled Clone Oracle Databases Online in Seconds with Oracle Clonedb and DirectNFS. The presentation for this session can be found here.

It may sound strange given EMC's reputation and history, but EMC has a strong partnership with Oracle in the area of NAS. We began working with Oracle on the Direct NFS client ("dNFS") way back in 2007, when dNFS was introduced as a major new feature in Oracle Database 11g Release 1. At that time, EMC, was a co-presenter (together with Oracle) at Oracle OpenWorld 2007 on this subject.

The goals of dNFS were simple, and well defined:

• Improve performance (in terms of latency and throughput) of NFS-mounted Oracle database I/O
• Reduce CPU cost of NFS-mounted Oracle database I/O
• Improve network port scalability and failover of NFS-mounted Oracle database I/O
• Simplify administration of NFS in terms of both network port scaling and mount point parameters
• Make administration of NFS uniform across all platforms

dNFS succeeds widely at all of these. dNFS dramatically improves latency of network I/O from Oracle by eliminating most context switches and making the code path to the disk much shorter. dNFS also provides better port scaling than kernel NFS, with much simpler administration. No fussy ether channel network switch configuration is required. All of the port scaling and port failover is provided within the Oracle environment.

 

dNFS also made administration completely uniform across all platforms. Amusingly, Windows is included! NFS now works on Windows, at least with an Oracle database.

In September 2010, Oracle announced that dNFS would be improved in Oracle Database 11g Release 2. One major improvement is the addition of dNFS clonedb, a thinly-provisioned, rapid database replication feature. This feature also works very well with storage-based replication.

 

The basis for dNFS clonedb is a database copy. This copy can be a backup, a storage-based snapshot or clone, or an operating system copy. (Of course, EMC likes to leverage our storage-based snaps and clones.)

 

Once a copy exists, it can be used to create a clonedb instance. The steps to do this are contained in My Oracle Support article 1210656.1 . Effectively, this creates a read / write virtual database which takes up minimal space, is created almost instantly, and contains only the space required to store any changes to the database. Given that storage-based snapshots are also space and time efficient (that is, they take up very little space, and can be created very quickly) there is a great deal of synergy between these technologies.

 

EMC did some performance testing with dNFS clonedb. The network diagram for the testbed is here:

First, we established a baseline in terms of performance. We ran an OLTP workload against the production database with no additional operations. This produced the following:

Notice the perfectly clean scaling of this performance chart. We then tested the performance of creating a snapshot to serve as the source for the clonedb database. This produced:

Note the slight response time hit when the snapshot was taken. However, transactional throughput was not affected. Basically, this is at most a minimal performance hit. Finally we tested performance while creating clonedb database instances from the storage snapshot. This produced:

Note that there was no performance hit during clonedb creation at all. One additional test was performed. We measured the storage space occupied by the clonedb when it was created. A 10 TB database was used as the source. The total space occupied by the clonedb was only 7 MB.

See the link above for the full presentation of this technical session. Further blog posts in this series will contain summaries of other EMC technical sessions at OOW 2011.

Thoughts on Oracle OpenWorld 2011 Keynotes

I attended Oracle OpenWorld 2011 as usual this year, at which I co-presented a technical session with Kevin Jernigan on the subject of Direct NFS clonedb. Since I had a full conference pass, I was able to attend many of the keynotes at this event.

I did not get to attend the Sunday Larry keynote, although I understand that it was pretty terrible. I did attend the Wednesday Larry keynote, and can tell you this: As a long-term Larry watcher (I have attended every OOW since 1998) it was hands down the worst Larry keynote I have ever seen.

Perhaps because Larry had just lost his best friend (Steve Jobs died of cancer the day of this keynote), he was off his game. That's the only reasonable explanation I have for such a spectacular failure of such a precious and expensive opportunity.

Larry's keynote consisted largely of a 20 minute-plus long demo of Fusion Applications, which while interesting in itself, was excrutiatingly dull, and definitely would have been much better if handled by someone other than Larry.

What Larry is great at (and what was largely lacking in his keynote) is the flamboyant, extravagant, and provocative way that he baits his competitors and makes outlandish claims concerning his own products' performance. He did a bit of sniping at (SalesForce.com Chairman and CEO) Marc Benioff. Aside from that, nothing.

The other Oracle keynotes were not much better. This was an almost content-free OOW for me. The ExaData and ExaLogic stuff was more of the same from last year. The big product launch was of course Fusion Applications, but that is just software. There was no big splash on the hardware side.

The best keynotes by far were Joe Tucci and John Chambers. Both provided a lot of content, much of which was competitive to Oracle, interestingly. Cisco is certainly competing with ExaData in the server market and EMC competes with Oracle in storage, virtualization and datawarehousing software. All of this was discussed extensively during these keynotes. Joe even said the "v" word (VMware) during his keynote, and Pat further elaborated on this subject during his follow on to Joe's talk. Pat also included a long demo of vCenter which was assisted by EMC's own Chad Sakac.

Chambers talk was very compelling to me. His vision for a new platform in the post-PC age was a serious challenge to both Microsoft and Apple in their ecosystems. I am a bit skeptical that Cisco can successfully compete with those vendors in their respective spaces, though. Microsoft owns the business productivity space, and Apple has a serious stake in the home and home office space. Cisco will need to penetrate both of these in order to fulfill Chamber's vision. We'll see.

I thought OOW was about Oracle. This year, not so much. This year, the interesting stuff at OOW was from other vendors. Score one for EMC and Cisco, and score zero for Oracle this year.

Announcing the Oracle Heretic Blog

Many of you have been following me for some time on the Oracle Storage Guy blog. I have been continuously receiving page hits and comments on blog posts I made years ago. I am grateful for all of the support you have given me.

As many of you have noticed, The Oracle Storage Guy blog has been quiet lately. I have been moving into a new role, reporting to Sam Lucido, and in that role I have taken on responsibilities relating to social networking. Specifically, I am now (together with Sam) responsible for administering the Everything Oracle at EMC website. As I have moved out of my former role within EMC's Global Solutions Organization (now Enterprise Solutions Group), I will be more involved with customers, speaking at tradeshows and the like. In that role, blogging is becoming more of a critical part of my role.

I am also actively on twitter, also at OracleHeretic.

This post announces the creation of the Oracle Heretic blog. The thrust of the new blog will be to discuss pressing issues of the Oracle technical space, especially as it relates to storage and virtualization. The tone of this new blog will be slightly stronger than the old Oracle Storage Guy blog as well. I will be more assertive here than I have been in the past. Hopefully, this fresh new approach will be welcome. Please watch the new blog, and let me know your thoughts! I will continue to post all of my new posts in both locations for the foreseeable future as well.

 

Up-Coming Opportunity to Present Oracle Direct NFS clonedb at OOW 2011

If you read my previous post, you realize how exciting this feature us for those of us who are attached to NAS storage for Oracle databases. I have submitted a technical session at Oracle OpenWorld 2011 on this subject, but I need your help to get it accepted.

Please log onto OracleMix and vote for my session at this location. Thanks very much for any help that you can provide, and please let me know if you are coming to OOW.

High Performance Database Cloning with Oracle Direct NFS Client clonedb

For those who attended EMCWorld in April, you are probably aware of the joint presentation that I made with Kevin Jernigan of Oracle. This session was sponsored by Oracle, and showcased the joint solution that EMC and Oracle have created together around the use of Oracle Direct NFS Client (dNFS) clonedb to create thinly provisioned, on-demand, lightweight clones of a running production Oracle database. Oracle and EMC are also jointly writing a white paper covering this solution, which will be published imminently on EMC's Powerlink website. I will advise you on this blog when this white paper is available.

In the meantime, the purpose of this post is to provide a summary of this solution, and give an overview of the results of the joint EMC / Oracle testing.

First a bit of history. Although EMC does not have the reputation in NAS storage for Oracle databases that some other vendors have, we have done a lot of work in this area, and I believe we have captured leadership.

EMC has been working on dNFS with Oracle since it went into beta in 2007 as part of Oracle Database 11g Release 1. We made our first joint presentation with Oracle on dNFS on EMC storage at Oracle OpenWorld in 2007.

We began work on a scalable performance solution in 2010, demonstrating performance within 5% of ASM over FC on equivalent server and storage hardware. This solution was also jointly published with Oracle.

Finally in this year, we have produced the dNFS clonedb solution which is the subject of this post, and which was presented jointly by Oracle and EMC and EMCWorld in April of this year. We also hope to jointly present this solution at Oracle OpenWorld 2011. More on this later.

Now some background. First on NFS. NFS was created by Sun in 1985. It is an RFC-based UNIX protocol based upon TCP/IP, and a sibling of FTP, SSH, POP, and so forth. NFS was undoubtedly the first widely deployed method for file sharing in the industry.

NFS and NAS are equivalent terms within the database storage industry. NAS simply refers to the storage of databases on a NAS array over NFS using an IP storage network. NFS became popular for database storage starting in the mid-1990s. I was personally involved in that movement during the period that I was working for NetApp. See my earlier posts in this blog for details on that.

NFS provided the following benefits relative to traditional SAN storage over FC:

• Cheap, low TCO
• Simple and fast to deploy, because of the underlying network technology, i.e. TCP/IP over Ethernet
• Easy storage virtualization
• Easy shared file system which supported grid computing

The challenges of NFS were:

• Relatively low performance compared to FC. This is because the I/O algorithms within NFS were optimized for file server I/O, i.e. a large number of lightweight processes using relatively low level I/O against a very large number of files. Database I/O is the exact opposite: A small number of very heavyweight processes doing a lot of high level I/O against a fairly small number of files.
• High CPU utilization of the protocol
• Management complexity across platforms. Each platform had its own wrinkles in terms of things like mount point parameters
• Poor scalability across Ethernet ports, requiring the use of complex, CPU intense port trunking at the Linux (or UNIX) kernel level

In response to these challenges, in 2007, Oracle introduced dNFS as part of Oracle Database 11g Release 1. dNFS was simply an implementation of the NFS protocol inside the Oracle database kernel. The benefits provided by dNFS were:

• Improved performance by tuning the protocol to match typical database I/O patterns
• Reduced CPU utilization
• Uniform management interface across all platforms (including, ironically, Microsoft Windows)
• Vastly superior transparent port scaling by implementing the ability to pool Ethernet ports within dNFS, rather than within the OS layer.

The results, as demonstrated by the testing performed by both Oracle and EMC have been exceptional: dNFS provides nearly equivalent performance to ASM over FC, while allowing the customer to use IP storage. This provides lower cost, better TCO, and simpler management. dNFS has been a wildly successful product for Oracle as a result.

Now to dNFS clonedb. In 2010, Oracle introduced dNFS clonedb as part of Oracle Database 11g Release 2. The idea behind dNFS clonedb is to create a feature which allows you to instantly clone a running Oracle production database which is mounted over dNFS. The clone is based upon a copy of the production database. This copy can be a storage-layer virtual copy, however, including a snapshots. Using the combination of this copy, plus the running production database, you can create an unlimited number of clones, each of which takes up minimal space, and has minimal performance impact on the production database.

Thus, the benefits of dNFS clonedb are:

• Clones are created instantly
• You can have an unlimited number of clones
• Clones take up minimal space
• No performance impact on the production database
• Performance of I/O against the clone is comparable to the production database

The objectives of the solution were simply to prove these benefits.

The solution has two major components:

• RAC production database with single instance dNFS clonedb (RAC testbed)
• Both production database and dNFS clonedb as single instance (SI testbed)

Two sets of testing were required because we wanted to test the impact of creating dNFS clonedb databases against a highly-scalable RAC implementation. Most of the testing was in this environment, so we will start with that one. However, we also wanted to compare the performance of the dNFS clonedb database itself to a normal dNFS mounted database. And since the dNFS clonedb database was running on a modest single-instance Oracle database server, this required for us to perform an additional performance test against a production server on the same hardware. This enabled us to directly compare the performance of dNFS clonedb against a normal production database.

Let's start with the testing of the RAC component. The network diagram for this component was as follows:

As shown in the diagram, the major features of the RAC environment are:

• Oracle Database (and RAC) 11g Release 2, which includes dNFS and clonedb
• EMC NS-960 unified storage array
• EMC Replication Manager (RM) which is responsible for managing the process of creating snapshots (using EMC Celerra SnapSure checkpoint) of the running Oracle production database

The first step of the testing was to create a baseline of normal performance on the production RAC database. This was done using an industry-standard OLTP benchmark. This produced the following result:

The next step was to test the performance impact of creating snapshots using EMC Celerra SnapSure checkpoint and RM. RM takes the database into hot backup mode while the snapshot is being created, and this results in a slight response time performance hit. However, TPS is not affected, and the response time effect is very short-lived (only during the brief period when the database is in hot backup). Here is that result:

And finally, on the last test in the RAC component, we created two clone databases using dNFS clonedb. These operations were run while the production database was under load using the OLTP performance benchmark, of course. The effect of these operations on performance was minuscule: We simply could not find any effect at all. The run was identical to the baseline for all practical purposes. We concluded that dNFS clonedb has no perceptible impact on the production database. Here is that result:

As you can see, as I said, this is basically identical to the baseline.

Finally, we need to add the single instance testing to show the performance impact of clonedb on the clone database itself. Here is the network diagram for that testing:

This is identical to the RAC testbed, with the exception that the production database is on a single instance server, instead of a RAC cluster. Also, the hardware configuration used for the production database was identical to that used for the clone database. This allowed us to run a baseline on the production system, and then compare that to the a performance run against the clone database. This produced the following result:

At the peak TPS, this result produced the following performance:

    clonedb:            400 TPS
    Non-clonedb:     445 TPS
    Difference:        10%

Thus, the clonedb feature under an OLTP workload is within about 10% of a normal dNFS mounted database, which I consider to be quite respectable.

We also timed the clonedb creation and snapshot creation operations. They were very brief. The snapshot creation operation, including a complete RM run with hot backup, mounting, and so forth, took about 7 minutes. Creating a clonedb database only took about 10 seconds.

The final benefit we tested was space. At creation, the clonedb database takes up virtually no space, as the following chart indicates:

Thus, at creation, a clonedb of an 11 TB production database took up only 7.4 MB of space. This is slightly misleading though. As edits are made to the clonedb database, it will begin to occupy space. The space overhead of a clonedb database can actually be easily determined as the number of database blocks which are different between the clonedb and the production database. Nonetheless, the typical clonedb database can be expected to take up far less space than a physical copy of the production database.

I believe that dNFS clonedb is an incredibly attractive and powerful addition to Oracle's already feature-rich database offering. I would recommend anyone interested in exploring NAS storage for Oracle databases to give dNFS a strong look, and especially to consider clonedb for creating clones of production databases.

What is Big Data?

I have been reading lots of material on the web about Big Data. As well as attending EMC World a couple of weeks ago, at which the major theme was around Big Data meets the Cloud. So I have been immersed in this area for a while now.

Unfortunately, I think folks don't really understand what Big Data really is. At least, the definitions that I read on the web (and see at tradeshow sessions) miss the essence of the problem.

I have always believed that if I am confused other folks are confused as well. Thus, I think that the issue of Big Data (which is undoubtedly one of the major IT issues of our time) needs to be clarified.

In this post, I will attempt to define what I think Big Data is, and in the process, hopefully spark some debate. Perhaps this will help clarify things a bit. In my view, most Big Data applications have several characteristics in common:

• Perhaps it goes without saying but the data being managed by Big Data is big. This is the one characteristic that most of the pundits seem to "get". Scaling to hundreds of terabytes is common, and petabyte scaling is not unheard of. This creates unique challenges, as you can imagine.
• The first challenge is performance. Big Data applications are volatile. An enormous percentage of the data gets overwritten very quickly. When you have terabytes of data being written very quickly, that means that traditional SAN storage and software like Oracle Database simply won't cut it in terms of the throughput required. A new way of thinking about the problem is required, and this new way of thinking primarily concerns major order-of-magnitude increases in throughput.
• The next challenge is cost. The hardware and software being used currently for most IT applications is simply too expensive and over-engineered for Big Data, because the cost per gigabyte is way too high. Most Big Data applications would be impossible if conventional, old-school hardware and software (like traditional SAN storage and Oracle Database) were used to manage the application. Thus, another new way of thinking is required in order to manage Big Data which revolves around dramatically reducing cost per gigabyte. This issue, combined with the previous issue, requires a new type of hardware and software, hence the use of large in-memory arrays and software like Hadoop.
• Traditional transactional properties like ACID simply do not apply to most Big Data applications. Hardware like traditional SAN storage and software like Oracle Database are oriented around one concept: Data is extremely precious and must be protected at all costs. In Big Data the use of "fuzzy logic" is very common: It is not any individual piece of data which is precious. Rather, it is the overall texture of the data which is important. Any individual piece of data is of trivial value. Many of my customers tell me that this change in mindset is where the costs savings come from. Once you divorce yourself from the concept of ACID properties and data which is guaranteed to be transactionally consistent, it becomes possible to create applications with orders of magnitude lower cost and higher performance than traditional transaction databases like Oracle.

The effect is that many Big Data applications are not even backed up. The current "state" of the Big Data is important: Preserving the data content historically is not important. Thus, Big Data applications are regularly polled and summarized. The summaries are then saved into a traditional transaction database like Oracle and stored on a traditional storage array. But of course this represents a minute fraction of the data managed by the Big Data application.

I will give you two examples of what I am talking about. The first concerns a major, multinational shipping corporation, which is a household name. I do not have permission to share the name of this customer, so I will not do so here. However, this is one of the largest and most well-established online shipping companies in the world.

One of the promises implicit in the business model of this company is promptness of delivery. If a customer ships a package overnight, the company guarantees that this package will be at the destination within that timeframe, and maintaining absolutely compliance with that promise is one of the company's core values.

This creates a tension: In the case of a peak in usage at a given location, the company can do one of the following two things:

1. Maintain a staff, facility, and equipment infrastructure sufficient to manage the peak usage that could ever be imagined at a given location, in the process wasting large amounts of money.
2. Maintain a reasonable level of staff, facility and equipment consistent with more common usage levels with the risk that the company will fail to meet its delivery promise during periods of peak usage.

A good example of peak usage occurs when a major event like the Superbowl happens. Superbowl 2011 drew over 100,000 people to a city called Arlington, Texas, which has a population of approximately 400,000. In the process, our online shipping company saw a major spike in shipping at the Arlington, Texas location. (A certain percentage of the population in any area use the company's shipping facilities at any given time. Thus, an increase in population in any area will result in an increase in shipping activity.) That represents an increase in about 25% in shipping during Superbowl weekend.

In order to avoid over provisioning staff, facilities and equipment while still maintaining its guarantee of prompt delivery, the company maintains a Big Data application in its corporate headquarters. This application contains an object for every package which is in the company's system at any time worldwide. This object maintains a property showing its current physical location in the system. At the moment an airbill is scanned into a company facility, this object shows up in the Big Data application at that location. Using this application, the company can see the number of packages in process within any location in the entire company.

As a result, within a matter of minutes, the company can detect a peak usage event. (There is a control center in the company headquarters where the state of the Big Data application is displayed on large screens, and personnel monitor this system continuously.) Once a peak usage event is detected, the control center personnel contact company facilities in the surrounding areas and have all of the spare drivers and sorters converge on the peak usage location. In our example, the folks at company headquarters would contact Irving, Carrollton, Dallas, Fort Worth, and so forth, and have all excess capacity shift to Arlington.

The effect of this Big Data application is as follows:

• A reasonable level of capacity can be maintained at any given location, in the process saving a lot of money.
• Excess capacity can quickly be shifted around the system so that peak usage can be handled without breaking the promise of prompt delivery.

As a result the Big Data application in the company headquarters has a huge ROI.

Another classic Big Data application is smart metering. I have a customer which is rolling out 30 million smart meters across its entire system. This is a major energy utility company on the East Coast of North America. Again, I will not use the company's name.

Each smart meter takes a usage measurement once an hour. At 30 million smart meters that's the following volume of data:

30,000,000 x 24 = 72,000,000 per day x 365 = 26,280,000,000 per year

This data is used by the company to measure the usage pattern over time within its system. In the process, the company can design its capacity more efficiently. The company can also assist its customers in the area of conservation. Peak users can be easily identified, and contacted to alleviate their peak usage patterns.

The result is that the company is greener and more efficient. The savings in improved efficiency in generation far exceeds the costs of the smart meters.

For billing purposes, the company only needs one measurement per month. That's a much more reasonable volume of data:

30,000,000 per month or 360,000,000 per year

Note the difference between the data which is used for capacity design, conservation and so forth, and the data which is used for billing:

• The hourly data is a huge volume, far exceeding that commonly managed by traditional database applications. The monthly data is easily within the range of traditional database application.
• The hourly data is only significant in total: Any individual hourly measurement is of trivial importance. What is important is the overall texture or "shape" of the hourly data at any given time. ACID properties do not need to be applied to the hourly measurements. They can be summarized in a Big Data application, and the resulting summaries can be saved into a traditional database application. The company would not even attempt to back up the enormous volume of data coming out of the hourly measurements, because doing so would be cost prohibitive and of dubious value. On the other hand, the monthly measurements are used for billing and thus have core importance to the business. This is OLTP data representing the lifeblood of the business. It must be transactionally consistent, comply with ACID, be backed up over time, and so forth.

These applications show the difference between traditional databases and Big Data, which I summarize as follows:

• Big Data has a data volume and data flow which exceeds the capacity of traditional database hardware and software.
• Big Data must be at a lower cost per gigabyte than traditional database applications by an order of magnitude or more.
• Big Data is not typically backed up: Rather it is summarized into a traditional database, but these summaries represent a minute fraction of the data managed by the Big Data application.
• Big Data applications do not comply with traditional ACID properties. Any item of data in a Big Data application is of minimal importance. Rather, the important thing is the overall "state" of the data.

I think my employer, i.e. EMC, "gets it" with respect to the challenges represented by Big Data. Certainly, the theme of EMC World this year reflects this. We face a huge challenge in coming up with the products to address the technical issues posed by Big Data. Unfortunately, I cannot tell that Oracle is similarly clued in with respect to Big Data. Oracle has few products which are focused on this area. It remains to be seen whether Oracle will become a major factor in this area.

Current meatspace coordinates are 36 degrees 7 minutes 13 seconds North 115 degrees 10 minutes 6 seconds West: location Las Vegas Nevada

Sorry. As I sit here in the blogger's lounge at EMC World I am channeling Randy Waterhouse from Neal Stephenson's classic Cryptonomicon which I am rereading for the I-can't-remember-how-many times. And Neal has got me thinking about technology, why we get up in the morning, and why we are still passionate about this stuff after decades.

Cryptonomicon remains oddly relevant today although written in 1999. It is about something which we are dealing with right now: Transformation.

The more I observe about the current wave in technology re-evolution the more I become convinced that this wave will transform our lives even more than any wave which has gone before, with the possible exception of the PC revolution in the late eighties.

More later.

VMworld 2010 Session Content Part 2 - Internal vs. External Cloud

Per my previous post, this is part 2 of a series of posts to provide the content of my recent VMworld technical session.

As I said in my previous post, the impending data explosion will produce huge challenges for the Oracle database community, in that the amount of scaling of Oracle database environments will an order of magnitude greater than the overall data explosion. My unofficial estimate is approximately a 200X increase in data volume managed by Oracle in the next 10 years (as opposed to about 44X overall). This creates the challenge of figuring out how to scale Oracle database environments in order to handle all of that data (as well as the transactional I/O that accompanies it).

In my view, there are two competing ways of scaling an IT workload presently, from a strictly structural standpoint: The external cloud and the internal cloud. (I will further break down the options on the internal cloud in my next post.)

The following diagram gives some insight in what I am talking about here:

 
Those of you who know me well know how fond I am of this type of diagram: The so-called "magic" diagram. Here the vertical dimension represents virtual vs. physical and the horizontal represents internal vs. external (from the perspective of the customer data center). A typical multi-layered application environment is shown with three layers: Web, App and Database. As is typical with most environments, the entire workload starts out in the customer's data center. Further, there is ratio of 9:3:1 for Web vs. App vs. Database. This is somewhat contrived, but fairly typical and real-world in my opinion. Thus, the initial workload when originally deployed looks like this:

For a while, the capacity of internal data center is able to handle the growth in this workload. I call this the period of normal growth. At the end of this period, the web server farm is beginning to exceed the capacity of the traditional data center. This looks like this:

At this point, the customer has a decision: They can either move the web server farm into the cloud, or they can move it out to a traditional external service provider (i.e. co-host location). Perhaps a progressive or farsighted customer would go ahead and move into the cloud. But most folks in my experience have not done so. Instead, they go through a period of scaling where the web server farm is located on a service provider's data center, and then continue to scale the rest of the workload internally. This enables the second period which I call accelerated growth. At the end of this period, the workload looks like this:

Growth has now become somewhat unmanageable. You can see that the co-location model is breaking down. Also, the capacity of the internal data center to absorb the App server farm is now being stressed. Again, the customer has to make some hard decisions. The easiest decision is to move the Web layer into the external cloud. This is easy, cost effective and very mainstream. The App server farm may be either moved into an external or internal cloud, or possibly moved out to the co-location facility. We will assume the latter.

At this point the customer begins the final period of growth, which I call explosive growth. The external cloud provides virtually infinite growth capacity for the Web server layer, but the co-location in an external service provider can now no longer handle the demands of the App server layer. And, finally, the Database server layer is now exceeding the capacity of the internal data center. At this point, the workload looks like this:

Another doubling of the workload, and the customer will be in big trouble. Hard decisions must be made. It is likely at this point that the customer will move the App server layer into the external cloud. Doing so is now fairly mainstream. The decision remains of what to do with the Database server layer.

Understand that I have personally engaged in these discussions with many customers. I think I know the mind of the DBA (as I have been one for many years). Most DBAs when confronted with this choice will make the following calculation:

Am I going to put my precious database, my business's crown jewels, which contain the source data of the business, our very lifeblood, into the hands of an external cloud provider, in what is certainly a very unstable and emerging technology space?

The answer for most DBAs is: No way! Thus, the decision is typically that the scaling of the database layer will be handled as an internal cloud. Many large customers are today launching exactly this type of project which goes by many names, but which is ultimately the idea of creating a Database as a Service (DBaaS) offering which is managed entirely by internal IT. Thus, the final evolution of our example looks like this:

The essence of the internal database cloud is the idea that we divorce the physical database server from any particular project. Rather, database access is provided as a utility service at a cost. The customers (internal IT projects in this case) contract with IT to have this provided. The billing may be on physical storage, transactional I/O, or some other measurement. But the effect is the same. The challenge for DBAs, system administrators, and storage administrators is to provide this service cost-effectively and reliably. But the payoff is huge. Only by providing this type of offering can the customer have any hope of scaling the workload in a period of explosive growth.

My next post will provide a breakdown of the approaches to creating an internal Oracle database cloud, and the choices that customers are making in that regard.

Finally, please remember that I will be at Oracle OpenWorld which is only a couple of weeks away. Please join me in the EMC booth, and I hope to see you there.