Oracle Storage Guy

NetApp's ExaData Backup Play Reprise

I have received a fair number of responses to my previous post on this subject (some via comments and some via email). I thought the discussion worthwhile enough to punch it up a bit more here.

Backup

As I pointed out in the previous post, EMC can easily match NetApp's play to back up ExaData with the following:

As Geoff Rosser so correctly pointed out, this answer is incomplete. Yes, Data Domain is an awesome Oracle backup solution. Yes, it provides incredible deduplication rates for Oracle database environments. (Thanks, dynamox.) However, it is not the only viable solution from EMC for backing up ExaData. For example, the following would also work:

As usual comments are welcome.

Oracle Licensing on VMware - no magic

Acknowledgements to Bart Sjerps for this content.

There seems to be a lot of confusion on licensing when customers consider running Oracle databases on VMware. Part of the confusion is caused by Oracle on purpose (classic FUD) by suggesting licensing is more expensive on VMware than on physical servers. The reality couldn't be more different - I strongly believe that many customers can actually *save* on database licenses by going virtual. But to understand how to achieve this, you need to know a few things - I hope I can clear this up in a short explanation. I will keep the discussion to Oracle database licenses and ignore application/middleware etc. for now.

License models

Customers typically license their basic database by one out of three options:

1. License by CPU (core) - the more CPU cores, the more licenses are needed. There is a processor core factor depending on the type of CPU and can be 0.25, 0.5, 0.75 or 1.0.
2. License by named user - the more named users, the more licenses are needed. The amount of CPU's is not important, neither the amount of total databases. Typically one license pack per 25 users.
3. Enterprise License - the customer negotiates a contract for the whole company and afterwards can deploy as many databases on as many servers/cpus as he wants.

If a customer uses 2 or 3, then it does not matter if they run virtual or physical. But there are also no license savings possible without re-negotiating their contracts. I don't want to go as far as suggesting to customers to change their license models so we leave this as-is for now.

In my experience, most enterprise customers use either cpu licensing or enterprise contracts. Some have different licensing methods for different business units. Oracle can be very creative in customer-specific contracts so expect to find a different situation for each individual customer.

But let's assume CPU licensing for the sake of this discussion.

Maintenance & support

Users typically buy the CPU licenses  but then have to pay maintenance for the time they use the licenses. Yearly maintenance cost is about 25% of license (list price). I have no information on typical discounts. I expect customers to get at least 50% discount off the price list (but only on licenses, not on maintenance AFAIK).

Database Edition and options

The plain database license comes in 3 versions (for servers):

1. Standard Edition One - Maximum 2 processors, no options allowed. Only used for testing and very small deployments
2. Standard Edition (SE) - Maximum 4 processors, no options allowed. Only used for smaller sizes and workloads (but stay tuned)
3. Enterprise Edition (EE) - No limitations and on top of EE, you can have many licensed features. Most customers will use this, at least for production databases

On top of the basic Database license, most customers use a set of options, each requiring additional licenses per CPU. The most common options are:

• Real Application Clusters (RAC) - allows many servers running the same database (active-active clustering) to allow scale out performance and high availability.
• Real Application Clusters One Node - same but one database can only run actively on one node. For high availability only.
• Active Data Guard - remote replication using log shipping. Note that standard Data Guard is free, but Active Data Guard allows the standby database to be opened for read-only purposes and offers some extra features.
• Partitioning - allows tables to be split up in smaller chunks. Absolutely required when running large databases and no downtime can be tolerated. Eases administration work and offers some performance benefits.
• Real Application Testing - allows workloads to be recorded and re-played on another database to do performance and functionality testing
• Advanced Compression - allows database blocks to be compressed - requiring less storage and boosting performance (in most cases).
• Diagnostics Pack / Tuning pack - provides automated reports. Oracle AWR (Advanced Workload Reports - a performance reporting tool) is part of Tuning Pack.

In my experience, nearly all customers have partitioning. Most customers have tuning/diagnostics pack. Some customers have RAC. Some customers have the other options. There are more options available but these are the most common.

Many customers have 3 or more options - sometimes the options cost more than the base database license - especially if they use RAC they will have most of the other options, too.

Running on a cluster

If a database runs on a cluster, then Oracle assumes the database can make use of any processor in the cluster. This is independent on what kind of cluster is used (so can be MSCS, HP MCSG, Vmware, Oracle RAC, etc).

This is basically the foundation for all FUD and confusion. For example, if you deploy a VMware farm (cluster) of 16 servers, and all virtual machines run all kinds of stuff (file/print, exchange, apps, etc etc) and only one tiny virtual machine in the corner, with only one virtual CPU runs a small Oracle database, you would expect only to pay for one CPU core - but Oracle's reasoning is that this tiny VM can be dynamically moved (VMotion) to all nodes in the cluster and on any processor. Therefore, all CPU's have to be fully licensed by Oracle. So in this case, running the single database on a (small) physical server would be cheaper than running on a VM in the farm.

Total cost of the stack

In a typical database server deployment, the cost of the database licensing is far greater than the cost of the hardware + OS licenses combined. I have no hard numbers but I assume the average DB license cost (plus options) is 10 times larger than the cost of the server + OS.

So a $5,000 server would typically require $50,000 on licenses. Then because maintenance is 25% yearly, the total cost of licenses over a 3 to 5 year period is even higher - so for a 5 year TCO the total license cost might be $75,000 (assumption - could also be closer to $100,000 - and no, I didn't make a mistake with an extra zero, Oracle *really* is this expensive).

Utilization

It is very hard to size a typical Oracle database based application. There are no good methods or calculations to figure out how much CPU power, disk I/O and memory is needed to run a given app. So historically, project teams size their database servers for peak loads, and because they cannot predict how big the peak load is, they double the resources "just in case". The end result is that most database servers are way oversized in terms of CPU and memory.

Most physical deployed database servers will average on about 10-15% CPU load (or less). However, they will peak to higher loads at certain times, such as monday morning when many users log in, or when month/quarter/year-end batch processing is started, etc.

Then, the utilization numbers can be influenced by other tasks of the processors. Some common causes of "artificially high" CPU loads on database servers:

• CPU is involved in storage mirroring (i.e. Host Level Mirroring - using Oracel ASM or a Unix volume manager)
• CPU is involved in file transfers over the IP network
• Backup (non-serverless, using CPU, Network and I/O bandwidth)
• Customers run the application server on the same machine driving up CPU load - This can drive up CPU load from 10% to 90% or more !!
• Same for Middleware and Enterprise Service Buses (Think Oracle BEA, IBM Websphere, SAP Netweaver, etc)
• A  bunch of monitoring/management agents burn CPU cycles (Tivoli, BMC, HP Openview, CA, etc). Each agent maybe consuming 1% but add it up and you have another 5-10% overhead.
• Administrators generate database dumps / exports and run their own reports, scripts and tools. They run ad-hoc queries as well that should not be on production.
• Poorly tuned database servers cause paging and other CPU overhead - hard to diagnose but driving up CPU and I/O significantly.
• Database admin tasks (table reorganizations, (re)building indexes, converting tablespaces, ...)
• And so on...

All of these cause the processors, expensively licensed for database processing, to do other stuff.

So if a server is running at 15% utilization, then the utilization caused by the database workload itself might only be 10% and the rest caused by other stuff (whether needed or not).

Needless to say that Oracle likes customers to use their expensive licensed CPU's for other tasks because it forces them to buy additional CPUs sooner and therefore drive their license revenues.

Isn't life great for an Oracle rep? ;-)

Number of databases

Most customers run many databases. For the average enterprise customer that I visit, 100+ databases is a normal number. A big global that I visited runs 3000+ Oracle databases worldwide (and this is only the scope of this specific project team). Imagine the cost of licensing all these databases on all individual servers...

Why so many? Well, customers do not like to share multiple applications on one database (and often this is not even supported). So if you run SAP ERP, Oracle JD Edwards, your own banking app and a few others, they all require their own production database.

For each production database, you might find an acceptance environment, test system, development server, maybe a staging area to load data into the data warehouse, maybe a firefighting environment, a standby for D/R, a training system and so on. Customers will rarely share production environments on the same server (unless virtualized or at least with workload management segregation). Sometimes they share a few databases for non-prod on a server. So for, say, 100 databases, the average customer runs between 30 and 50 (physical) servers.

Power of big numbers

It does not require rocket science to understand that many of these databases do not require peak performance at the same time. A development system typically drives workload during daytime (when developers are coding new application features). A data warehouse runs queries during the day and loads in the evening. For a production system it depends on the business process. An acceptance system might sit idle for weeks and then suddenly peak for a few days preparing for a new version deployment into the live production system. And so on.

So what if you could share resources across databases - without influencing code levels, security, stability and so on?

If that would be possible - you would not size for "peak load times two" anymore. You would size for what you expect and assume an average utilization of, say, 70% over the whole landscape. If one database needs extra horsepower, there is enough available in the landscape.

How much license cost would you save by bringing down the number of CPU's so that utilization goes up from 10% to 70%?

What would be the effect on power, cooling, floor space, hardware investments, time-to-market?

What would be the business advantage of not limiting production performance of a single server, by whatever was sized during initial deployment? Risk avoidance?

What would be the business advantage of solving future performance issues by just adding the latest and greatest Intel server in the cluster and Vmotion the troubled database over?

Wasn't this exactly why we started server virtualization in the first place about 8 years ago? And why EMC aquired VMware?

Wouldn't you think the average Oracle sales rep is scared to death when his customer starts considering to run his databases on a virtual (cloud) platform? Would it make sense for him to drive his customers mad with FUD around licensing, support issues and whatever he can think of to prevent his customers going this way? Even threatening to drop all support if they continue to go in that direction?

If Oracle is scared of losing license revenue, wouldn't you think there is a huge potential for savings for our customers here?

The journey to the private database cloud

So how should we deal with this?

A few starting points:

• Oracle supports VMware. Period. Any other claim of Oracle reps can be taken with a grain of salt (to be more specific: it's nonsense).
• Oracle does NOT certify VMware. Then again, Oracle does not certify anything except their own hard- and software. But IMO, support is all you need and the discussion around certification leads nowhere.
• Oracle might ask the customer to recreate issues on a physical server if they suspect problems with the hypervisor. Isn't it great that we can do this easily with Replication Manager? ;-)
• Oracle only supports Oracle RAC on VMware for one specific version (11.2.0.2). Any other version with RAC is not recommended on VMware because of support issues. Expected to change in the future.
• Both EMC and VMware offer additional support guarantees for customers deploying Oracle on Vmware. So where Oracle pulls back, EMC and VMware will fix any issue anyway.
• Performance is no longer an issue. With Vsphere 5, a single virtual machine can have 32 virtual processors, 1 TB ram and drive 1 million iops. Only the most demanding workloads would not fit in this footprint. But with customers running hundreds of databases, maybe we should start with the 95% + that DO fit and make significant savings there. By the time we're done, VMware will have Vsphere 6 and who knows what happens then.

How to get around the licensing issue

As I said, Oracle requires licenses for all servers in a cluster. So how do you limit the number of licenses? By deploying an Oracle-only VMware cluster. Only run Oracle databases here. No apps, no middleware, no fileservers, and try to move everything off that does not relate to database processing. No host replication, no storage mirroring, etc.

Say you have a legacy environment with 10 servers, each with 16 cores, so you have 160 cores licensed with oracle EE and a bunch of options. Average CPU load is 15% but let's assume 20% to be conservative.

I claim that a single VMware cluster with 3 servers each with 32 cores will easily do the job. Now we have 3 * 32 = 96 cores to be licensed. 96/160 = 0.6 = 60% so we saved 40% on licensing right away. Probably the average CPU load on the whole cluster will still be much less than 70% so we can gradually add a bunch more databases until we average out on 70%.

If the old system was not running Intel x86 but SPARC, PA-RISC or POWER cpu's then the processor factor was probably 1.0 or 0.75. Intel has 0.5. So for 96 cores (Intel) you would need to pay 48 full licenses. Another 33% savings.

The savings of 40% on licensing will easily justify an investment of a nice new EMC storage infrastructure with EFDs, FAST-VP and all other goodies. Do you think the customer will push us hard for a $0.01 lower GB price competing HDS or Netapp if we just saved them millions in Oracle licenses?

But the story does not end here.

Additional savings

Let's assume the customer needed high availability and scale-out performance and was running Oracle RAC. RAC is the most expensive licensed option and you need at least two for a two-node cluster. But VMware allows for HA (High Availaiblity clustering) as well. Using VMware HA instead of RAC, you would have to fail-over and recover the database in case of an outage - if the customer cannot tolerate this then he needs to stick with RAC (only for mission critical databases!). But most customers can live with 5 minutes of downtime in case a server CPU fails and in that case, replacing RAC with VMware HA can save them another big bunch of dollars.

Let's assume that with virtualization you justified the investment in a nice EMC infrastructure with Flash drives to replace the competitive gear. Now the Oracle cluster is no longer limited by storage I/O's and can drive more workload out of the same 3 VMware servers in the cluster. But you can also replace host mirroring (where applicable). You can implement snapshot backups to get the I/O load away from the production servers. You removed the middleware and apps stuff from the database servers - reducing CPU utilization and allowing even more headroom for DB consolidation - all without buying extra licenses from Oracle.

You have a customer who wants even more?

What if they create TWO database clusters for VMware? One for production (running Oracle Enterprise Edition (EE) with all the options they need) and one for Non-prod (running Oracle Standard Edition (SE) without options - good enough for test/dev and smaller, non-mission critical workloads). I bet the number of non-prod databases will be much more than prod. By removing the expensive options AND moving from Enterprise to Standard Edition, you saved another ton of money on Oracle licensing as SE is much cheaper than EE. But be aware - the devil is in the details and using Standard Edition is not for the faint-of-heart (for example, you could no longer clone a partitioned database to a SE enabled server because of the missing license and functionality). Still if the customer is keen on saving as much as possible, then this might be the final silver bullet...

Do they run a huge Enterprise Data warehouse? Carefully find out if they have troubles with it and see if you can position Greenplum - saving another bag of money and speeding up their BI queries. But be careful, in an Oracle-religious shop it might backfire on you...

Reality Check

I had this discussion already with a few enterprise customers. And found that although the story is easy in theory, the reality is different. If a customer already has the 160 CPU licenses purchased from Oracle, then the Oracle rep will not happily give a money-back in return of the shelfware licenses. So in that case the customer can only save on maintenance and support. But having enough licenses on the shelf, he would not have to purchase any more for the next 5 to 10 years. So talk cost avoidance instead of immediate savings. And again, if they are licensed by user or have a site license, then saving on licenses will be a tough discussion. Still, the savings on power/cooling/hardware/floorspace would still be significant enough to proceed anyway.

And don't forget the other benefits of private cloud of which we all know how to position: they are no different for Oracle than for other business applications.

Final thought

For this to work you need a customer that is willing to work with you and be open on how they negociated with Oracle, and a team of DB engineers to work with you to make it happen. If internal politics cause significant roadblocks then you will get nowhere.

It's not an easy sell but the rewards can be massive. We're only just starting to figure out how to convince customers and drive this approach. Feedback welcome and let me know if you need support.

Resources

The online Everything Oracle at EMC community has lots of information on this subject. See in particular this presentation which I co-presented (with Sam Lucido) at this year's VMworld.

Landing page with Oracle 's price list: http://www.oracle.com/us/corporate/pricing/price-lists/index.html

Download "US Oracle Technology Commercial Price List" for the database license document. Read the fine print because it's not always as simple as it seems.

Processor Core Factor information:http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf

 

Oracle OpenWorld 2011 Technical Sessions: Part 1

This begins a series of blog posts on the subject of technical sessions that EMC presented at OOW2011. To begin with, we will examine the session that I co-presented with Kevin Jernigan entitled Clone Oracle Databases Online in Seconds with Oracle Clonedb and DirectNFS. The presentation for this session can be found here.

It may sound strange given EMC's reputation and history, but EMC has a strong partnership with Oracle in the area of NAS. We began working with Oracle on the Direct NFS client ("dNFS") way back in 2007, when dNFS was introduced as a major new feature in Oracle Database 11g Release 1. At that time, EMC, was a co-presenter (together with Oracle) at Oracle OpenWorld 2007 on this subject.

The goals of dNFS were simple, and well defined:

• Improve performance (in terms of latency and throughput) of NFS-mounted Oracle database I/O
• Reduce CPU cost of NFS-mounted Oracle database I/O
• Improve network port scalability and failover of NFS-mounted Oracle database I/O
• Simplify administration of NFS in terms of both network port scaling and mount point parameters
• Make administration of NFS uniform across all platforms

dNFS succeeds widely at all of these. dNFS dramatically improves latency of network I/O from Oracle by eliminating most context switches and making the code path to the disk much shorter. dNFS also provides better port scaling than kernel NFS, with much simpler administration. No fussy ether channel network switch configuration is required. All of the port scaling and port failover is provided within the Oracle environment.

 

dNFS also made administration completely uniform across all platforms. Amusingly, Windows is included! NFS now works on Windows, at least with an Oracle database.

In September 2010, Oracle announced that dNFS would be improved in Oracle Database 11g Release 2. One major improvement is the addition of dNFS clonedb, a thinly-provisioned, rapid database replication feature. This feature also works very well with storage-based replication.

 

The basis for dNFS clonedb is a database copy. This copy can be a backup, a storage-based snapshot or clone, or an operating system copy. (Of course, EMC likes to leverage our storage-based snaps and clones.)

 

Once a copy exists, it can be used to create a clonedb instance. The steps to do this are contained in My Oracle Support article 1210656.1 . Effectively, this creates a read / write virtual database which takes up minimal space, is created almost instantly, and contains only the space required to store any changes to the database. Given that storage-based snapshots are also space and time efficient (that is, they take up very little space, and can be created very quickly) there is a great deal of synergy between these technologies.

 

EMC did some performance testing with dNFS clonedb. The network diagram for the testbed is here:

First, we established a baseline in terms of performance. We ran an OLTP workload against the production database with no additional operations. This produced the following:

Notice the perfectly clean scaling of this performance chart. We then tested the performance of creating a snapshot to serve as the source for the clonedb database. This produced:

Note the slight response time hit when the snapshot was taken. However, transactional throughput was not affected. Basically, this is at most a minimal performance hit. Finally we tested performance while creating clonedb database instances from the storage snapshot. This produced:

Note that there was no performance hit during clonedb creation at all. One additional test was performed. We measured the storage space occupied by the clonedb when it was created. A 10 TB database was used as the source. The total space occupied by the clonedb was only 7 MB.

See the link above for the full presentation of this technical session. Further blog posts in this series will contain summaries of other EMC technical sessions at OOW 2011.

Thoughts on Oracle OpenWorld 2011 Keynotes

I attended Oracle OpenWorld 2011 as usual this year, at which I co-presented a technical session with Kevin Jernigan on the subject of Direct NFS clonedb. Since I had a full conference pass, I was able to attend many of the keynotes at this event.

I did not get to attend the Sunday Larry keynote, although I understand that it was pretty terrible. I did attend the Wednesday Larry keynote, and can tell you this: As a long-term Larry watcher (I have attended every OOW since 1998) it was hands down the worst Larry keynote I have ever seen.

Perhaps because Larry had just lost his best friend (Steve Jobs died of cancer the day of this keynote), he was off his game. That's the only reasonable explanation I have for such a spectacular failure of such a precious and expensive opportunity.

Larry's keynote consisted largely of a 20 minute-plus long demo of Fusion Applications, which while interesting in itself, was excrutiatingly dull, and definitely would have been much better if handled by someone other than Larry.

What Larry is great at (and what was largely lacking in his keynote) is the flamboyant, extravagant, and provocative way that he baits his competitors and makes outlandish claims concerning his own products' performance. He did a bit of sniping at (SalesForce.com Chairman and CEO) Marc Benioff. Aside from that, nothing.

The other Oracle keynotes were not much better. This was an almost content-free OOW for me. The ExaData and ExaLogic stuff was more of the same from last year. The big product launch was of course Fusion Applications, but that is just software. There was no big splash on the hardware side.

The best keynotes by far were Joe Tucci and John Chambers. Both provided a lot of content, much of which was competitive to Oracle, interestingly. Cisco is certainly competing with ExaData in the server market and EMC competes with Oracle in storage, virtualization and datawarehousing software. All of this was discussed extensively during these keynotes. Joe even said the "v" word (VMware) during his keynote, and Pat further elaborated on this subject during his follow on to Joe's talk. Pat also included a long demo of vCenter which was assisted by EMC's own Chad Sakac.

Chambers talk was very compelling to me. His vision for a new platform in the post-PC age was a serious challenge to both Microsoft and Apple in their ecosystems. I am a bit skeptical that Cisco can successfully compete with those vendors in their respective spaces, though. Microsoft owns the business productivity space, and Apple has a serious stake in the home and home office space. Cisco will need to penetrate both of these in order to fulfill Chamber's vision. We'll see.

I thought OOW was about Oracle. This year, not so much. This year, the interesting stuff at OOW was from other vendors. Score one for EMC and Cisco, and score zero for Oracle this year.

VMworld 2010 Session Content Part 2 - Internal vs. External Cloud

Per my previous post, this is part 2 of a series of posts to provide the content of my recent VMworld technical session.

As I said in my previous post, the impending data explosion will produce huge challenges for the Oracle database community, in that the amount of scaling of Oracle database environments will an order of magnitude greater than the overall data explosion. My unofficial estimate is approximately a 200X increase in data volume managed by Oracle in the next 10 years (as opposed to about 44X overall). This creates the challenge of figuring out how to scale Oracle database environments in order to handle all of that data (as well as the transactional I/O that accompanies it).

In my view, there are two competing ways of scaling an IT workload presently, from a strictly structural standpoint: The external cloud and the internal cloud. (I will further break down the options on the internal cloud in my next post.)

The following diagram gives some insight in what I am talking about here:

 
Those of you who know me well know how fond I am of this type of diagram: The so-called "magic" diagram. Here the vertical dimension represents virtual vs. physical and the horizontal represents internal vs. external (from the perspective of the customer data center). A typical multi-layered application environment is shown with three layers: Web, App and Database. As is typical with most environments, the entire workload starts out in the customer's data center. Further, there is ratio of 9:3:1 for Web vs. App vs. Database. This is somewhat contrived, but fairly typical and real-world in my opinion. Thus, the initial workload when originally deployed looks like this:

For a while, the capacity of internal data center is able to handle the growth in this workload. I call this the period of normal growth. At the end of this period, the web server farm is beginning to exceed the capacity of the traditional data center. This looks like this:

At this point, the customer has a decision: They can either move the web server farm into the cloud, or they can move it out to a traditional external service provider (i.e. co-host location). Perhaps a progressive or farsighted customer would go ahead and move into the cloud. But most folks in my experience have not done so. Instead, they go through a period of scaling where the web server farm is located on a service provider's data center, and then continue to scale the rest of the workload internally. This enables the second period which I call accelerated growth. At the end of this period, the workload looks like this:

Growth has now become somewhat unmanageable. You can see that the co-location model is breaking down. Also, the capacity of the internal data center to absorb the App server farm is now being stressed. Again, the customer has to make some hard decisions. The easiest decision is to move the Web layer into the external cloud. This is easy, cost effective and very mainstream. The App server farm may be either moved into an external or internal cloud, or possibly moved out to the co-location facility. We will assume the latter.

At this point the customer begins the final period of growth, which I call explosive growth. The external cloud provides virtually infinite growth capacity for the Web server layer, but the co-location in an external service provider can now no longer handle the demands of the App server layer. And, finally, the Database server layer is now exceeding the capacity of the internal data center. At this point, the workload looks like this:

Another doubling of the workload, and the customer will be in big trouble. Hard decisions must be made. It is likely at this point that the customer will move the App server layer into the external cloud. Doing so is now fairly mainstream. The decision remains of what to do with the Database server layer.

Understand that I have personally engaged in these discussions with many customers. I think I know the mind of the DBA (as I have been one for many years). Most DBAs when confronted with this choice will make the following calculation:

Am I going to put my precious database, my business's crown jewels, which contain the source data of the business, our very lifeblood, into the hands of an external cloud provider, in what is certainly a very unstable and emerging technology space?

The answer for most DBAs is: No way! Thus, the decision is typically that the scaling of the database layer will be handled as an internal cloud. Many large customers are today launching exactly this type of project which goes by many names, but which is ultimately the idea of creating a Database as a Service (DBaaS) offering which is managed entirely by internal IT. Thus, the final evolution of our example looks like this:

The essence of the internal database cloud is the idea that we divorce the physical database server from any particular project. Rather, database access is provided as a utility service at a cost. The customers (internal IT projects in this case) contract with IT to have this provided. The billing may be on physical storage, transactional I/O, or some other measurement. But the effect is the same. The challenge for DBAs, system administrators, and storage administrators is to provide this service cost-effectively and reliably. But the payoff is huge. Only by providing this type of offering can the customer have any hope of scaling the workload in a period of explosive growth.

My next post will provide a breakdown of the approaches to creating an internal Oracle database cloud, and the choices that customers are making in that regard.

Finally, please remember that I will be at Oracle OpenWorld which is only a couple of weeks away. Please join me in the EMC booth, and I hope to see you there.

VMworld 2010 Session Content Part 1 - Impending Data Explosion

I had a very successful breakout session at VMworld this year, presenting to a total of around 200 people. The topic was:

Creating an Internal Oracle Database Cloud Using vSphere

I will now share much of the substance of my session here on this blog, for those of you who could not attend VMworld.

The first section (which will constitute this first post) was on the coming data explosion that will soon hit (indeed is already hitting) Oracle DBAs all over the industry.

Many of you are familiar with the Joe Tucci keynote from EMC World this spring. In that talk, Joe pointed out the recent IDC / Dataquest study, which points out the coming world wide data explosion which can be simply stated as follows:

Year   Worldwide Data Set
2009   0.8 Zettabytes
2020   35 Zettabytes

Where a Zettabyte is 1024 Exabytes, an Exabyte is 1024 Petabytes and a Petabyte is 1024 Terabytes. A Zettabyte is therefore approximately 10 ^ 21 bytes.

That's a data explosion of 44x in approximately 10 years.

What Joe does not point out is the shift from unstructured data to structured data. Presently, 90% of worldwide data is unstructured. However, almost all of the sources of new data which are causing the explosion are structured. (These include social networking, blogging, Twitter, and all of the Web 2.0 content, plus sources like e-readers, PDAs, smart phones, medical / dental digital imaging, online security, smart energy metering and the like.) The result is this graphic showing the relationship between structured and unstructured data going into the future:

Undoubtedly much of this data will end up being owned by the so-called Database 2.0 vendors. Certainly, that will be true for the bulk of the social networking and Web 2.0 content, where those vendors have formed their roots. But this is not true for all of this data. Look again at the sources of data shown above. Is it likely that a medical or dental image (which may have critical consequences for actual patient care and confidentiality) will end up on a relatively untested and emerging Database 2.0 technology? I would regard that is very unlikely. It is much more likely that this data will end up in the known-to-be-reliable good enough technology in the Database 1.0 space we know as Oracle.

The same thing is true for smart metering. Again, the output of these meters will be used to calculate actual customer bills, not to mention determining the energy output for homes and businesses. Is the external cloud Database 2.0 environment ready to absorb this data? I doubt that. Folks like Progress Energy (my energy provider) are going to place that on Oracle, most likely.

Even making the incredibly depressing (for Oracle) prediction that Oracle's market share in the database market falls by 50% in the next 10 years, the result for Oracle can be calculated simply as follows:

44 x (9 / 2)

or about 200X. This means that the growth in Oracle data volume worldwide will increase by 200X in the next 10 years, by my conservative estimate.

!!!!!!

I compare this to the movie Princess Bride, one of my favorite films by Rob Reiner. In this film, as many of you know, the main character Wesley leaves his beloved Buttercup to find his fortune. Very soon, he is captured by the Dread Pirate Roberts who makes him his cabin boy. During his years of service, the Dread Pirate Roberts tells Wesley the same thing everday:

"Hello, Wesley. I'll probably kill you in the morning"

I maintain that this is what the impending data explosion means to Oracle DBAs. The day will come when Oracle DBAs live in the "Wesley State". Coming to work everyday, they will look at their screens and their Oracle database will grin back, saying to them, in effect:

"Hello, DBA. I'll probably kill you in the morning."

Those DBAs who survive this challenge will do so because they have absorbed a very different way of thinking about their Oracle database environments. That will require many changes, among them the willingness to scale their database infrastructure in a completely new and radical way.

More on this later. In up-coming blogs, I will explain how I think that change will come about, what choices the Oracle DBA has to meet this challenge, and so forth.

To end, many of you are coming to OOW this month. I will be there as well, and will be co-presenting on the subject of RMAN backup with my good friend and former manager, Bruce Clarke. Bruce was my second boss at NetApp, and is now with Data Domain. Please come by the EMC booth and say hello, and I will post the logistics of my talk on this blog.

Also, a big thanks to Steve Tout with VMware, who co-presented with me at VMworld and provided my customer case study. Steve is responsible for all identities which exist on VMware.com and VMworld.com. Please see his blog as well. It contains much of the content he presented at VMworld.

 

 

 

Come see me at VMworld

As I have posted here, my talk at VMworld is on! I will be speaking on Monday at noon and Thursday at 10:30 a.m., so you have two opportunities to come see my talk.

Again, the session is titled:

Creating an Internal Oracle Database Cloud Using vSphere

The general thrust of the session is to lay out the advantages of VMware vSphere for creating an internal Oracle database cloud, compared to other methods (notably Oracle RAC) for doing so. As I will point out, each have their place, and I will layout the usage cases where RAC makes sense, as well as those where vSphere is a better call.

I will also look at the market space for Oracle compared to the so-called Database 2.0 vendors, and generally discuss the internal (also known as private) cloud as compared to the external cloud.

I will have a surprise as well: There will be a customer case study presented during my talk. I am not going to tell you here who the customer is; you will have to come see the session to find that out! But it will be very enlightening to say the least. An actual customer experience in deploying a rich application environment on vSphere, and how that experience produced real benefits for their business.

I look forward to seeing you all there.

Data Domain deduplication plus RMAN incremental backup saves space and time

Bruce Clarke, my good friend and former boss, and I have have been collaborating recently on RMAN backup to Data Domain deduplication storage arrays. In the process I have learned from Bruce (who I gratefully acknowledge for the technical content of this post, including the scripts shown below) about a very valuable approach to RMAN backup.

Most DBAs do full RMAN backups of all databases which are small enough to fit into a backup window. Full backups have several advantages:

1. They are very simple to run.
2. They optimize the restore operation, as no redundant blocks need to be applied.
3. They reduce dependencies in terms of the number of backup pieces that need to be cataloged and maintained in order to do a given restore, thereby reducing risk.

However, fulls have two strong disadvantages:

1. They waste space since the entire database's content is stored in each full backup.
2. They are time-consuming to run, because all active blocks in the database must be transferred to the backup target.

The first disadvantage is neatly avoided by the use of a deduplication array, such as Data Domain. My recent testing indicates that this is certainly true. More on this in a later post.

The second disadvantage is unavoidable at present. (Future developments in the area of source-based deduplication on the Data Domain side may mitigate this later on, but I will ignore that for now.) For most DBAs, the trade-off is acceptable as long as the full backup can fit into the available backup window. This is somewhat of a moving target as well, since recent developments in the area of storage networking (such as 10 GbE) have made the bandwidth increase, thereby increasing the size of the database that can fit into a backup window.

Unfortunately, the size of the typical database is also exploding as we all know. Thus, the issue remains: For a very significant number of large databases, the full backup cannot fit into the backup window. For these databases, the DBA has no real choice: He or she must use RMAN incremental backup.

Incremental backup (absent incremental update, which I will discuss later) provides the DBA with the option to only back up blocks which have been updated since the last backup. This provides a dramatic savings in terms of backup time, but with a significant manageability cost:

1. The number of backup pieces which must be cataloged and maintained increases significantly, making the dependencies greater, and thus increases the risk of a failed restore.
2. The restore time increases, as blocks must be restored multiple times, as the same block may be updated many times and thus stored in separate incremental backups.

Because of these issues, Oracle introduced incremental update for RMAN with 10g. There is an excellent article on the way this feature works in the Oracle documentation on OTN. Effectively, incremental update applies the incremental backup to the previous full backup. This neatly solves all of the issues of RMAN backup, except one: Instead of having a set of full backups, you end up with a single rolling full backup. In other words, once you apply the incremental backup to the previous full backup, you no longer have that previous full backup. Instead, you have a new full backup as of the point in time when you took the incremental backup.

This issue can be solved, obviously, by making a copy of the full backup prior to the incremental update operation. However, making this copy is time consuming, causes I/O, and takes up space.

Enter Bruce's solution: By using the Data Domain fast copy feature, you can make a copy of the previous full backup very quickly, and with no additional space. Further, the blocks which will be applied to the full backup copy have already been stored on the Data Domain in the incremental backup. Thus, they take up very little space as well. The end result is tantamount to RMAN backup nirvana:

1. You can take your backups in a fraction of the time it would require to do a full backup.
2. At the end of the process, you have a set of full backups for each time you run your RMAN backup.
3. All backups are fully deduplicated, thus saving you lots of space.
4. Restore operations are optimized as there are no dependencies other than a single full backup, and thus no redundant blocks to apply.

The scripts to accomplish this are below. The caveats all apply: This is provided as is with no warranty or support obligations. You are on your own in terms of all that. However, the solution does work, and you should definitely give it a try. Also, note that this is Data Domain copyrighted material. However, I have received permission from Bruce Clarke, the author of this script, to publish it here.

Example script for weekly backups

This was created and tested on Sun Solaris so slight changes for other platforms might be required.

#!/bin/sh -f # version 1.0 - Copyright (C) 2009, Data Domain, Inc - Nov, 2009
#
# Perform backup of an Oracle database in preparation using the
# incremental update procedure
#
# This script performs periodic full backups using the fact that
# a requested to the existing full creating an updated
# full in case recovery is required.
#
# Arguments: $1 - "full" or "incremental"
# $2 - instance name to be backed - Default: "orcl"
#
# specifying "full" starts a new "tag" thus creating a new full sequence
#
# The script also keeps everything in separate subdirectories. Because
# the underlying storage is assumed to be deduplicated, the actual storage
# consumed is similar to just running a series of incrementals
#
# Archivelogs and controlfiles are also backed up in the sequence
#
# directory naming scheme being followed
#             $BACKUP_DIR :: path to make backup directory
#             $BACKUP_DIR/full-$INSTANCE-date :: full backup directory for today
#
# IMPORTANT NOTE: THIS SCRIPT IS PROVIDED WITHOUT WARRANTY OR SUPPORT.
# FEEDBACK IS ALWAYS
# WELCOME, HOWEVER, IT SHOULD BE CLEARLY UNDERSTOOD THAT USE OF THIS SCRIPT
# SHOULD ONLY
# BE MADE AFTER AN EXPERIENCED ORACLE DBA CLEARLY UNDERSTANDS ALL STEPS IN THIS
# SCRIPT
# AND HAS MADE NECESSARY MODIFICATIONS AND PERFORMED SUFFICIENT TESTING TO BE
# ASSURED OF ITS
# PROPER OPERATION IN THEIR PARTICULAR ENVIRONMENT
#
# edit variables below as necessary
#
#DRYRUN=1 # Remove this line to allow script to run
                    # with this line, the script will only echo
                    # operations it would have performed
BACKUP_MOUNT="/dd/dd690a" # where "backup" directory on DD is mounted
BACKUP_DD_PATH="/backup/oracle"
BACKUP_DDSYS="[email protected]" # used to perform ssh fastcopy after
# completion
INSTANCE="orcl"
RMANID="sys"
RMANPW="oracle"
RMANCAT="NOCATALOG" # Modify if a catalog is being used
RETENTIONLOCK=no # Yes, says change the atime to lock the file, otherwise don't
DELETE_OLDER=21 # how old a directory needs to be before it expires
                                # only relevant if RETENTIONLOCK is set to "yes"
MAXSIZE="10 g"   # Maximum backup filesize - smaller allows replication to start
                                # sooner
# only relevant to archivelog backupsets. All other backups are "copy"
#### END OF VARIABLES
#### CHANGING VARIABLES BELOW THIS LINE WILL ALTER BEHAVIOR OF THIS SCRIPT
#
INCR_DATE="date_plus_3_weeks.pl" # script to give us a date in the future for
# "touch -atime <date> <file> to give us a lock date
case $1 in
full)
        NEWBACKUP=Y
        INCRBACKUP=N
        ;;
FULL)
        NEWBACKUP=Y
        INCRBACKUP=N
        ;;
incremental)
        INCRBACKUP=Y
        NEWBACKUP=N
        ;;
INCREMENTAL)
        INCRBACKUP=Y
        NEWBACKUP=N
        ;;
*)
        echo "$0 - required argument \"full\" or \"incremental\" not specified"
        exit 1
        ;;
esac
if [ "X$2" != "X" ]
then
        INSTANCE=$2
fi
echo "-------------------------------------------"
if [ ${NEWBACKUP} = "Y" ]
then
        echo "Performing New Level 1 image backup of Instance \"${INSTANCE}\" "
else
        echo "Performing Incremental Level 1 image backup of Instance \"${INSTANCE}\" "
fi
echo "-------------------------------------------"
DATE_SUFFIX=`/usr/bin/date '+%e%b%y'`
BACKUP_DIR=${BACKUP_MOUNT}${BACKUP_DD_PATH}
FULL_BACKUP_DIR=${BACKUP_DIR}/full-${INSTANCE}-${DATE_SUFFIX}
INCR_BACKUP_DIR=${BACKUP_DIR}/incr-${INSTANCE}-${DATE_SUFFIX}
# set the backup directory that we care about for today
if [ ${NEWBACKUP} = Y ] then
        THIS_BACKUP_DIR=${FULL_BACKUP_DIR}
        FASTCOPYPATH=${BACKUP_DD_PATH}/full-${INSTANCE}-${DATE_SUFFIX}
else
        THIS_BACKUP_DIR=${INCR_BACKUP_DIR}
        FULL_BACKUP_DIR=`cat ${BACKUP_DIR}/most_recent_full`
        # derive fastcopypath
        FASTCOPYPATH=""
        Y=${FULL_BACKUP_DIR}
        X=$Y
        Z=""
        while [ ${X} != "backup" ]; do
                X=`basename ${Y}`
                Y=`dirname ${Y}`
                if [ x$Z = "x" ]
                then
                        Z=$X
                else
                        Z=$X/$Z
                fi
        done
        FASTCOPYPATH=$FASTCOPYPATH/$Z
fi
#
if [ ! X${DRYRUN} = X ]
then
        set -vn
fi
# check if directory exists
if [ ! -d ${BACKUP_DIR} ]
then
        echo "Directory \"${BACKUP_DIR}\" not presently available - aborting"
        exit 2
fi
# directory already exists? If so, delete it -- OPTIONAL
if [ -d ${THIS_BACKUP_DIR} ]
then
        echo "Directory \"${THIS_BACKUP_DIR}\" exists: Deleting"
        # rm -rf ${THIS_BACKUP_DIR}
fi
# Make sure we can create the new directory
mkdir -p ${THIS_BACKUP_DIR}
if [ $? != 0 ]
then
        echo "Failed to create \"${THIS_BACKUP_DIR}\" - aborting"
        exit 3
fi
# Delete old backups - OPTIONAL
# Shouldn't be necessary if RMAN is maintaining the Retention policy
#find ${BACKUP_DIR} -name 'full*' -type d -mtime ${DELETE_OLDER} -exec rm -rf {} \;
# Get the appropriate TAG for use with RMAN. Create a new one if full, else use the existing one
if [ ${NEWBACKUP} = "Y" ]
then
        echo "${INSTANCE}_${DATE_SUFFIX}" > ${FULL_BACKUP_DIR}/tagfile
fi
# Now set it here
TAGTODAY=`cat ${FULL_BACKUP_DIR}/tagfile`
# Echo the tag for the record
echo "-----------------------------------------------"
echo "TAG that will be used is ${TAGTODAY}"
echo "-----------------------------------------------"
# create the subdirectory that holds just the archivelogs mkdir -p ${THIS_BACKUP_DIR}/archivesonly
#
# Now finally get down to running the backup # # The command we want RMAN to run is:
# backup incremental level 1 for recover of copy with tag ${TAGTODAY} database;
rman target=${RMANID}/${RMANPW}@${INSTANCE} ${NOCATALOG} << FULLBACK
run {
        configure controlfile autobackup on;
        configure controlfile autobackup format for device type disk to '${THIS_BACKUP_DIR}/%F';
        allocate channel c1 device type disk format '${THIS_BACKUP_DIR}/%U';
        allocate channel c2 device type disk format '${THIS_BACKUP_DIR}/%U';
        allocate channel c3 device type disk format '${THIS_BACKUP_DIR}/%U';
        backup incremental level 1 for recover of copy with tag ${TAGTODAY} database ;
        release channel c1 ;
        release channel c2 ;
        release channel c3 ;
        allocate channel c1 device type disk format '${THIS_BACKUP_DIR}/archivesonly/%U' maxpiecesize ${MAXSIZE};
        allocate channel c2 device type disk format '${THIS_BACKUP_DIR}/archivesonly/%U' maxpiecesize ${MAXSIZE};
        allocate channel c3 device type disk format '${THIS_BACKUP_DIR}/archivesonly/%U' maxpiecesize ${MAXSIZE};
        backup tag ${TAGTODAY} archivelog all delete all input;
        recover copy of database with tag ${TAGTODAY};
}
quit
FULLBACK
# Done
echo Image backup of instance \"${INSTANCE}\" complete
echo --------------------------------------------
echo " Deleting expired backups from RMAN catalog"
echo --------------------------------------------
# Delete expired backups
rman target=${RMANID}/${RMANPW}@${INSTANCE} ${NOCATALOG} << DELEXP
delete noprompt expired backup;
delete noprompt expired archivelog all;
delete noprompt expired copy;
quit
DELEXP
#
# Fastcopy the full directory we just updated so it is retained # NEWDIR=${FASTCOPYPATH}-retain-${DATE_SUFFIX}
ssh ${BACKUP_DDSYS} "filesys fastcopy force source ${FASTCOPYPATH} \
destination ${NEWDIR} "
# now bring the second copy into the RMAN catalog
#
rman target=${RMANID}/${RMANPW}@${INSTANCE} ${NOCATALOG} << CATHERE
catalog start with '${BACKUP_MOUNT}${NEWDIR}';
yes
quit;
CATHERE
# record the backup directory so the incremental can find it
#
if [ ${NEWBACKUP} = Y ]
then
        rm -f ${BACKUP_DIR}/most_recent_full
        echo ${FULL_BACKUP_DIR} > ${BACKUP_DIR}/most_recent_full
fi
#
# If Retention Lock is going to be used, set the access time on the new files
#
if [ ${RETENTIONLOCK} = "yes" ]
then
        RETAINUNTIL=`date_plus_3_weeks.pl`
        export RETAINUNTIL
        echo --------------------------------------------
        echo Locking file until ${RETAINUNTIL}
        echo --------------------------------------------
        find ${FULL_BACKUP_DIR} -name '*' -exec touch -a ${RETAINUNTIL} \{\} \;
fi
# Script complete

Perl script that provides a date string three weeks out

This script is called by the weekly full script shown above. The actual desired retention should be altered here or else the script parameterized and the desired retention passed as an argument.

#!/usr/bin/perl
use strict;
use DateTime;
my $dt2 = DateTime->today();
# get today's date $dt2->add (weeks => 3);
print ($dt2->year(),$dt2->month(),$dt2->day(),"\n");

Interpreting the results

Below are some space reporting results from the Data Domain system used in the development of the script shown above. The first output was taken before the script was invoked with the “incremental” option and the second after it completed. Please note that the database was open but undergoing little change, so a significant amount of the incremental backup storage was consumed in backing up the handful of archive logfiles that had been created. Nonetheless, the report shows that 4.5GB of additional data was introduced to the Data Domain but only an additional 16MB of space has been consumed.

Upcoming Webcast for Oracle Database Virtualization using VMware

Please join me in my upcoming EMC Live! event. I will be presenting on Efficient and High Performance Virtualization of Oracle Database Environments using vSphere, similar to my presentation at VMworld in September. However, I have added significant material, so those who saw the VMworld presentation should still join this webcast. Anyone who is interested in virtualization of Oracle database servers should find this presentation interesting as well.

This event will be presented live on October 29 at 11:00 a.m. EDT. Details can be found here.

Charles Phillips' response to customer input

For information on the response to Charles Phillips request for customer input on VMware, see Chad Sakac's blog. I heartily agree with Chad's comments. In addition, I will add the following point, which I think is key to the discussion.

Charles Phillips makes effectively the same statement that is contained in the metalink statement on support, i.e.:

What we've said on other VMs is that we will respond to support issues as they arise but so far we've elected not to formally certify any third party VMs....

The metalink support statement is very similar. It says:

Oracle has not certified any of its products on VMware virtualized environments.

In my opinion, both statements are slightly disingenuous. Certainly, they are misleading.

Other than RAC, Oracle simply does not certify things like this. There is no Oracle certification, so far as I can tell, for any hardware platform, other than RAC certified configurations. And we are not talking about RAC here.

As Chad points out, Oracle does not maintain a Hardware Compatibility Program for its software. For this reason, telling customers not to run VMware because it is not "certified" to run with Oracle software is simply meaningless. The same can be said of Dell servers, Intel CPUs, Emulex HBAs, Broadcom NICs and almost all of the rest of the stack on which Oracle software commonly runs.

Ask yourself this simple question: What is it that VMware provides? Folks think of VMware as a software company. In many respects this is wrong headed thinking. VMware is a platform vendor. What VMware provides is virtualized hardware. Thus, VMware provides a virtualized CPU, network connection, SAN connectivity, etc.

In that respect, VMware is far more similar to, say, Dell than it is to, say, Red Hat. Certainly, VMware does not provide an OS environment upon which the Oracle software runs directly. Rather, VMware provides a hardware (albeit virtualized hardware) stack upon which an OS environment like Red Hat Enterprise Linux (RHEL) runs.

Oracle does provide certification of things like RHEL. But, as I have pointed out before, RHEL runs just fine on VMware, and is actually certified to do so by Red Hat.

Perhaps it would help if VMware behaved more like a platform vendor and less like a software developer. That is another conversation. Personally, I would welcome that. Such a change would require VMware to be more involved in solutions development for applications vendors like Oracle, which I think would be a very welcome development.

In the meantime, I respectfully disagree with the assertion by Charles Phillips that Oracle software should not be run on VMware because it is not "certified" by Oracle. That assertion is simply inaccurate and misleading in my view.

Your comments are welcome as usual.