Oracle Storage Guy

NetApp's ExaData Backup Play Reprise

I have received a fair number of responses to my previous post on this subject (some via comments and some via email). I thought the discussion worthwhile enough to punch it up a bit more here.

Backup

As I pointed out in the previous post, EMC can easily match NetApp's play to back up ExaData with the following:

As Geoff Rosser so correctly pointed out, this answer is incomplete. Yes, Data Domain is an awesome Oracle backup solution. Yes, it provides incredible deduplication rates for Oracle database environments. (Thanks, dynamox.) However, it is not the only viable solution from EMC for backing up ExaData. For example, the following would also work:

As usual comments are welcome.

NetApp's ExaData Backup Play

There has been lots of material on the web recently concerning NetApp being able to backup ExaData. The purpose of this blog is to respond to that content, and state why NetApp's offering is rather lame, and actually offers nothing new.

The items on the web produced by NetApp are easy to find. I will not increase their Google hit rate by linking to them here. Suffice it to say, Neil Gerren's blog contains the principle content to which I will respond here. There is also NetApp technical report TR 4022, a 34 page tome, which I have read thoroughly. I think I completely understand what NetApp has produced (which they thought so much of to produce a press release on the subject), and, believe me, there is nothing new or unique in NetApp's offering.

I will summarize the solution here. To avoid any issues with NetApp copyrights, I have recreated the graphics describing the solution. However, the gist is identical to that contained in TR 4022. The solution consists of two parts:

• Backup
• Disaster recovery / remote replication

Backup

Let's start with the backup component. What NetApp proposes is the following:

The fundamental gist of the solution is:

• Connect ExaData to a NetApp filer using 10 GbE.
• Backup the Oracle database on the ExaData to the NetApp filer using RMAN.

I ask you: Is there anything interesting or unique here? In fact, I would state for the record that it would be manifestly better to do the following:

In other words, an EMC Data Domain deduplication array is capable of connecting to an ExaData box and backing it up using RMAN in exactly the same manner as a NetApp filer. I find this statement so obvious that it seems self-evident. But, again, NetApp's material on the subject makes it necessary to point this out.

Data Domain has many advantages over NetApp in the area of Oracle backup. But I will not belabor that point. Suffice it to say, we can do the same thing that NetApp can in the area of backing up Oracle ExaData via 10 GbE. And we certainly have a product which is very suited to backing up Oracle database data, and commonly used for this purpose.

Disaster Recovery / Remote Replication

Let's turn now to the disaster recovery / remote replication solution. What NetApp proposes for a disaster recovery / remote replication solution looks like this:

The fundamental gist of the solution is: 

• Replicate the production Oracle databse on ExaData using Oracle Data Guard with physical standby. The target platform consists of generic Intel x86 servers running Linux. (Remember that the RAC compute nodes in the ExaData are nothing more than relatively typical x86 / Linux servers. Thus, Oracle Data Guard with physical standby to a similar non-Oracle server will work just fine, it being an identical platform from an Oracle point of view.)
• The target database server is connected to a NetApp filer using standard protocols (e.g. FC or NFS).
• Once the database copy is on the NetApp filer, normal NetApp tools can be used to snap, clone, etc. the target database.

Again. it seems rather patently obvious that the following is also possible:

And in terms of the ability to perform snaps, clones, and so forth of the Oracle database once it is on our array, EMC's arrays contain features that easily meet those offered by NetApp.

So, again, NetApp's highly touted offering concerning storing ExaData offers nothing new or unique. Each and every feature offered by NetApp is available from EMC, and many features offered by EMC are superior to those provided by NetApp.

Comments to this post are welcome.

 

VMworld 2010 Session Content Part 1 - Impending Data Explosion

I had a very successful breakout session at VMworld this year, presenting to a total of around 200 people. The topic was:

Creating an Internal Oracle Database Cloud Using vSphere

I will now share much of the substance of my session here on this blog, for those of you who could not attend VMworld.

The first section (which will constitute this first post) was on the coming data explosion that will soon hit (indeed is already hitting) Oracle DBAs all over the industry.

Many of you are familiar with the Joe Tucci keynote from EMC World this spring. In that talk, Joe pointed out the recent IDC / Dataquest study, which points out the coming world wide data explosion which can be simply stated as follows:

Year   Worldwide Data Set
2009   0.8 Zettabytes
2020   35 Zettabytes

Where a Zettabyte is 1024 Exabytes, an Exabyte is 1024 Petabytes and a Petabyte is 1024 Terabytes. A Zettabyte is therefore approximately 10 ^ 21 bytes.

That's a data explosion of 44x in approximately 10 years.

What Joe does not point out is the shift from unstructured data to structured data. Presently, 90% of worldwide data is unstructured. However, almost all of the sources of new data which are causing the explosion are structured. (These include social networking, blogging, Twitter, and all of the Web 2.0 content, plus sources like e-readers, PDAs, smart phones, medical / dental digital imaging, online security, smart energy metering and the like.) The result is this graphic showing the relationship between structured and unstructured data going into the future:

Undoubtedly much of this data will end up being owned by the so-called Database 2.0 vendors. Certainly, that will be true for the bulk of the social networking and Web 2.0 content, where those vendors have formed their roots. But this is not true for all of this data. Look again at the sources of data shown above. Is it likely that a medical or dental image (which may have critical consequences for actual patient care and confidentiality) will end up on a relatively untested and emerging Database 2.0 technology? I would regard that is very unlikely. It is much more likely that this data will end up in the known-to-be-reliable good enough technology in the Database 1.0 space we know as Oracle.

The same thing is true for smart metering. Again, the output of these meters will be used to calculate actual customer bills, not to mention determining the energy output for homes and businesses. Is the external cloud Database 2.0 environment ready to absorb this data? I doubt that. Folks like Progress Energy (my energy provider) are going to place that on Oracle, most likely.

Even making the incredibly depressing (for Oracle) prediction that Oracle's market share in the database market falls by 50% in the next 10 years, the result for Oracle can be calculated simply as follows:

44 x (9 / 2)

or about 200X. This means that the growth in Oracle data volume worldwide will increase by 200X in the next 10 years, by my conservative estimate.

!!!!!!

I compare this to the movie Princess Bride, one of my favorite films by Rob Reiner. In this film, as many of you know, the main character Wesley leaves his beloved Buttercup to find his fortune. Very soon, he is captured by the Dread Pirate Roberts who makes him his cabin boy. During his years of service, the Dread Pirate Roberts tells Wesley the same thing everday:

"Hello, Wesley. I'll probably kill you in the morning"

I maintain that this is what the impending data explosion means to Oracle DBAs. The day will come when Oracle DBAs live in the "Wesley State". Coming to work everyday, they will look at their screens and their Oracle database will grin back, saying to them, in effect:

"Hello, DBA. I'll probably kill you in the morning."

Those DBAs who survive this challenge will do so because they have absorbed a very different way of thinking about their Oracle database environments. That will require many changes, among them the willingness to scale their database infrastructure in a completely new and radical way.

More on this later. In up-coming blogs, I will explain how I think that change will come about, what choices the Oracle DBA has to meet this challenge, and so forth.

To end, many of you are coming to OOW this month. I will be there as well, and will be co-presenting on the subject of RMAN backup with my good friend and former manager, Bruce Clarke. Bruce was my second boss at NetApp, and is now with Data Domain. Please come by the EMC booth and say hello, and I will post the logistics of my talk on this blog.

Also, a big thanks to Steve Tout with VMware, who co-presented with me at VMworld and provided my customer case study. Steve is responsible for all identities which exist on VMware.com and VMworld.com. Please see his blog as well. It contains much of the content he presented at VMworld.

 

 

 

Data Domain deduplication plus RMAN incremental backup saves space and time

Bruce Clarke, my good friend and former boss, and I have have been collaborating recently on RMAN backup to Data Domain deduplication storage arrays. In the process I have learned from Bruce (who I gratefully acknowledge for the technical content of this post, including the scripts shown below) about a very valuable approach to RMAN backup.

Most DBAs do full RMAN backups of all databases which are small enough to fit into a backup window. Full backups have several advantages:

1. They are very simple to run.
2. They optimize the restore operation, as no redundant blocks need to be applied.
3. They reduce dependencies in terms of the number of backup pieces that need to be cataloged and maintained in order to do a given restore, thereby reducing risk.

However, fulls have two strong disadvantages:

1. They waste space since the entire database's content is stored in each full backup.
2. They are time-consuming to run, because all active blocks in the database must be transferred to the backup target.

The first disadvantage is neatly avoided by the use of a deduplication array, such as Data Domain. My recent testing indicates that this is certainly true. More on this in a later post.

The second disadvantage is unavoidable at present. (Future developments in the area of source-based deduplication on the Data Domain side may mitigate this later on, but I will ignore that for now.) For most DBAs, the trade-off is acceptable as long as the full backup can fit into the available backup window. This is somewhat of a moving target as well, since recent developments in the area of storage networking (such as 10 GbE) have made the bandwidth increase, thereby increasing the size of the database that can fit into a backup window.

Unfortunately, the size of the typical database is also exploding as we all know. Thus, the issue remains: For a very significant number of large databases, the full backup cannot fit into the backup window. For these databases, the DBA has no real choice: He or she must use RMAN incremental backup.

Incremental backup (absent incremental update, which I will discuss later) provides the DBA with the option to only back up blocks which have been updated since the last backup. This provides a dramatic savings in terms of backup time, but with a significant manageability cost:

1. The number of backup pieces which must be cataloged and maintained increases significantly, making the dependencies greater, and thus increases the risk of a failed restore.
2. The restore time increases, as blocks must be restored multiple times, as the same block may be updated many times and thus stored in separate incremental backups.

Because of these issues, Oracle introduced incremental update for RMAN with 10g. There is an excellent article on the way this feature works in the Oracle documentation on OTN. Effectively, incremental update applies the incremental backup to the previous full backup. This neatly solves all of the issues of RMAN backup, except one: Instead of having a set of full backups, you end up with a single rolling full backup. In other words, once you apply the incremental backup to the previous full backup, you no longer have that previous full backup. Instead, you have a new full backup as of the point in time when you took the incremental backup.

This issue can be solved, obviously, by making a copy of the full backup prior to the incremental update operation. However, making this copy is time consuming, causes I/O, and takes up space.

Enter Bruce's solution: By using the Data Domain fast copy feature, you can make a copy of the previous full backup very quickly, and with no additional space. Further, the blocks which will be applied to the full backup copy have already been stored on the Data Domain in the incremental backup. Thus, they take up very little space as well. The end result is tantamount to RMAN backup nirvana:

1. You can take your backups in a fraction of the time it would require to do a full backup.
2. At the end of the process, you have a set of full backups for each time you run your RMAN backup.
3. All backups are fully deduplicated, thus saving you lots of space.
4. Restore operations are optimized as there are no dependencies other than a single full backup, and thus no redundant blocks to apply.

The scripts to accomplish this are below. The caveats all apply: This is provided as is with no warranty or support obligations. You are on your own in terms of all that. However, the solution does work, and you should definitely give it a try. Also, note that this is Data Domain copyrighted material. However, I have received permission from Bruce Clarke, the author of this script, to publish it here.

Example script for weekly backups

This was created and tested on Sun Solaris so slight changes for other platforms might be required.

#!/bin/sh -f # version 1.0 - Copyright (C) 2009, Data Domain, Inc - Nov, 2009
#
# Perform backup of an Oracle database in preparation using the
# incremental update procedure
#
# This script performs periodic full backups using the fact that
# a requested to the existing full creating an updated
# full in case recovery is required.
#
# Arguments: $1 - "full" or "incremental"
# $2 - instance name to be backed - Default: "orcl"
#
# specifying "full" starts a new "tag" thus creating a new full sequence
#
# The script also keeps everything in separate subdirectories. Because
# the underlying storage is assumed to be deduplicated, the actual storage
# consumed is similar to just running a series of incrementals
#
# Archivelogs and controlfiles are also backed up in the sequence
#
# directory naming scheme being followed
#             $BACKUP_DIR :: path to make backup directory
#             $BACKUP_DIR/full-$INSTANCE-date :: full backup directory for today
#
# IMPORTANT NOTE: THIS SCRIPT IS PROVIDED WITHOUT WARRANTY OR SUPPORT.
# FEEDBACK IS ALWAYS
# WELCOME, HOWEVER, IT SHOULD BE CLEARLY UNDERSTOOD THAT USE OF THIS SCRIPT
# SHOULD ONLY
# BE MADE AFTER AN EXPERIENCED ORACLE DBA CLEARLY UNDERSTANDS ALL STEPS IN THIS
# SCRIPT
# AND HAS MADE NECESSARY MODIFICATIONS AND PERFORMED SUFFICIENT TESTING TO BE
# ASSURED OF ITS
# PROPER OPERATION IN THEIR PARTICULAR ENVIRONMENT
#
# edit variables below as necessary
#
#DRYRUN=1 # Remove this line to allow script to run
                    # with this line, the script will only echo
                    # operations it would have performed
BACKUP_MOUNT="/dd/dd690a" # where "backup" directory on DD is mounted
BACKUP_DD_PATH="/backup/oracle"
BACKUP_DDSYS="[email protected]" # used to perform ssh fastcopy after
# completion
INSTANCE="orcl"
RMANID="sys"
RMANPW="oracle"
RMANCAT="NOCATALOG" # Modify if a catalog is being used
RETENTIONLOCK=no # Yes, says change the atime to lock the file, otherwise don't
DELETE_OLDER=21 # how old a directory needs to be before it expires
                                # only relevant if RETENTIONLOCK is set to "yes"
MAXSIZE="10 g"   # Maximum backup filesize - smaller allows replication to start
                                # sooner
# only relevant to archivelog backupsets. All other backups are "copy"
#### END OF VARIABLES
#### CHANGING VARIABLES BELOW THIS LINE WILL ALTER BEHAVIOR OF THIS SCRIPT
#
INCR_DATE="date_plus_3_weeks.pl" # script to give us a date in the future for
# "touch -atime <date> <file> to give us a lock date
case $1 in
full)
        NEWBACKUP=Y
        INCRBACKUP=N
        ;;
FULL)
        NEWBACKUP=Y
        INCRBACKUP=N
        ;;
incremental)
        INCRBACKUP=Y
        NEWBACKUP=N
        ;;
INCREMENTAL)
        INCRBACKUP=Y
        NEWBACKUP=N
        ;;
*)
        echo "$0 - required argument \"full\" or \"incremental\" not specified"
        exit 1
        ;;
esac
if [ "X$2" != "X" ]
then
        INSTANCE=$2
fi
echo "-------------------------------------------"
if [ ${NEWBACKUP} = "Y" ]
then
        echo "Performing New Level 1 image backup of Instance \"${INSTANCE}\" "
else
        echo "Performing Incremental Level 1 image backup of Instance \"${INSTANCE}\" "
fi
echo "-------------------------------------------"
DATE_SUFFIX=`/usr/bin/date '+%e%b%y'`
BACKUP_DIR=${BACKUP_MOUNT}${BACKUP_DD_PATH}
FULL_BACKUP_DIR=${BACKUP_DIR}/full-${INSTANCE}-${DATE_SUFFIX}
INCR_BACKUP_DIR=${BACKUP_DIR}/incr-${INSTANCE}-${DATE_SUFFIX}
# set the backup directory that we care about for today
if [ ${NEWBACKUP} = Y ] then
        THIS_BACKUP_DIR=${FULL_BACKUP_DIR}
        FASTCOPYPATH=${BACKUP_DD_PATH}/full-${INSTANCE}-${DATE_SUFFIX}
else
        THIS_BACKUP_DIR=${INCR_BACKUP_DIR}
        FULL_BACKUP_DIR=`cat ${BACKUP_DIR}/most_recent_full`
        # derive fastcopypath
        FASTCOPYPATH=""
        Y=${FULL_BACKUP_DIR}
        X=$Y
        Z=""
        while [ ${X} != "backup" ]; do
                X=`basename ${Y}`
                Y=`dirname ${Y}`
                if [ x$Z = "x" ]
                then
                        Z=$X
                else
                        Z=$X/$Z
                fi
        done
        FASTCOPYPATH=$FASTCOPYPATH/$Z
fi
#
if [ ! X${DRYRUN} = X ]
then
        set -vn
fi
# check if directory exists
if [ ! -d ${BACKUP_DIR} ]
then
        echo "Directory \"${BACKUP_DIR}\" not presently available - aborting"
        exit 2
fi
# directory already exists? If so, delete it -- OPTIONAL
if [ -d ${THIS_BACKUP_DIR} ]
then
        echo "Directory \"${THIS_BACKUP_DIR}\" exists: Deleting"
        # rm -rf ${THIS_BACKUP_DIR}
fi
# Make sure we can create the new directory
mkdir -p ${THIS_BACKUP_DIR}
if [ $? != 0 ]
then
        echo "Failed to create \"${THIS_BACKUP_DIR}\" - aborting"
        exit 3
fi
# Delete old backups - OPTIONAL
# Shouldn't be necessary if RMAN is maintaining the Retention policy
#find ${BACKUP_DIR} -name 'full*' -type d -mtime ${DELETE_OLDER} -exec rm -rf {} \;
# Get the appropriate TAG for use with RMAN. Create a new one if full, else use the existing one
if [ ${NEWBACKUP} = "Y" ]
then
        echo "${INSTANCE}_${DATE_SUFFIX}" > ${FULL_BACKUP_DIR}/tagfile
fi
# Now set it here
TAGTODAY=`cat ${FULL_BACKUP_DIR}/tagfile`
# Echo the tag for the record
echo "-----------------------------------------------"
echo "TAG that will be used is ${TAGTODAY}"
echo "-----------------------------------------------"
# create the subdirectory that holds just the archivelogs mkdir -p ${THIS_BACKUP_DIR}/archivesonly
#
# Now finally get down to running the backup # # The command we want RMAN to run is:
# backup incremental level 1 for recover of copy with tag ${TAGTODAY} database;
rman target=${RMANID}/${RMANPW}@${INSTANCE} ${NOCATALOG} << FULLBACK
run {
        configure controlfile autobackup on;
        configure controlfile autobackup format for device type disk to '${THIS_BACKUP_DIR}/%F';
        allocate channel c1 device type disk format '${THIS_BACKUP_DIR}/%U';
        allocate channel c2 device type disk format '${THIS_BACKUP_DIR}/%U';
        allocate channel c3 device type disk format '${THIS_BACKUP_DIR}/%U';
        backup incremental level 1 for recover of copy with tag ${TAGTODAY} database ;
        release channel c1 ;
        release channel c2 ;
        release channel c3 ;
        allocate channel c1 device type disk format '${THIS_BACKUP_DIR}/archivesonly/%U' maxpiecesize ${MAXSIZE};
        allocate channel c2 device type disk format '${THIS_BACKUP_DIR}/archivesonly/%U' maxpiecesize ${MAXSIZE};
        allocate channel c3 device type disk format '${THIS_BACKUP_DIR}/archivesonly/%U' maxpiecesize ${MAXSIZE};
        backup tag ${TAGTODAY} archivelog all delete all input;
        recover copy of database with tag ${TAGTODAY};
}
quit
FULLBACK
# Done
echo Image backup of instance \"${INSTANCE}\" complete
echo --------------------------------------------
echo " Deleting expired backups from RMAN catalog"
echo --------------------------------------------
# Delete expired backups
rman target=${RMANID}/${RMANPW}@${INSTANCE} ${NOCATALOG} << DELEXP
delete noprompt expired backup;
delete noprompt expired archivelog all;
delete noprompt expired copy;
quit
DELEXP
#
# Fastcopy the full directory we just updated so it is retained # NEWDIR=${FASTCOPYPATH}-retain-${DATE_SUFFIX}
ssh ${BACKUP_DDSYS} "filesys fastcopy force source ${FASTCOPYPATH} \
destination ${NEWDIR} "
# now bring the second copy into the RMAN catalog
#
rman target=${RMANID}/${RMANPW}@${INSTANCE} ${NOCATALOG} << CATHERE
catalog start with '${BACKUP_MOUNT}${NEWDIR}';
yes
quit;
CATHERE
# record the backup directory so the incremental can find it
#
if [ ${NEWBACKUP} = Y ]
then
        rm -f ${BACKUP_DIR}/most_recent_full
        echo ${FULL_BACKUP_DIR} > ${BACKUP_DIR}/most_recent_full
fi
#
# If Retention Lock is going to be used, set the access time on the new files
#
if [ ${RETENTIONLOCK} = "yes" ]
then
        RETAINUNTIL=`date_plus_3_weeks.pl`
        export RETAINUNTIL
        echo --------------------------------------------
        echo Locking file until ${RETAINUNTIL}
        echo --------------------------------------------
        find ${FULL_BACKUP_DIR} -name '*' -exec touch -a ${RETAINUNTIL} \{\} \;
fi
# Script complete

Perl script that provides a date string three weeks out

This script is called by the weekly full script shown above. The actual desired retention should be altered here or else the script parameterized and the desired retention passed as an argument.

#!/usr/bin/perl
use strict;
use DateTime;
my $dt2 = DateTime->today();
# get today's date $dt2->add (weeks => 3);
print ($dt2->year(),$dt2->month(),$dt2->day(),"\n");

Interpreting the results

Below are some space reporting results from the Data Domain system used in the development of the script shown above. The first output was taken before the script was invoked with the “incremental” option and the second after it completed. Please note that the database was open but undergoing little change, so a significant amount of the incremental backup storage was consumed in backing up the handful of archive logfiles that had been created. Nonetheless, the report shows that 4.5GB of additional data was introduced to the Data Domain but only an additional 16MB of space has been consumed.