Javascript required
Skip to content Skip to sidebar Skip to footer
Home / Oracle Database Backup to Aws S3

Oracle Database Backup to Aws S3

Background

Historically most of the conversations I have regarding Oracle Recovery Manager (RMAN) are around the use of NFS storage such as a Pure Storage FlashBlade, and Oracle Direct NFS (dNFS).

However, with more Oracle customers deploying databases in Oracle Cloud Infrastructure (OCI) and / or using Object Storage for backups I thought it was about time I had a look using RMAN with S3.

Preparation

Before we can start using the Oracle OSB (Oracle Secure Backup) Module for Oracle RMAN (Recovery Manager) backups we need to have the following to hand:

  • OTN account details
    • You may register for a free OTN account at http://otn.oracle.com
  • AWS Access Key ID and Secret Access Key.
    • You can create a free AWS account with 5GB of S3 storage https://aws.amazon.com
  • The OBS installer (osbws_installer.jar)
    • Oracle 19c ships the osbws_installer.zip within the $ORACLE_HOME/lib directory.
    • Alternatively you can find the Oracle OSB Web Services Module for Amazon S3 here.
Oracle Secure Backup Cloud Module for Amazon S3

Create AWS S3 User Account

Logon to the AWS console and create a dedicated AWS user for your RMAN backups using the 'Add User' wizard, you can find this in the Identity and Access Management (IAM) area.

In this example I have specified Programmatic access, as this account will not be connecting via the console.

Create User

At step 2, create a group and attach the AmazonS3FullAccess policy.

Create Group

Complete the wizard to create the user account.

Check AWS S3 Bucket

From the AWS Console, select S3 and create a new bucket.

Create s3 Bucket

Test S3 Access

Install and configure AWS CLI to test access if not already available.

The AWS CLI provides a rich set of options which you read-up on here, we can list use the AWS to show our newly created bucket with the aws s3api list-buckets command.

$          aws s3api list-buckets --query "Buckets[].Name" --endpoint-url http://s3.amazonaws.com --profile aws          ------------------- |   ListBuckets   | +-----------------+ |                      rontestbucket            |     +-----------------+

Prepare Oracle Environment

You will need to create an Oracle wallet in your Oracle Home is you don't already have one. e.g.

$          cd $ORACLE_HOME/dbs          $          mkdir osbws_wallet

The Oracle Secure Cloud Backup Module requires a Java version of 1.7 or higher, you can check this with java -version e.g.

$          java -version          openjdk version "1.8.0_212" OpenJDK Runtime Environment (build 1.8.0_212-b04) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

Unzip the Oracle 19c osbws_installer.zip or the file previously downloaded.

Please note the installer needs internet access.

          The installer will download the library appropriate to the platform it is running on. It will also create the library configuration file and the Oracle Wallet where the S3 credentials are stored.

Let's run the installer without any options to find a list of command line parameters.

$          java -jar osbws_install.jar          Oracle Secure Backup Web Service Install Tool, build 19.3.0.0.0DBRU_2019-04-17 No arguments supplied Usage: java -jar osbws_install.jar -AWSID:          AWS Access Key ID -AWSKey:         AWS Secret Access Key -IAMRole:        AWS IAM role name -IAMRoleMetaUri: Metadata URI for the specified IAM role -awsEndPoint:    non default host name -awsPort:        non default HTTP/HTTPS port -location:       Location to store backups -useHttps:       setup HTTPS -useSigV2:       setup authentication scheme -walletDir:      Directory to store wallet -trustedCerts:   SSL certificates to be imported -import-all-trustcerts:                  Import all certificates from Java truststore -configFile:     File name of config file -libDir:         Directory to store library -libPlatform:    Platform of library to download          -lib-download-only:                  Download library only          -proxyHost:      HTTP proxy host -proxyPort:      HTTP proxy port -proxyID:        HTTP proxy userid, if needed -proxyPass:      HTTP proxy password, if needed -argFile:        File name of arguments file -help:           Print this usage information and exit

Oracle OSB Installer

The osbws_install Java utility will try to perform the following:

  • Install an osbws library file into '-libDir' location e.g. '$ORACLE_HOME/lib'
  • Create a configuration file in '$ORACLE_HOME/dbs'
  • Create a wallet file in the '-walletDir' location e.g. '$ORACLE_HOME/dbs/osbs_wallet'

Below is my Oracle Secure Backup script for AWS.

          #!/bin/bash export AWSID=<AWS ID> export AWSKey=<AWS Secret Key>   java -jar osbws_install.jar \  -AWSID ${AWSID} \  -AWSKey ${AWSKey} \  -walletDir ${ORACLE_HOME}/dbs/osbws_wallet \  -location ${location} \  -libDir $ORACLE_HOME/lib

Ok, now let's now install OBS

$          ./osbws_install_AWS.sh          Install OSB Library  Oracle Secure Backup Web Service Install Tool, build 19.3.0.0.0DBRU_2019-04-17  Debug: os.name        = Linux  Debug: os.arch        = amd64  Debug: os.version     = 4.14.35-1902.3.1.el7uek.x86_64  Debug: file.separator = /  Debug: Platform = PLATFORM_LINUX64  Debug: Verifying AWS account using endpoint s3.amazonaws.com  Debug: Canoical Request: ...  Oracle Secure Backup Web Service wallet created in directory          /u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbws_wallet.          Oracle Secure Backup Web Service initialization file          /u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbwsDEMO1.ora          created.  Downloading Oracle Secure Backup Web Service Software Library from file osbws_linux64.zip.  Debug: Temp zip file = /tmp/osbws_linux644593000752882300726.zip  Debug: Downloaded 27721116 bytes in 19 seconds.  Debug: Transfer rate was 1459006 bytes/second.  Download complete.

OSB Database file

From the above we can see a osbw<ORACLE_SID>.ora has been created in ${ORACLE_HOME}/dbs.

Before we perform a backup let's have a look at the file created.

$          cat $ORACLE_HOME/dbs/osbwsDEMO1.ora  #OSB_WS_HOST=http://s3.amazonaws.com          OSB_WS_WALLET='location=file:/u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbws_wallet CREDENTIAL_ALIAS=rekins_aws'          OSB_WS_BUCKET=rontestbucket          OSB_WS_LOCATION=eu-west-2

I have customised the OSB parameter file to use the S3 bucket previously created, if you do not specify OSB_WS_BUCKET Oracle will automatically create an s3 bucket on your behalf.

OSB supports a number of other parameters and a few undocumented, non-supported underscore parameters which may be useful for debugging purposes e.g. _OBS_WS_TRACE_LEVEL

Set this to 100 to enable tracing and 0 to disable e.g. _OSB_WS_TRACE_LEVEL=100

If used the trace information can be found in sbtio.log file in $ORACLE_BASE/diag/rdbms/../../trace/

RMAN Backup to AWS s3

Now we have every thing in place, let's try a performing a test backup of the user tablespace.

$          rman target=/          RMAN> RUN 2> { 3> allocate channel c1_s3 device type sbt 4> parms='SBT_LIBRARY=/u01/app/oracle/product/19.0.0/dbhome_1/lib/libosbws.so,SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbwsDEMO1.ora)'; 5>          backup tablespace users; 6> }   using target database control file instead of recovery catalog allocated channel: c1_s3 channel c1_s3: SID=5691 instance=DEMO1 device type=SBT_TAPE channel c1_s3: Oracle Secure Backup Web Services Library VER=19.0.0.1   Starting backup at 25-JAN-2021 150728 channel c1_s3: starting full datafile backup set channel c1_s3: specifying datafile(s) in backup set input datafile file number=00007 name=+DATA/DEMO/DATAFILE/users.379.1040572917          channel c1_s3: starting piece 1 at 25-JAN-2021 150728 channel c1_s3: finished piece 1 at 25-JAN-2021 150731 piece handle=c6vlh9lg_1_1          tag=TAG20210125T150728 comment=API Version 2.0,MMS Version 19.0.0.1 channel c1_s3: backup set complete, elapsed time: 00:00:03 Finished backup at 25-JAN-2021 150731   Starting Control File and SPFILE Autobackup at 25-JAN-2021 150731 piece handle=c-3784643325-20210125-01 comment=API Version 2.0,MMS Version 19.0.0.1 Finished Control File and SPFILE Autobackup at 25-JAN-2021 150735 released channel: c1_s3        
RMAN>          list backup of tablespace users;          List of Backup Sets =================== BS Key  Type LV Size       Device Type Elapsed Time Completion Time    ------- ---- -- ---------- ----------- ------------ ------------------ 1892    Full    2.50M      SBT_TAPE    00:00:01     25-JAN-2021 150729         BP Key: 3489   Status: AVAILABLE  Compressed: NO  Tag: TAG20210125T150728         Handle:          c6vlh9lg_1_1          Media:          s3-eu-west-2.amazonaws.com/rontestbucket          List of Datafiles in backup set 1892   File LV Type Ckp SCN    Ckp Time           Abs Fuz SCN Sparse Name   ---- -- ---- ---------- ------------------ ----------- ------ ----   7       Full 69186257   25-JAN-2021 150728              NO          +DATA/DEMO/DATAFILE/users.379.1040572917

Finally let's log back into the AWS Console to confirm our RMAN backup has arrived in our specified AWS S3 bucket.

AWS s3 Bucket

From the above we can see our RMAN backup piece handle c6vlh9lg_1_1 has been created in the rontestbucket S3 bucket.

OSB Parameters

Below I have listed the supported and undocumented Oracle 19c OSB parameters for troubleshooting / educational purposes.

$ strings $ORACLE_HOME/lib/libosbws.so  | grep '^_OSB_'  $ strings $ORACLE_HOME/lib/libosbws.so  | grep '^OSB_'
Documented / Supported Undocumented / Unsupported
OSB_WS_PFILE _OSB_WS_ALLOCBUF_DISABLE
OSB_WS_HOST _OSB_WS_AUTH_SCHEME
OSB_WS_PROXY _OSB_WS_SEND_BUF_SIZE
OSB_WS_BUCKET _OSB_WS_RECV_BUF_SIZE
OSB_WS_LOCATION _OSB_WS_SDU_SIZE
OSB_WS_CHUNK_SIZE _OSB_WS_TDU_SIZE
OSB_WS_LICENSE_ID _OSB_WS_REUSE_CONNECTION
OSB_WS_LICENSE_MAX_SESSIONS _OSB_WS_SESSION_RETENTION_TIME
OSB_WS_WALLET _OSB_WS_CHECKER_RETENTION_TIME
OSB_WS_VIRTUAL_HOST _OSB_WS_PURGE_LIMIT
OSB_WS_IAM_ROLE _OSB_WS_CLEANER
OSB_WS_IAM_ROLE_META_URI _OSB_WS_CLEANER_RETENTION_TIME
OSB_WS_CREDENTIAL_OBJECT _OSB_WS_RUN_CLEANER
_OSB_WS_CONNECT_TIMEOUT
_OSB_WS_RESPONSE_TIMEOUT
_OSB_WS_SEND_TIMEOUT
_OSB_WS_RETRY_WAIT_TIME
_OSB_WS_UPLOAD_DELAY
_OSB_WS_EVENT
_OSB_WS_TRACE_LEVEL
_OSB_WS_100_CONTINUE
_OSB_WS_DEFERRED_DELETE
_OSB_WS_FOLLOW_REDIRECT
_OSB_WS_BUFFER_WRITE
_OSB_WS_BUFFER_READ
_OSB_WS_USE_IPV6
_OSB_WS_NO_SSL
_OSB_WS_VALIDATE_CERT
_OSB_WS_UNLOAD_DLL
_OSB_WS_NO_PROXY
_OSB_WS_USE_BULK_DELETE
_OSB_WS_CHUNK_PARTS
Please Note underscore parameters should only be used for trouble shooting or under the direction of Oracle Support

Summary

In Part 1 I have shown how we can use RMAN and Oracle Secure Backup (OSB) to backup an on-premises database to an AWS S3 Object Store.

In Part 2 I will show how we can do the same but this time to an on-premises Pure Storage FlashBlade S3 Object Storage.

Follow @RonEkins

Oracle Database Backup to Aws S3

Source: https://ronekins.com/2021/01/25/performing-oracle-rman-backups-to-s3-part-1-amazon-s3/