Arquitectura y Setup de FileSystems ACFS en Oracle 12c RAC de 3 Nodos con Flex ASM

Introducción:

ACFS (Oracle Automatic Storage Management Cluster File System) es un File System escalable y multi-plataforma y está estrechamente acoplado con Oracle Clusterware. Al ACFS se accede con las herramientas nativas de Sistema Operativo para Filesystems y Storage (NAS) y con File Access Protocols como NFS y Common Internet File System (CIFS). Podemos considerar ACFS como un File System en Cluster con Alta Disponibilidad.

ACFS Soporta los siguientes archivos:

Podemos almacenar archivos de la Base de Datos Oracle (Datafiles, Redo log, Control files, RMAN backups, DataPump , archivos relacionados con Oracle GoldenGate, etc.)
Podemos almacenar archivos de Aplicación (los binaries de RDBMS, los archivos de trace de la base, alert logs, reports de los aplicativos, BFILEs,y archivos de configuración)
Podemos almacenar archivos como video, audio, texto, imagenes, planos de ingenieria, y otros archivos de datos de propósito general de aplicativos.

ACFS soporta diferentes plataformas como Linux, Oracle Solaris, IBM AIX y MS Windows y se puede usar tanto como file system para propósitos generales en plataformas standalone como en cluster. ACFS no tiene límite de cantidad de directorios.

Figura: Arquitectura y Componentes de Oracle ACFS

ACFS/ADVM se administra usando los siguientes utilitarios:

ASMCA
ASMCMD
SQL*Plus
EM Cloud Control
ACFS Plug-in (API)

Nuevas Funcionalidades ACFS en Oracle 12c (12.1.0.2.0)

Oracle Exadata soporta Oracle ACFS
Oracle ACFS soporta más de 1000 Snapshots
Oracle ACFS soporta Oracle Database File Mapping Views

Ejemplo Setup del Ambiente para Oracle 12c (12.1.0.2.0) ACFS en un RAC de 3 Nodos

Environment	RAC de 3 Nodos en Oracle Enterprise Linux 6.5 (rac1-12c, rac2-12c and rac3-12c)
Oracle GI	Oracle 12c (12.1.0.2.0) con Oct 2016 PSU Patch
Oracle RDBMS	Oracle 12c (12.1.0.2.0) con Oct 2016 PSU Patch
ASM and Database Instances	+ASM1 en rac1-12c +ASM2 en rac2-12c +ASM3 en rac3-12c
ASM Device File	/dev/asm/dbhome_1-285
ACFS Disk Group	ACFSDG

Paso 1: Agregar dos discos y formatearlos para crear el diskgroup ACFSDG

[root@rac1-12c ~]# fdisk -l | grep "Disk /dev/sd"
Disk /dev/sdn: 12.9 GB, 12884901888 bytes
Disk /dev/sdo: 12.9 GB, 12884901888 bytes

[root@rac1-12c ~]# fdisk /dev/sdn
[root@rac1-12c ~]# fdisk /dev/sdo

Paso 2: Crear el directorio compartido para ACFS en todos los nodos (rac1-12c, rac2-12c and rac3-12c)

[oracle@rac1-12c ~]$ su - root
[root@rac1-12c ~]# mkdir -p /u01/acfs
[root@rac1-12c ~]# chown -R oracle:oinstall /u01/acfs/
[root@rac1-12c ~]# chmod -R 777 /u01/acfs/

[root@rac1-12c ~]# ssh rac2-12c
[root@rac2-12c ~]# mkdir -p /u01/acfs
[root@rac2-12c ~]# chown -R oracle:oinstall /u01/acfs/
[root@rac2-12c ~]# chmod -R 777 /u01/acfs/

[root@rac2-12c ~]# ssh rac3-12c
[root@rac3-12c ~]# mkdir -p /u01/acfs
[root@rac3-12c ~]# chown -R oracle:oinstall /u01/acfs/
[root@rac3-12c ~]# chmod -R 777 /u01/acfs/

Paso 3: Crear el diskgroup ACFSDG usando ASMCA y crear ACFS para Uso de Base de Datos

Specify ASM Compatibility:           12.1.0.0.0
Database Compatibility:              12.1.0.0.0
ADVM Compatibility:                  12.1.0.0.0

Clickear la Opción “Create ACFS for Database Use” y especificar “Volume Name”, “Mount Point”, “Size”, “Owner Name” y “Owner Group”.

Paso 4: Ejecutar el script como usuario ‘root’ solamente en el nodo 1 (rac1-12c)

Paso 5: Verificar el Volumen ACFS (‘/u01/acfs’) en todos los nodos del cluster (rac1-12c, rac2-12c and rac3-12c)

[oracle@rac1-12c acfs]$ df -m /u01/acfs/
Filesystem                    1M-blocks         Used    Available     Use%   Mounted on
/dev/asm/dbhome_1-285         20480             379      20102        2%     /u01/acfs

[oracle@rac2-12c acfs]$ df -m /u01/acfs/
Filesystem                    1M-blocks         Used    Available     Use%   Mounted on
/dev/asm/dbhome_1-285         20480             379      20102        2%     /u01/acfs

[oracle@rac3-12c acfs]$ df -m /u01/acfs/
Filesystem                    1M-blocks         Used    Available     Use%   Mounted on
/dev/asm/dbhome_1-285         20480             379      20102        2%     /u01/acfs

Validación de ACFS/ADVM

[oracle@rac1-12c ~]$ srvctl status asm -detail
ASM is running on rac3-12c,rac2-12c,rac1-12c
ASM is enabled.

[oracle@rac1-12c ~]$ srvctl config asm -detail
ASM home: <crs home="">
Password file: +ocrvote/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:</crs>

Como usuario root ejecutar “acfsload start”

[root@rac1-12c grid]# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed

[root@rac1-12c grid]# lsmod |grep oracle
oracleacfs                      3307969         2
oracleadvm                      506254          7
oracleoks                       505749          2 oracleacfs,oracleadvm

Note: Antes de verificar el asm –proxy hay que convertir de Standard ASM a FLEX ASM

[root@rac1-12c grid]# srvctl status asm -proxy
PRCR-1001 : Resource ora.proxy_advm does not exist

[root@rac1-12c grid]# srvctl config asm -proxy -detail
PRCR-1001 : Resource ora.proxy_advm does not exist

[root@rac1-12c grid]# srvctl add asm -proxy
PRCA-1063 : Failed to create ADVM proxy resource proxy_advm
PRCR-1035 : Failed to look up CRS resource ora.asm for ora.cluster_asm.type
[root@rac1-12c grid]# exit

Verificar el Cluster si es Flex ASM or Standard Cluster. Si es Standard Cluster lo convertimos a Flex ASM usando ASMCA

[oracle@rac1-12c grid]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle

ASMCMD> showclustermode
ASM cluster : Flex mode disabled

Clickear en la opción : “Convert to Oracle Flex ASM”, especificar ASM Listener Port (1522) y seleccionar Interface Name (Nombre de la Placa de Red)

[root@rac1-12c ~]# sh /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1-12c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.acfsdg.dbhome_1.acfs' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1-12c'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac3-12c'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.OCRVOTE.dg' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.rac1-12c.vip' on 'rac1-12c'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac1-12c'
CRS-2677: Stop of 'ora.rac1-12c.vip' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.rac1-12c.vip' on 'rac3-12c'
CRS-2677: Stop of 'ora.scan3.vip' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac2-12c'
CRS-2677: Stop of 'ora.orcl.db' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rac1-12c'
CRS-2676: Start of 'ora.rac1-12c.vip' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rac3-12c'
CRS-2676: Start of 'ora.scan3.vip' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.cvu' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac2-12c'
CRS-2677: Stop of 'ora.acfsdg.dbhome_1.acfs' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.DBHOME_1.advm' on 'rac1-12c'
CRS-2677: Stop of 'ora.ACFSDG.DBHOME_1.advm' on 'rac1-12c' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rac3-12c'
CRS-2677: Stop of 'ora.oc4j' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac3-12c'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rac3-12c' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1-12c'
CRS-2677: Stop of 'ora.ons' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1-12c'
CRS-2677: Stop of 'ora.net1.network' on 'rac1-12c' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1-12c' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.storage' on 'rac1-12c'
CRS-2677: Stop of 'ora.storage' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1-12c'
CRS-2677: Stop of 'ora.ctssd' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1-12c'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1-12c'
CRS-2677: Stop of 'ora.cssd' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1-12c'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1-12c'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1-12c'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1-12c'
CRS-2676: Start of 'ora.diskmon' on 'rac1-12c' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1-12c' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1-12c'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1-12c'
CRS-2676: Start of 'ora.ctssd' on 'rac1-12c' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1-12c'
CRS-2676: Start of 'ora.asm' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac1-12c'
CRS-2676: Start of 'ora.storage' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1-12c'
CRS-2676: Start of 'ora.crsd' on 'rac1-12c' succeeded
Oracle Grid Infrastructure restarted in node rac1-12c
PRCC-1014 : ASMNET1LSNR_ASM was already running
PRCR-1004 : Resource ora.ASMNET1LSNR_ASM.lsnr is already running
PRCR-1079 : Failed to start resource ora.ASMNET1LSNR_ASM.lsnr
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rac1-12c'
ASM listener ASMNET1LSNR_ASM running already
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2-12c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac2-12c'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac2-12c'
CRS-2677: Stop of 'ora.scan3.vip' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac1-12c'
CRS-2677: Stop of 'ora.orcl.db' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac1-12c'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.acfsdg.dbhome_1.acfs' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2-12c'
CRS-2677: Stop of 'ora.OCRVOTE.dg' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac2-12c'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'rac2-12c' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac2-12c' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2-12c'
CRS-2677: Stop of 'ora.asm' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2-12c'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.rac2-12c.vip' on 'rac2-12c'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2-12c' succeeded
CRS-2677: Stop of 'ora.rac2-12c.vip' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.rac2-12c.vip' on 'rac3-12c'
CRS-2676: Start of 'ora.rac2-12c.vip' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.acfsdg.dbhome_1.acfs' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.DBHOME_1.advm' on 'rac2-12c'
CRS-2677: Stop of 'ora.ACFSDG.DBHOME_1.advm' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2-12c'
CRS-2677: Stop of 'ora.ons' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2-12c'
CRS-2677: Stop of 'ora.net1.network' on 'rac2-12c' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2-12c' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.storage' on 'rac2-12c'
CRS-2677: Stop of 'ora.storage' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2-12c'
CRS-2677: Stop of 'ora.ctssd' on 'rac2-12c' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac2-12c' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2-12c'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2-12c'
CRS-2677: Stop of 'ora.cssd' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac2-12c'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2-12c'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2-12c'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2-12c'
CRS-2676: Start of 'ora.diskmon' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2-12c'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2-12c'
CRS-2676: Start of 'ora.ctssd' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2-12c'
CRS-2676: Start of 'ora.asm' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac2-12c'
CRS-2676: Start of 'ora.storage' on 'rac2-12c' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2-12c'
CRS-2676: Start of 'ora.crsd' on 'rac2-12c' succeeded
Oracle Grid Infrastructure restarted in node rac2-12c
CRS-2673: Attempting to stop 'ora.crsd' on 'rac3-12c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac3-12c'
CRS-2677: Stop of 'ora.cvu' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac3-12c'
CRS-2677: Stop of 'ora.orcl.db' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac3-12c'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac2-12c'
CRS-2676: Start of 'ora.scan2.vip' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.cvu' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac2-12c'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac2-12c' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rac3-12c'
CRS-2672: Attempting to start 'ora.OCRVOTE.dg' on 'rac2-12c'
CRS-2676: Start of 'ora.OCRVOTE.dg' on 'rac2-12c' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac3-12c'
CRS-2677: Stop of 'ora.FRA.dg' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac3-12c' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rac1-12c' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rac3-12c'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rac1-12c'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rac1-12c' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac3-12c'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.OCRVOTE.dg' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac3-12c'
CRS-2677: Stop of 'ora.asm' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac3-12c'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.rac3-12c.vip' on 'rac3-12c'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.rac3-12c.vip' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.rac3-12c.vip' on 'rac2-12c'
CRS-2676: Start of 'ora.rac3-12c.vip' on 'rac2-12c' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rac1-12c' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac3-12c'
CRS-2677: Stop of 'ora.ons' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac3-12c'
CRS-2677: Stop of 'ora.net1.network' on 'rac3-12c' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac3-12c' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.storage' on 'rac3-12c'
CRS-2677: Stop of 'ora.storage' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac3-12c'
CRS-2677: Stop of 'ora.ctssd' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac3-12c' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac3-12c'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac3-12c' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac3-12c'
CRS-2677: Stop of 'ora.cssd' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac3-12c'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3-12c'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac3-12c'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac3-12c'
CRS-2676: Start of 'ora.diskmon' on 'rac3-12c' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac3-12c' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac3-12c'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac3-12c'
CRS-2676: Start of 'ora.ctssd' on 'rac3-12c' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac3-12c'
CRS-2676: Start of 'ora.asm' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac3-12c'
CRS-2676: Start of 'ora.storage' on 'rac3-12c' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac3-12c'
CRS-2676: Start of 'ora.crsd' on 'rac3-12c' succeeded
Oracle Grid Infrastructure restarted in node rac3-12c
[root@rac1-12c ~]#

[oracle@rac1-12c grid]$ ps -ef | grep pmon
oracle    5032     1  0 21:49 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   10543 20985  0 21:52 pts/1    00:00:00 grep pmon
oracle   31231     1  0 21:47 ?        00:00:00 asm_pmon_+ASM1
oracle   31948     1  0 21:48 ?        00:00:00 ora_pmon_orcl1
oracle   31977     1  0 21:48 ?        00:00:00 apx_pmon_+APX1

Nota: +ASM1 esta corriendo, ASM proxy esta corriendo y el diskgroup ACFSDG esta montado

[oracle@rac1-12c grid]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-12c grid]$ asmcmd

ASMCMD> showclustermode
ASM cluster : Flex mode enabled

[root@rac1-12c grid]# srvctl add asm -proxy
PRCR-1086 : resource ora.proxy_advm is already registered

[root@rac1-12c grid]# srvctl start asm -proxy
PRCC-1014 : proxy_advm was already running
PRCR-1004 : Resource ora.proxy_advm is already running
PRCR-1079 : Failed to start resource ora.proxy_advm
CRS-5702: Resource 'ora.proxy_advm' is already running on 'rac1-12c'

[root@rac1-12c grid]# srvctl status asm -proxy
ADVM proxy is running on node rac3-12c,rac2-12c,rac1-12c

Nota: Una vez convertido a Flex ASM verificar el estado del componente asm –proxy

ASMCMD> volinfo --all
Diskgroup Name: ACFSDG
Volume Name: DBHOME_1
Volume Device: /dev/asm/dbhome_1-285
State: ENABLED
Size (MB): 20480
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage: ACFS
Mountpath: /u01/acfs
ASMCMD> exit

[oracle@rac1-12c grid]$ ssh rac1-12c ls -l /dev/asm/*
brwxrwx--- 1 root oinstall 251, 145921 Nov 10 21:48 /dev/asm/dbhome_1-285

[oracle@rac1-12c grid]$ ssh rac2-12c ls -l /dev/asm/*
brwxrwx--- 1 root oinstall 251, 145921 Nov 10 21:49 /dev/asm/dbhome_1-285

[oracle@rac1-12c grid]$ ssh rac3-12c ls -l /dev/asm/*
brwxrwx--- 1 root oinstall 251, 145921 Nov 10 21:51 /dev/asm/dbhome_1-285

[oracle@rac1-12c grid]$ /sbin/mkfs -t acfs /dev/asm/dbhome_1-285
mkfs.acfs: version                         = 12.1.0.2.0
mkfs.acfs: on-disk version                 = 39.0
mkfs.acfs: volume                          = /dev/asm/dbhome_1-285
mkfs.acfs: ACFS-01010: Volume already contains an ACFS file system.  
To reformat the volume, reissue mkfs.acfs with the -f option.
mkfs.acfs: ACFS-01004: /dev/asm/dbhome_1-285 was not formatted.
[oracle@rac1-12c grid]$

Podemos desmontar el file system ACFS usando el commando ‘srvctl’ y verificar el estado con el commando ‘crsctl’

[root@rac1-12c grid]# srvctl stop filesystem -d /dev/asm/dbhome_1-285
[root@rac1-12c grid]# srvctl status filesystem -d /dev/asm/dbhome_1-285
ACFS file system /u01/acfs is not mounted
[root@rac1-12c grid]#

[root@rac1-12c grid]# /u01/app/12.1.0.2/grid/bin/srvctl add filesystem -d /dev/asm/dbhome_1-285 -m /u01/acfs -u oracle -fstype ACFS -description '"/u01/acfs ACFS Filesystem"' -autostart ALWAYS

PRCA-1022 : ACFS file system resource already exists for disk group acfsdg and volume dbhome_1

Nota: Cuando reiniciamos el CRS automaticamente monta el file system compartido ACFS en los tres nodos (rac1-12c, rac2-12c & rac3-12c). Para verificar esta funcionalidad bajamos el CRS en los 3 nodos (rac1-12c, rac2-12c and rac3-12c) y reiniciamos el CRS en los tres nodos.

[root@rac1-12c grid]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1-12c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1-12c'
CRS-2673: Attempting to stop 'ora.ACFSDG.DBHOME_1.advm' on 'rac1-12c'
-------------------------output truncated--------------------------------------------------

[root@rac2-12c grid]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2-12c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2-12c'
CRS-2673: Attempting to stop 'ora.ACFSDG.DBHOME_1.advm' on 'rac2-12c'
-------------------------output truncated--------------------------------------------------

[root@rac3-12c grid]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac3-12c'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac3-12c'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac3-12c'
-------------------------output truncated--------------------------------------------------

Nota: Verificamos en los tres nodos que el filesystem “/u01/acfs” no esta montado

Reiniciamos el servicio de CRS en los tres nodos (rac1-12c, rac2-12c and rac3-12c)

[root@rac3-12c grid]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac3-12c grid]#

[root@rac2-12c grid]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2-12c grid]#

[root@rac1-12c grid]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1-12c grid]#

[root@rac1-12c grid]# srvctl start asm -proxy
PRCC-1014 : proxy_advm was already running
PRCR-1004 : Resource ora.proxy_advm is already running
PRCR-1079 : Failed to start resource ora.proxy_advm
CRS-5702: Resource 'ora.proxy_advm' is already running on 'rac1-12c'
[root@rac1-12c grid]#

Nota: Verificamos que el filesystem “/u01/acfs” esta montado en los tres nodos(rac1-12c, rac2-12c and rac3-12c)

Creando Redo Logs en un file system compartido ACFS tanto como en el diskgroup ACFSDG quedan visibles en los tres nodos (rac1-12c, rac2-12c & rac3-12c)

SQL> ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 7 ('+ACFSDG', '/u01/acfs/redo07b.log') size 50M;
Database altered.
SQL> ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 8 ('+ACFSDG', '/u01/acfs/redo08b.log') size 50M;
Database altered.
SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 9 ('+ACFSDG', '/u01/acfs/redo09b.log') size 50M;
Database altered.

SQL> select group#,member from v$logfile;

GROUP# MEMBER
------ ---------------------------------------------
2     +DATA/ORCL/ONLINELOG/group_2.278.927387563
2     +FRA/ORCL/ONLINELOG/group_2.268.927387563
1     +DATA/ORCL/ONLINELOG/group_1.282.927387561
1     +FRA/ORCL/ONLINELOG/group_1.269.927387563
5     +DATA/ORCL/ONLINELOG/group_5.274.927387799
5     +FRA/ORCL/ONLINELOG/group_5.267.927387799
6     +DATA/ORCL/ONLINELOG/group_6.271.927387799
6     +FRA/ORCL/ONLINELOG/group_6.266.927387799
3     +DATA/ORCL/ONLINELOG/group_3.281.927387801
3     +FRA/ORCL/ONLINELOG/group_3.265.927387801
4     +DATA/ORCL/ONLINELOG/group_4.272.927387801
4     +FRA/ORCL/ONLINELOG/group_4.264.927387801
7     +ACFSDG/ORCL/ONLINELOG/group_7.258.927588919
7     /u01/acfs/redo07b.log
8     +ACFSDG/ORCL/ONLINELOG/group_8.259.927588923
8     /u01/acfs/redo08b.log
9     +ACFSDG/ORCL/ONLINELOG/group_9.260.927588927
9     /u01/acfs/redo09b.log

18 rows selected.

Nota: Verificamos que los miembros del Grupo de Redo Log (7b, 8b y 9b) estan almacenados en “/u01/acfs”

[oracle@rac1-12c ~]$ ls –lrth /u01/acfs/
total 161M
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo07b.log
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo08b.log
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo09b.log

[oracle@rac2-12c ~]$ ls –lrth /u01/acfs/
total 161M
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo07b.log
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo08b.log
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo09b.log

[oracle@rac3-12c ~]$ ls –lrth /u01/acfs/
total 161M
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo07b.log
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo08b.log
-rw-r----- 1 oracle oinstall 51M Nov 10 23:35 redo09b.log

Creando tablespaces (tbsorcl y tbstest) en un file system compartido ACFS y en un diskgroup ACFSDG

Creando un backup de control file en ACFS queda visible en los tres nodos (rac1-12c, rac2-12c & rac3-12c)

Nota: Al igual que la instancias de ASM, las instancias de proxy ASM estan corriendo en los tres nodos (rac1-12c, rac2-12c & rac3-12c) y el file system ACFS esta montado.

[oracle@rac1-12c ~]$ ps -fea | egrep 'APX'; df -m /u01/acfs
oracle   10924     1  0 17:23 ?        00:00:00 apx_pmon_+APX1
oracle   10926     1  0 17:23 ?        00:00:01 apx_psp0_+APX1
oracle   11052     1  1 17:23 ?        00:02:03 apx_vktm_+APX1
oracle   11056     1  0 17:23 ?        00:00:01 apx_gen0_+APX1
oracle   11058     1  0 17:23 ?        00:00:00 apx_mman_+APX1
oracle   11062     1  0 17:23 ?        00:00:00 apx_diag_+APX1
oracle   11064     1  0 17:23 ?        00:00:02 apx_dia0_+APX1
oracle   11067     1  0 17:23 ?        00:00:00 apx_lreg_+APX1
oracle   11070     1  0 17:23 ?        00:00:00 apx_pxmn_+APX1
oracle   11072     1  0 17:23 ?        00:00:00 apx_rbal_+APX1
oracle   11080     1  0 17:23 ?        00:00:00 apx_vdbg_+APX1
oracle   11089     1  0 17:23 ?        00:00:00 apx_vubg_+APX1
oracle   11974    1  0 17:24 ?        00:00:00 oracle+APX1root (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   11983     1  0 17:24 ?        00:00:00 apx_vbg0_+APX1
oracle   11991     1  0 17:24 ?        00:00:07 apx_acfs_+APX1
oracle   12030     1  0 17:24 ?       00:00:00 apx_vbg1_+APX1
oracle   12034     1  0 17:24 ?        00:00:00 apx_vbg2_+APX1
oracle   12040     1  0 17:24 ?        00:00:00 apx_vbg3_+APX1
oracle   12042     1  0 17:24 ?        00:00:00 apx_vmb0_+APX1
oracle   18362 16733  0 20:14 pts/1    00:00:00 egrep APX
Filesystem            1M-blocks  Used Available Use% Mounted on
/dev/asm/dbhome_1-285     20480   462     20019   3% /u01/acfs

[oracle@rac1-12c ~]$ ssh rac2-12c ps -fea | egrep 'APX'; df -m /u01/acfs
oracle   11099     1  0 17:23 ?        00:00:00 apx_pmon_+APX2
oracle   11101     1  0 17:23 ?        00:00:01 apx_psp0_+APX2
oracle   11236     1  0 17:23 ?        00:00:01 apx_vktm_+APX2
oracle   11244     1  0 17:23 ?        00:00:01 apx_gen0_+APX2
oracle  11246     1  0 17:23 ?        00:00:00 apx_mman_+APX2
oracle   11250     1  0 17:23 ?        00:00:00 apx_diag_+APX2
oracle   11252     1  0 17:23 ?        00:00:02 apx_dia0_+APX2
oracle   11254     1  0 17:23 ?        00:00:00 apx_lreg_+APX2
oracle   11256     1  0 17:23 ?        00:00:00 apx_pxmn_+APX2
oracle   11260     1  0 17:23 ?        00:00:00 apx_rbal_+APX2
oracle   11265     1  0 17:23 ?        00:00:00 apx_vdbg_+APX2
oracle   11280     1  0 17:23 ?        00:00:00 apx_vubg_+APX2
oracle   12158    1  0 17:24 ?        00:00:00 oracle+APX2root (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle   12167     1  0 17:24 ?        00:00:00 apx_vbg0_+APX2
oracle   12175     1  0 17:24 ?        00:00:07 apx_acfs_+APX2
oracle   12211     1  0 17:24 ?        00:00:00 apx_vmb0_+APX2
oracle   12217     1  0 17:24 ?        00:00:00 apx_vbg1_+APX2
oracle   12219     1  0 17:24 ?        00:00:00 apx_vbg2_+APX2
oracle   12224     1  0 17:24 ?        00:00:00 apx_vbg3_+APX2
Filesystem            1M-blocks  Used Available Use% Mounted on
/dev/asm/dbhome_1-285     20480   462     20019   3% /u01/acfs
[oracle@rac1-12c ~]$

[oracle@rac1-12c ~]$ ssh rac3-12c ps -fea | egrep 'APX'; df -m /u01/acfs
oracle    6287     1  0 17:22 ?        00:00:00 apx_pmon_+APX3
oracle    6289     1  0 17:22 ?        00:00:01 apx_psp0_+APX3
oracle    6293     1  0 17:22 ?        00:00:01 apx_vktm_+APX3
oracle    6297     1  0 17:22 ?        00:00:01 apx_gen0_+APX3
oracle    6299     1  0 17:22 ?        00:00:00 apx_mman_+APX3
oracle    6303     1  0 17:22 ?        00:00:00 apx_diag_+APX3
oracle    6305     1  0 17:22 ?        00:00:02 apx_dia0_+APX3
oracle    6307     1  0 17:22 ?        00:00:00 apx_lreg_+APX3
oracle    6309     1  0 17:22 ?        00:00:00 apx_pxmn_+APX3
oracle    6311     1  0 17:22 ?        00:00:00 apx_rbal_+APX3
oracle    6313     1  0 17:22 ?        00:00:00 apx_vdbg_+APX3
oracle    6315     1  0 17:22 ?        00:00:00 apx_vubg_+APX3
oracle    6727     1  0 17:22 ?        00:00:00 oracle+APX3root (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    6742     1  0 17:22 ?        00:00:00 apx_vbg0_+APX3
oracle    6754     1  0 17:22 ?        00:00:07 apx_acfs_+APX3
oracle    7634     1  0 17:23 ?        00:00:00 apx_vbg1_+APX3
oracle    7641     1  0 17:23 ?       00:00:00 apx_vbg2_+APX3
oracle    7648     1  0 17:23 ?        00:00:00 apx_vbg3_+APX3
oracle    7652     1  0 17:23 ?        00:00:00 apx_vmb0_+APX3
Filesystem            1M-blocks  Used Available Use% Mounted on
/dev/asm/dbhome_1-285     20480   462     20019   3% /u01/acfs
[oracle@rac1-12c ~]$

Resumen: Oracle ACFS (ASM Cluster File System) puede ser usado para almacenar archivos de la base de datos (Data files, Redo Log files, Control Files and Recovery Manager (RMAN) backups etc.), ejecutables y otros archivos, como un file system multi-plataforma. Hemos visto también como almacenar Redo Logs y Tablespaces tanto en el file system ACFS como en el diskgroup ACFSDG.

Y V Ravi Kumar es un Oracle ACE y Oracle Certified Master (OCM) con 18 años de experiencia en Bancos, Financieras y Aseguradoras (BFSI) trabajando en varios roles como Senior Database Architect y DBA de Producción. Es también OCP en Oracle 8i, 9i, 10g, 11g y 12c y Certificado en Oracle GoldenGate, RAC, Performance Tuning y Oracle Exadata. Frecuentemente motiva a varios DBAs y apoya a la Comunidad Oracle publicando sus tips, ideas, sugerencias y soluciones en su blog. Ravi ha escrito más de 40 artículos para OTN de Oracle Exadata, Oracle RAC and Oracle GoldenGate tanto para OTN en Español, OTN en Portugues y OTN en Inglés, 19 artículos para TOAD World, 2 Artículos para UKOUG, 3 Artículos para OTech Magazine y 2 Artículos para Redgate. Es miembro de NYOUG y AIOUG y es Oracle Speaker frecuente en @NYOUG, @OTN, AIOUG, Sangam y IOUG. Ha diseñado e implementado la Base de Datos de Core Banking System (CBS) para Central Banks de dos paises - India y Mahe, Seychelles. Ravi es Co-Fundador de OraWorld (www.oraworld.com). Mas detalles en su perfil en LaserSoft

Rita Nuñez es un Oracle ACE Associate y DBA con más de 15 años de experiencia. Ingeniera en Sistemas. Instructora de Cursos Oficiales de Oracle University. CEO de Tecnix Solutions lidera los servicios de Consultoría y Capacitación en Productos Oracle. Colabora con la Comunidad Oracle escribiendo artículos para OTN y en su blog, además dando presentaciones en eventos de OTN desde 2013 tanto en Argentina como en Brasil, Uruguay y Chile, y también este año en el Oracle Open World 2016. Es Coordinadora del SIG Database & RAC de ArOUG.

Archana Durai tiene más de 7 años de experiencia como DBA Oracle teniendo también conocimientos y experiencia en otros motores RDBMS. Ella tiene experiencia en areas como Life Sciences (Medicina, Biología, Biomedicina, Bioquimica) y Medios y Comunicaciones. Es Oracle Certified Professional en 9i, 10g, 11g y12c y certificada en Oracle GoldenGate y Real Application Clusters (RAC), IBM DB2 Certified Administrator (v10.1). Además es RedHat Certified Engineer (RHCE) y Certificada en ITIL V3 Foundation.

Este artículo ha sido revisado por el equipo de productos Oracle y se encuentra en cumplimiento de las normas y prácticas para el uso de los productos Oracle.