DBA

My server is configured with HugePage, it's swapping, this is the shell script to find out how much shared memory used by each database, how much memory is using normal page or HugePage, this script only list memory bigger than 100,000kB. for line in `ps -ef|grep ckpt|grep -v "+ASM"|grep -v "grep"|awk '{print $2"_"$8}'`; do #echo $line PID=$(echo $line|cut  -d "_" -f 1) DBNAME=$(echo $line|cut  -d "_" -f 4) SMAPS=$"/proc/"$PID"/smaps" echo "#########"$DBNAME"########"$SMAPS"########" awk '/^Size:/ {if ($2>100000) i=1};i && i++ <= 12' $SMAPS done Sample output: This example shows that 268MB on 4kB (normal page), 10GB on 2048kB page (HugePage) #########FFDE2########/proc/64294/smaps######## Size:             268332 kB Rss:               12300 kB Pss:                 135 kB Shared_Clean:      12300 kB Shared_Dirty:          0 kB Private_

Our RAC crash when we have storage issue, after storage is fixed, starting RAC using "crsctl start crs" failed on ASM start. Did not think it could be caused by GNS but looks like it was. GNS was offline because the VIP somehow ran away on one of the RAC node, I can ping and ssh to the VIP but cannot run "srvctl stop/start GNS". This is the evidence that ASM rely on GNS to be online, if GNS is not online, then the remote_listener in +ASM's parameter file is invalid and asm is killed. In alert_ASM1.log: Using parameter settings in server-side spfile +OCR_VOTE/wwwracprdcrs00/asmparameterfile/registry.253.830014321 ... Sat Aug 03 00:49:10 2019 USER (ospid: 16060): terminating the instance due to error 119 The ASM log does not say why error 119 occur, but checking ohasd_oraagent_grid.trc, it's the remote_listener that causing problem: In ohasd_oraagent_grid.trc: 2019-08-03 00:49:10.295345 :CLSDYNAM:2912564992: [ ora.asm]{0:5:3} [start] ORA-00119: i