database not registered to scan_listener

created a new database COGSTD, but it's not registered on scan_listener, when connect, the error is:

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

In the alert log, when the database started, I usually see the remote_listener and local_listener are set dynamically before database is mount, but on this problematic database, I only see local_listener are set, but not remote_listener

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=xx.xx.xx.xx0)(PORT=1521))' SCOPE=MEMORY SID='COGSTD3';

Troubleshooting:

1. The service is registered with local listener:
node1>lsnrctl status listener|grep COGSTD
Service "COGSTD" has 1 instance(s).
  Instance "COGSTD1", status READY, has 1 handler(s) for this service...

Similar output on node2 and node 3 with the local instnance name

2. The service is not registered with scan_listener:

-bash-4.1>srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node node3

SCAN Listener LISTENER_SCAN2 is enabled

SCAN listener LISTENER_SCAN2 is running on node node2

SCAN Listener LISTENER_SCAN3 is enabled

SCAN listener LISTENER_SCAN3 is running on node node1

node3>lsnrctl status listener_scan1|grep COGSTD

node2>lsnrctl status listener_scan2|grep COGSTD
node1>lsnrctl status listener_scan3|grep COGSTD

nothing!!!

google and metalink search suggest that it's an oracle bug.

Bug 12887685  - INSTANCE CAN'T BE REGISTERED IN LISTENER_SCAN1
Bug 12665692  - PMON DOES NOT REGISTER TO ONE OF THE SCAN LISTENER.
Bug 12392343  - INSTANCE/SERVICE DOES NOT REGISTER TO ALL SCAN LISTENERS
Bug 12659561  - INSTANCE DOES NOT REGISTER SERVICES WHEN SCAN FAILOVERED
Bug 10194101  - SOME INSTANCES INTERMITTENTLY DO NOT CROSS-REGISTER WITH THE 
SCAN LISTENER
Bug 7420414   - SCAN LISTENER LOSE SERVICE REGISTRATION INFORMATION AFTER 
SERVICE RELOCATION

The following note is the workaround:
SCAN Registration Issue: Services Fail to Register to SCAN Listeners (Doc ID 1448717.1)

Now implement the workaround:
Try to use the scan name, but it does not accept it at first time
SQL>  alter system set remote_listener='scan.devxx.racxx.company.corp:1521';
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name
'scan.devxx.racxx.company.corp:1521'

then try

1. add  SCAN IP to tnsnames.ora for RDBMS $ORACLE_HOME on all three RAC nodes.
REMOTE_LISTENERS_SCAN=
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST =  xx.xx.xx.x3)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST =  xx.xx.xx.x4)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST =  xx.xx.xx.x5)(PORT = 1521))

    )

2. modify the sqlnet.ora to allow ezconnect

3. modify the remote_listener parameter

SQL> alter system set REMOTE_LISTENER='REMOTE_LISTENERS_SCAN' scope=both sid='*';

System altered.

this time, it accept it.

SQL> alter system register;

System altered.

node1>lsnrctl status listener_scan3|grep COGSTD

Service "COGSTD" has 3 instance(s).

  Instance "COGSTD1", status READY, has 1 handler(s) for this service...

  Instance "COGSTD2", status READY, has 1 handler(s) for this service...

  Instance "COGSTD3", status READY, has 1 handler(s) for this service...

it works on all three nodes. I can connect to database from remote.

4. Try to use the scan name that failed before, it's ok this time. WTF!

SQL> alter system set REMOTE_LISTENER='scan.devxx.racxx.company.corp:1521' scope=both sid='*';

System altered.

SQL> alter system register;

System altered.

To avoid future problem, I relocate the scan_listener

srvctl relocate scan -scannumber 1 -node node1

srvctl relocate scan -scannumber 3 -node node3

Remove the scan_listener parameter from spfile and bounce the database, check the alert log, noticed that the local_listener and remote_listener are both setup dynamically before mount database:

Tue Jul 11 14:38:13 2017

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=xx.xx.xx.xx0)(PORT=1521))' SCOPE=MEMORY SID='COGSTD3';

Tue Jul 11 14:38:13 2017

ALTER SYSTEM SET remote_listener=' scan.xxx.xxx.xxx.xxx:1521' SCOPE=MEMORY SID='COGSTD3';

ALTER DATABASE MOUNT /* db agent *//* {3:56066:29347} */

So what's the problem? does it has something to do with listener.ora and  endpoints_listener.ora.

or it's because of the tnsnames.ora does not have ezconnect, to did not trace the LREG|PMON process, but let me revoke all the changes but only leave ezconnect as the only change.

relocate scan_listener as it was

remove remote_listener parameter

bounce database

Everything work as normal, so I think the problem is the missing ezconnect in sqlnet.ora on some of the nodes. if sqlnet.ora is missing, by default, it using

NAMES.DIRECTORY_PATH=(tnsnames, ldap, ezconnect)

But on some of the RAC nodes, NAMES.DIRECTORY_PATH only configured to use tnsnames, not ezconnect.

So this problem may have nothing to do with bug, it's just ezconnect is missing.