Missing properties - password (00151) in emagent.log

So, I could not start the agent the other day. The error in $ORACLE_HOME/sysman/log/emagent.log was:

Missing properties - password (00151)

Background

At one point, this machine had an agent on it. Everything was working fine. We decided to use this machine as a test bed for some other databases so we wiped it out and reinstalled Oracle 11gR2. This is a Sun SPARC Solaris 10 box by the way. Our mistake was we did not remove the agent from Grid before we wiped out the box.

Once our testing was done, we decided to re-install the agent. Problem was, the agent would not start. It was failing with the error message above in the emagent.log file.

Here is how we got the problem fixed:

1. Go to the agent $ORACLE_HOME/bin
2. Run:

./agentca -d

The will get your agent started by the way.

3. When this is complete, log into Grid.
4. Go to Setup > Agents

5. Click on the agent link.
6. Run Agent resyncronization
7. Click on the UNBLOCK button. Click on this button even if the agent does not say it is blocked.

Wait a few minutes and the agent should clear. You will have to re-configure your targets.

RMAN-06026: some targets not found - RMAN-06024: no backup or copy of the control file found

Interesting situation I ran into and I have not quite figured out the why yet...This has to do with duplicating a database with RMAN on version 11gR2 (11.2.0.2).

Background: Every six months at one of my clients, we do a tape restore recovery test. Normally, we back up the database to the FRA, then remove the backups from disk every couple of weeks. Nightly, the backupsets are copied to tape. To perform the test, I choose a backup period that is beyond when we removed the backups from disk. Therefore, I need to have the sys admin restore from tape to the FRA. Upon doing so, I "CATALOG START WITH" to put the backup pieces back into the catalog.

The way we have the backups configured is we do a weekly level 0 and daily level 1's. To do the recovery test, I tell the SA to restore the level 0 directory (date) and the level 1 directory (date) as well as the corresponding control files autobackup directories.

Then I choose a time that falls right in between the two of them.

For example, this time, my level 0 completed at 11:23 PM on 11/20, and my level 1 completed at 6:45 AM on 11/21. So my recovery time was set to 3:45 AM on 11/21. I use this setup then to duplicate the database to a new host with a new database name. Of course, since this is point-in-time, I have to copy the backups to the new machine.

On the new machine, I configure the TNS, a dedicated listener, and an init.ora...

I start up the AUXILIARY in nomount mode...

From the TARGET machine I:

rman target / auxiliary sys@AUXDUP nocatalog

Here is my script:

run {
set until time = "to_date('2011-11-21:03:45:00','YYYY-MM-DD:HH24:MI:SS')";
duplicate target database to AUXDUP
DB_FILE_NAME_CONVERT=('/ora01/oradata/PROD/','/ora01/oradata/AUXDUP/')
LOGFILE
group 1 ('/ora01/oradata/AUXDUP/redo01a.log') size 256M reuse,
group 2 ('/ora01/oradata/AUXDUP/redo02a.log') size 256M reuse,
group 3 ('/ora01/oradata/AUXDUP/redo03a.log') size 256M reuse
nofilenamecheck;
}

Today, when I ran this, I received the following error:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 12/20/2011 11:25:47
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
RMAN-06026: some targets not found - aborting restore
RMAN-06024: no backup or copy of the control file found to restore

Now, there are many reasons that you will find on Metalink and the internet for this error. None of those things applied to me. Most of them centered around duplicating for standby.

What makes it particularly confusing is that many of the resources you find will say this error is not really about the controlfile but about the times you choose and backup availability. I knew my time was good and backups were all in place. My problem WAS with the controlfile.

Essentially, this error is telling you it cannot find a copy of the controlfile to restore. I know for a fact I restored and cataloged my controlfile backups as well as copied them to the new host.

However, when I tried to list my controlfile backups, I got:

RMAN> list backup of controlfile completed before '22-NOV-2011';

specification does not match any backup in the repository

What? I know I cataloged them.

Let's catalog them again, just to be sure:

RMAN> catalog start with '/ora01/flash_recovery_area/PROD';

searching for all files that match the pattern /ora01/flash_recovery_area/PROD

List of Files Unknown to the Database
=====================================
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_20/o1_mf_s_767748081_7dmng414_.bkp
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_20/o1_mf_s_767748205_7dmnkzny_.bkp
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774118_7dnfvrhl_.bkp

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_20/o1_mf_s_767748081_7dmng414_.bkp
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_20/o1_mf_s_767748205_7dmnkzny_.bkp
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp
File Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774118_7dnfvrhl_.bkp

So, you can see, I cataloged them again with no specific errors.

Try listing them:

RMAN> list backup of controlfile completed before '22-NOV-2011';

specification does not match any backup in the repository

So, for some reason, RMAN is not cataloging the files.

I checked my controlfile_record_keep_time (not sure that this would make a difference) and it was set to 60 days. Well within the recovery window.

In a short amount of searching, I could find no evidence that this was a reported bug or experienced by others before.

How about if I try listing the specific pieces?

RMAN> list backuppiece '/ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp';

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of list command at 12/20/2011 11:26:33
RMAN-20260: backup piece not found in the repository
RMAN-06092: error while looking up backup piece

Interesting...clearly it is not recognized by my backup metadata.

How about if I try cataloging the piece itself?

RMAN> catalog backuppiece '/ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp';

cataloged backup piece
backup piece handle=/ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp RECID=42367 STAMP=770383607

RMAN> list backuppiece '/ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp';


List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
42367 17202 1 1 AVAILABLE DISK /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp

Now we are getting somewhere...

Let me catalog the rest of the pieces...

Relist (backup this time, not backuppiece):

RMAN> list backup of controlfile completed before '22-NOV-2011';

using target database control file instead of recovery catalog

List of Backup Sets
===================


BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
17202 Full 32.23M DISK 00:00:00 21-NOV-2011 06:33:39
BP Key: 42367 Status: AVAILABLE Compressed: NO Tag: TAG20111121T063339
Piece Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774019_7dnfroro_.bkp
Control File Included: Ckp SCN: 30756382041 Ckp time: 21-NOV-2011 06:33:39

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
17203 Full 32.23M DISK 00:00:00 21-NOV-2011 06:35:18
BP Key: 42368 Status: AVAILABLE Compressed: NO Tag: TAG20111121T063518
Piece Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_21/o1_mf_s_767774118_7dnfvrhl_.bkp
Control File Included: Ckp SCN: 30756382106 Ckp time: 21-NOV-2011 06:35:18

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
17204 Full 32.23M DISK 00:00:00 20-NOV-2011 23:21:21
BP Key: 42369 Status: AVAILABLE Compressed: NO Tag: TAG20111120T232121
Piece Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_20/o1_mf_s_767748081_7dmng414_.bkp
Control File Included: Ckp SCN: 30754959584 Ckp time: 20-NOV-2011 23:21:21

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
17205 Full 32.23M DISK 00:00:00 20-NOV-2011 23:23:25
BP Key: 42370 Status: AVAILABLE Compressed: NO Tag: TAG20111120T232325
Piece Name: /ora01/flash_recovery_area/PROD/autobackup/2011_11_20/o1_mf_s_767748205_7dmnkzny_.bkp
Control File Included: Ckp SCN: 30754964327 Ckp time: 20-NOV-2011 23:23:25

Voila!

So, in conclusion, I am not sure if this is a bug or what, but, I most certainly did not have to go through this before we upgraded our 10.2.0.4 database to 11.2.0.2 a couple months ago.
Apparently, I have to specifically catalog each controlfile autobackup piece for the duplicate to work.

BTW, this did not apply to the Level 0's and Level 1's. They cataloged just fine.

Upon doing this, my duplicate ran perfectly.

Oracle OracleNet Tracing in 11g+

Don't forget, in 11g+, you just need to set the TRACE_LEVEL_* to start tracing.

No longer is there a need to set:

TRACE_DIRECTORY_*
or
TRACE_FILE_*

as all files go to the DIAGNOSTIC_DEST. Yes, even the client has a diagnostic dest. Looking under the client ORACLE_BASE.

Needed to install Oracle Application Express today. It was not installed with the database creation so I had to do it manually.

Step 1: created a tablespace for the APEX data:

conn / as sysdba
Connected.
create tablespace apex_data datafile '+DATA1';

exit

Step 2: ran APEX install script from $ORACLE_HOME/apex:

cd $ORACLE_HOME/apex
sqlplus / as sysdba

@apexins password apex_data apex_data temp /i/

Install went pretty well but I found that in the end, DBA_REGISTRY showed Application Express as INVALID.

Checked DBA_ERRORS and found the following:

OWNER NAME TYPE SEQUENCE LINE POSITION TEXT ATTRIBUTE
----------- --------------- --------------- ---------- ---------- ---------- -------------------------------------------------- ----------
MESSAGE_NUMBER
--------------
APEX_030200 WWV_FLOW_HELP PACKAGE BODY 16 884 9 PL/SQL: Statement ignored ERROR
0

APEX_030200 WWV_FLOW_HELP PACKAGE BODY 15 884 9 PLS-00201: identifier 'CTX_DDL' must be declared ERROR

I searched that error on Metalink and found this document: 1312938.1

Basically, to get APEX valid, you have to:

SQL> conn ctxsys
Enter password:
Connected.
SQL> grant execute on ctx_ddl to APEX_030200;

Grant succeeded.

SQL> grant execute on ctx_doc to APEX_030200;

Grant succeeded.

SQL> connect / as sysdba

SQL> alter package APEX_030200.WWV_FLOW_HELP compile;

Package altered.

SQL> alter package APEX_030200.WWV_FLOW_HELP compile body;

Package body altered.

SQL> conn / as sysdba
Connected.
SQL> @?/rdbms/admin/utlrp

That seemed to fix my issue just fine!