ADRCI (AUTOMATIC DIAGNOSTIC REPOSITORY COMMAND INTERPRETER) :-
ADRCI is a command-line tool that is part of the fault diagnosability infrastructure introduced in Oracle Database Release 11g. ADRCI enables you to:
- View diagnostic data within the Automatic Diagnostic Repository (ADR).
- View Health Monitor reports.
- Package incident and problem information into a zip file for transmission to Oracle Support.
- The ADR is a file-based repository for database diagnostic data such as traces, dumps, the alert log, health monitor reports, and more.
- It has a unified directory structure across multiple instances and multiple products.
- Beginning with Release 11g, the database, Automatic Storage Management (ASM), and other Oracle products or components store all diagnostic data in the ADR.
- Each instance of each product stores diagnostic data underneath its own ADR home directory.
- For example,In an Oracle Real Application Clusters environment with shared storage and ASM, each database instance and each ASM instance has a home directory within the ADR. The ADR’s unified directory structure enables customers and Oracle Support to correlate and analyze diagnostic data across multiple instances and multiple products.
- Most of the diagnostic logs stored in the admin directory in Oracle 10g and earlier has now been moved to the Automatic Diagnostic Repository,or ADR directory, in Oracle 11g. The ADR directory is created in $ORACLE_BASE/diag and contains a subdirectory for each different Oracle installation type on the system and may also contain some empty directories for products which are not installed. Within the database, the ADR location is defined by the diagnostic_dest initialization parameter, and the background_dump_dest, user_dump_dest and similar parameters have been deprecated.
Definitions :
- Automatic Diagnostic Repository (ADR)
- Problem
- Incident
- Problem Key
- Incident Package
- Finalizing
- ADR Home
- ADR Base
- Homepath
ADR directory structure :
Subdirectories Under ADR Base :
ADR repository location using V$DIAG_INFO view :
Diagnostic_dest location :-
SQL> show parameter diagnostic_dest; NAME TYPE VALUE --------------- ------- ------------------ diagnostic_dest string /oradb/app/oracle
ADRCI command interpreter :
Viewing the alert log:-
One is present in the conventional text format, per OFA in $ORACLE_BASE/diag/rdbms/name of the db/name of the instance/trace This location is determined by the new initialization parameter DIAGNOSTIC_DEST, while BACKGROUND_DUMP_DEST is deprecated in 11g.
The other one is in XML format placed in $ORACLE_BASE/diag/rdbms/name of the db/name of the instance/alert.
[oracle@orcl:~ orcldemo] adrci ADRCI: Release 12.2.0.1.0 - Production on Wed Oct 24 21:15:43 2018 Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved. ADR base = "/oradb/app/oracle" adrci> show homes; ADR Homes: diag/clients/user_oracle/host_680780456_107 diag/rdbms/orcldemo/orcldemo diag/rdbms/orcl/orcl diag/tnslsnr/orcl/listener adrci> set home diag/rdbms/orcldemo/orcldemo adrci> show alert -tail 20 2018-10-24 03:36:06.135000 +05:30 Warning: VKTM detected a forward time drift. Time drifts can result in unexpected behavior such as time-outs. Please see the VKTM trace file for more details: /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc 2018-10-24 05:33:24.412000 +05:30 Warning: VKTM detected a forward time drift. Please see the VKTM trace file for more details: /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc 2018-10-24 12:41:59.003000 +05:30 Warning: VKTM detected a forward time drift. Time drifts can result in unexpected behavior such as time-outs. Please see the VKTM trace file for more details: /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc 2018-10-24 15:31:50.376000 +05:30 Resize operation completed for file# 3, old size 1085440K, new size 1095680K 2018-10-24 16:59:56.481000 +05:30 Warning: VKTM detected a forward time drift. Time drifts can result in unexpected behavior such as time-outs. Please see the VKTM trace file for more details: /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc 2018-10-24 17:59:55.271000 +05:30 Warning: VKTM detected a forward time drift. Time drifts can result in unexpected behavior such as time-outs. Please see the VKTM trace file for more details: /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc adrci> show alert -p "message_text like '%incident%'" ADR Home = /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo: **************************************************************** Output the results to file: /tmp/alert_47719_1400_orcldemo_3.ado
Eg:-
show alert
show alert -p "message_text like '%incident%'"
show alert -tail 20
Viewing Trace files :
To directly viewed the trace file using the SHOW_TRACE command.
adrci> show trace /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc Output the results to file: /tmp/utsout_47719_14007_5.ado /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc ---------------------------------------------------------- LEVEL PAYLOAD ----- ------------------------------------------------------------------------------------------------------------------------------------------------ Trace file /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_vktm_4748.trc Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production Build label: RDBMS_12.2.0.1.0_LINUX.X64_170125 ORACLE_HOME: /oradb/app/oracle/product/12.2.0.1/db_1 System name: Linux Node name: orcl.localdomain.com Release: 2.6.32-696.el6.x86_64 Version: #1 SMP Tue Mar 21 19:29:05 UTC 2017 Machine: x86_64 Instance name: orcldemo Redo thread mounted by this instance: 0 <none> Oracle process number: 5 Unix process pid: 4748, image: oracle@orcl.localdomain.com (VKTM) *** 2018-10-15T23:33:55.572017+05:30 *** SESSION ID:(5.6196) 2018-10-15T23:33:55.572050+05:30 *** CLIENT ID:() 2018-10-15T23:33:55.572056+05:30 *** SERVICE NAME:() 2018-10-15T23:33:55.572061+05:30 *** MODULE NAME:() 2018-10-15T23:33:55.572068+05:30 *** ACTION NAME:() 2018-10-15T23:33:55.572073+05:30 *** CLIENT DRIVER:() 2018-10-15T23:33:55.572077+05:30 kstmmainvktm: succeeded in setting elevated priority highres_disabled VKTM running at (100ms) precision kstmrmtickcntkeeper: param _dbrm_quantum will not be effective [Start] HighResTick = 743299503 kstmrmtickcnt = 0, ksudbrmseccnt[0] = 1539626635 kstmchkdrift (kstmrmtickcntkeeper:highres): Time jumped forward by (1796157)usec at (15368631714) whereas (1000000) is allowed
Purging tracefiles :
Database will creates lots of tracefiles that need to be purged from time to time. In fact, this is done automatically, but you may want to change the default purge policy:
The ordinary tracefiles will stay for 30 days (720 hours), while files like incident files stay one year (8760 hours) by default. We can change that policy,
Purging tracefiles manually :
adrci> purge -age 1440 (for older than 1 days)
Problem :
- A problem is a critical error in the database. Critical errors include internal errors such as ORA-00600 and other severe errors such as ORA-07445 (operating system exception) or ORA-04031 (out of memory in the shared pool).
- Problems are tracked in the ADR. Each problem has a problem key and a unique problem ID.
To view the problem occurred in database from ADRCI command line,type SHOW PROBLEM
INCIDENT :
- An incident is a single occurrence of a problem. When a problem occurs multiple times, an incident is created for each occurrence. Incidents are tracked in the ADR. Each incident is identified by a numeric incident ID, which is unique within the ADR. When an incident occurs, the database makes an entry in the alert log, sends an incident alert to Oracle Enterprise Manager, gathers diagnostic data about the incident in the form of dump files (incident dumps), tags the incident dumps with the incident ID, and stores the incident dumps in an ADR subdirectory created for that incident.
- Diagnosis and resolution of a critical error usually starts with an incident alert. You can obtain a list of all incidents in the ADR with an ADRCI command. Each incident is mapped to a single problem only.
- Incidents are flood-controlled so that a single problem does not generate too many incidents and incident dumps.
adrci> show incident -mode detail -p "incident_id=12209";
ADR Home = /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo:
**********************************************************
**********************************************************
INCIDENT INFO RECORD 1
**********************************************************
INCIDENT_ID 12209
STATUS ready
CREATE_TIME 2018-09-19 05:07:14.674000 +05:30
PROBLEM_ID 1
CLOSE_TIME <NULL>
FLOOD_CONTROLLED none
ERROR_FACILITY ORA
ERROR_NUMBER 600
ERROR_ARG1 ktcrcm: caller passed invalid xcb
ERROR_ARG2 4
ERROR_ARG3 0x06CF235C0
ERROR_ARG4 0x000000000
ERROR_ARG5 1
ERROR_ARG6 4232
ERROR_ARG7 <NULL>
ERROR_ARG8 <NULL>
ERROR_ARG9 <NULL>
ERROR_ARG10 <NULL>
ERROR_ARG11 <NULL>
ERROR_ARG12 <NULL>
SIGNALLING_COMPONENT <NULL>
SIGNALLING_SUBCOMPONENT <NULL>
SUSPECT_COMPONENT <NULL>
SUSPECT_SUBCOMPONENT <NULL>
ECID <NULL>
IMPACTS 0
CON_UID 0
PROBLEM_KEY ORA 600 [ktcrcm: caller passed invalid xcb]
FIRST_INCIDENT 12209
FIRSTINC_TIME 2018-09-19 05:07:14.674000 +05:30
LAST_INCIDENT 14407
LASTINC_TIME 2018-09-19 05:56:05.333000 +05:30
IMPACT1 0
IMPACT2 0
IMPACT3 0
IMPACT4 0
KEY_NAME Service
KEY_VALUE orcldemo.localdomain.com
KEY_NAME Module
KEY_VALUE oracle@test.localdomain.com (TNS V1-V3)
KEY_NAME ProcId
KEY_VALUE 26.141
KEY_NAME Client ProcId
KEY_VALUE oracle@orcl.localdomain.com (TNS V1-V3).13694_139900975931168
KEY_NAME PQ
KEY_VALUE (0, 1537313832)
KEY_NAME SID
KEY_VALUE 65.20850
OWNER_ID 1
INCIDENT_FILE /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_ora_13694.trc
OWNER_ID 1
INCIDENT_FILE /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/incident/incdir_12209/orcldemo_ora_13694_i 12209.trc
1 row fetched
Show incident tracefile
adrci> show tracefile -t /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/incident/incdir_12209/orcldemo_ora_13694_i12209.trc DIA-48415: Syntax error found in string [show tracefile -t /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/incident/incdir_12209/orcldemo_ora_13694_i12209.trc] at column [116] adrci> show trace /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/incident/incdir_12209/orcldemo_ora_13694_i12209.trc Output the results to file: /tmp/utsout_49795_13979_4.ado /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/incident/incdir_12209/orcldemo_ora_13694_i12209.trc ---------------------------------------------------------- LEVEL PAYLOAD ----- ------------------------------------------------------------------------------------------------------------------------------------------------ Dump file /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/incident/incdir_12209/orcldemo_ora_13694_i12209.trc Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production Build label: RDBMS_12.2.0.1.0_LINUX.X64_170125 ORACLE_HOME: /oradb/app/oracle/product/12.2.0.1/db_1 System name: Linux Node name: orcl.localdomain.com Release: 2.6.32-696.el6.x86_64 Version: #1 SMP Tue Mar 21 19:29:05 UTC 2017 Machine: x86_64 Instance name: orcldemo Redo thread mounted by this instance: 1 Oracle process number: 26 Unix process pid: 13694, image: oracle@orcl.localdomain.com *** 2018-09-19T05:07:14.933565+05:30 *** SESSION ID:(65.20850) 2018-09-19T05:07:14.933582+05:30 *** CLIENT ID:() 2018-09-19T05:07:14.933587+05:30 *** SERVICE NAME:(orcldemo.localdomain.com) 2018-09-19T05:07:14.933593+05:30 *** MODULE NAME:(oracle@test.localdomain.com (TNS V1-V3)) 2018-09-19T05:07:14.933599+05:30 *** ACTION NAME:() 2018-09-19T05:07:14.933604+05:30 *** CLIENT DRIVER:() 2018-09-19T05:07:14.933608+05:30 [TOC00000] Jump to table of contents Dump continued from file: /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo/trace/orcldemo_ora_13694.trc [TOC00001] 1> ***** Error Stack ***** ORA-00600: internal error code, arguments: [ktcrcm: caller passed invalid xcb], [4], [0x06CF235C0], [0x000000000], [1], [4232], [], [], [], [], [], []
Incident Package :
- An incident package (package) is a collection of data about incidents for one or more problems. Before sending incident data to Oracle Support it must be collected into a package using the Incident Packaging Service (IPS). After a package is created, you can add external files to the package, remove selected files from the package, or scrub (edit) selected files in the package to remove sensitive data.
- A package is a logical construct only, until you create a physical file from the package contents. That is, an incident package starts out as a collection of metadata in the Automatic Diagnostic Repository (ADR). As you add and remove package contents, only the metadata is modified. When you are ready to upload the data to Oracle Support, you create a physical package using ADRCI, which saves the data into a zip file.
Creation of Packages & ZIP files to send to Oracle Support :
Get Problem ID from incident output.
adrci> show incident -mode detail -p "incident_id=12209";
ADR Home = /oradb/app/oracle/diag/rdbms/orcldemo/orcldemo:
*************************************************************************
**********************************************************
INCIDENT INFO RECORD 1
**********************************************************
INCIDENT_ID 12209
STATUS ready
CREATE_TIME 2018-09-19 05:07:14.674000 +05:30
PROBLEM_ID 1
Gather all the required information with a method called “Incident Packaging Service” (IPS).
adrci> ips create package problem 1 correlate all Created package 1 based on problem id 1, correlation level all
Create a ZIP file using logical package
adrci> ips generate package 1 in "/home/oracle" Generated package 1 in file /home/oracle/ORA600ktc_20181024223141_COM_1.zip, mode complete
The package is now zipped and ready to upload to Oracle Support.
[oracle@orcl:~ orcldemo] ls -lrt ORA600*
-rw-r--r-- 1 oracle oinstall 5452175 Oct 24 22:33 /home/oracle/ORA600ktc_20181024223141_COM_1.zip
Examples :-
- At the ADRCI prompt, enter the following command:
IPS CREATE PACKAGE INCIDENT incident_number
For example, the following command creates a package based on incident 3:
IPS CREATE PACKAGE INCIDENT 3
ADRCI generates output similar to the following:
Created package 10 based on incident id 3, correlation level typical The package number assigned to this logical package is 10.
The following are variations on the IPS CREATE PACKAGE command:
IPS CREATE PACKAGE
This creates an empty package. You must use the IPS ADD INCIDENT or IPS ADD FILE commands to add diagnostic data to the package before generating it.
IPS CREATE PACKAGE PROBLEM problem_ID
This creates a package and includes diagnostic information for incidents that reference the specified problem ID.
IPS CREATE PACKAGE SECONDS sec
This creates a package and includes diagnostic information for all incidents that occurred from sec seconds ago until now. sec must be an integer.
IPS CREATE PACKAGE TIME ‘start_time’ TO ‘end_time’
This creates a package and includes diagnostic information for all incidents that occurred within the specified time range. start_time and end_time must be in the format ‘YYYY-MM-DD HH24:MI:SS.FF TZR’. This is a valid format string for the NLS_TIMESTAMP_TZ_FORMAT initialization parameter.
Finalizing :
- Before ADRCI can generate a physical package from a logical package, the package must be finalized. This means that other components are called to add any correlated diagnostic data files to the incidents already in this package.
- Finalizing also adds recent trace files, alert log entries, Health Monitor reports, SQL test cases, and configuration information. This step is run automatically when a physical package is generated, and can also be run manually using the ADRCI utility. After manually finalizing a package, you can review the files that were added and then remove or edit any that contain sensitive information.
ADR Home :
- An ADR home is the root directory for all diagnostic data—traces, dumps, alert log, and so on—for a particular instance of a particular Oracle product or component.
- For example, in a Real Application Clusters environment with ASM, each database instance and each ASM instance has an ADR home. All ADR homes share the same hierarchical directory structure. Some of the standard subdirectories in each ADR home include alert (for the alert log), trace (for trace files), and incident (for incident information). All ADR homes are located within the ADR base directory.
- Some ADRCI commands can work with multiple ADR homes simultaneously. The current ADRCI homepath determines the ADR homes that are searched for diagnostic data when an ADRCI command is issued.
ADR Base :
- To permit correlation of diagnostic data across multiple ADR homes, ADR homes are grouped together under the same root directory called the ADR base.
- For example, in an Oracle Real Application Clusters (RAC) environment, the ADR base could be on a shared disk, and the ADR home for each Oracle RAC instance could be located under this ADR base.
- The location of the ADR base for a database instance is set by the DIAGNOSTIC_DEST initialization parameter. If this parameter is omitted or is null, the database sets it to a default value.
- When multiple database instances share an Oracle home, whether they are multiple single instances or the instances of an Oracle Real Application Clusters database, and when one or more of these instances set ADR base in different locations, the last instance to start up determines the default ADR base for ADRCI.
Homepath :
- All ADRCI commands operate on diagnostic data in the “current” ADR homes. More than one ADR home can be current at any one time. Some ADRCI commands (such as SHOW INCIDENT) search for and display diagnostic data from all current ADR homes, while other commands require that only one ADR home be current, and display an error message if more than one is current.
- The ADRCI homepath determines the ADR homes that are current. It does so by pointing to a directory within the ADR base hierarchy. If it points to a single ADR home directory, that ADR home is the only current ADR home. If the homepath points to a directory that is above the ADR home directory level in the hierarchy, all ADR homes that are below the directory that is pointed to become current.
- The homepath is null by default when ADRCI starts. This means that all ADR homes under ADR base are current.
- The SHOW HOME and SHOW HOMEPATH commands display a list of the ADR homes that are current, and the SET HOMEPATH command sets the homepath.
Catch Me On:- Hariprasath Rajaram LinkedIn: https://www.linkedin.com/in/hari-prasath-aa65bb19/ Facebook: https://www.facebook.com/HariPrasathdba Facebook Group:https://www.facebook.com/groups/894402327369506/ Facebook Page: https://www.facebook.com/dbahariprasath/? Twitter: https://twitter.com/hariprasathdba