Diagnosing and Resolving Performance Problems Using ADDM

Purpose

This tutorial describes how you can use the Automatic Database Diagnostics Monitor (ADDM) to diagnose and resolve performance problems.

Approximately 45 minutes

Topics

This tutorial covers the following topics:

Overview

Place the cursor over this icon to load and view all the screenshots for this tutorial. (Caution: This action loads all screenshots simultaneously, so response time may be slow depending on your Internet connection.)

Note: Alternatively, you can place the cursor over an individual icon in the following steps to load and view only the screenshot associated with that step. You can hide an individual screenshot by clicking it.

Oracle Database 10g includes a self-diagnostic engine built right into the Oracle Database 10g kernel, called the Automatic Database Diagnostic Monitor (ADDM). ADDM periodically examines the state of the database, automatically identifies potential database performance bottlenecks, and recommends corrective actions. Oracle Enterprise Manager presents ADDM's findings and recommendations in a convenient and intuitive fashion, and guides administrators step-by-step to quickly resolve performance problems by implementing ADDM's recommendations. ADDM starts its analysis by focusing on the activities that the database is spending most time on and then drills down through a sophisticated problem classification tree to determine the root causes of problems.

In all cases, these problems are flagged by the Automatic Database Diagnostics Monitor (ADDM), which does a top-down system analysis every hour by default and reports its findings on several Enterprise Manager pages. ADDM runs automatically to coincide with the snapshots taken by the Automatic Workload Repository (AWR). Its output consists of a description of each problem and a recommended action.

Back to Topic List

Before starting this tutorial, you should:

1.

Perform the Installing Oracle Database 10g on Windows tutorial.

2.

Download and unzip the addm.zip into your working directory (c:\wkdir).

3. Ensure that the latest SVG plug-in is installed. (This is required for the proper display of links and icons on the performance pages.)

Back to Topic List

Before starting a workload on your system, make sure that you have completed the following tasks:

1.

Open your browser and enter the following URL (Replace <hostname> with your own host name or IP address):

http://<hostname>:1158/em

Enter sys as User Name, oracle as Password, SYSDBA in the Connect As field, and click Login.

Move your mouse over this icon to see the image

 

2.

You are now on the Database Home page. Click the Performance tab.

Move your mouse over this icon to see the image

 

3.

On the Performance page, make sure that the Real Time: 15 Second Refresh value is selected from the View Data field. Wait for a few seconds until the graphic appears. Scroll down to the Average Active Sessions graphic.

Move your mouse over this icon to see the image

 

4.

You should see a graphic similar to the following one, although there may be larger or smaller spikes and more or less activity displayed.

Move your mouse over this icon to see the image

 

5.

To start a SQL*Plus session, select Start > Programs > Oracle - OraDb10g_home1 > Application Development > SQL Plus. Enter hr as the User Name, hr as Password and click OK.

Move your mouse over this icon to see the image

 

6.

Execute the following commands to create a sequence, a simple table, and an index in the HR schema:

@c:\wkdir\addm01

The addm01.sql file includes the following commands:

drop table t purge;
drop sequence s;
create table t(c number,d varchar2(20));
create index it on t(c);
create sequence s
start with 1
increment by 1
nomaxvalue;

Move your mouse over this icon to see the image

These objects are used to show the ADDM functionality. Keep the SQL*Plus session open throughout this tutorial.

Editing AWR Snapshot Settings

ADDM is automatically triggered every hour right after an AWR snapshot is created. For the purposes of this tutorial, you set the snapshot interval to 10 minutes by using Database Control, so that you do not have to wait an hour to view the performance findings. Perform the following steps to change the snapshot frequency:

1.

Click the Administration tab.

Move your mouse over this icon to see the image

 

2. On the Administration page, click the Automatic Workload Repository link in the Statistics Management section.

Move your mouse over this icon to see the image

 

3.

Click Edit to change the snapshot interval.

Move your mouse over this icon to see the image

 

4.

Select 10 minutes from the Interval drop down list and click OK.

Move your mouse over this icon to see the image

 

5.

When returned to the Automatic Workload Repository page, check the number of snapshots for your system. In the example shown here, the number of snapshots is currently 168. Click Refresh.

Move your mouse over this icon to see the image

 

6.

Wait until you see that the number of snapshots is increased by one. You may have to click Refresh more than once. After a new snapshot has been generated, click the Database Instance: orcl locator link on the top left corner of the page.

Move your mouse over this icon to see the image

 

Identifying Performance Issues

Start a workload in two sessions to generate performance issues and then identify the cause of the problem. The performance issue, which you are about to create, depends on the size of your RAM. The 654321 value in the FOR loop of the addm02.sql script is tested for 1 GB of RAM. You may need to decrease or increase this value, depending on your hardware and system configuration. Perform the following steps:

1.

Return to your SQL*Plus session and type in the following text but do not press [Enter]. You press [Enter] in a later step to execute the script.

@c:\wkdir\addm02

The addm02.sql script contains the following commands:

set echo on
set serveroutput on

begin
for i in 1..654321 loop
insert into t values(s.nextval,'first');
commit;
end loop;
end;
/

Move your mouse over this icon to see the image

 

2.

To start a second SQL*Plus session, select Start > Programs > Oracle - OraDb10g_home1 > Application Development > SQL Plus. Enter hr as the User Name, hr as Password and click OK.

Move your mouse over this icon to see the image

Enter the same command as in the step above and press [Enter] to execute it.

@c:\wkdir\addm02

Move your mouse over this icon to see the image

3.

Immediately switch to the first SQL*Plus session and press [Enter] to execute the script. Each addm02.sql script now inserts rows from a different session.

Move your mouse over this icon to see the image

 

4.

Switch back to your browser window where the Administration page is displayed. Click the Performance tab.

Move your mouse over this icon to see the image

 

5.

Scroll down to view the graphs on the page. In all graphs, you should see a noticable increase, a rise in activity.

Note: In all following examples, your actual performance data and graphics look, most likely, different from the ones displayed here. This depends on hardware, software configuration and system activity.

Move your mouse over this icon to see the image

 

6.

Click the Top Activity link located in the legend area for the Average Active Sessions graphic.

Move your mouse over this icon to see the image

 

7.

On the Top Activity page, if needed, drag the shaded box over the peak in the graph which you created with the execution of the two addm02.sql scripts.

Move your mouse over this icon to see the image

Notice in the Top Sessions section (bottom right), the two HR users executing SQL scripts from their SQL*Plus sessions. Click the SQL ID link for the INSERT statement in the Top SQL section (bottom left).

Move your mouse over this icon to see the image

 

8.

On the SQL Details page, you can see the text of the SQL statement, that is responsible for the peak in the performance graph. You see this statement again in the next section through an AWR snapshot. ADDM can identify these performance issues automatically and provide recommended solutions. To proceed, click the Database Instance: orcl locator link.

Move your mouse over this icon to see the image

 

Using ADDM to Automatically Identify Performance Issues

ADDM automatically analyze the database performance when the next AWR snapshot is created. ADDM analyzes the period between the current and its previous snapshot. To view the latest ADDM analysis findings, perform the following steps:

1.

On the Performance page, scroll down to the Average Active Session graph. Notice the icon at the bottom of the graph that indicates the AWR snapshot taken before the workload was started.

Wait until a new AWR snapshot icon appears on the graph, about 10 minutes after the time of the previous snapshot. When you see two snapshot icons, one before the workload started and one after the workload started, click the AWR snapshot icon to the right of the spike which was created by your workload.

Move your mouse over this icon to see the image

 

2.

The Automatic Database Diagnostic Monitor (ADDM) page appears. Scroll down to the Performance Analysis section.

Tip: If you receive a finding with a Host Configuration Recommendation, as shown in the next two screenshots, consider modifying the addm02.sql script and setting the 654321 loop value to a lower value. Then execute the addm01.sql script before going back to step 1 of this section. (Your findings should then be more similar to the screenshot for step 3, but this is not essential for the successful execution of this tutorial.)

Move your mouse over this icon to see the image

 

Move your mouse over this icon to see the image

Your Performance Analysis section should look similar to the screeenshot below, containing the following findings:

  • SQL statements consuming significant database time were found.
  • Sequence cache misses were consuming significant database time.

 

3.

Click the SQL statements consuming significant database time were found. link that has the largest percentage of Impact (%).

Move your mouse over this icon to see the image

 

4.

On the corresponding Performance Finding Details page, click Show All Details.

Move your mouse over this icon to see the image

 

5.

You should now see the INSERT statement you found manually in the previous topic. This time, ADDM identified the statement automatically for you. Click the Automatic Database Diagnostic Monitor (ADDM) locator link.

Move your mouse over this icon to see the image

 

6.

When returned to the Automatic Database Diagnostic Monitor page, scroll down again to the list of findings. This time, click the Sequence cache misses were consuming significant database time. link.

Move your mouse over this icon to see the image

 

7.

The Performance Finding Details page appears. Here you see that ADDM recommends to use a larger cache value for frequently accessed sequences. Now fix the problem by raising the cache value of the S sequence. Click the Database Instance: orcl locator link.

Move your mouse over this icon to see the image

 

Manually Running ADDM

ADDM analyzes the performance for the period between the current AWR snapshot and the previous one. To make sure ADDM evaluates performance for the period after the sequence caching problem was fixed, you manually create a new snapshot and ADDM report. Perform the following steps:

1.

On the Performance page, scroll down to the Average Active Session graph. If any scripts are still running, wait until they finished and the database activity returns to normal, then click Run ADDM Now.

Move your mouse over this icon to see the image

 

2.

Click Yes to create a new AWR snapshot and run ADDM on this and the previous snapshot.

Move your mouse over this icon to see the image

 

3.

The Processing: Run ADDM Now page appears. It may display for one minute depending on your system activity.

Move your mouse over this icon to see the image

 

4. When returned to the Automatic Database Diagnostic Monitor (ADDM) page, click the Database Instance: orcl locator link.

Move your mouse over this icon to see the image

 

Implementing ADDM Recommendations

This script creates the same objects as the addm01.sql script, but the S sequence now has a cache value of 700,000 instead of the default is 20.

Return to your SQL*Plus session and enter the following command:

@c:\wkdir\addm03

The addm03.sql script contains the following commands:

drop table t purge;
drop sequence s;
create table t(c number,d varchar2(20));
create index it on t(c);
create sequence s
start with 1
increment by 1
nomaxvalue

cache 700000;
/

Move your mouse over this icon to see the image

You have now resolved this issue. You had also noticed, that the HR user executed the addm02.sql script from two sessions. To further improve performance, you encourage the user the execute this script sequentially.

 

Checking Whether the Performance Issue is Resolved

Now, you must check whether the sequence issue is no longer a performance problem.

1.

Return to your SQL*Plus session and enter the following command:

@c:\wkdir\addm02

Move your mouse over this icon to see the image

 

2.

Switch back to your browser window. On the Performance page scroll down to the Average Active Sessions graph. After a few minutes you should see some activity but much less than during the first workload period.

Move your mouse over this icon to see the image

Wait until a new snapshot icon appears at the bottom of the Average Active Session graph, then click the snapshot icon to view the ADDM findings.

 

3.

You may not have any findings at all, but this depends on your overall system activity. Click the Icon Key link and familiarize yourself with the meaning of the icons.

Move your mouse over this icon to see the image

 

4.

Click the Database tab in the top right corner.

Move your mouse over this icon to see the image

You are on the Database home page.

Back to Topic List

You can view the history of database session activity for both real-time (less than one hour) and historical time contexts.
Perform the following steps to view your workload in historical mode:

1.

On the Database home page, click the Performance tab.

Move your mouse over this icon to see the image

 

2.

Select Historical from the View Data drop down list.

Move your mouse over this icon to see the image

 

3.

The Performance page for the ORCL instance now displays historical data and the Historical Interval Selection chart. The database activity graphs can show several days worth of performance measurements, depending upon how long the database instance has been running.

Move your mouse over this icon to see the image

 

For the selected time interval, you can view theCPU Utilization for your Host, Average Active Sessions, Instance Disk I/O and Instance Throughput charts.

 

4.

If the instance has been running for more than 24 hours, you can drag the shaded box to the desired 24-hour interval for which you want to view historical data, as shown here.

Move your mouse over this icon to see the image

 

Back to Topic List

 

1.

Click the Administration tab.

Move your mouse over this icon to see the image

 

2. On the Administration page, click the Automatic Workload Repository link in the Statistics Management section.

Move your mouse over this icon to see the image

 

3.

Click Edit to change the snapshot interval.

Move your mouse over this icon to see the image

 

4.

Select 1 Hour (or whatever you standard value is) from the Interval drop down list and click OK.

Move your mouse over this icon to see the image

 

5.

Return to one of your SQL*Plus sessions and enter the following command:

@c:\wkdir\addm_cleanup

The addm_cleanup.sql script contains the following commands:

drop table t purge;
drop sequence s;

Move your mouse over this icon to see the image

 

6.

Close all tutorial windows.

Back to Topic List

In this tutorial, you learned how to:

Create an ADDM snapshot
Identify performance issues within the database
Identify and resolve the performance issues by using ADDM
View the history of activity for a database session

Back to Topic List