diff --git a/in-memory-23ai/README.md b/in-memory-23ai/README.md new file mode 100644 index 000000000..14b40338d --- /dev/null +++ b/in-memory-23ai/README.md @@ -0,0 +1,36 @@ +# Oracle Database In-Memory 23ai Github + + +Oracle Database In-Memory (Database In-Memory) is a product, first introduced in Oracle Database 12c, that improves the performance of analytic queries by orders of magnitude. It makes it possible to enable real-time analytics in mixed workload environments. Since it is fully integrated into Oracle Database and requires no application changes to use, it can be used in any Oracle Database environment where existing reporting or analytic workloads exist to provide orders of magnitude performance improvements. + +The In-Memory Column Store (IM column store) is a key feature of Database In-Memory. The In-Memory Column Store (IM column store) stores tables and partitions in memory using a columnar format optimized for rapid scans. Oracle Database uses a sophisticated architecture to manage data in columnar and row formats simultaneously. This lab will explore how Database In-Memory works and the newer features that have been added in Oracle Database 19c and 21c. + +[Run a workshop now!](http://developer.oracle.com/livelabs) + +## Get an Oracle Cloud Trial Account for Free! +If you don't have an Oracle Cloud account then you can quickly and easily sign up for a free trial account that provides: +- $300 of free credits good for up to 3500 hours of Oracle Cloud usage +- Credits can be used on all eligible Cloud Platform and Infrastructure services for the next 30 days +- Your credit card will only be used for verification purposes and will not be charged unless you 'Upgrade to Paid' in My Services + +Click here to request your trial account: [https://www.oracle.com/cloud/free](https://www.oracle.com/cloud/free) + + +## Product Pages +- [Oracle In-Memory](https://www.oracle.com/database/technologies/in-memory.html) +- [Oracle Database 21c](https://www.oracle.com/database/) + +## Documentation +- [Introduction to the Oracle Database In-Memory](https://docs.oracle.com/en/database/oracle/oracle-database/21/inmem/intro-to-in-memory-column-store.html#GUID-BFA53515-7643-41E5-A296-654AB4A9F9E7) +- [In-Memory Column Store Architecture](https://docs.oracle.com/en/database/oracle/oracle-database/21/inmem/in-memory-column-store-architecture.html#GUID-EEA265EE-8FBA-4457-8C3F-315B9EEA2224) + +## Videos +- [Oracle Database In-Memory YouTube Channel](https://www.youtube.com/channel/UCSYHgTG68nrHa5aTGfFH4pA) +- [Database Product Management YouTube Channel](https://www.youtube.com/channel/UCr6mzwq_gcdsefQWBI72wIQ) +- [Real Time Analytics with Oracle Database In-Memory](https://www.youtube.com/watch?v=eToO3PRIs8k) +- [Managing Oracle Database In-Memory](https://www.youtube.com/watch?v=IZ7UMoQxtLo) +- [Oracle Database In-Memory Demo](https://www.youtube.com/watch?v=mF-h26iKTYY) +- [Enabling the In-Memory Column Store](https://www.youtube.com/watch?v=dZ9cnIL6KKw) +- [Exadata In-Memory High Availability](https://www.youtube.com/watch?v=j3n5ZjUvcD0) +- [Oracle Database](https://www.youtube.com/watch?v=EVPNyL2vAVI) +- [Upgrading to Oracle Database](https://www.youtube.com/watch?v=lOzL5irmuJo) diff --git a/in-memory-23ai/aim-23ai/aim-high.md b/in-memory-23ai/aim-23ai/aim-high.md new file mode 100644 index 000000000..f40be4fc8 --- /dev/null +++ b/in-memory-23ai/aim-23ai/aim-high.md @@ -0,0 +1,682 @@ +# Automatic In-Memory Level High + +## Introduction +Watch the video below to get an overview of Automatic In-Memory: + +[YouTube video](youtube:pFWjl1G7uDI) + +Watch the video below for a quick walk-through of this lab. +[Automatic In-Memory High](videohub:1_0rzwly4i) + +*Estimated Lab Time:* 15 Minutes. + +### Objectives + +- Learn how Automatic In-Memory (AIM) level HIGH works +- Perform various queries invoking AIM with INMEMORY_AUTOMATIC_LEVEL set to HIGH + +### Prerequisites + +This lab assumes you have: +- A Free Tier, Paid or LiveLabs Oracle Cloud account +- You have completed: + - Get Started with noVNC Remote Desktop + - Lab: Initialize Environment + - Lab: Setting up the In-Memory Column Store + +**NOTE:** *When doing Copy/Paste using the convenient* **Copy** *function used throughout the guide, you must hit the* **ENTER** *key after pasting. Otherwise the last line will remain in the buffer until you hit* **ENTER!** + +## Task 1: AIM Level High + +In Oracle Database 18c a feature called Automatic In-Memory (AIM) was added. The goal of AIM is to manage the contents of the IM column store based on usage. AIM initially had two levels, LOW and MEDIUM, that enabled automatic management of IM column store contents once the IM column store became full. In Oracle Database 21c a third level was added that automatically manages all non-system segments without having to first enable the objects for in-memory. + +This Lab will explore the new AIM level HIGH and how it works. We will also take a look at two new features in Oracle Database 23ai, AIM Performance Features and Automatic In-Memory Sizing. Note that AIM works as the column store experiences "memory pressure" (i.e. gets full). The SSB schema will be used to help "fill up" the IM column store and then other schema objects will help show how AIM can manage the total number of objects for maximum benefit. + +Reload the environment variables for **CDB1** if you exited the terminal after the previous lab + +``` +. ~/.set-env-db.sh CDB1 +``` + +Let's switch to the aim23 folder and log back in to the PDB: + +``` + +cd /home/oracle/labs/inmemory/aim23 +sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + +``` + +And adjust the sqlplus display: + +``` + +set pages 9999 +set lines 150 + +``` + +Query result: + +``` +[CDB1:oracle@dbhol:~/labs/inmemory]$ cd /home/oracle/labs/inmemory/aim23 +[CDB1:oracle@dbhol:~/labs/inmemory/aim23]$ sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + +SQL*Plus: Release 23.0.0.0.0 - Production on Tue Jun 4 15:31:23 2024 +Version 23.4.0.24.05 + +Copyright (c) 1982, 2024, Oracle. All rights reserved. + +Last Successful login time: Tue Jun 4 2022 15:31:23 2024 + +Connected to: +Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production +Version 23.4.0.24.05 + +SQL> set pages 9999 +SQL> set lines 150 +SQL> +``` + +1. First let's check the inmemory status of the objects in the SSB schema: + + Run the script *01\_aim\_attributes.sql* + + ``` + + @01_aim_attributes.sql + + ``` + + or run the query below: + + Query result: + + ``` + SQL> @01_aim_attributes.sql + Connected. + SQL> + SQL> -- Show table attributes + SQL> + SQL> select owner, table_name, NULL as partition_name, inmemory, + 2 inmemory_priority, inmemory_distribute, inmemory_compression + 3 from dba_tables + 4 where owner in ('AIM','SSB') + 5 UNION ALL + 6 select table_owner as owner, table_name, partition_name, inmemory, + 7 inmemory_priority, inmemory_distribute, inmemory_compression + 8 from dba_tab_partitions + 9 where table_owner in ('AIM','SSB') + 10 order by owner, table_name, partition_name; + + + INMEMORY INMEMORY INMEMORY + OWNER TABLE_NAME PARTITION_NAME INMEMORY PRIORITY DISTRIBUTE COMPRESSION + ---------- -------------------- --------------- ---------- ---------- ------------ -------------- + AIM LRGTAB1 ENABLED NONE AUTO AUTO + AIM LRGTAB2 ENABLED NONE AUTO AUTO + AIM LRGTAB3 ENABLED NONE AUTO AUTO + AIM MEDTAB1 ENABLED NONE AUTO AUTO + AIM MEDTAB2 ENABLED NONE AUTO AUTO + AIM MEDTAB3 ENABLED NONE AUTO AUTO + AIM SMTAB1 ENABLED NONE AUTO AUTO + AIM SMTAB2 ENABLED NONE AUTO AUTO + AIM SMTAB3 ENABLED NONE AUTO AUTO + SSB CHICAGO_DATA ENABLED NONE AUTO AUTO + SSB CUSTOMER ENABLED NONE AUTO AUTO + SSB DATE_DIM ENABLED NONE AUTO AUTO + SSB EXT_CUST_BULGARIA ENABLED NONE AUTO AUTO + SSB EXT_CUST_NORWAY ENABLED NONE AUTO AUTO + SSB JSON_PURCHASEORDER ENABLED NONE AUTO AUTO + SSB LINEORDER PART_1994 ENABLED NONE AUTO AUTO + SSB LINEORDER PART_1995 ENABLED NONE AUTO AUTO + SSB LINEORDER PART_1996 ENABLED NONE AUTO AUTO + SSB LINEORDER PART_1997 ENABLED NONE AUTO AUTO + SSB LINEORDER PART_1998 ENABLED NONE AUTO AUTO + SSB LINEORDER + SSB PART ENABLED NONE AUTO AUTO + SSB SUPPLIER ENABLED NONE AUTO AUTO + + 23 rows selected. + + SQL> + ``` + + Note the inmemory status of the tables. Here they are all enabled since AIM level is set to high. + +2. Next we will take a look at what has happened with SGA sizing. With the IM Auto Sizing feature the IM column store is now part of Automatic Shared Memory Managment (ASMM). Based on our activity in this lab lets take a look at what SGA sizing operations have taken place. + + Run the script *02\_sga\_sizing.sql* + + ``` + + @02_sga_sizing.sql + + ``` + + or run the query below: + + ``` + + column component format a20; + column parameter format a21; + select + component, oper_type, oper_mode, parameter, initial_size, + target_size, final_size, + to_char(start_time,'MM/DD/YYYY HH24:MI:SS') start_time, + to_char(end_time,'MM/DD/YYYY HH24:MI:SS') end_time + from + v$sga_resize_ops + where + component in ('shared_pool','DEFAULT buffer cache','In-Memory Area') + order by + start_time + / + + ``` + + Query result: + + ``` + SQL> set echo off + SQL> @02_sga_sizing.txt + Connected. + + COMPONENT OPER_TYPE OPER_MODE PARAMETER INITIAL_SIZE TARGET_SIZE FINAL_SIZE START_TIME END_TIME + -------------------- ------------- --------- --------------------- ------------ ----------- ---------- ------------------- ------------------- + DEFAULT buffer cache INITIALIZING db_cache_size 1912602624 1912602624 1912602624 06/12/2024 14:48:05 06/12/2024 14:48:05 + DEFAULT buffer cache STATIC db_cache_size 0 1912602624 1912602624 06/12/2024 14:48:05 06/12/2024 14:48:05 + In-Memory Area STATIC _datamemory_area_size 0 2432696320 2432696320 06/12/2024 14:48:05 06/12/2024 14:48:05 + In-Memory Area GROW DEFERRED _datamemory_area_size 2432696320 2566914048 2566914048 06/12/2024 14:58:18 06/12/2024 14:58:18 + DEFAULT buffer cache SHRINK DEFERRED db_cache_size 1912602624 1778384896 1778384896 06/12/2024 14:58:18 06/12/2024 14:58:18 + In-Memory Area GROW DEFERRED _datamemory_area_size 2566914048 2835349504 2835349504 06/12/2024 15:00:18 06/12/2024 15:00:18 + DEFAULT buffer cache SHRINK DEFERRED db_cache_size 1778384896 1509949440 1509949440 06/12/2024 15:00:18 06/12/2024 15:00:18 + + 7 rows selected. + + SQL> + ``` + + Note that now that In-Memory Area is managed as part of ASMM you see that the buffer cache is shrinking and the In-Memory Area is growing. + +3. Now lets look at what objects are populated. + + Run the script *03\_im\_populated.sql* + + ``` + + @03_im_populated.sql + + ``` + + or run the queries below: + + ``` + + column owner format a10; + column segment_name format a20; + column partition_name format a15; + column populate_status format a15; + column bytes heading 'Disk Size' format 999,999,999,999 + column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 + column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 + select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated + from v$im_segments + where owner not in ('AUDSYS','SYS') + order by owner, segment_name, partition_name; + + select * from v$inmemory_area; + + ``` + + Query result: + + ``` + SQL> @03_im_populated.sql + Connected. + SQL> + SQL> -- Query the view v$IM_SEGMENTS to shows what objects are in the column store + SQL> -- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 + SQL> -- it indicates the entire table was populated. + SQL> + SQL> select owner, segment_name, partition_name, populate_status, bytes, + 2 inmemory_size, bytes_not_populated + 3 from v$im_segments + 4 where owner not in ('AUDSYS','SYS') + 5 order by owner, segment_name, partition_name; + + In-Memory Bytes + OWNER SEGMENT_NAME PARTITION_NAME POPULATE_STATUS Disk Size Size Not Populated + ---------- -------------------- --------------- --------------- ---------------- ---------------- ---------------- + AIM MEDTAB1 COMPLETED 38,322,176 25,559,040 0 + AIM MEDTAB2 COMPLETED 38,322,176 23,199,744 0 + AIM MEDTAB3 COMPLETED 38,322,176 23,199,744 0 + SSB CHICAGO_DATA STARTED 1,154,768,896 241,696,768 735,420,416 + SSB CUSTOMER COMPLETED 24,928,256 23,199,744 0 + SSB DATE_DIM COMPLETED 122,880 1,179,648 0 + SSB LINEORDER PART_1994 COMPLETED 565,338,112 479,330,304 0 + SSB LINEORDER PART_1995 COMPLETED 565,354,496 479,330,304 0 + SSB LINEORDER PART_1996 COMPLETED 567,484,416 480,378,880 0 + SSB LINEORDER PART_1997 COMPLETED 564,281,344 479,330,304 0 + SSB LINEORDER PART_1998 COMPLETED 330,407,936 279,642,112 0 + SSB PART COMPLETED 56,893,440 16,973,824 0 + SSB SUPPLIER COMPLETED 1,769,472 2,228,224 0 + + 13 rows selected. + + SQL> + SQL> select * from v$inmemory_area; + + POOL ALLOC_BYTES USED_BYTES POPULATE_STATUS CON_ID + -------------------------- ----------- ---------- --------------- ---------- + 1MB POOL 2667577344 2548039680 POPULATING 3 + 64KB POOL 134217728 7208960 POPULATING 3 + IM POOL METADATA 16777216 16777216 POPULATING 3 + + SQL> + ``` + + Notice that s. + +4. Now let's see if we can figure out what has happened with the AIM processing. First we will look at the tasks that are running as part of AIM. + + Run the script *04\_aim\_actions.sql* + + ``` + + @04_aim\_actions.sql + + ``` + + or run the queries below: + + ``` + + set head off; + set echo off; + set verify off; + set term off; + column max_id new_value max_id format 99999999; + select max(task_id) max_id + from dba_inmemory_aimtasks; + set term on; + -- + column max_task new_value max_task; + accept max_task default &&max_id prompt 'Enter max task id > '; + prompt ; + column min_task new_value min_task; + accept min_task default &&max_id-5 prompt 'Enter min task id > '; + prompt ; + set head on; + select &&min_task min_task, &&max_task max_task from dual; + -- + col object_owner format a15; + col object_name format a30; + col subobject_name format a30; + select * from dba_inmemory_aimtaskdetails + where task_id between &&min_task and &&max_task + and object_owner not in ('SYS','AUDSYS') + -- and action != 'NO ACTION' + order by task_id, object_owner, object_name, subobject_name, action; + set verify on; + set echo on; + + ``` + + Query result: + + ``` + SQL> @04_aim_actions.sql + Connected. + Enter max task id > + + Enter min task id > + + + MIN_TASK MAX_TASK + ---------- ---------- + 4588 4593 + + + TASK_ID OBJECT_OWNER OBJECT_NAME SUBOBJECT_NAME ACTION STATE + ---------- --------------- ------------------------------ ------------------------------ ---------------- ---------- + 4589 AIM LRGTAB1 NO ACTION DONE + 4589 AIM LRGTAB2 NO ACTION DONE + 4589 AIM LRGTAB3 NO ACTION DONE + 4589 AIM MEDTAB1 POPULATE DONE + 4589 AIM MEDTAB2 POPULATE DONE + 4589 AIM MEDTAB3 POPULATE DONE + 4589 AIM SMTAB1 NO ACTION DONE + 4589 AIM SMTAB2 NO ACTION DONE + 4589 AIM SMTAB3 NO ACTION DONE + 4589 SSB CHICAGO_DATA PARTIAL POPULATE DONE + 4589 SSB CUSTOMER POPULATE DONE + 4589 SSB DATE_DIM POPULATE DONE + 4589 SSB EXT_CUST_BULGARIA NO ACTION DONE + 4589 SSB EXT_CUST_NORWAY NO ACTION DONE + 4589 SSB JSON_PURCHASEORDER NO ACTION DONE + 4589 SSB LINEORDER PART_1994 POPULATE DONE + 4589 SSB LINEORDER PART_1995 POPULATE DONE + 4589 SSB LINEORDER PART_1996 POPULATE DONE + 4589 SSB LINEORDER PART_1997 POPULATE DONE + 4589 SSB LINEORDER PART_1998 POPULATE DONE + 4589 SSB PART POPULATE DONE + 4589 SSB SUPPLIER POPULATE DONE + 4589 VECTOR DM$P5DOC_MODEL NO ACTION DONE + 4589 VECTOR DM$P8DOC_MODEL NO ACTION DONE + 4589 VECTOR DM$P9DOC_MODEL NO ACTION DONE + 4589 VECTOR DM$PADOC_MODEL NO ACTION DONE + 4589 VECTOR SEARCH_DATA NO ACTION DONE + 4589 VECTOR SEARCH_DATA10K NO ACTION DONE + 4590 AIM LRGTAB1 NO ACTION DONE + 4590 AIM LRGTAB2 NO ACTION DONE + 4590 AIM LRGTAB3 NO ACTION DONE + 4590 AIM MEDTAB1 POPULATE DONE + 4590 AIM MEDTAB2 POPULATE DONE + 4590 AIM MEDTAB3 POPULATE DONE + 4590 AIM SMTAB1 NO ACTION DONE + 4590 AIM SMTAB2 NO ACTION DONE + 4590 AIM SMTAB3 NO ACTION DONE + 4590 SSB CHICAGO_DATA PARTIAL POPULATE DONE + 4590 SSB CUSTOMER POPULATE DONE + 4590 SSB DATE_DIM POPULATE DONE + 4590 SSB EXT_CUST_BULGARIA NO ACTION DONE + 4590 SSB EXT_CUST_NORWAY NO ACTION DONE + 4590 SSB JSON_PURCHASEORDER NO ACTION DONE + 4590 SSB LINEORDER PART_1994 POPULATE DONE + 4590 SSB LINEORDER PART_1995 POPULATE DONE + 4590 SSB LINEORDER PART_1996 POPULATE DONE + 4590 SSB LINEORDER PART_1997 POPULATE DONE + 4590 SSB LINEORDER PART_1998 POPULATE DONE + 4590 SSB PART POPULATE DONE + 4590 SSB SUPPLIER POPULATE DONE + 4590 VECTOR DM$P5DOC_MODEL NO ACTION DONE + 4590 VECTOR DM$P8DOC_MODEL NO ACTION DONE + 4590 VECTOR DM$P9DOC_MODEL NO ACTION DONE + 4590 VECTOR DM$PADOC_MODEL NO ACTION DONE + 4590 VECTOR SEARCH_DATA NO ACTION DONE + 4590 VECTOR SEARCH_DATA10K NO ACTION DONE + 4591 AIM LRGTAB1 NO ACTION DONE + 4591 AIM LRGTAB2 NO ACTION DONE + 4591 AIM LRGTAB3 NO ACTION DONE + 4591 AIM MEDTAB1 POPULATE DONE + 4591 AIM MEDTAB2 POPULATE DONE + 4591 AIM MEDTAB3 POPULATE DONE + 4591 AIM SMTAB1 NO ACTION DONE + 4591 AIM SMTAB2 NO ACTION DONE + 4591 AIM SMTAB3 NO ACTION DONE + 4591 SSB CHICAGO_DATA PARTIAL POPULATE DONE + 4591 SSB CUSTOMER POPULATE DONE + 4591 SSB DATE_DIM POPULATE DONE + 4591 SSB EXT_CUST_BULGARIA NO ACTION DONE + 4591 SSB EXT_CUST_NORWAY NO ACTION DONE + 4591 SSB JSON_PURCHASEORDER NO ACTION DONE + 4591 SSB LINEORDER PART_1994 POPULATE DONE + 4591 SSB LINEORDER PART_1995 POPULATE DONE + 4591 SSB LINEORDER PART_1996 POPULATE DONE + 4591 SSB LINEORDER PART_1997 POPULATE DONE + 4591 SSB LINEORDER PART_1998 POPULATE DONE + 4591 SSB PART POPULATE DONE + 4591 SSB SUPPLIER POPULATE DONE + 4591 VECTOR DM$P5DOC_MODEL NO ACTION DONE + 4591 VECTOR DM$P8DOC_MODEL NO ACTION DONE + 4591 VECTOR DM$P9DOC_MODEL NO ACTION DONE + 4591 VECTOR DM$PADOC_MODEL NO ACTION DONE + 4591 VECTOR SEARCH_DATA NO ACTION DONE + 4591 VECTOR SEARCH_DATA10K NO ACTION DONE + 4592 AIM LRGTAB1 NO ACTION DONE + 4592 AIM LRGTAB2 NO ACTION DONE + 4592 AIM LRGTAB3 NO ACTION DONE + 4592 AIM MEDTAB1 POPULATE DONE + 4592 AIM MEDTAB2 POPULATE DONE + 4592 AIM MEDTAB3 POPULATE DONE + 4592 AIM SMTAB1 NO ACTION DONE + 4592 AIM SMTAB2 NO ACTION DONE + 4592 AIM SMTAB3 NO ACTION DONE + 4592 SSB CHICAGO_DATA PARTIAL POPULATE DONE + 4592 SSB CUSTOMER POPULATE DONE + 4592 SSB DATE_DIM POPULATE DONE + 4592 SSB EXT_CUST_BULGARIA NO ACTION DONE + 4592 SSB EXT_CUST_NORWAY NO ACTION DONE + 4592 SSB JSON_PURCHASEORDER NO ACTION DONE + 4592 SSB LINEORDER PART_1994 POPULATE DONE + 4592 SSB LINEORDER PART_1995 POPULATE DONE + 4592 SSB LINEORDER PART_1996 POPULATE DONE + 4592 SSB LINEORDER PART_1997 POPULATE DONE + 4592 SSB LINEORDER PART_1998 POPULATE DONE + 4592 SSB PART POPULATE DONE + 4592 SSB SUPPLIER POPULATE DONE + 4592 VECTOR DM$P5DOC_MODEL NO ACTION DONE + 4592 VECTOR DM$P8DOC_MODEL NO ACTION DONE + 4592 VECTOR DM$P9DOC_MODEL NO ACTION DONE + 4592 VECTOR DM$PADOC_MODEL NO ACTION DONE + 4592 VECTOR SEARCH_DATA NO ACTION DONE + 4592 VECTOR SEARCH_DATA10K NO ACTION DONE + 4593 AIM LRGTAB1 NO ACTION DONE + 4593 AIM LRGTAB2 NO ACTION DONE + 4593 AIM LRGTAB3 NO ACTION DONE + 4593 AIM MEDTAB1 POPULATE DONE + 4593 AIM MEDTAB2 POPULATE DONE + 4593 AIM MEDTAB3 POPULATE DONE + 4593 AIM SMTAB1 NO ACTION DONE + 4593 AIM SMTAB2 NO ACTION DONE + 4593 AIM SMTAB3 NO ACTION DONE + 4593 SSB CHICAGO_DATA PARTIAL POPULATE DONE + 4593 SSB CUSTOMER POPULATE DONE + 4593 SSB DATE_DIM POPULATE DONE + 4593 SSB EXT_CUST_BULGARIA NO ACTION DONE + 4593 SSB EXT_CUST_NORWAY NO ACTION DONE + 4593 SSB JSON_PURCHASEORDER NO ACTION DONE + 4593 SSB LINEORDER PART_1994 POPULATE DONE + 4593 SSB LINEORDER PART_1995 POPULATE DONE + 4593 SSB LINEORDER PART_1996 POPULATE DONE + 4593 SSB LINEORDER PART_1997 POPULATE DONE + 4593 SSB LINEORDER PART_1998 POPULATE DONE + 4593 SSB PART POPULATE DONE + 4593 SSB SUPPLIER POPULATE DONE + 4593 VECTOR DM$P5DOC_MODEL NO ACTION DONE + 4593 VECTOR DM$P8DOC_MODEL NO ACTION DONE + 4593 VECTOR DM$P9DOC_MODEL NO ACTION DONE + 4593 VECTOR DM$PADOC_MODEL NO ACTION DONE + 4593 VECTOR SEARCH_DATA NO ACTION DONE + 4593 VECTOR SEARCH_DATA10K NO ACTION DONE + + 140 rows selected. + + SQL> + ``` + + Make note of the last task_id. We will use this as input in the next step. Also note that the tasks are being run approximately every 2 minutes. AIM tasks will be scheduled during each IMCO cycle, which is approximately every 2 minutes, when the IM column store is under memory pressure. This means that it may take a couple of cycles before an object is populated by AIM in the IM column store. + + +5. Let's take a look at the Heat Map statistics for the segments. Although Heat Map is not used directly by AIM, and does not have to be enabled for AIM to work, it does give us an easy way to look at the usage statistics that AIM does base its decisions on. + + Run the script *05\_hm\_stats.sql* + + ``` + + @05_hm_stats.sql + + ``` + + or run the queries below: + + ``` + + col owner format a10; + col object_name format a20; + col subobject_name format a15; + col track_time format a16; + col segment_write heading 'SEG|WRITE' format a10; + col segment_read heading 'SEG|READ' format a10; + col full_scan heading 'FULL|SCAN' format a10; + col lookup_scan heading 'LOOKUP|SCAN' format a10; + col n_fts heading 'NUM FULL|SCAN' format 99999999; + col n_lookup heading 'NUM LOOKUP|SCAN' format 99999999; + col n_write heading 'NUM SEG|WRITE' format 99999999; + select + OWNER, + OBJECT_NAME, + SUBOBJECT_NAME, + to_char(TRACK_TIME,'MM/DD/YYYY HH24:MI') track_time, + SEGMENT_WRITE, + SEGMENT_READ, + FULL_SCAN, + LOOKUP_SCAN, + N_FTS, + N_LOOKUP, + N_WRITE + from + sys."_SYS_HEAT_MAP_SEG_HISTOGRAM" h, + dba_objects o + where + o.object_id = h.obj# + and track_time >= sysdate-1 + order by + track_time, + OWNER, + OBJECT_NAME, + SUBOBJECT_NAME; + + ``` + + Query result: + + ``` + SQL> @05_hm_stats.sql + Connected. + + SEG SEG FULL LOOKUP NUM FULL NUM LOOKUP NUM SEG + OWNER OBJECT_NAME SUBOBJECT_NAME TRACK_TIME WRITE READ SCAN SCAN SCAN SCAN WRITE + ---------- -------------------- --------------- ---------------- ---------- ---------- ---------- ---------- --------- ---------- --------- + SSB CUSTOMER 06/12/2024 13:10 NO YES YES NO 8 0 0 + SSB DATE_DIM 06/12/2024 13:10 NO YES YES NO 15 0 0 + SSB LINEORDER PART_1994 06/12/2024 13:10 NO YES YES NO 26 0 0 + SSB LINEORDER PART_1995 06/12/2024 13:10 NO YES YES NO 26 0 0 + SSB LINEORDER PART_1996 06/12/2024 13:10 NO YES YES YES 29 24540 0 + SSB LINEORDER PART_1997 06/12/2024 13:10 NO YES YES NO 29 0 0 + SSB LINEORDER PART_1998 06/12/2024 13:10 NO YES YES NO 26 0 0 + SSB LINEORDER_I1 06/12/2024 13:10 NO YES NO YES 0 1 0 + SSB LINEORDER_I2 06/12/2024 13:10 NO YES NO YES 0 1 0 + SSB PART 06/12/2024 13:10 NO YES YES NO 11 0 0 + SSB SUPPLIER 06/12/2024 13:10 NO YES YES NO 11 0 0 + AIM MEDTAB1 06/12/2024 15:48 NO YES YES NO 4 0 0 + AIM MEDTAB2 06/12/2024 15:48 NO YES YES NO 4 0 0 + AIM MEDTAB3 06/12/2024 15:48 NO YES YES NO 3 0 0 + + 14 rows selected. + + SQL> + ``` + + Note that your values may be different than what is shown above. The values shown will be based on the usage that has occurred in your database. + +6. Now let's look any performance features were created by AIM. + + Run the script *06\_auto\_im\_features.sql* + + ``` + + @06_auto_im_features.sql + + ``` + + or run the queries below: + + ``` + + col owner_name format a10; + col table_name format a15; + col column_name format a20; + col optimized_arithmetic heading opt|arith format a5; + col bloomfilter_optimization heading bloom|opt format a5; + col vector_optimization heading vector|opt format a6; + col join_group heading join|group format a5; + col creation_date heading create|date; + set lines 150 + set tab off + select owner_name, table_name, column_name, optimized_arithmetic, + bloomfilter_optimization, vector_optimization, join_group, + creation_date + from dba_aim_perf_features; + + ``` + + Query result: + + ``` + SQL> @06_auto_im_features.sql + Connected. + + opt bloom vector join create + OWNER_NAME TABLE_NAME COLUMN_NAME arith opt opt group date + ---------- --------------- -------------------- ----- ----- ------ ----- --------- + SSB LINEORDER LO_ORDERDATE N Y N N 16-MAY-24 + + SQL> + ``` + + Take a look at the OBJECT_NAME and the ACTION. Now that the IM column store is under memory pressure AIM has taken over control of population and there is a lot going on. Based on usage statistics AIM will populate the objects that will result in the most benefit to queries being run. You may want to take a look at some of the other task details to get a better picture of what has happened. Also note that now that AIM is controlling population the PRIORITY level will be ignored and AIM will decide which objects to populate and which to evict. + +7. Now let's run an AIM activity report and see what decisions AIM ... + + Run the script *07\_auto\_im\_activity.sql* + + ``` + + @07_auto_im_activity.sql + + ``` + + or run the queries below: + + ``` + + set serveroutput on + declare + report clob := null; + begin + report := DBMS_AUTOIM.activity_report(level=>'DETAILED'); + dbms_output.put_line(report); + end; + / + + ``` + + Query result: + + ``` + SQL> @07_auto_im_activity.sql + Connected. + REPORT SUMMARY + ------------------------------------------------------------------------------- + Start time : 11-JUN-2024 15:58:31 + End time : 12-JUN-2024 15:58:31 + No. of times auto task + scheduled : 15 + Statements Analyzed : 0 + IM Performance Candidates Identified : 0 + Statements Verified : 0 + IM Performance + Candidates Accepted : + ------------------------------------------------------------------------------- + + + + PL/SQL procedure successfully completed. + + SQL> + ``` + +## Conclusion + +This lab demonstrated how the new INMEMORY\_AUTOMATIC\_LEVEL = HIGH feature works and how AIM level high can enable the automatic management of the contents of IM column store. This means no more having to try and figure out which objects would get the most benefit from being populated. Now the database will do it for you. + +You may now **proceed to the next lab**. + +## Acknowledgements + +- **Author** - Andy Rivenes, Product Manager, Database In-Memory +- **Contributors** - +- **Last Updated By/Date** - Andy Rivenes, June 2024 diff --git a/in-memory-23ai/initialize-environment/images/check-db-service-up.png b/in-memory-23ai/initialize-environment/images/check-db-service-up.png new file mode 100644 index 000000000..cbec590b4 Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/check-db-service-up.png differ diff --git a/in-memory-23ai/initialize-environment/images/check-dblistner-service-up.png b/in-memory-23ai/initialize-environment/images/check-dblistner-service-up.png new file mode 100644 index 000000000..5bde89b44 Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/check-dblistner-service-up.png differ diff --git a/in-memory-23ai/initialize-environment/images/check-pmon-up.png b/in-memory-23ai/initialize-environment/images/check-pmon-up.png new file mode 100644 index 000000000..22fe17480 Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/check-pmon-up.png differ diff --git a/in-memory-23ai/initialize-environment/images/check-tns-up.png b/in-memory-23ai/initialize-environment/images/check-tns-up.png new file mode 100644 index 000000000..347ca6f83 Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/check-tns-up.png differ diff --git a/in-memory-23ai/initialize-environment/images/init-inmemory.png b/in-memory-23ai/initialize-environment/images/init-inmemory.png new file mode 100644 index 000000000..794f7140e Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/init-inmemory.png differ diff --git a/in-memory-23ai/initialize-environment/images/launch-sqldeveloper.png b/in-memory-23ai/initialize-environment/images/launch-sqldeveloper.png new file mode 100644 index 000000000..46bf233b4 Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/launch-sqldeveloper.png differ diff --git a/in-memory-23ai/initialize-environment/images/remote-desktop-landing.png b/in-memory-23ai/initialize-environment/images/remote-desktop-landing.png new file mode 100644 index 000000000..b7e0c916f Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/remote-desktop-landing.png differ diff --git a/in-memory-23ai/initialize-environment/images/test-database-connections.png b/in-memory-23ai/initialize-environment/images/test-database-connections.png new file mode 100644 index 000000000..660853dfa Binary files /dev/null and b/in-memory-23ai/initialize-environment/images/test-database-connections.png differ diff --git a/in-memory-23ai/initialize-environment/initialize-environment.md b/in-memory-23ai/initialize-environment/initialize-environment.md new file mode 100644 index 000000000..d094646d9 --- /dev/null +++ b/in-memory-23ai/initialize-environment/initialize-environment.md @@ -0,0 +1,320 @@ +# Initialize Environment + +## Introduction + +In this lab we will review and startup all components required to successfully run this workshop. + +*Estimated Lab Time:* 15 Minutes. + +### Video Preview +Watch the video below to get an explanation of enabling the In-Memory column store. + +[Youtube video](youtube:dZ9cnIL6KKw) + +Watch the video below for a walk through of the lab. +[Youtube video](youtube:7rbgF8Z6hc4) + + +### Objectives +- Initialize the workshop environment. + +### Prerequisites +This lab assumes you have: +- A Free Tier, Paid or LiveLabs Oracle Cloud account +- You have completed: + - Lab: Prepare Setup (*Free-tier* and *Paid Tenants* only) + - Lab: Environment Setup + +**NOTE:** *When doing Copy/Paste using the convenient* **Copy** *function used throughout the guide, you must hit the* **ENTER** *key after pasting. Otherwise the last line will remain in the buffer until you hit* **ENTER!** + +## Task 1: Validate That Required Processes are Up and Running. + +1. Now with access to your remote desktop session, proceed as indicated below to validate your environment before you start executing the subsequent labs. The following Processes should be up and running: + + - Database Listeners + - LISTENER (1521) + - LISTCDB2 (1522) + - Database Server Instances + - CDB1 + - CDB2 + + ![In-Memory landing page](./images/remote-desktop-landing.png " ") + +2. Click the *Terminal* icon on the desktop to launch a session, then run the following to validate that expected processes are up. + + ``` + + ps -ef|grep LIST|grep -v grep + ps -ef|grep ora_|grep pmon|grep -v grep + systemctl status oracle-database oracle-db-listener + + ``` + + ![Check PMON Database process status](./images/check-pmon-up.png "check PMON Database process status") + ![Check database service status](./images/check-db-service-up.png "Check database service status") + ![Check DB listener service status](./images/check-dblistner-service-up.png "Check DB listener service status") + + If all expected processes are shown in your output as seen above, then your environment is ready for the next task. + +3. If you see questionable output(s), failure or down component(s), refer to the appendix section to restart the service accordingly + +4. Follow the (3) steps shown below to launch *SQL Developer*: + + ![Launch SQL Developer](./images/launch-sqldeveloper.png "Launch SQL Developer") + +5. Test database connectivity by clicking on the *+* sign next to each CDB or PDB listed as shown below + + ![Test Database Connections](./images/test-database-connections.png "Test Database Connections") + + +## Task 2: Initialize Database for In-Memory Use Cases + +1. From your remote desktop session as user *oracle*, run the block below + + ``` + + clear + cd ~oracle/labs + rm -rf ~oracle/labs/* + wget -O novnc-inmemory-21c.zip https://c4u04.objectstorage.us-ashburn-1.oci.customer-oci.com/p/EcTjWk2IuZPZeNnD_fYMcgUhdNDIDA6rt9gaFj_WZMiL7VvxPBNMY60837hu5hga/n/c4u04/b/livelabsfiles/o/labfiles/novnc-inmemory-21c.zip + unzip -qo novnc-inmemory-21c.zip + rm -f novnc-inmemory-21c.zip + cd inmemory + ls -ltrh + + ``` + + ![query image](./images/init-inmemory.png " ") + +2. This workshop will use CDB1 exclusively. As a result, run the following to disable *CDB2* from auto-startup and shut it down. + + ``` + + sudo systemctl stop oracle-database + sudo sed -i -e 's|CDB2.*$|CDB2:/opt/oracle/product/21c/dbhome_1:N|g' /etc/oratab + sudo systemctl start oracle-database + + ``` + +3. Confirm that only **CDB1** database is running on the host. + + ``` + + ps -ef|grep ora_|grep pmon|grep -v grep + + ``` + +## Task 3: Enable In-Memory + +1. Set your oracle environment and connect to **CDB1** database using SQLcl. + + Run the commands below + + ``` + . ~/.set-env-db.sh CDB1 + ``` + + ``` + + sql / as sysdba + + ``` + + Output: + ``` + [CDB1:oracle@dbhol:~]$ . ~/.set-env-db.sh CDB1 + ================================================================================ + ___ _ _ _ _ _ + / _ \ _ __ __ _ ___| | ___ | | (_)_ _____| | __ _| |__ ___ + | | | | '__/ _` |/ __| |/ _ \ | | | \ \ / / _ \ | / _` | '_ \/ __| + | |_| | | | (_| | (__| | __/ | |___| |\ V / __/ |__| (_| | |_) \__ \ + \___/|_| \__,_|\___|_|\___| |_____|_| \_/ \___|_____\__,_|_.__/|___/ + + ================================================================================ + ENV VARIABLES + -------------------------------------------------------------------------------- + . ORACLE_BASE = /opt/oracle + . ORACLE_BASE_HOME = /opt/oracle/homes/OraDBHome21cEE + . ORACLE_HOME = /opt/oracle/product/21c/dbhome_1 + . ORACLE_SID = CDB1 + . PRIVATE_IP = 10.0.0.54 + . PUBLIC_IP = xxx.xxx.88.238 + . HOSTNAME = dbhol.livelabs.oraclevcn.com + -------------------------------------------------------------------------------- + Database ENV set for CDB1 + + Run this to reload/setup the Database ENV: source /usr/local/bin/.set-env-db.sh + -------------------------------------------------------------------------------- + ================================================================================ + + [CDB1:oracle@dbhol:~]$ sql / as sysdba + + SQLcl: Release 21.2 Production on Wed Oct 05 02:50:39 2022 + + Copyright (c) 1982, 2022, Oracle. All rights reserved. + + Connected to: + Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production + Version 21.7.0.0.0 + + SQL> + ``` +2. Update **CDB1** database system parameters + + Run the commands below + + ``` + + alter system set heat_map=ON scope=spfile; + alter system set sga_max_size=8G scope=spfile; + alter system set sga_target=8G scope=spfile; + alter system set db_keep_cache_size=3000M scope=spfile; + alter system set pga_aggregate_target=2500M scope=spfile; + alter system set inmemory_size=3300M scope=spfile; + alter system set inmemory_max_populate_servers=4 scope=spfile; + alter system set inmemory_virtual_columns=enable scope=spfile; + alter system set "_inmemory_64k_percent"=5 scope=spfile; + alter system set "_inmemory_small_segment_threshold"=0 scope=spfile; + alter system set "_optimizer_use_feedback"=FALSE scope=spfile; + alter system set "_imado_enable_coloptim"=FALSE scope=spfile; + + ``` + + Output: + + ``` + SQL> alter system set heat_map=ON scope=spfile; + 2 alter system set sga_max_size=8G scope=spfile; + 3 alter system set sga_target=8G scope=spfile; + 4 alter system set db_keep_cache_size=3000M scope=spfile; + 5 alter system set pga_aggregate_target=2500M scope=spfile; + 6 alter system set inmemory_size=3300M scope=spfile; + 7 alter system set inmemory_max_populate_servers=4 scope=spfile; + 8 alter system set inmemory_virtual_columns=enable scope=spfile; + 9 alter system set "_inmemory_64k_percent"=5 scope=spfile; + 10 alter system set "_inmemory_small_segment_threshold"=0 scope=spfile; + 11 alter system set "_optimizer_use_feedback"=FALSE scope=spfile; + 12* alter system set "_imado_enable_coloptim"=FALSE scope=spfile; + + System SET altered. + System SET altered. + System SET altered. + System SET altered. + System SET altered. + System SET altered. + System SET altered. + System SET altered. + System SET altered. + ``` + +3. Restart **CDB1** database + + Run the commands below + + ``` + + shutdown immediate + startup + exit + + ``` + + Output: + + ``` + SQL> shutdown immediate + 2 startup + 3* exit + Database closed. + Database dismounted. + ORACLE instance shut down. + ORACLE instance started. + + Total System Global Area 8589933480 bytes + Fixed Size 9706408 bytes + Variable Size 553648128 bytes + Database Buffers 4546625536 bytes + Redo Buffers 7069696 bytes + In-Memory Area 3472883712 bytes + Database mounted. + Database opened. + Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production + Version 21.7.0.0.0 + ``` + +You may now **proceed to the next lab**. + +## Appendix 1: Managing Startup Services + +The following is a list of the commands to start, stop and determine the status of each of the Lab database services. + +1. Database service (All databases and Standard Listener). + + - Start + + ``` + + sudo systemctl start oracle-database + + ``` + - Stop + + ``` + + sudo systemctl stop oracle-database + + ``` + + - Status + + ``` + + systemctl status oracle-database + + ``` + + - Restart + + ``` + + sudo systemctl restart oracle-database + + ``` + +2. Database service (Non-Standard Listeners). + + - Start + + ``` + + sudo systemctl start oracle-db-listener + + ``` + - Stop + + ``` + + sudo systemctl stop oracle-db-listener + + ``` + + - Status + + ``` + + systemctl status oracle-db-listener + + ``` + + - Restart + + ``` + + sudo systemctl restart oracle-db-listener + + ``` + +## Acknowledgements +* **Author** - Rene Fontcha, LiveLabs Platform Lead, NA Technology +* **Contributors** - Kay Malcolm, Didi Han, Andy Rivenes +* **Last Updated By/Date** - Rene Fontcha, LiveLabs Platform Lead, NA Technology, October 2022 diff --git a/in-memory-23ai/intro/images/DBIM.png b/in-memory-23ai/intro/images/DBIM.png new file mode 100644 index 000000000..e76700f3b Binary files /dev/null and b/in-memory-23ai/intro/images/DBIM.png differ diff --git a/in-memory-23ai/intro/images/arch.png b/in-memory-23ai/intro/images/arch.png new file mode 100644 index 000000000..b942eefc6 Binary files /dev/null and b/in-memory-23ai/intro/images/arch.png differ diff --git a/in-memory-23ai/intro/images/inmem.png b/in-memory-23ai/intro/images/inmem.png new file mode 100644 index 000000000..3b22ad56e Binary files /dev/null and b/in-memory-23ai/intro/images/inmem.png differ diff --git a/in-memory-23ai/intro/images/youtube.png b/in-memory-23ai/intro/images/youtube.png new file mode 100644 index 000000000..c5b2edcc6 Binary files /dev/null and b/in-memory-23ai/intro/images/youtube.png differ diff --git a/in-memory-23ai/intro/intro.md b/in-memory-23ai/intro/intro.md new file mode 100644 index 000000000..41e76cd00 --- /dev/null +++ b/in-memory-23ai/intro/intro.md @@ -0,0 +1,51 @@ +# Introduction + +## About this workshop + +*Database In-Memory* features a highly optimized In-Memory Column Store (IM column store) maintained alongside the existing row formatted buffer cache as depicted below. The primary purpose of the IM column store is to accelerate column-oriented data accesses made by analytics operations. + +It is similar in spirit to having a conventional index (for analytics) on every column in a table. However, it is much more lightweight than a conventional index, requiring no logging, or any writes to the database. Just as the performance benefit to an application from conventional indexes depends on the amount of time the application spends accessing data in the tables that are indexed, the benefit from the IM column store also depends on the amount of time the application spends on data access for analytic operations. + +*Estimated Workshop Time*: 90 minutes + +Watch the video below for an overview of Oracle In-Memory. + +[Youtube video](youtube:JGM1taVRZHs) + +If you would like to watch us do the workshop, click [here](https://youtu.be/QzCnO_Me97g). + +## Overview +### Database In-Memory and Performance + +There are four basic architectural elements of the column store that enable orders of magnitude faster analytic query processing: + +1. *Compressed columnar storage*: Storing data contiguously in compressed column units allows an analytic query to scan only data within the required columns, instead of having to skip past unneeded data in other columns as would be needed for a row major format. Columnar storage therefore allows a query to perform highly efficient sequential memory references while compression allows the query to optimize its use of the available system (processor to memory) bandwidth. + ![In-Memory dual format](./images/DBIM.png " ") + +2. *Vector Processing*: In addition to being able to process data sequentially, column organized storage also enables the use of vector processing. Modern CPUs feature highly parallel instructions known as SIMD or vector instructions, for example Intel AVX. These instructions can process multiple values in one instruction –for instance, they allow multiple values to be compared with a given value, for example find sales with State = “California”, in one instruction. Vector processing of compressed columnar data further multiplies the scan speed obtained with columnar storage, resulting in scan speeds exceeding tens of billions of rows per second, per CPU core. + +3. *In-Memory Storage Indexes*: The IM column store for a given table is divided into units known as In-Memory Compression Units(IMCUs) that represent many rows, typically several hundred thousand. Each IMCU automatically records the min and max values for the data within each column in the IMCU, as well as other summary information regarding the data. This metadata serves as an In-Memory Storage Index: For instance, it allows an entire IMCU to be skipped during a scan when it is known from the scan predicates that no matching value will be found within the IMCU. + +4. *In-Memory Optimized Joins and Reporting*: Because of massive increases in scan speeds, the Bloom filter optimization (introduced earlier in Oracle Database 10g) can be commonly selected by the optimizer. With the Bloom filter optimization, the scan of the outer (dimension) table generates a compact Bloom filter which can then be used to greatly reduce the amount of data processed by the join from the scan of the inner (fact) table. Similarly, an optimization known as Vector Group By can be used to reduce a complex aggregation query on a typical star schema into a series of filtered scans against the dimension and fact tables. + +### In-Memory Architecture + +The following figure shows a sample IM column store. The database stores the sh.sales table on disk in traditional row format. The SGA stores the data in columnar format in the IM column store, and in row format in the database buffer cache. + +![In-Memory architecture](./images/arch.png " ") + +## Learn More + +Database In-Memory Channel +[YouTube video](youtube:UCSYHgTG68nrHa5aTGfFH4pA) + +Oracle Database Product Management Videos on In-Memory +[YouTube video](youtube:UCr6mzwq_gcdsefQWBI72wIQ) + +You may now **proceed to the next lab**. + +## Acknowledgements + +- **Authors/Contributors** - Andy Rivenes, Product Manager, Database In-Memory +- **Contributors** - Kay Malcolm, Anoosha Pilli, Rene Fontcha +- **Last Updated By/Date** - Rene Fontcha, LiveLabs Platform Lead, NA Technology, October 2021 diff --git a/in-memory-23ai/joins-aggregations/joins-aggregations.md b/in-memory-23ai/joins-aggregations/joins-aggregations.md new file mode 100644 index 000000000..f11c77d9f --- /dev/null +++ b/in-memory-23ai/joins-aggregations/joins-aggregations.md @@ -0,0 +1,1494 @@ +# In-Memory Joins and Aggregations + +## Introduction + +Watch the video below for a quick walk-through of the In-memory Joins and Aggregations lab: + +[In-Memory Joins and Aggregations](videohub:1_gx8ajh93) + +*Estimated Lab Time:* 15 Minutes. + +### Objectives + +- Learn how to enable In-Memory on the Oracle Database +- Perform various queries on the In-Memory Column Store + +### Prerequisites + +This lab assumes you have: +- A Free Tier, Paid or LiveLabs Oracle Cloud account +- You have completed: + - Get Started with noVNC Remote Desktop + - Lab: Initialize Environment + - Lab: Setting up the In-Memory Column Store + +**NOTE:** *When doing Copy/Paste using the convenient* **Copy** *function used throughout the guide, you must hit the* **ENTER** *key after pasting. Otherwise the last line will remain in the buffer until you hit* **ENTER!** + +## Task 1: In-Memory Joins and Aggregation + +Up until now we have been focused on queries that scan only one table, the LINEORDER table. Let’s broaden the scope of our investigation to include joins and parallel execution. This section executes a series of queries that begin with a single join between the fact table, LINEORDER, and one or more dimension tables and works up to a 5 table join. The queries will be executed in both the buffer cache and the column store, to demonstrate the different ways the column store can improve query performance above and beyond just the basic performance benefits of scanning data in a columnar format. + +Reload the environment variables for **CDB1** if you exited the terminal after the previous lab + +``` +. ~/.set-env-db.sh CDB1 +``` + +Let's switch to the joins-aggr folder and log back in to the PDB: + +``` + +cd /home/oracle/labs/inmemory/joins-aggr +sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + +``` + +And adjust the sqlplus display: + +``` + +set pages 9999 +set lines 150 + +``` + +Query result: + +``` +[CDB1:oracle@dbhol:~/labs/inmemory]$ cd /home/oracle/labs/inmemory/joins-aggr +[CDB1:oracle@dbhol:~/labs/inmemory/joins-aggr]$ sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + +SQL*Plus: Release 23.0.0.0.0 - Production on Tue Jun 4 15:26:39 2024 +Version 23.4.0.24.05 + +Copyright (c) 1982, 2024, Oracle. All rights reserved. + +Last Successful login time: Thu Aug 18 2022 21:37:24 +00:00 + +Connected to: +Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production +Version 21.7.0.0.0 + +SQL> set pages 9999 +SQL> set lines 150 +SQL> +``` + +1. We will start by joining the LINEORDER and DATE\_DIM tables in a "What If" style query that calculates the amount of revenue increase that would have resulted from eliminating certain company-wide discounts in a given percentage range for products shipped on a given day (Christmas eve 1996). + + Run the script *01\_join\_im.sql* + + ``` + + @01_join_im.sql + + ``` + + or run the query below: + + ``` + + set timing on + SELECT SUM(lo_extendedprice * lo_discount) revenue + FROM lineorder l, date_dim d + WHERE l.lo_orderdate = d.d_datekey + AND l.lo_discount BETWEEN 2 AND 3 + AND l.lo_quantity < 24 + AND d.d_date='December 24, 1996'; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @01_join_im.sql + Connected. + SQL> + SQL> -- Demonstrate an in-memory join query + SQL> + SQL> Select sum(lo_extendedprice * lo_discount) revenue + 2 From LINEORDER l, DATE_DIM d + 3 Where l.lo_orderdate = d.d_datekey + 4 And l.lo_discount between 2 and 3 + 5 And l.lo_quantity < 24 + 6 And d.d_date='December 24, 1996'; + + REVENUE + ---------------- + 9710699495 + + Elapsed: 00:00:00.04 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID b2jysvyzbss5p, child number 0 + ------------------------------------- + Select sum(lo_extendedprice * lo_discount) revenue From LINEORDER l, + DATE_DIM d Where l.lo_orderdate = d.d_datekey And l.lo_discount + between 2 and 3 And l.lo_quantity < 24 And d.d_date='December 24, + 1996' + + Plan hash value: 2500197027 + + ------------------------------------------------------------------------------------------------------------ + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + ------------------------------------------------------------------------------------------------------------ + | 0 | SELECT STATEMENT | | | | 4058 (100)| | | | + | 1 | SORT AGGREGATE | | 1 | 47 | | | | | + |* 2 | HASH JOIN | | 2085 | 97995 | 4058 (21)| 00:00:01 | | | + | 3 | JOIN FILTER CREATE | :BF0001 | 1 | 27 | 1 (0)| 00:00:01 | | | + | 4 | PART JOIN FILTER CREATE | :BF0000 | 1 | 27 | 1 (0)| 00:00:01 | | | + |* 5 | TABLE ACCESS INMEMORY FULL| DATE_DIM | 1 | 27 | 1 (0)| 00:00:01 | | | + | 6 | JOIN FILTER USE | :BF0001 | 3492K| 66M| 4045 (21)| 00:00:01 | | | + | 7 | PARTITION RANGE JOIN-FILTER| | 3492K| 66M| 4045 (21)| 00:00:01 |:BF0000|:BF0000| + |* 8 | TABLE ACCESS INMEMORY FULL| LINEORDER | 3492K| 66M| 4045 (21)| 00:00:01 |:BF0000|:BF0000| + ------------------------------------------------------------------------------------------------------------ + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + 5 - inmemory("D"."D_DATE"='December 24, 1996') + filter("D"."D_DATE"='December 24, 1996') + 8 - inmemory(("L"."LO_DISCOUNT"<=3 AND "L"."LO_QUANTITY"<24 AND "L"."LO_DISCOUNT">=2 AND + SYS_OP_BLOOM_FILTER(:BF0001,"L"."LO_ORDERDATE"))) + filter(("L"."LO_DISCOUNT"<=3 AND "L"."LO_QUANTITY"<24 AND "L"."LO_DISCOUNT">=2 AND + SYS_OP_BLOOM_FILTER(:BF0001,"L"."LO_ORDERDATE"))) + + + 34 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 5 + IM scan CUs columns accessed 70 + IM scan CUs pcode aggregation pushdown 17 + IM scan rows 9128918 + IM scan rows pcode aggregated 2131 + IM scan rows valid 9128918 + IM scan segments minmax eligible 18 + physical reads 5 + session logical reads 69402 + session logical reads - IM 69288 + session pga memory 17897656 + table scans (IM) 2 + + 12 rows selected. + + SQL> + ``` + + Database In-Memory has no problem executing a query with a join, and in fact can optimize hash joins by being able to take advantage of Bloom filters. It’s easy to identify Bloom filters in the execution plan. They will appear in two places, at creation time (i.e. JOIN FILTER CREATE) and again when they are applied (i.e. JOIN FILTER USE). Look at Id 3 and Id 6 in the plan above. You can also see what join condition was used to build the Bloom filter by looking at the predicate information under the plan. + + You may also note that there is another use of a Bloom filter at line Id 4. This is not a Database In-Memory feature but simply Oracle Database optimizing access to the partitions that make up the LINEORDER table. A takeaway here is that the use of Database In-Memory does not prevent the use of the other Oracle Database features. + +2. Let's run the query using the buffer cache. + + Run the script *02\_join\_buffer.sql* + + ``` + + @02_join_buffer.sql + + ``` + + or run the query below: + + ``` + + set timing on + select /*+ NO_INMEMORY NO_VECTOR_TRANSFORM */ + sum(lo_extendedprice * lo_discount) revenue + from LINEORDER l, DATE_DIM d + where l.lo_orderdate = d.d_datekey + and l.lo_discount between 2 and 3 + and l.lo_quantity < 24 + and d.d_date='December 24, 1996'; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @02_join_buffer.sql + Connected. + SQL> + SQL> -- Buffer Cache query with the column store disabled using a NO_INMEMORY hint + SQL> -- + SQL> + SQL> SELECT /*+ NO_INMEMORY NO_VECTOR_TRANSFORM */ + 2 SUM(lo_extendedprice * lo_discount) revenue + 3 FROM lineorder l, + 4 date_dim d + 5 WHERE l.lo_orderdate = d.d_datekey + 6 AND d.d_date='December 24, 1996'; + + REVENUE + ---------------- + 467915827233 + + Elapsed: 00:00:00.85 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 8xkbya444k6g5, child number 0 + ------------------------------------- + SELECT /*+ NO_INMEMORY NO_VECTOR_TRANSFORM */ + SUM(lo_extendedprice * lo_discount) revenue FROM lineorder l, + date_dim d WHERE l.lo_orderdate = d.d_datekey AND + d.d_date='December 24, 1996' + + Plan hash value: 2848862642 + + ----------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + ----------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 86646 (100)| | | | + | 1 | SORT AGGREGATE | | 1 | 44 | | | | | + |* 2 | HASH JOIN | | 24932 | 1071K| 86646 (1)| 00:00:04 | | | + | 3 | PART JOIN FILTER CREATE | :BF0000 | 1 | 27 | 7 (0)| 00:00:01 | | | + |* 4 | TABLE ACCESS FULL | DATE_DIM | 1 | 27 | 7 (0)| 00:00:01 | | | + | 5 | PARTITION RANGE JOIN-FILTER| | 41M| 677M| 86498 (1)| 00:00:04 |:BF0000|:BF0000| + | 6 | TABLE ACCESS FULL | LINEORDER | 41M| 677M| 86498 (1)| 00:00:04 |:BF0000|:BF0000| + ----------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + 4 - filter("D"."D_DATE"='December 24, 1996') + + + 27 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 89 + IM scan segments disk 2 + session logical reads 69104 + session pga memory 19204344 + + SQL> + ``` + + You might notice that we added a hint to specify NO\_INMEMORY. This is an easy way to tell the optimizer to not use the IM column store. You might also notice that there is a NO\_VECTOR\_TRANSFORM hint as well. Vector transformation is available when Database In-Memory is enabled, and we will cover its advantages later in this Lab. For now, we have disabled it to make it easier to compare this execution plan with the execution plan from the previous step. + +3. Now let’s try a more complex query that encompasses three joins and an aggregation. This time our query will compare the revenue for different product classes, from suppliers in a certain region for the year 1997. + + Run the script *03\_3join\_im.sql* + + ``` + + @03_3join_im.sql + + ``` + + or run the query below: + + ``` + + set timing on + SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, p.p_brand1,SUM(lo_revenue) rev + FROM lineorder l, + date_dim d, + part p, + supplier s + WHERE l.lo_orderdate = d.d_datekey + AND l.lo_partkey = p.p_partkey + AND l.lo_suppkey = s.s_suppkey + AND p.p_category = 'MFGR#12' + AND s.s_region = 'AMERICA' + AND d.d_year = 1997 + GROUP BY d.d_year,p.p_brand1; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @03_3join_im.sql + Connected. + SQL> + SQL> -- Demonstrate an in-memory join query + SQL> + SQL> SELECT /*+ NO_VECTOR_TRANSFORM */ + 2 d.d_year, p.p_brand1,SUM(lo_revenue) rev + 3 FROM lineorder l, + 4 date_dim d, + 5 part p, + 6 supplier s + 7 WHERE l.lo_orderdate = d.d_datekey + 8 AND l.lo_partkey = p.p_partkey + 9 AND l.lo_suppkey = s.s_suppkey + 10 AND p.p_category = 'MFGR#12' + 11 AND s.s_region = 'AMERICA' + 12 AND d.d_year = 1997 + 13 GROUP BY d.d_year,p.p_brand1; + + D_YEAR P_BRAND1 REV + ---------------- --------- ---------------- + 1997 MFGR#128 6597639363 + 1997 MFGR#126 6213627043 + 1997 MFGR#1214 6630127600 + 1997 MFGR#1234 6695984533 + 1997 MFGR#122 7101221796 + 1997 MFGR#1217 7053041453 + 1997 MFGR#1221 6892277556 + 1997 MFGR#127 6765391794 + 1997 MFGR#1230 6596531127 + 1997 MFGR#1211 6852509575 + 1997 MFGR#1225 6804217225 + 1997 MFGR#1231 6839363437 + 1997 MFGR#1213 6686343443 + 1997 MFGR#1232 7404918843 + 1997 MFGR#1227 6713851455 + 1997 MFGR#1220 6613283998 + 1997 MFGR#1219 6651946261 + 1997 MFGR#1237 7041724061 + 1997 MFGR#1218 6841323272 + 1997 MFGR#1210 6795372926 + 1997 MFGR#123 6280463233 + 1997 MFGR#121 6409702180 + 1997 MFGR#1240 7056019394 + 1997 MFGR#1215 7079477060 + 1997 MFGR#124 6817087386 + 1997 MFGR#1224 7373166413 + 1997 MFGR#1212 6573951551 + 1997 MFGR#1233 6914572704 + 1997 MFGR#129 6164317597 + 1997 MFGR#1223 6465041111 + 1997 MFGR#1236 6827320374 + 1997 MFGR#1238 6443513085 + 1997 MFGR#1228 7038686432 + 1997 MFGR#1235 7075150948 + 1997 MFGR#1239 6227318243 + 1997 MFGR#1216 6877592440 + 1997 MFGR#125 6416763567 + 1997 MFGR#1229 6481227038 + 1997 MFGR#1222 6551372899 + 1997 MFGR#1226 7185780867 + + 40 rows selected. + + Elapsed: 00:00:00.24 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 6ygn2wqbv5syt, child number 0 + ------------------------------------- + SELECT /*+ NO_VECTOR_TRANSFORM */ d.d_year, + p.p_brand1,SUM(lo_revenue) rev FROM lineorder l, date_dim d, + part p, supplier s WHERE l.lo_orderdate = d.d_datekey AND + l.lo_partkey = p.p_partkey AND l.lo_suppkey = s.s_suppkey AND + p.p_category = 'MFGR#12' AND s.s_region = 'AMERICA' AND + d.d_year = 1997 GROUP BY d.d_year,p.p_brand1 + + Plan hash value: 4224806115 + + ---------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + ---------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 4140 (100)| | | | + | 1 | HASH GROUP BY | | 1000 | 77000 | 4140 (21)| 00:00:01 | | | + |* 2 | HASH JOIN | | 98430 | 7401K| 4136 (21)| 00:00:01 | | | + | 3 | JOIN FILTER CREATE | :BF0001 | 365 | 4380 | 1 (0)| 00:00:01 | | | + | 4 | PART JOIN FILTER CREATE | :BF0000 | 365 | 4380 | 1 (0)| 00:00:01 | | | + |* 5 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 365 | 4380 | 1 (0)| 00:00:01 | | | + |* 6 | HASH JOIN | | 451K| 28M| 4134 (21)| 00:00:01 | | | + | 7 | JOIN FILTER CREATE | :BF0002 | 4102 | 73836 | 3 (0)| 00:00:01 | | | + |* 8 | TABLE ACCESS INMEMORY FULL | SUPPLIER | 4102 | 73836 | 3 (0)| 00:00:01 | | | + |* 9 | HASH JOIN | | 2216K| 99M| 4123 (21)| 00:00:01 | | | + | 10 | JOIN FILTER CREATE | :BF0003 | 31882 | 716K| 87 (19)| 00:00:01 | | | + |* 11 | TABLE ACCESS INMEMORY FULL | PART | 31882 | 716K| 87 (19)| 00:00:01 | | | + | 12 | JOIN FILTER USE | :BF0001 | 41M| 955M| 3895 (18)| 00:00:01 | | | + | 13 | JOIN FILTER USE | :BF0002 | 41M| 955M| 3895 (18)| 00:00:01 | | | + | 14 | JOIN FILTER USE | :BF0003 | 41M| 955M| 3895 (18)| 00:00:01 | | | + | 15 | PARTITION RANGE JOIN-FILTER| | 41M| 955M| 3895 (18)| 00:00:01 |:BF0000|:BF0000| + |* 16 | TABLE ACCESS INMEMORY FULL| LINEORDER | 41M| 955M| 3895 (18)| 00:00:01 |:BF0000|:BF0000| + ---------------------------------------------------------------------------------------------------------------- + + Outline Data + ------------- + + /*+ + BEGIN_OUTLINE_DATA + IGNORE_OPTIM_EMBEDDED_HINTS + OPTIMIZER_FEATURES_ENABLE('23.1.0') + DB_VERSION('23.1.0') + OPT_PARAM('_optimizer_use_feedback' 'false') + ALL_ROWS + OUTLINE_LEAF(@"SEL$1") + FULL(@"SEL$1" "P"@"SEL$1") + FULL(@"SEL$1" "L"@"SEL$1") + FULL(@"SEL$1" "S"@"SEL$1") + FULL(@"SEL$1" "D"@"SEL$1") + LEADING(@"SEL$1" "P"@"SEL$1" "L"@"SEL$1" "S"@"SEL$1" "D"@"SEL$1") + USE_HASH(@"SEL$1" "L"@"SEL$1") + USE_HASH(@"SEL$1" "S"@"SEL$1") + USE_HASH(@"SEL$1" "D"@"SEL$1") + PX_JOIN_FILTER(@"SEL$1" "L"@"SEL$1") + PX_JOIN_FILTER(@"SEL$1" "S"@"SEL$1") + PX_JOIN_FILTER(@"SEL$1" "D"@"SEL$1") + SWAP_JOIN_INPUTS(@"SEL$1" "S"@"SEL$1") + SWAP_JOIN_INPUTS(@"SEL$1" "D"@"SEL$1") + USE_HASH_AGGREGATION(@"SEL$1" GROUP_BY) + END_OUTLINE_DATA + */ + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + 5 - inmemory("D"."D_YEAR"=1997) + filter("D"."D_YEAR"=1997) + 6 - access("L"."LO_SUPPKEY"="S"."S_SUPPKEY") + 8 - inmemory("S"."S_REGION"='AMERICA') + filter("S"."S_REGION"='AMERICA') + 9 - access("L"."LO_PARTKEY"="P"."P_PARTKEY") + 11 - inmemory("P"."P_CATEGORY"='MFGR#12') + filter("P"."P_CATEGORY"='MFGR#12') + 16 - inmemory(SYS_OP_BLOOM_FILTER_LIST(SYS_OP_BLOOM_FILTER(:BF0003,"L"."LO_PARTKEY"),SYS_OP_BLOOM_FILT + ER(:BF0002,"L"."LO_SUPPKEY"),SYS_OP_BLOOM_FILTER(:BF0001,"L"."LO_ORDERDATE"))) + filter(SYS_OP_BLOOM_FILTER_LIST(SYS_OP_BLOOM_FILTER(:BF0003,"L"."LO_PARTKEY"),SYS_OP_BLOOM_FILTER + (:BF0002,"L"."LO_SUPPKEY"),SYS_OP_BLOOM_FILTER(:BF0001,"L"."LO_ORDERDATE"))) + + 78 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 8 + IM scan CUs columns accessed 78 + IM scan CUs pcode aggregation pushdown 17 + IM scan rows 9920979 + IM scan rows pcode aggregated 74532 + IM scan rows valid 9920979 + IM scan segments minmax eligible 21 + physical reads 14 + session logical reads 76542 + session logical reads - IM 76058 + session pga memory 19208376 + table scans (IM) 4 + + 12 rows selected. + + SQL> + ``` + + In this query, three Bloom filters have been created and applied to the scan of the LINEORDER table, one for the join to the DATE\_DIM table, one for the join to the PART table, and one for the join to the SUPPLIER table. How is Oracle able to apply three Bloom filters when normally a join only involves two tables at a time? + + This is where Oracle’s 30 plus years of database innovation kick in. By embedding the column store into Oracle Database we can take advantage of all of the optimizations that have been added to the database. In this case, the Optimizer has switched from its typical left deep tree execution to a right deep tree execution plan using an optimization called ‘swap\_join\_inputs’. You can see this in the Outline Data above. What this means for the IM column store is that we are able to generate multiple Bloom filters by scanning the three "dimension" tables before we scan the necessary columns in the "fact" table, meaning we are able to benefit by eliminating rows during the scan rather than waiting for the join to do it. + + +4. Now let’s execute the same query using the buffer cache. + + Run the script *04\_3join\_buffer.sql* + + ``` + + @04_3join_buffer.sql + + ``` + + or run the query below: + + ``` + + set timing on + alter session set inmemory_query = disable; + SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, p.p_brand1,SUM(lo_revenue) rev + FROM lineorder l, + date_dim d, + part p, + supplier s + WHERE l.lo_orderdate = d.d_datekey + AND l.lo_partkey = p.p_partkey + AND l.lo_suppkey = s.s_suppkey + AND p.p_category = 'MFGR#12' + AND s.s_region = 'AMERICA' + AND d.d_year = 1997 + GROUP BY d.d_year,p.p_brand1; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @04_3join_buffer.sql + Connected. + SQL> + SQL> alter session set inmemory_query = disable; + + Session altered. + + Elapsed: 00:00:00.00 + SQL> + SQL> -- Demonstrate a buffer cache three way join query + SQL> + SQL> SELECT /*+ NO_VECTOR_TRANSFORM */ + 2 d.d_year, p.p_brand1,SUM(lo_revenue) rev + 3 FROM lineorder l, + 4 date_dim d, + 5 part p, + 6 supplier s + 7 WHERE l.lo_orderdate = d.d_datekey + 8 AND l.lo_partkey = p.p_partkey + 9 AND l.lo_suppkey = s.s_suppkey + 10 AND p.p_category = 'MFGR#12' + 11 AND s.s_region = 'AMERICA' + 12 AND d.d_year = 1997 + 13 GROUP BY d.d_year,p.p_brand1; + + D_YEAR P_BRAND1 REV + ---------------- --------- ---------------- + 1997 MFGR#128 6597639363 + 1997 MFGR#126 6213627043 + 1997 MFGR#1214 6630127600 + 1997 MFGR#1234 6695984533 + 1997 MFGR#122 7101221796 + 1997 MFGR#1217 7053041453 + 1997 MFGR#1221 6892277556 + 1997 MFGR#127 6765391794 + 1997 MFGR#1230 6596531127 + 1997 MFGR#1211 6852509575 + 1997 MFGR#1225 6804217225 + 1997 MFGR#1231 6839363437 + 1997 MFGR#1213 6686343443 + 1997 MFGR#1232 7404918843 + 1997 MFGR#1227 6713851455 + 1997 MFGR#1220 6613283998 + 1997 MFGR#1219 6651946261 + 1997 MFGR#1237 7041724061 + 1997 MFGR#1218 6841323272 + 1997 MFGR#1210 6795372926 + 1997 MFGR#123 6280463233 + 1997 MFGR#121 6409702180 + 1997 MFGR#1240 7056019394 + 1997 MFGR#1215 7079477060 + 1997 MFGR#124 6817087386 + 1997 MFGR#1224 7373166413 + 1997 MFGR#1212 6573951551 + 1997 MFGR#1233 6914572704 + 1997 MFGR#129 6164317597 + 1997 MFGR#1223 6465041111 + 1997 MFGR#1236 6827320374 + 1997 MFGR#1238 6443513085 + 1997 MFGR#1228 7038686432 + 1997 MFGR#1235 7075150948 + 1997 MFGR#1239 6227318243 + 1997 MFGR#1216 6877592440 + 1997 MFGR#125 6416763567 + 1997 MFGR#1229 6481227038 + 1997 MFGR#1222 6551372899 + 1997 MFGR#1226 7185780867 + + 40 rows selected. + + Elapsed: 00:00:01.26 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 6ygn2wqbv5syt, child number 1 + ------------------------------------- + SELECT /*+ NO_VECTOR_TRANSFORM */ d.d_year, + p.p_brand1,SUM(lo_revenue) rev FROM lineorder l, date_dim d, + part p, supplier s WHERE l.lo_orderdate = d.d_datekey AND + l.lo_partkey = p.p_partkey AND l.lo_suppkey = s.s_suppkey AND + p.p_category = 'MFGR#12' AND s.s_region = 'AMERICA' AND + d.d_year = 1997 GROUP BY d.d_year,p.p_brand1 + + Plan hash value: 3074123985 + + ------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + ------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 88663 (100)| | | | + | 1 | HASH GROUP BY | | 1000 | 77000 | 88663 (1)| 00:00:04 | | | + |* 2 | HASH JOIN | | 98430 | 7401K| 88659 (1)| 00:00:04 | | | + | 3 | PART JOIN FILTER CREATE | :BF0000 | 365 | 4380 | 7 (0)| 00:00:01 | | | + |* 4 | TABLE ACCESS FULL | DATE_DIM | 365 | 4380 | 7 (0)| 00:00:01 | | | + |* 5 | HASH JOIN | | 451K| 28M| 88651 (1)| 00:00:04 | | | + |* 6 | TABLE ACCESS FULL | SUPPLIER | 4102 | 73836 | 69 (0)| 00:00:01 | | | + |* 7 | HASH JOIN | | 2216K| 99M| 88574 (1)| 00:00:04 | | | + |* 8 | TABLE ACCESS FULL | PART | 31882 | 716K| 1906 (1)| 00:00:01 | | | + | 9 | PARTITION RANGE JOIN-FILTER| | 41M| 955M| 86526 (1)| 00:00:04 |:BF0000|:BF0000| + | 10 | TABLE ACCESS FULL | LINEORDER | 41M| 955M| 86526 (1)| 00:00:04 |:BF0000|:BF0000| + ------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + 4 - filter("D"."D_YEAR"=1997) + 5 - access("L"."LO_SUPPKEY"="S"."S_SUPPKEY") + 6 - filter("S"."S_REGION"='AMERICA') + 7 - access("L"."LO_PARTKEY"="P"."P_PARTKEY") + 8 - filter("P"."P_CATEGORY"='MFGR#12') + + + 37 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 144 + IM scan segments disk 4 + session logical reads 78647 + session pga memory 20318456 + + SQL> + ``` + + As you can see, the IM column store continues to out-perform the buffer cache query. + +5. Up until this point we have only seen hash joins used with our in-memory queries. While it is true that hash joins are a further optimization with Database In-Memory and it's ability to use Bloom filters to effectively perform the join as a scan and filter operation, but what about a nested loops join? Is it possible for Database In-Memory to work with a nested loops join? Perhaps one table is not in memory or an index access will have less cost. Let's see how that might work. + + Run the script *05\_join\_nl\_im.sql* + + ``` + + @05_join_nl_im.sql + + ``` + + or run the query below: + + ``` + + set timing on + -- Enable the use of invisible indexes + alter session set optimizer_use_invisible_indexes=true; + -- Execute query + select /*+ NO_VECTOR_TRANSFORM INDEX(l, lineorder_i1) */ + sum(lo_extendedprice * lo_discount) revenue + from LINEORDER l, DATE_DIM d + where l.lo_orderdate = d.d_datekey + and d.d_date='December 24, 1996'; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @05_join_nl_im.sql + Connected. + SQL> + SQL> -- Show the use of in-memory with a Nested Loops join + SQL> + SQL> -- Enable the use of invisible indexes + SQL> + SQL> alter session set optimizer_use_invisible_indexes=true; + + Session altered. + + SQL> + SQL> -- Execute query + SQL> + SQL> set timing on + SQL> + SQL> select /*+ NO_VECTOR_TRANSFORM INDEX(l, lineorder_i1) */ + 2 sum(lo_extendedprice * lo_discount) revenue + 3 from LINEORDER l, DATE_DIM d + 4 where l.lo_orderdate = d.d_datekey + 5 and d.d_date='December 24, 1996'; + + REVENUE + -------------------- + 467915827233 + + Elapsed: 00:00:00.07 + SQL> + SQL> set timing off + SQL> + SQL> pause Hit enter ... + Hit enter ... + + SQL> + SQL> select * from table(dbms_xplan.display_cursor()); + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 9qr4mbmr1jyx2, child number 0 + ------------------------------------- + select /*+ NO_VECTOR_TRANSFORM INDEX(l, lineorder_i1) */ + sum(lo_extendedprice * lo_discount) revenue from LINEORDER l, + DATE_DIM d where l.lo_orderdate = d.d_datekey and + d.d_date='December 24, 1996' + + Plan hash value: 1478637188 + + --------------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 6223 (100)| | | | + | 1 | SORT AGGREGATE | | 1 | 44 | | | | | + | 2 | NESTED LOOPS | | 24932 | 1071K| 6223 (1)| 00:00:01 | | | + | 3 | NESTED LOOPS | | 24932 | 1071K| 6223 (1)| 00:00:01 | | | + |* 4 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 1 | 27 | 1 (0)| 00:00:01 | | | + |* 5 | INDEX RANGE SCAN | LINEORDER_I1 | 24932 | | 80 (0)| 00:00:01 | | | + | 6 | TABLE ACCESS BY GLOBAL INDEX ROWID| LINEORDER | 24932 | 413K| 6222 (1)| 00:00:01 | ROWID | ROWID | + --------------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 4 - inmemory("D"."D_DATE"='December 24, 1996') + filter("D"."D_DATE"='December 24, 1996') + 5 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + + Note + ----- + - this is an adaptive plan + + + 32 rows selected. + + SQL> + SQL> alter session set optimizer_use_invisible_indexes=false; + + Session altered. + + SQL> + ``` + + Notice that we have told the optimizer that it can use invisible indexes and we have added an index hint that can be used on the LINEORDER table. This results in the optimizer choosing to perform a nested loops join by first accessing the DATE\_DIM table in-memory and then accessing the LINEORDER table through an index. In this example, the cost is lower to access the LINEORDER table in the IM column store, but we wanted to show you that it is possible for the optimizer to choose different join types when using in-memory. If you're adventerous you can edit the script and remove the index hint. Don't worry, this is your own environment so it won't affect anyone else. If you then run the query again you can compare the cost to accessing the data through the index with a nested loops join. The thing to remember is the ability of the optimizer to choose the lowest cost methods to run queries with or without accessing object(s) in-memory. + + +6. Up until this point we have been focused on joins and how the IM column store can execute them incredibly efficiently. Let’s now turn our attention to more OLAP style “What If” queries. In this case our query examines the yearly profits from a specific region and manufacturer over our complete data set. + + Oracle has introduced a new optimizer transformation, called vector transformation. This is also known as In-Memory Aggregation and results in a new group by method called Vector Group By. This transformation is a two-part process not dissimilar to that of star transformation. First, the dimension tables are scanned and any WHERE clause predicates are applied. A new data structure called a key vector is created based on the results of these scans. The key vector is similar to a Bloom filter as it allows the join predicates to be applied as additional filter predicates during the scan of the fact table, but it also enables us to conduct the group by or aggregation during the scan of the fact table instead of having to do it afterwards. + + The second part of the execution plan sees the results of the fact table scan being joined back to the temporary tables created as part of the scan of the dimension tables, that is defined by the lines that start with LOAD AS SELECT. These temporary tables contain the payload columns (columns needed in the select list) from the dimension table(s). In Release 12.2 and above these tables are now pure in-memory tables as evidenced by the addition of the (CURSOR DURATION MEMORY) phrase that is appended to the LOAD AS SELECT phrases. The combination of these two features dramatically improves the efficiency of a multiple table join with complex aggregations. Both features are visible in the execution plan of our queries. + + To see this in action execute the query *06\_vgb\_im.sql* + + ``` + + @06_vgb_im.sql + + ``` + + or run the query below: + + ``` + + set timing on + SELECT + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + from + LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + where + l.lo_orderdate = d.d_datekey + and l.lo_partkey = p.p_partkey + and l.lo_suppkey = s.s_suppkey + and l.lo_custkey = c.c_custkey + and s.s_region = 'AMERICA' + and c.c_region = 'AMERICA' + group by + d.d_year, c.c_nation + order by + d.d_year, c.c_nation; + set timing off + select * from table(dbms_xplan.display_cursor()); + + ``` + + Query result: + + ``` + SQL> @06_vgb_im.sql + Connected. + SQL> set numwidth 20 + SQL> + SQL> -- In-Memory query with In-Memory Aggregation enabled + SQL> + SQL> select d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + 2 from LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + 3 where l.lo_orderdate = d.d_datekey + 4 and l.lo_partkey = p.p_partkey + 5 and l.lo_suppkey = s.s_suppkey + 6 and l.lo_custkey = c.c_custkey + 7 and s.s_region = 'AMERICA' + 8 and c.c_region = 'AMERICA' + 9 group by d.d_year, c.c_nation + 10 order by d.d_year, c.c_nation; + + D_YEAR C_NATION PROFIT + -------------------- --------------- -------------------- + 1994 ARGENTINA 261149015641 + 1994 BRAZIL 263808033983 + 1994 CANADA 264598150413 + 1994 PERU 258595600981 + 1994 UNITED STATES 265282504206 + 1995 ARGENTINA 258498976118 + 1995 BRAZIL 269135848643 + 1995 CANADA 264654265482 + 1995 PERU 257451709833 + 1995 UNITED STATES 259660457396 + 1996 ARGENTINA 259361903850 + 1996 BRAZIL 265970119048 + 1996 CANADA 265333193889 + 1996 PERU 260916013039 + 1996 UNITED STATES 262339293224 + 1997 ARGENTINA 261099548066 + 1997 BRAZIL 266353055971 + 1997 CANADA 265036379243 + 1997 PERU 259114682243 + 1997 UNITED STATES 262208356128 + 1998 ARGENTINA 151054449013 + 1998 BRAZIL 153632348378 + 1998 CANADA 156899052279 + 1998 PERU 152297126350 + 1998 UNITED STATES 153937088695 + + 25 rows selected. + + Elapsed: 00:00:00.73 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID c3f4u823f89t3, child number 0 + ------------------------------------- + select d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + from LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C where + l.lo_orderdate = d.d_datekey and l.lo_partkey = p.p_partkey and + l.lo_suppkey = s.s_suppkey and l.lo_custkey = c.c_custkey and + s.s_region = 'AMERICA' and c.c_region = 'AMERICA' group by + d.d_year, c.c_nation order by d.d_year, c.c_nation + + Plan hash value: 3030148494 + + --------------------------------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 4336 (100)| | | | + | 1 | TEMP TABLE TRANSFORMATION | | | | | | | | + | 2 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D666C_2308E0F | | | | | | | + | 3 | HASH GROUP BY | | 7 | 112 | 2 (50)| 00:00:01 | | | + | 4 | KEY VECTOR CREATE BUFFERED | :KV0000 | 7 | 112 | 1 (0)| 00:00:01 | | | + | 5 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 2556 | 30672 | 1 (0)| 00:00:01 | | | + | 6 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D6669_2308E0F | | | | | | | + | 7 | HASH GROUP BY | | 1 | 9 | 101 (30)| 00:00:01 | | | + | 8 | KEY VECTOR CREATE BUFFERED | :KV0001 | 1 | 9 | 76 (7)| 00:00:01 | | | + | 9 | TABLE ACCESS INMEMORY FULL | PART | 800K| 3906K| 75 (6)| 00:00:01 | | | + | 10 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D666A_2308E0F | | | | | | | + | 11 | HASH GROUP BY | | 1 | 22 | 4 (25)| 00:00:01 | | | + | 12 | KEY VECTOR CREATE BUFFERED | :KV0002 | 1 | 22 | 3 (0)| 00:00:01 | | | + |* 13 | TABLE ACCESS INMEMORY FULL | SUPPLIER | 4102 | 73836 | 3 (0)| 00:00:01 | | | + | 14 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D666B_2308E0F | | | | | | | + | 15 | HASH GROUP BY | | 25 | 950 | 40 (23)| 00:00:01 | | | + | 16 | KEY VECTOR CREATE BUFFERED | :KV0003 | 25 | 950 | 37 (17)| 00:00:01 | | | + |* 17 | TABLE ACCESS INMEMORY FULL | CUSTOMER | 59761 | 1984K| 37 (17)| 00:00:01 | | | + | 18 | SORT GROUP BY | | 62 | 6510 | 4189 (24)| 00:00:01 | | | + |* 19 | HASH JOIN | | 62 | 6510 | 4188 (24)| 00:00:01 | | | + |* 20 | HASH JOIN | | 62 | 4712 | 4186 (24)| 00:00:01 | | | + | 21 | MERGE JOIN CARTESIAN | | 7 | 329 | 6 (0)| 00:00:01 | | | + | 22 | MERGE JOIN CARTESIAN | | 1 | 31 | 4 (0)| 00:00:01 | | | + | 23 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6669_2308E0F | 1 | 9 | 2 (0)| 00:00:01 | | | + | 24 | BUFFER SORT | | 1 | 22 | 2 (0)| 00:00:01 | | | + | 25 | TABLE ACCESS FULL | SYS_TEMP_0FD9D666A_2308E0F | 1 | 22 | 2 (0)| 00:00:01 | | | + | 26 | BUFFER SORT | | 7 | 112 | 4 (0)| 00:00:01 | | | + | 27 | TABLE ACCESS FULL | SYS_TEMP_0FD9D666C_2308E0F | 7 | 112 | 2 (0)| 00:00:01 | | | + | 28 | VIEW | VW_VT_80F21617 | 62 | 1798 | 4180 (24)| 00:00:01 | | | + | 29 | VECTOR GROUP BY | | 62 | 3100 | 4180 (24)| 00:00:01 | | | + | 30 | HASH GROUP BY | | 62 | 3100 | 4180 (24)| 00:00:01 | | | + | 31 | KEY VECTOR USE | :KV0000 | 2535K| 120M| 4178 (24)| 00:00:01 | | | + | 32 | KEY VECTOR USE | :KV0001 | 2535K| 111M| 4178 (24)| 00:00:01 | | | + | 33 | KEY VECTOR USE | :KV0003 | 2535K| 101M| 4178 (24)| 00:00:01 | | | + | 34 | KEY VECTOR USE | :KV0002 | 8510K| 308M| 4178 (24)| 00:00:01 | | | + | 35 | PARTITION RANGE ITERATOR | | 41M| 1354M| 4177 (24)| 00:00:01 |:KV0000|:KV0000| + |* 36 | TABLE ACCESS INMEMORY FULL | LINEORDER | 41M| 1354M| 4177 (24)| 00:00:01 |:KV0000|:KV0000| + | 37 | TABLE ACCESS FULL | SYS_TEMP_0FD9D666B_2308E0F | 25 | 725 | 2 (0)| 00:00:01 | | | + --------------------------------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 13 - inmemory("S"."S_REGION"='AMERICA') + filter("S"."S_REGION"='AMERICA') + 17 - inmemory("C"."C_REGION"='AMERICA') + filter("C"."C_REGION"='AMERICA') + 19 - access("ITEM_14"=INTERNAL_FUNCTION("C0")) + 20 - access("ITEM_12"=INTERNAL_FUNCTION("C0") AND "ITEM_13"=INTERNAL_FUNCTION("C0") AND "ITEM_15"=INTERNAL_FUNCTION("C0")) + 36 - inmemory((SYS_OP_KEY_VECTOR_FILTER("L"."LO_SUPPKEY",:KV0002) AND SYS_OP_KEY_VECTOR_FILTER("L"."LO_CUSTKEY",:KV0003) AND + SYS_OP_KEY_VECTOR_FILTER("L"."LO_PARTKEY",:KV0001) AND SYS_OP_KEY_VECTOR_FILTER("L"."LO_ORDERDATE",:KV0000))) + filter((SYS_OP_KEY_VECTOR_FILTER("L"."LO_SUPPKEY",:KV0002) AND SYS_OP_KEY_VECTOR_FILTER("L"."LO_CUSTKEY",:KV0003) AND + SYS_OP_KEY_VECTOR_FILTER("L"."LO_PARTKEY",:KV0001) AND SYS_OP_KEY_VECTOR_FILTER("L"."LO_ORDERDATE",:KV0000))) + + Note + ----- + - vector transformation used for this statement + + + 72 rows selected. + + SQL> + ``` + + Our query is more complex now and if you look closely at the execution plan you will see the creation and use of :KV000n structures which are the Key Vectors along with the Vector Group By operation. + +7. To see how dramatic the difference really is we can run the same query but we will disable the vector group by operation. + + Run the script *07\_novgb\_im.sql* + + ``` + + @07_novgb_im.sql + + ``` + + or run the query below: + + ``` + + set timing on + SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + from + LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + where + l.lo_orderdate = d.d_datekey + and l.lo_partkey = p.p_partkey + and l.lo_suppkey = s.s_suppkey + and l.lo_custkey = c.c_custkey + and s.s_region = 'AMERICA' + and c.c_region = 'AMERICA' + group by + d.d_year, c.c_nation + order by + d.d_year, c.c_nation; + set timing off + select * from table(dbms_xplan.display_cursor()); + + ``` + + Query result: + + ``` + SQL> @07_novgb_im.sql + Connected. + SQL> + SQL> -- In-Memory query with In-Memory Aggregation disabled + SQL> + SQL> SELECT /*+ NO_VECTOR_TRANSFORM */ + 2 d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + 3 From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + 4 Where l.lo_orderdate = d.d_datekey + 5 And l.lo_partkey = p.p_partkey + 6 And l.lo_suppkey = s.s_suppkey + 7 And l.lo_custkey = c.c_custkey + 8 And s.s_region = 'AMERICA' + 9 And c.c_region = 'AMERICA' + 10 Group by d.d_year, c.c_nation + 11 Order by d.d_year, c.c_nation; + + D_YEAR C_NATION PROFIT + -------------------- --------------- -------------------- + 1994 ARGENTINA 261149015641 + 1994 BRAZIL 263808033983 + 1994 CANADA 264598150413 + 1994 PERU 258595600981 + 1994 UNITED STATES 265282504206 + 1995 ARGENTINA 258498976118 + 1995 BRAZIL 269135848643 + 1995 CANADA 264654265482 + 1995 PERU 257451709833 + 1995 UNITED STATES 259660457396 + 1996 ARGENTINA 259361903850 + 1996 BRAZIL 265970119048 + 1996 CANADA 265333193889 + 1996 PERU 260916013039 + 1996 UNITED STATES 262339293224 + 1997 ARGENTINA 261099548066 + 1997 BRAZIL 266353055971 + 1997 CANADA 265036379243 + 1997 PERU 259114682243 + 1997 UNITED STATES 262208356128 + 1998 ARGENTINA 151054449013 + 1998 BRAZIL 153632348378 + 1998 CANADA 156899052279 + 1998 PERU 152297126350 + 1998 UNITED STATES 153937088695 + + 25 rows selected. + + Elapsed: 00:00:01.47 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID bgm86vz7ghjsj, child number 0 + ------------------------------------- + SELECT /*+ NO_VECTOR_TRANSFORM */ d.d_year, c.c_nation, + sum(lo_revenue - lo_supplycost) profit From LINEORDER l, + DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C Where + l.lo_orderdate = d.d_datekey And l.lo_partkey = + p.p_partkey And l.lo_suppkey = s.s_suppkey + And l.lo_custkey = c.c_custkey And + s.s_region = 'AMERICA' And c.c_region + = 'AMERICA' Group by d.d_year, c.c_nation + Order by d.d_year, c.c_nation + + Plan hash value: 2398965824 + + ------------------------------------------------------------------------------------------------------------------------ + | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop | + ------------------------------------------------------------------------------------------------------------------------ + | 0 | SELECT STATEMENT | | | | | 42874 (100)| | | | + | 1 | SORT GROUP BY | | 124 | 12772 | | 42874 (4)| 00:00:02 | | | + |* 2 | HASH JOIN | | 2535K| 249M| | 42791 (3)| 00:00:02 | | | + | 3 | PART JOIN FILTER CREATE | :BF0000 | 2556 | 30672 | | 1 (0)| 00:00:01 | | | + | 4 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 2556 | 30672 | | 1 (0)| 00:00:01 | | | + |* 5 | HASH JOIN | | 2535K| 220M| 12M| 42782 (3)| 00:00:02 | | | + | 6 | TABLE ACCESS INMEMORY FULL | PART | 800K| 3906K| | 75 (6)| 00:00:01 | | | + |* 7 | HASH JOIN | | 2535K| 207M| 2688K| 30295 (4)| 00:00:02 | | | + | 8 | JOIN FILTER CREATE | :BF0001 | 59761 | 1984K| | 37 (17)| 00:00:01 | | | + |* 9 | TABLE ACCESS INMEMORY FULL | CUSTOMER | 59761 | 1984K| | 37 (17)| 00:00:01 | | | + |* 10 | HASH JOIN | | 8510K| 422M| | 4321 (26)| 00:00:01 | | | + | 11 | JOIN FILTER CREATE | :BF0002 | 4102 | 73836 | | 3 (0)| 00:00:01 | | | + |* 12 | TABLE ACCESS INMEMORY FULL | SUPPLIER | 4102 | 73836 | | 3 (0)| 00:00:01 | | | + | 13 | JOIN FILTER USE | :BF0001 | 41M| 1354M| | 4177 (24)| 00:00:01 | | | + | 14 | JOIN FILTER USE | :BF0002 | 41M| 1354M| | 4177 (24)| 00:00:01 | | | + | 15 | PARTITION RANGE JOIN-FILTER| | 41M| 1354M| | 4177 (24)| 00:00:01 |:BF0000|:BF0000| + |* 16 | TABLE ACCESS INMEMORY FULL| LINEORDER | 41M| 1354M| | 4177 (24)| 00:00:01 |:BF0000|:BF0000| + ------------------------------------------------------------------------------------------------------------------------ + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + 5 - access("L"."LO_PARTKEY"="P"."P_PARTKEY") + 7 - access("L"."LO_CUSTKEY"="C"."C_CUSTKEY") + 9 - inmemory("C"."C_REGION"='AMERICA') + filter("C"."C_REGION"='AMERICA') + 10 - access("L"."LO_SUPPKEY"="S"."S_SUPPKEY") + 12 - inmemory("S"."S_REGION"='AMERICA') + filter("S"."S_REGION"='AMERICA') + 16 - inmemory(SYS_OP_BLOOM_FILTER_LIST(SYS_OP_BLOOM_FILTER(:BF0002,"L"."LO_SUPPKEY"),SYS_OP_BLOOM_FILTER(:BF00 + 01,"L"."LO_CUSTKEY"))) + filter(SYS_OP_BLOOM_FILTER_LIST(SYS_OP_BLOOM_FILTER(:BF0002,"L"."LO_SUPPKEY"),SYS_OP_BLOOM_FILTER(:BF0001 + ,"L"."LO_CUSTKEY"))) + + + 52 rows selected. + + SQL> + ``` + + Notice how much slower this second query ran even though it still ran in-memory, and even took advantage of Bloom filters. In this Lab our tables are pretty small, but as the tables grow in size the performance difference is typically much greater. This is why we say that you can normally expect at least a 3-8x performance improvement with In-Memory Aggregation. + +8. With Database In-Memory enabled the optimizer can even take advantage of vector transformation when the tables are not in-memory. To see this in action execute the same query against the buffer cache. + + Run the script *08\_vgb\_buffer.sql* + + ``` + + @08_vgb_buffer.sql + + ``` + + or run the query below: + + ``` + + set timing on + alter session set inmemory_query = disable; + SELECT + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + from + LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + where + l.lo_orderdate = d.d_datekey + and l.lo_partkey = p.p_partkey + and l.lo_suppkey = s.s_suppkey + and l.lo_custkey = c.c_custkey + and s.s_region = 'AMERICA' + and c.c_region = 'AMERICA' + group by + d.d_year, c.c_nation + order by + d.d_year, c.c_nation; + set timing off + select * from table(dbms_xplan.display_cursor()); + + ``` + + Query result: + + ``` + SQL> @08_vgb_buffer.sql + Connected. + SQL> + SQL> alter session set inmemory_query = disable; + + Session altered. + + Elapsed: 00:00:00.00 + SQL> + SQL> -- Query with In-Memory Aggregation enabled and in-memory disabled + SQL> + SQL> SELECT + 2 d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + 3 From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + 4 Where l.lo_orderdate = d.d_datekey + 5 And l.lo_partkey = p.p_partkey + 6 And l.lo_suppkey = s.s_suppkey + 7 And l.lo_custkey = c.c_custkey + 8 And s.s_region = 'AMERICA' + 9 And c.c_region = 'AMERICA' + 10 Group by d.d_year, c.c_nation + 11 Order by d.d_year, c.c_nation; + + D_YEAR C_NATION PROFIT + -------------------- --------------- -------------------- + 1994 ARGENTINA 261149015641 + 1994 BRAZIL 263808033983 + 1994 CANADA 264598150413 + 1994 PERU 258595600981 + 1994 UNITED STATES 265282504206 + 1995 ARGENTINA 258498976118 + 1995 BRAZIL 269135848643 + 1995 CANADA 264654265482 + 1995 PERU 257451709833 + 1995 UNITED STATES 259660457396 + 1996 ARGENTINA 259361903850 + 1996 BRAZIL 265970119048 + 1996 CANADA 265333193889 + 1996 PERU 260916013039 + 1996 UNITED STATES 262339293224 + 1997 ARGENTINA 261099548066 + 1997 BRAZIL 266353055971 + 1997 CANADA 265036379243 + 1997 PERU 259114682243 + 1997 UNITED STATES 262208356128 + 1998 ARGENTINA 151054449013 + 1998 BRAZIL 153632348378 + 1998 CANADA 156899052279 + 1998 PERU 152297126350 + 1998 UNITED STATES 153937088695 + + 25 rows selected. + + Elapsed: 00:00:05.55 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID a0x06yzufd67k, child number 0 + ------------------------------------- + SELECT d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) + profit From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, + CUSTOMER C Where l.lo_orderdate = d.d_datekey + And l.lo_partkey = p.p_partkey And + l.lo_suppkey = s.s_suppkey And l.lo_custkey + = c.c_custkey And s.s_region = 'AMERICA' + And c.c_region = 'AMERICA' Group by + d.d_year, c.c_nation Order by d.d_year, c.c_nation + + Plan hash value: 3030148494 + + --------------------------------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 89412 (100)| | | | + | 1 | TEMP TABLE TRANSFORMATION | | | | | | | | + | 2 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D6671_2308E0F | | | | | | | + | 3 | HASH GROUP BY | | 7 | 112 | 8 (13)| 00:00:01 | | | + | 4 | KEY VECTOR CREATE BUFFERED | :KV0000 | 7 | 112 | 7 (0)| 00:00:01 | | | + | 5 | TABLE ACCESS FULL | DATE_DIM | 2556 | 30672 | 7 (0)| 00:00:01 | | | + | 6 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D666E_2308E0F | | | | | | | + | 7 | HASH GROUP BY | | 1 | 9 | 1924 (2)| 00:00:01 | | | + | 8 | KEY VECTOR CREATE BUFFERED | :KV0001 | 1 | 9 | 1899 (1)| 00:00:01 | | | + | 9 | TABLE ACCESS FULL | PART | 800K| 3906K| 1898 (1)| 00:00:01 | | | + | 10 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D666F_2308E0F | | | | | | | + | 11 | HASH GROUP BY | | 1 | 22 | 70 (2)| 00:00:01 | | | + | 12 | KEY VECTOR CREATE BUFFERED | :KV0002 | 1 | 22 | 69 (0)| 00:00:01 | | | + |* 13 | TABLE ACCESS FULL | SUPPLIER | 4102 | 73836 | 69 (0)| 00:00:01 | | | + | 14 | LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_0FD9D6670_2308E0F | | | | | | | + | 15 | HASH GROUP BY | | 25 | 950 | 843 (1)| 00:00:01 | | | + | 16 | KEY VECTOR CREATE BUFFERED | :KV0003 | 25 | 950 | 841 (1)| 00:00:01 | | | + |* 17 | TABLE ACCESS FULL | CUSTOMER | 59761 | 1984K| 841 (1)| 00:00:01 | | | + | 18 | SORT GROUP BY | | 62 | 6510 | 86566 (1)| 00:00:04 | | | + |* 19 | HASH JOIN | | 62 | 6510 | 86565 (1)| 00:00:04 | | | + |* 20 | HASH JOIN | | 62 | 4712 | 86563 (1)| 00:00:04 | | | + | 21 | MERGE JOIN CARTESIAN | | 7 | 329 | 6 (0)| 00:00:01 | | | + | 22 | MERGE JOIN CARTESIAN | | 1 | 31 | 4 (0)| 00:00:01 | | | + | 23 | TABLE ACCESS FULL | SYS_TEMP_0FD9D666E_2308E0F | 1 | 9 | 2 (0)| 00:00:01 | | | + | 24 | BUFFER SORT | | 1 | 22 | 2 (0)| 00:00:01 | | | + | 25 | TABLE ACCESS FULL | SYS_TEMP_0FD9D666F_2308E0F | 1 | 22 | 2 (0)| 00:00:01 | | | + | 26 | BUFFER SORT | | 7 | 112 | 4 (0)| 00:00:01 | | | + | 27 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6671_2308E0F | 7 | 112 | 2 (0)| 00:00:01 | | | + | 28 | VIEW | VW_VT_80F21617 | 62 | 1798 | 86557 (1)| 00:00:04 | | | + | 29 | VECTOR GROUP BY | | 62 | 3100 | 86557 (1)| 00:00:04 | | | + | 30 | HASH GROUP BY | | 62 | 3100 | 86557 (1)| 00:00:04 | | | + | 31 | KEY VECTOR USE | :KV0000 | 2535K| 120M| 86556 (1)| 00:00:04 | | | + | 32 | KEY VECTOR USE | :KV0001 | 2535K| 111M| 86556 (1)| 00:00:04 | | | + | 33 | KEY VECTOR USE | :KV0003 | 2535K| 101M| 86556 (1)| 00:00:04 | | | + | 34 | KEY VECTOR USE | :KV0002 | 8510K| 308M| 86555 (1)| 00:00:04 | | | + | 35 | PARTITION RANGE ITERATOR | | 41M| 1354M| 86555 (1)| 00:00:04 |:KV0000|:KV0000| + |* 36 | TABLE ACCESS FULL | LINEORDER | 41M| 1354M| 86555 (1)| 00:00:04 |:KV0000|:KV0000| + | 37 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6670_2308E0F | 25 | 725 | 2 (0)| 00:00:01 | | | + --------------------------------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 13 - filter("S"."S_REGION"='AMERICA') + 17 - filter("C"."C_REGION"='AMERICA') + 19 - access("ITEM_14"=INTERNAL_FUNCTION("C0")) + 20 - access("ITEM_12"=INTERNAL_FUNCTION("C0") AND "ITEM_13"=INTERNAL_FUNCTION("C0") AND "ITEM_15"=INTERNAL_FUNCTION("C0")) + 36 - filter((SYS_OP_KEY_VECTOR_FILTER("L"."LO_SUPPKEY",:KV0002) AND SYS_OP_KEY_VECTOR_FILTER("L"."LO_CUSTKEY",:KV0003) AND + SYS_OP_KEY_VECTOR_FILTER("L"."LO_PARTKEY",:KV0001) AND SYS_OP_KEY_VECTOR_FILTER("L"."LO_ORDERDATE",:KV0000))) + + Note + ----- + - vector transformation used for this statement + + + 70 rows selected. + + SQL> + ``` + + Note that like hash joins with Bloom filters, vector group by with key vectors can be performed on row-store objects. This only requires that Database In-Memory be enabled and shows just how flexible Database In-Memory is. You don't have to have all objects populated in the IM column store. Of course, the row store objects cannot be accessed as fast as column-store objects and cannot take advantage of all of the other performance features available to IM column store scans. + +9. And finally let's take a look at how our query runs in the row-store with no Database In-Memory optimizations. + + Run the script *09\_novgb\_buffer.sql* + + ``` + + @09_novgb_buffer.sql + + ``` + + or run the query below: + + ``` + + set timing on + alter session set inmemory_query = disable; + SELECT /*+ NO_VECTOR_TRANSFORM */ + d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + from + LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + where + l.lo_orderdate = d.d_datekey + and l.lo_partkey = p.p_partkey + and l.lo_suppkey = s.s_suppkey + and l.lo_custkey = c.c_custkey + and s.s_region = 'AMERICA' + and c.c_region = 'AMERICA' + group by + d.d_year, c.c_nation + order by + d.d_year, c.c_nation; + set timing off + select * from table(dbms_xplan.display_cursor()); + + ``` + + Query result: + + ``` + SQL> @09_novgb_buffer.sql + Connected. + SQL> + SQL> alter session set inmemory_query = disable; + + Session altered. + + Elapsed: 00:00:00.00 + SQL> + SQL> -- Query with In-Memory Aggregation enabled and in-memory disabled + SQL> + SQL> SELECT /*+ NO_VECTOR_TRANSFORM */ + 2 d.d_year, c.c_nation, sum(lo_revenue - lo_supplycost) profit + 3 From LINEORDER l, DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C + 4 Where l.lo_orderdate = d.d_datekey + 5 And l.lo_partkey = p.p_partkey + 6 And l.lo_suppkey = s.s_suppkey + 7 And l.lo_custkey = c.c_custkey + 8 And s.s_region = 'AMERICA' + 9 And c.c_region = 'AMERICA' + 10 Group by d.d_year, c.c_nation + 11 Order by d.d_year, c.c_nation; + + D_YEAR C_NATION PROFIT + -------------------- --------------- -------------------- + 1994 ARGENTINA 261149015641 + 1994 BRAZIL 263808033983 + 1994 CANADA 264598150413 + 1994 PERU 258595600981 + 1994 UNITED STATES 265282504206 + 1995 ARGENTINA 258498976118 + 1995 BRAZIL 269135848643 + 1995 CANADA 264654265482 + 1995 PERU 257451709833 + 1995 UNITED STATES 259660457396 + 1996 ARGENTINA 259361903850 + 1996 BRAZIL 265970119048 + 1996 CANADA 265333193889 + 1996 PERU 260916013039 + 1996 UNITED STATES 262339293224 + 1997 ARGENTINA 261099548066 + 1997 BRAZIL 266353055971 + 1997 CANADA 265036379243 + 1997 PERU 259114682243 + 1997 UNITED STATES 262208356128 + 1998 ARGENTINA 151054449013 + 1998 BRAZIL 153632348378 + 1998 CANADA 156899052279 + 1998 PERU 152297126350 + 1998 UNITED STATES 153937088695 + + 25 rows selected. + + Elapsed: 00:00:06.72 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID bgm86vz7ghjsj, child number 1 + ------------------------------------- + SELECT /*+ NO_VECTOR_TRANSFORM */ d.d_year, c.c_nation, + sum(lo_revenue - lo_supplycost) profit From LINEORDER l, + DATE_DIM d, PART p, SUPPLIER s, CUSTOMER C Where + l.lo_orderdate = d.d_datekey And l.lo_partkey = + p.p_partkey And l.lo_suppkey = s.s_suppkey + And l.lo_custkey = c.c_custkey And + s.s_region = 'AMERICA' And c.c_region + = 'AMERICA' Group by d.d_year, c.c_nation + Order by d.d_year, c.c_nation + + Plan hash value: 916746023 + + ---------------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop | + ---------------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | | 127K(100)| | | | + | 1 | SORT GROUP BY | | 124 | 12772 | | 127K (1)| 00:00:05 | | | + |* 2 | HASH JOIN | | 2535K| 249M| | 127K (1)| 00:00:05 | | | + | 3 | PART JOIN FILTER CREATE | :BF0000 | 2556 | 30672 | | 7 (0)| 00:00:01 | | | + | 4 | TABLE ACCESS FULL | DATE_DIM | 2556 | 30672 | | 7 (0)| 00:00:01 | | | + |* 5 | HASH JOIN | | 2535K| 220M| 12M| 127K (1)| 00:00:05 | | | + | 6 | TABLE ACCESS FULL | PART | 800K| 3906K| | 1898 (1)| 00:00:01 | | | + |* 7 | HASH JOIN | | 2535K| 207M| 2688K| 113K (1)| 00:00:05 | | | + |* 8 | TABLE ACCESS FULL | CUSTOMER | 59761 | 1984K| | 841 (1)| 00:00:01 | | | + |* 9 | HASH JOIN | | 8510K| 422M| | 86765 (1)| 00:00:04 | | | + |* 10 | TABLE ACCESS FULL | SUPPLIER | 4102 | 73836 | | 69 (0)| 00:00:01 | | | + | 11 | PARTITION RANGE JOIN-FILTER| | 41M| 1354M| | 86555 (1)| 00:00:04 |:BF0000|:BF0000| + | 12 | TABLE ACCESS FULL | LINEORDER | 41M| 1354M| | 86555 (1)| 00:00:04 |:BF0000|:BF0000| + ---------------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - access("L"."LO_ORDERDATE"="D"."D_DATEKEY") + 5 - access("L"."LO_PARTKEY"="P"."P_PARTKEY") + 7 - access("L"."LO_CUSTKEY"="C"."C_CUSTKEY") + 8 - filter("C"."C_REGION"='AMERICA') + 9 - access("L"."LO_SUPPKEY"="S"."S_SUPPKEY") + 10 - filter("S"."S_REGION"='AMERICA') + + + 42 rows selected. + + SQL> + ``` + + Compare the time from Step 7 (06\_vgb\_im.sql) to the time running the query in the row-store with no vector group by enabled. Quite a dramatic difference. + +10. Exit lab + + Type commands below: + + ``` + + exit + cd .. + + ``` + + Command results: + + ``` + SQL> exit + Disconnected from Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production + Version 23.4.0.24.05 + [CDB1:oracle@dbhol:~/labs/inmemory/joins-aggr]$ cd .. + [CDB1:oracle@dbhol:~/labs/inmemory]$ + ``` + + +## Conclusion + +This lab saw our performance comparison expanded to queries with both joins and aggregations. You had an opportunity to see just how efficiently an in-memory hash join with Bloom filters can be executed in the IM column store. + +You also got to see just how sophisticated the Oracle Optimizer has become over the last 30 plus years, when it used a combination of complex query transformations to find the optimal execution plan for a star query. + +Oracle Database adds in-memory database functionality to existing databases, and transparently accelerates analytics by orders of magnitude while simultaneously speeding up mixed-workload OLTP. With Oracle Database In-Memory, users get immediate answers to business questions that previously took hours or days to run. + +You may now **proceed to the next lab**. + +## Acknowledgements + +- **Author** - Andy Rivenes, Product Manager, Database In-Memory +- **Contributors** - Maria Colgan, Distinguished Product Manager +- **Last Updated By/Date** - Andy Rivenes, June 2024 diff --git a/in-memory-23ai/prepare-setup/prepare-setup.md b/in-memory-23ai/prepare-setup/prepare-setup.md new file mode 100644 index 000000000..2f287f598 --- /dev/null +++ b/in-memory-23ai/prepare-setup/prepare-setup.md @@ -0,0 +1,76 @@ +# Prepare Setup + +## Introduction +This lab will show you how to download the Oracle Resource Manager (ORM) stack zip file needed to set up the resource needed to run this workshop. + +*Estimated Lab Time:* 10 minutes + +### Objectives +- Download ORM stack +- Configure an existing Virtual Cloud Network (VCN) + +### Prerequisites +This lab assumes you have: +- An Oracle Cloud account + +## Task 1: Download Oracle Resource Manager (ORM) stack zip file +1. Click on the link below to download the Resource Manager zip file you need to build your environment: + + + - [db-inmemory-mkplc-freetier.zip](https://objectstorage.us-ashburn-1.oraclecloud.com/p/YXh0XbJGk0fvsZ2GFKSGPyDUzJ2ywCp6NBF56_UnDj6mZtjvidxULIXwI2JXFE1x/n/c4u02/b/hosted_workshops/o/stacks/db-inmemory-mkplc-freetier.zip) + + + - [db-inmemory-advanced-mkplc-freetier.zip](https://objectstorage.us-ashburn-1.oraclecloud.com/p/ywqTs1seIu1gYTENJDUXG63N3-UWXJChyTZAPtF5XVE3InHBPHmwzM1mkm0RUFjE/n/c4u02/b/hosted_workshops/o/stacks/db-inmemory-advanced-mkplc-freetier.zip) + + +2. Save in your downloads folder. + +We strongly recommend using this stack to create a self-contained/dedicated VCN with your instance(s). Skip to *Task 3* to follow our recommendations. If you would rather use an existing VCN then proceed to the next task to update your existing VCN with the required Ingress rules. + +## Task 2: Adding security rules to an existing VCN + +This workshop requires a certain number of ports to be available, a requirement that can be met by using the default ORM stack execution that creates a dedicated VCN. In order to use an existing VCN/subnet, the following rules should be added to the security list. + +| Type | Source Port | Source CIDR | Destination Port | Protocol | Description | +| :----------- | :--------: | :--------: | :----------: | :----: | :------------------------------------ | +| Ingress | All | 0.0.0.0/0 | 22 | TCP | SSH | +| Ingress | All | 0.0.0.0/0 | 80 | TCP | Remote Desktop using noVNC | +| Egress | All | N/A | 80 | TCP | Outbound HTTP access | +| Egress | All | N/A | 443 | TCP | Outbound HTTPS access | +{: title="List of Required Network Security Rules"} + + + + + +1. Go to *Networking >> Virtual Cloud Networks* +2. Choose your network +3. Under Resources, select Security Lists +4. Click on Default Security Lists under the Create Security List button +5. Click Add Ingress Rule button +6. Enter the following: + - Source Type: CIDR + - Source CIDR: 0.0.0.0/0 + - IP Protocol: TCP + - Source Port Range: All (Keep Default) + - Destination Port Range: *Select from the above table* + - Description: *Select the corresponding description from the above table* +7. Click the Add Ingress Rules button +8. Repeat steps [5-7] until a rule is created for each port listed in the table + +## Task 3: Setup Compute +Using the details from the two Tasks above, proceed to the lab *Environment Setup* to set up your workshop environment using Oracle Resource Manager (ORM) and one of the following options: +- Create Stack: *Compute + Networking* +- Create Stack: *Compute only* with an existing VCN where security lists have been updated as per *Task 2* above + +You may now proceed to the next lab. + +## Acknowledgements +* **Author** - Rene Fontcha, LiveLabs Platform Lead, NA Technology +* **Contributors** - Meghana Banka +* **Last Updated By/Date** - Rene Fontcha, LiveLabs Platform Lead, NA Technology, August 2022 diff --git a/in-memory-23ai/queries/queries.md b/in-memory-23ai/queries/queries.md new file mode 100644 index 000000000..06314b088 --- /dev/null +++ b/in-memory-23ai/queries/queries.md @@ -0,0 +1,909 @@ +# In-Memory Queries + +## Introduction +Watch a preview video of querying the In-Memory Column Store + +[YouTube video](youtube:U9BmS53KuGs) + +Watch the video below for a walk through of the In-Memory Queries lab: + +[In-Memory Queries](videohub:1_ohs9hpw0) + +*Estimated Lab Time:* 15 Minutes. + +### Objectives + +- Perform various queries on the In-Memory Column Store + +### Prerequisites + +This lab assumes you have: +- A Free Tier, Paid or LiveLabs Oracle Cloud account +- You have completed: + - Get Started with noVNC Remote Desktop + - Lab: Initialize Environment + - Lab: Setting up the In-Memory Column Store + +**NOTE:** *When doing Copy/Paste using the convenient* **Copy** *function used throughout the guide, you must hit the* **ENTER** *key after pasting. Otherwise the last line will remain in the buffer until you hit* **ENTER!** + +## Task 1: Querying the In-Memory Column Store + +Now that you’ve gotten familiar with the IM column store let’s look at the benefits of using it. You will execute a series of queries against the large fact table LINEORDER, in both the buffer cache and the IM column store, to demonstrate the different ways the IM column store can improve query performance above and beyond the basic performance benefits of accessing data in memory only. + +Reload the environment variables for **CDB1** if you exited the terminal after the previous lab + +``` +. ~/.set-env-db.sh CDB1 +``` + +Let's switch to the queries folder and log back in to the PDB. + +``` + +cd /home/oracle/labs/inmemory/queries +sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + +``` + +And adjust the sqlplus display: + +``` + +set pages 9999 +set lines 150 + +``` + +Query result: + +``` +[CDB1:oracle@dbhol:~/labs/inmemory]$ cd /home/oracle/labs/inmemory/queries +[CDB1:oracle@dbhol:~/labs/inmemory/queries]$ sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + +SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 31 17:21:31 2024 +Version 23.4.0.24.05 + +Copyright (c) 1982, 2024, Oracle. All rights reserved. + +Last Successful login time: Fri May 31 2024 17:19:47 -07:00 + +Connected to: +Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production +Version 23.4.0.24.05 + +SQL> set pages 9999 +SQL> set lines 150 +SQL> +``` + +1. Let's begin with a simple query: *What is the most expensive order and total quantity we have received to date?* There are no indexes or views set up for this so the execution plan will be to do a full table scan of the LINEORDER table. Note the elapsed time. + + Run the script *01\_im\_query\_stats.sql* + + ``` + + @01_im_query_stats.sql + + ``` + + or run the query below: + + ``` + + set timing on + select + max(lo_ordtotalprice) most_expensive_order, + sum(lo_quantity) total_items + from LINEORDER; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @01_im_query_stats.sql + Connected. + SQL> + SQL> -- In-Memory query + SQL> + SQL> select + 2 max(lo_ordtotalprice) most_expensive_order, + 3 sum(lo_quantity) total_items + 4 from LINEORDER; + + MOST_EXPENSIVE_ORDER TOTAL_ITEMS + -------------------- ----------- + 57346348 1064978115 + + Elapsed: 00:00:00.02 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID abxsukctvzc8u, child number 0 + ------------------------------------- + select max(lo_ordtotalprice) most_expensive_order, sum(lo_quantity) + total_items from LINEORDER + + Plan hash value: 4085810105 + + ---------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + ---------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 3613 (100)| | | | + | 1 | SORT AGGREGATE | | 1 | 9 | | | | | + | 2 | PARTITION RANGE ALL | | 41M| 358M| 3613 (12)| 00:00:01 | 1 | 5 | + | 3 | TABLE ACCESS INMEMORY FULL| LINEORDER | 41M| 358M| 3613 (12)| 00:00:01 | 1 | 5 | + ---------------------------------------------------------------------------------------------------------- + + + 16 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 16 + IM scan CUs columns accessed 156 + IM scan CUs pcode aggregation pushdown 156 + IM scan rows 41760941 + IM scan rows pcode aggregated 41760941 + IM scan rows projected 78 + IM scan rows valid 41760941 + physical reads 30 + session logical reads 322465 + session logical reads - IM 316512 + session pga memory 19601592 + table scans (IM) 5 + + 12 rows selected. + + SQL> + ``` + + The execution plan shows that we performed a TABLE ACCESS INMEMORY FULL of the LINEORDER table. + +2. Now we will run the same query using the buffer cache. Remember that the LINEORDER table has been fully cached in the KEEP pool. You will notice that we have added a hint called *NO\_INMEMORY*. This will tell the optimizer not to access the data from the IM column store. Compare the query run time with the time from Step 1. + + Run the script *02\_buffer\_query\_stats.sql* + + ``` + + @02_buffer_query_stats.sql + + ``` + + or run the query below: + + ``` + + set timing on + select /*+ NO_INMEMORY */ + max(lo_ordtotalprice) most_expensive_order, + sum(lo_quantity) total_items + from LINEORDER; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @02_buffer_query_stats.sql + Connected. + SQL> + SQL> -- Buffer Cache query with the column store disabled via NO_INMEMORY hint + SQL> + SQL> select /*+ NO_INMEMORY */ + 2 max(lo_ordtotalprice) most_expensive_order, + 3 sum(lo_quantity) total_items + 4 from LINEORDER; + + MOST_EXPENSIVE_ORDER TOTAL_ITEMS + -------------------- -------------------- + 57346348 1064978115 + + Elapsed: 00:00:04.58 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID ar655fzccnk1c, child number 0 + ------------------------------------- + select /*+ NO_INMEMORY */ max(lo_ordtotalprice) most_expensive_order, + sum(lo_quantity) total_items from LINEORDER + + Plan hash value: 4085810105 + + -------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + -------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 86470 (100)| | | | + | 1 | SORT AGGREGATE | | 1 | 9 | | | | | + | 2 | PARTITION RANGE ALL| | 41M| 358M| 86470 (1)| 00:00:04 | 1 | 5 | + | 3 | TABLE ACCESS FULL | LINEORDER | 41M| 358M| 86470 (1)| 00:00:04 | 1 | 5 | + -------------------------------------------------------------------------------------------------- + + + 16 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 462 + IM scan segments disk 5 + session logical reads 315628 + session pga memory 18680056 + + SQL> + ``` + + As you can see the query executed extremely quickly in both cases because this is purely an in-memory scan. However, the performance of the query against the IM column store was significantly faster than the traditional buffer cache - why? + + The IM column store only has to scan two columns - LO\_ORDTOTALPRICE and LO\_QUANTITY - while the row store has to scan all of the columns in each of the rows until it reaches the LO\_ORDTOTALPRICE and LO\_QUANTITY columns. The IM column store also benefits from the fact that the data is compressed so the volume of data scanned is much less. Finally, the column format can take advantage of SIMD vector processing (Single Instruction processing Multiple Data values). Instead of evaluating each entry in the column one at a time, SIMD vector processing allows a set of column values to be evaluated together in a single CPU instruction. + + The execution plan shows that the optimizer has chosen an in-memory scan, but to confirm that the IM column store was used, we need to examine the session level statistics. Notice that in the in-memory query several IM statistics show up (for this lab we have only displayed some key statistics – there are lots more!). The only one we are really interested in now is the "IM scan rows". + + IM scan rows: Number of in-memory rows scanned. + + As our query did a full table scan of the LINEORDER table, that session statistic shows that we scanned approximately 41 million rows from the IM column store. Notice that in the second buffer cache query that statistic does not show up. Only one in-memory statistic shows up, "IM scan segments disk" with a value of 1. This means that even though the LINEORDER table is in the IM column store (IM segment) we actually scan that segment outside of the column store from the buffer cache. Since we fully cached the tables in the KEEP pool we are making a memory to memory comparison, and in this case we can verify that the query did no physical IO (if a small number of physical IOs show then try running it again to ensure that it is fully cached in the KEEP pool). + +3. Let's look for a specific order in the LINEORDER table based on the order key. Typically, a full table scan is not an efficient execution plan when looking for a specific entry in a table, but there are some things going on in the execution plan that we will take a look at. + + Run the script *03\_single\_key\_im.sql* + + ``` + + @03_single_key_im.sql + + ``` + + or run the query below: + + ``` + + set timing on + select lo_orderkey, lo_custkey, lo_revenue + from LINEORDER + where lo_orderkey = 5000000; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @03_single_key_im.sql + Connected. + SQL> + SQL> -- In-Memory Column Store query + SQL> + SQL> select lo_orderkey, lo_custkey, lo_revenue + 2 from LINEORDER + 3 where lo_orderkey = 5000000; + + LO_ORDERKEY LO_CUSTKEY LO_REVENUE + -------------------- -------------------- -------------------- + 5000000 48647 2456268 + + Elapsed: 00:00:00.01 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 513g163sj3cv2, child number 0 + ------------------------------------- + select lo_orderkey, lo_custkey, lo_revenue from LINEORDER where + lo_orderkey = 5000000 + + Plan hash value: 2881531378 + + --------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 3371 (100)| | | | + | 1 | PARTITION RANGE ALL | | 4 | 68 | 3371 (7)| 00:00:01 | 1 | 5 | + |* 2 | TABLE ACCESS INMEMORY FULL| LINEORDER | 4 | 68 | 3371 (7)| 00:00:01 | 1 | 5 | + --------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - inmemory("LO_ORDERKEY"=5000000) + filter("LO_ORDERKEY"=5000000) + + + 21 rows selected. + + SQL> + ``` + + Notice that in the Predicate Information of the execution plan you see both "inmemory" and "filter". This indicates that the filter predicate (i.e. "LO_ORDERKEY"=5000000) was "pushed" into the scan of the LINEORDER table (i.e. inmemory) rather than having to be evaluated after the value was retrieved like it is in the row store. This is one of the significant ways that Database In-Memory speeds up queries. + + ``` + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - inmemory("LO_ORDERKEY"=5000000) + filter("LO_ORDERKEY"=5000000) + ``` + +4. Let's run the same query in the row format. Typically, a full table scan is not an efficient execution plan when looking for a specific entry in a table, but not all columns are going to be indexed. If we don't have an index how long does it take to run this query in the row store versus the column store? + + Run the script *04\_single\_key\_buffer.sql* + + ``` + + @04_single_key_buffer.sql + + ``` + + or run the query below: + + ``` + + set timing on + select /*+ NO_INMEMORY */ + lo_orderkey, lo_custkey, lo_revenue + from LINEORDER + where lo_orderkey = 5000000; + set timing off + select * from table(dbms_xplan.display_cursor()); + + ``` + + Query result: + + ``` + SQL> @04_single_key_buffer.sql + Connected. + SQL> + SQL> -- Buffer Cache query with the column store disables via INMEMORY_QUERY parameter + SQL> + SQL> select /*+ NO_INMEMORY */ + 2 lo_orderkey, lo_custkey, lo_revenue + 3 from LINEORDER + 4 where lo_orderkey = 5000000; + + LO_ORDERKEY LO_CUSTKEY LO_REVENUE + -------------------- -------------------- -------------------- + 5000000 48647 2456268 + + Elapsed: 00:00:02.35 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 4rcm8pc7xktun, child number 0 + ------------------------------------- + select /*+ NO_INMEMORY */ lo_orderkey, lo_custkey, lo_revenue + from LINEORDER where lo_orderkey = 5000000 + + Plan hash value: 2881531378 + + ------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + ------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 86258 (100)| | | | + | 1 | PARTITION RANGE ALL| | 4 | 68 | 86258 (1)| 00:00:04 | 1 | 5 | + |* 2 | TABLE ACCESS FULL | LINEORDER | 4 | 68 | 86258 (1)| 00:00:04 | 1 | 5 | + ------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - filter("LO_ORDERKEY"=5000000) + + + 20 rows selected. + + SQL> + ``` + + Notice that in the Predicate Information section of the execution plan you only see "filter". This indicates that the filter predicate (i.e. "LO_ORDERKEY"=5000000) was evaluated after the value was retrieved. This is another advantage of Database In-Memory, being able to do filtering during the scan of the data. + + ``` + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - filter("LO_ORDERKEY"=5000000) + ``` + +5. Think indexing the LO\_ORDERKEY column would provide the same performance as the IM column store? There is an invisible index already created on the LO\_ORDERKEY column of the LINEORDER table. By using the parameter OPTIMIZER\_USE\_INVISIBLE\_INDEXES we can compare the performance of the IM column store to using an index. Let's see how well the index performs. + + Run the script *05\_index\_comparison.sql* + + ``` + + @05_index_comparison.sql + + ``` + + or run the query below: + + ``` + + alter session set optimizer_use_invisible_indexes=true; + set timing on + Select /* With index */ lo_orderkey, lo_custkey, lo_revenue + From LINEORDER + Where lo_orderkey = 5000000; + set timing off + select * from table(dbms_xplan.display_cursor()); + alter session set optimizer_use_invisible_indexes=false; + + ``` + + Query result: + + ``` + SQL> @05_index_comparison.sql + Connected. + SQL> + SQL> -- Show query using an index based on cost + SQL> + SQL> -- Enable the use of invisible indexes + SQL> + SQL> alter session set optimizer_use_invisible_indexes=true; + + Session altered. + + SQL> + SQL> set timing on + SQL> + SQL> Select /* With index */ lo_orderkey, lo_custkey, lo_revenue + 2 From LINEORDER + 3 Where lo_orderkey = 5000000; + + LO_ORDERKEY LO_CUSTKEY LO_REVENUE + -------------------- -------------------- -------------------- + 5000000 48647 2456268 + + Elapsed: 00:00:00.03 + SQL> + SQL> set timing off + SQL> + SQL> pause Hit enter ... + Hit enter ... + + SQL> + SQL> select * from table(dbms_xplan.display_cursor()); + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 8kgqurq43dgq1, child number 0 + ------------------------------------- + Select /* With index */ lo_orderkey, lo_custkey, lo_revenue From + LINEORDER Where lo_orderkey = 5000000 + + Plan hash value: 3247970186 + + --------------------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 12 (100)| | | | + | 1 | PARTITION RANGE ALL | | 4 | 68 | 12 (0)| 00:00:01 | 1 | 5 | + | 2 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| LINEORDER | 4 | 68 | 12 (0)| 00:00:01 | 1 | 5 | + |* 3 | INDEX RANGE SCAN | LINEORDER_I2 | 4 | | 11 (0)| 00:00:01 | 1 | 5 | + --------------------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 3 - access("LO_ORDERKEY"=5000000) + + + 21 rows selected. + + SQL> + SQL> alter session set optimizer_use_invisible_indexes=false; + + Session altered. + + SQL> + ``` + + Notice that in the execution plan an INDEX RANGE SCAN was performed on the LINEORDER_I2 index. This is another big benefit of the way that Oracle implemented Database In-Memory. The optimizer, based on cost, is able to determine the most efficient way to access data. In this example, with an appropriate index available, the optimizer decided it was more efficient to use the index. This is why no application SQL has to be changed to use Database In-Memory and also why Database In-Memory can be used in mixed workload environments (i.e. transactions and analytics together). + +6. What if you don't have an index that the optimizer can use, but you have populated your data in the IM column store? Will performance suffer? Another feature of Database In-Memory is called In-Memory Storage Indexes, and they are created automatically during population. We're going to repeat the query from Step 3, but this time we're going to include a bit more information to see if we can figure out what is going on under the covers. + + Run the script *06\_storage\_index.sql* + + ``` + + @06_storage_index.sql + + ``` + + or run the query below: + + ``` + + set timing on + select lo_orderkey, lo_custkey, lo_revenue + from LINEORDER + where lo_orderkey = 5000000; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @06_storage_index.sql + Connected. + SQL> + SQL> -- Demonstrate the use of In-Memory Storage Indexes + SQL> + SQL> select lo_orderkey, lo_custkey, lo_revenue + 2 from LINEORDER + 3 where lo_orderkey = 5000000; + + LO_ORDERKEY LO_CUSTKEY LO_REVENUE + -------------------- -------------------- -------------------- + 5000000 48647 2456268 + + Elapsed: 00:00:00.01 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID 513g163sj3cv2, child number 0 + ------------------------------------- + select lo_orderkey, lo_custkey, lo_revenue from LINEORDER where + lo_orderkey = 5000000 + + Plan hash value: 2881531378 + + --------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 3371 (100)| | | | + | 1 | PARTITION RANGE ALL | | 4 | 68 | 3371 (7)| 00:00:01 | 1 | 5 | + |* 2 | TABLE ACCESS INMEMORY FULL| LINEORDER | 4 | 68 | 3371 (7)| 00:00:01 | 1 | 5 | + --------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - inmemory("LO_ORDERKEY"=5000000) + filter("LO_ORDERKEY"=5000000) + + + 21 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 3 + IM scan CUs columns accessed 9 + IM scan CUs pruned 71 + IM scan rows 41760941 + IM scan rows projected 1 + IM scan rows valid 3744224 + IM scan segments minmax eligible 78 + session logical reads 316626 + session logical reads - IM 316512 + session pga memory 17618448 + table scans (IM) 5 + + 11 rows selected. + + SQL> + ``` + + Note that we are now back to an inmemory query. This time we included the session statistics for the query. Take note of two key statistics. The first is "IM scan segments minmax eligible". This tells us how many IMCUs where eligible for minmax pruning, which is how In-Memory Storage Indexes work. The second important statistic is "IM scan CUs pruned". Notice that this number is almost as large as the total number of IMCUs. This means that Database In-Memory was able to avoid scanning almost all of the data. This is because at population time In-Memory storage indexes are created for each set of column values in each IMCU with the MIN and MAX values. During the scan these MIN and MAX values can be compared with filter predicates and can possibly result in not having to scan the actual columnar data, thereby improving performance. After all, the fastest way to do something is to not do it at all. + +7. Analytical queries typically have more than one WHERE clause predicate. What happens when there are multiple single column predicates on a table? Traditionally you would create a multi-column index. Can the IM column store compete with that? + + Let’s change our query to look for a specific line item in an order and monitor the session statistics: + + Run the script *07\_multi\_preds.sql* + + ``` + + @07_multi_preds.sql + + ``` + + or run the query below: + + ``` + + set timing on + select lo_orderkey, lo_custkey, lo_revenue + from LINEORDER + where lo_custkey = 5641 + and lo_shipmode = 'SHIP' + and lo_orderpriority = '5-LOW'; + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @07_multi_preds.sql + Connected. + SQL> + SQL> -- In-Memory query + SQL> + SQL> select lo_orderkey, lo_custkey, lo_revenue + 2 from LINEORDER + 3 where lo_custkey = 5641 + 4 and lo_shipmode = 'SHIP' + 5 and lo_orderpriority = '5-LOW'; + + LO_ORDERKEY LO_CUSTKEY LO_REVENUE + -------------------- -------------------- -------------------- + 16925125 5641 5711620 + 16925125 5641 6400967 + 18139779 5641 4508300 + 27479847 5641 7247038 + 27479847 5641 4341522 + 22534688 5641 705728 + 22534688 5641 3400905 + 28390534 5641 486770 + 59428583 5641 6083192 + 59428583 5641 3904048 + 40154336 5641 3052806 + 13644419 5641 3786727 + 51805731 5641 4164048 + + 13 rows selected. + + Elapsed: 00:00:00.05 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID gcrg8wtsj5wxh, child number 0 + ------------------------------------- + select lo_orderkey, lo_custkey, lo_revenue from LINEORDER where + lo_custkey = 5641 and lo_shipmode = 'SHIP' and + lo_orderpriority = '5-LOW' + + Plan hash value: 2881531378 + + --------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 3281 (100)| | | | + | 1 | PARTITION RANGE ALL | | 6 | 264 | 3281 (3)| 00:00:01 | 1 | 5 | + |* 2 | TABLE ACCESS INMEMORY FULL| LINEORDER | 6 | 264 | 3281 (3)| 00:00:01 | 1 | 5 | + --------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - inmemory(("LO_CUSTKEY"=5641 AND "LO_SHIPMODE"='SHIP' AND "LO_ORDERPRIORITY"='5-LOW')) + filter(("LO_CUSTKEY"=5641 AND "LO_SHIPMODE"='SHIP' AND "LO_ORDERPRIORITY"='5-LOW')) + + + 22 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 18 + IM scan CUs columns accessed 167 + IM scan rows 41760941 + IM scan rows projected 13 + IM scan rows valid 41760941 + IM scan segments minmax eligible 78 + physical reads 2 + session logical reads 317065 + session logical reads - IM 316512 + session pga memory 18290872 + table scans (IM) 5 + + 11 rows selected. + + SQL> + ``` + + In this example you can see in the Predicate Information section that multiple filter predicates were included in the "inmemory" function. Database In-Memory is not limited to pushing only single predicates into a scan. There is another important statistic that can be used to measure how much work is saved by pushing predicates into the scan. Notice the statistic "IM scan rows projected". The value is 13, which is precisely the number of rows returned by the query. This statistic shows that even though we scanned 41 million rows we only returned 13 rows. That is another reason Database In-Memory is so much faster than the row store. + + +8. Let’s get a bit more complicated and see what happens when we have more complex where clause predicates that include multiple columns and subselects. The query this time is to determine which of the expensive bulk orders generated the least amount of revenue for the company when shipped by truck. + + Run the script *08\_multi\_col.sql* + + ``` + + @08_multi_col.sql + + ``` + + or run the query below: + + ``` + + set timing on + select lo_orderkey, lo_revenue + From LINEORDER + Where lo_revenue = (Select min(lo_revenue) + From LINEORDER + Where lo_supplycost = (Select max(lo_supplycost) + From LINEORDER + Where lo_quantity > 10) + And lo_shipmode LIKE 'TRUCK%' + And lo_discount between 2 and 5 + ); + set timing off + select * from table(dbms_xplan.display_cursor()); + @../imstats.sql + + ``` + + Query result: + + ``` + SQL> @08_multi_col.sql + Connected. + SQL> + SQL> -- In-Memory query + SQL> + SQL> Select lo_orderkey, lo_revenue + 2 From LINEORDER + 3 Where lo_revenue = (Select min(lo_revenue) + 4 From LINEORDER + 5 Where lo_supplycost = (Select max(lo_supplycost) + 6 From LINEORDER + 7 Where lo_quantity > 10) + 8 And lo_shipmode LIKE 'TRUCK%' + 9 And lo_discount between 2 and 5 + 10 ); + + LO_ORDERKEY LO_REVENUE + -------------------- -------------------- + 5335335 199404 + 3842596 199404 + 21888516 199404 + 49976640 199404 + + Elapsed: 00:00:00.06 + SQL> + SQL> set echo off + Hit enter ... + + + PLAN_TABLE_OUTPUT + ------------------------------------------------------------------------------------------------------------------------------------------------------ + SQL_ID bnysz8dqbamnd, child number 0 + ------------------------------------- + Select lo_orderkey, lo_revenue From LINEORDER Where lo_revenue = + (Select min(lo_revenue) From LINEORDER + Where lo_supplycost = (Select max(lo_supplycost) + From LINEORDER + Where lo_quantity > 10) And + lo_shipmode LIKE 'TRUCK%' And lo_discount between + 2 and 5 ) + + Plan hash value: 230990116 + + --------------------------------------------------------------------------------------------------------------- + | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | + --------------------------------------------------------------------------------------------------------------- + | 0 | SELECT STATEMENT | | | | 11416 (100)| | | | + | 1 | PARTITION RANGE ALL | | 7 | 84 | 3697 (14)| 00:00:01 | 1 | 5 | + |* 2 | TABLE ACCESS INMEMORY FULL | LINEORDER | 7 | 84 | 3697 (14)| 00:00:01 | 1 | 5 | + | 3 | SORT AGGREGATE | | 1 | 25 | | | | | + | 4 | PARTITION RANGE ALL | | 75 | 1875 | 4022 (21)| 00:00:01 | 1 | 5 | + |* 5 | TABLE ACCESS INMEMORY FULL | LINEORDER | 75 | 1875 | 4022 (21)| 00:00:01 | 1 | 5 | + | 6 | SORT AGGREGATE | | 1 | 8 | | | | | + | 7 | PARTITION RANGE ALL | | 33M| 254M| 3697 (14)| 00:00:01 | 1 | 5 | + |* 8 | TABLE ACCESS INMEMORY FULL| LINEORDER | 33M| 254M| 3697 (14)| 00:00:01 | 1 | 5 | + --------------------------------------------------------------------------------------------------------------- + + Predicate Information (identified by operation id): + --------------------------------------------------- + + 2 - inmemory("LO_REVENUE"=) + filter("LO_REVENUE"=) + 5 - inmemory(("LO_SHIPMODE" LIKE 'TRUCK%' AND "LO_DISCOUNT"<=5 AND "LO_DISCOUNT">=2 AND + "LO_SUPPLYCOST"=)) + filter(("LO_SHIPMODE" LIKE 'TRUCK%' AND "LO_DISCOUNT"<=5 AND "LO_DISCOUNT">=2 AND + "LO_SUPPLYCOST"=)) + 8 - inmemory("LO_QUANTITY">10) + filter("LO_QUANTITY">10) + + + 38 rows selected. + + Hit enter ... + + + NAME VALUE + -------------------------------------------------- -------------------- + CPU used by this session 33 + IM scan CUs columns accessed 391 + IM scan CUs pcode aggregation pushdown 81 + IM scan CUs pruned 28 + IM scan rows 125282823 + IM scan rows pcode aggregated 33408482 + IM scan rows projected 85 + IM scan rows valid 110207880 + IM scan segments minmax eligible 234 + session logical reads 949739 + session logical reads - IM 949536 + session pga memory 17897656 + table scans (IM) 15 + + 13 rows selected. + + SQL> + ``` + + Even with all of these complex predicates the optimizer chose an in-memory query, showing that for large scan operations it is the most efficient approach. + +9. Exit lab + + Type commands below: + + ``` + + exit + cd .. + + ``` + + Command results: + + ``` + SQL> exit + Disconnected from Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production + Version 23.4.0.24.05 + [CDB1:oracle@dbhol:~/labs/inmemory/queries]$ cd .. + [CDB1:oracle@dbhol:~/labs/inmemory]$ + ``` + +## Conclusion + +In this lab you had an opportunity to try out Oracle’s In-Memory performance claims with queries that run against a table with over 41 million rows, the LINEORDER table, which resides in both the IM column store and the buffer cache KEEP pool. From a very simple aggregation, to more complex queries with multiple columns and filter predicates, the IM column store was able to out perform the buffer cache queries. Remember both sets of queries are executing completely within memory, so that’s quite an impressive improvement. + +These significant performance improvements are possible because of Oracle’s unique in-memory columnar format that allows us to only scan the columns we need and to take full advantage of SIMD vector processing. We also got a little help from our new in-memory storage indexes, which allow us to prune out unnecessary data. Remember that with the IM column store, every column has a storage index that is automatically maintained for you. + +You may now **proceed to the next lab**. + +## Acknowledgements + +- **Author** - Andy Rivenes, Product Manager, Database In-Memory +- **Contributors** - Maria Colgan, Distinguished Product Manager +- **Last Updated By/Date** - Andy Rivenes, June 2024 diff --git a/in-memory-23ai/setup/setup.md b/in-memory-23ai/setup/setup.md new file mode 100644 index 000000000..ef050b5c2 --- /dev/null +++ b/in-memory-23ai/setup/setup.md @@ -0,0 +1,598 @@ +# Set up the In-Memory column store + +## Introduction + +In this lab, you will explore how to enable the In-Memory column store, populate objects and query various views to monitor Database In-Memory. + +Watch the video below to get an explanation of enabling the In-Memory column store. + +[Youtube video](youtube:dZ9cnIL6KKw) + +Quick walk through on how to enable In-Memory. + +[Setting up the In-Memory column store](videohub:1_dg318frc) + +*Estimated Lab Time:* 15 Minutes. + +### Objectives + +- Learn how to enable Database In-Memory and populate objects in the IM column store +- Explore various views to monitor Database In-Memory + +### Prerequisites + +This lab assumes you have: +- A Free Tier, Paid or LiveLabs Oracle Cloud account +- You have completed: + - Get Started with noVNC Remote Desktop + - Lab: Initialize Environment + +**NOTE:** *When doing Copy/Paste using the convenient* **Copy** *function used throughout the guide, you must hit the* **ENTER** *key after pasting. Otherwise the last line will remain in the buffer until you hit* **ENTER!** + +## Task 1: Logging In and Enabling In-Memory + +In this Lab we will explore how the In-Memory column store is enabled in Oracle Database, and then how to enable and populate objects and verify the population of those objects in the In-Memory column store. + +1. Let's switch to the setup folder and log back in to the PDB: + + Reload the environment variables for **CDB1** if you exited the terminal after the previous lab + + ``` + . ~/.set-env-db.sh CDB1 + ``` + + Connect to **PDB1** + + ``` + + cd /home/oracle/labs/inmemory/setup + sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + + ``` + + And adjust the sqlplus display: + + ``` + + set pages 9999 + set lines 150 + + ``` + + Query result: + + ``` + [CDB1:oracle@dbhol:~/labs/inmemory]$ cd /home/oracle/labs/inmemory/setup + [CDB1:oracle@dbhol:~/labs/inmemory/setup]$ sqlplus ssb/Ora_DB4U@localhost:1521/pdb1 + + SQL*Plus: Release 23.0.0.0.0 - Production on Mon May 20 14:02:47 2024 + Version 23.4.0.24.05 + + Copyright (c) 1982, 2024, Oracle. All rights reserved. + + Last Successful login time: Mon May 20 2024 13:50:04 -07:00 + + Connected to: + Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production + Version 23.4.0.24.05 + + SQL> set pages 9999 + SQL> set lines 150 + SQL> + ``` + +2. Database In-Memory is integrated into Oracle Database 12c (12.1.0.2) and higher. The IM column store is not enabled by default, but can be easily enabled via a few steps. In this lab we have set the following parameters: + + Run the script *01\_show\_parms.sql* + + ``` + + @01_show_parms.sql + + ``` + + or run the query below: + + ``` + + show parameter sga + show parameter db_keep_cache_size + show parameter heat_map + show parameter inmemory_size + show parameter inmemory_automatic_level + + ``` + + Query result: + + ``` + SQL> @01_show_parms.sql + Connected. + SQL> + SQL> -- Shows the SGA init.ora parameters + SQL> + SQL> show parameter sga + + NAME TYPE VALUE + ------------------------------------ ----------- ------------------------------ + allow_group_access_to_sga boolean FALSE + lock_sga boolean FALSE + pre_page_sga boolean TRUE + sga_max_size big integer 8G + sga_min_size big integer 0 + sga_target big integer 0 + SQL> + SQL> show parameter db_keep_cache_size + + NAME TYPE VALUE + ------------------------------------ ----------- ------------------------------ + db_keep_cache_size big integer 3008M + SQL> + SQL> show parameter heat_map + + NAME TYPE VALUE + ------------------------------------ ----------- ------------------------------ + heat_map string ON + SQL> + SQL> show parameter inmemory_size + + NAME TYPE VALUE + ------------------------------------ ----------- ------------------------------ + inmemory_size big integer 3312M + SQL> + SQL> show parameter inmemory_automatic_level + + NAME TYPE VALUE + ------------------------------------ ----------- ------------------------------ + inmemory_automatic_level string high + SQL> + ``` + + Since the IM column store is not enabled by default (i.e. INMEMORY\_SIZE=0), we have set it to a size that will work for this Lab. HEAT\_MAP defaults to OFF, but it has been enabled for one of the later labs. The KEEP pool (i.e. DB\_KEEP\_CACHE\_SIZE) is set to 0 by default. We have defined it for this Lab so that you can compare the performance of objects populated in the IM column store with the same objects fully cached in the buffer cache and compare the difference in performance for yourself. We have also set SGA\_TARGET, as opposed to defining individual SGA components or MEMORY\_TARGET, in order to enable Automatic Shared Memory Management (ASMM) to enable a new feature in Database In-Memory called Automatic In-Memory Sizing. + +3. Since Database In-Memory is fully integrated into Oracle Database the IM column store is allocated within the System Global Area (SGA) and can be easily displayed using normal database commands. + + Run the script *02\_show\_sga.sql* + + ``` + + @02_show_sga.sql + + ``` + + or run the query below: + + ``` + + show sga + + ``` + + Query result: + + ``` + SQL> @02_show_sga.sql + Connected. + SQL> set numwidth 20 + SQL> + SQL> -- Show SGA memory allocation + SQL> + SQL> show sga + + Total System Global Area 8587393600 bytes + Fixed Size 5380672 bytes + Variable Size 469762048 bytes + Database Buffers 4630511616 bytes + Redo Buffers 8855552 bytes + In-Memory Area 3472883712 bytes + SQL> + ``` + + Notice that the SGA is made up of Fixed Size, Variable Size, Database Buffers and Redo Buffers. And since we have set the INEMMORY\_SIZE parameter we also see the In-Memory Area allocated within the SGA. + + +4. In 23ai the In-Memory area is sub-divided into three pools: a 1MB POOL used to store actual columnar formatted data populated in the IM column store, a 64KB POOL to store metadata about the objects populated in the IM column store and a IM POOL METADATA to manage objects populated in the IM column store. The view V$INMEMORY\_AREA shows the total memory allocated and used in the IM column store. + + Run the script *03\_im\_usage.sql* + + ``` + + @03_im_usage.sql + + ``` + + or run the query below: + + ``` + + column pool format a10; + column alloc_bytes format 999,999,999,999,999 + column used_bytes format 999,999,999,999,999 + SELECT pool, alloc_bytes, used_bytes, populate_status, con_id + FROM v$inmemory_area; + + ``` + + Query result: + + ``` + SQL> @03_im_usage.sql + Connected. + SQL> column pool format a10; + SQL> column alloc_bytes format 999,999,999,999,999 + SQL> column used_bytes format 999,999,999,999,999 + SQL> + SQL> -- Show total column store usage + SQL> + SQL> SELECT pool, alloc_bytes, used_bytes, populate_status, con_id + 2 FROM v$inmemory_area; + + POOL ALLOC_BYTES USED_BYTES POPULATE_STATUS CON_ID + ---------------- -------------------- -------------------- -------------------------- ---------- + 1MB POOL 3,288,334,336 0 DONE 3 + 64KB POOL 167,772,160 0 DONE 3 + IM POOL METADATA 16,777,216 16,777,216 DONE 3 + + SQL> + ``` + +5. The following query accesses the USER_TABLES view and displays attributes of the tables in the SSB schema. + + Run the script *04\_im\_attributes.sql* + + ``` + + @04_im_attributes.sql + + ``` + + or run the query below: + + ``` + + column table_name format a12; + column partition_name format a15; + column buffer_pool format a11; + column compression heading 'DISK|COMPRESSION' format a11; + column compress_for format a12; + column INMEMORY_PRIORITY heading 'INMEMORY|PRIORITY' format a10; + column INMEMORY_DISTRIBUTE heading 'INMEMORY|DISTRIBUTE' format a12; + column INMEMORY_COMPRESSION heading 'INMEMORY|COMPRESSION' format a14; + select table_name, NULL as partition_name, buffer_pool, compression, compress_for, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression + from user_tables + where table_name in ('DATE_DIM','PART','SUPPLIER','CUSTOMER') + UNION ALL + select table_name, partition_name, buffer_pool, compression, compress_for, inmemory, + inmemory_priority, inmemory_distribute, inmemory_compression + from user_tab_partitions + where table_name = 'LINEORDER'; + + ``` + + Query result: + + ``` + SQL> @04_im_attributes.sql + Connected. + SQL> + SQL> -- Show table attributes + SQL> + SQL> select table_name, NULL as partition_name, buffer_pool, compression, compress_for, inmemory, + 2 inmemory_priority, inmemory_distribute, inmemory_compression + 3 from user_tables + 4 where table_name in ('DATE_DIM','PART','SUPPLIER','CUSTOMER') + 5 UNION ALL + 6 select table_name, partition_name, buffer_pool, compression, compress_for, inmemory, + 7 inmemory_priority, inmemory_distribute, inmemory_compression + 8 from user_tab_partitions + 9 where table_name = 'LINEORDER'; + + DISK INMEMORY INMEMORY INMEMORY + TABLE_NAME PARTITION_NAME BUFFER_POOL COMPRESSION COMPRESS_FOR INMEMORY PRIORITY DISTRIBUTE COMPRESSION + ------------ --------------- ----------- ----------- ------------ -------- ---------- ------------ -------------- + CUSTOMER KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + DATE_DIM KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + PART KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + SUPPLIER KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + LINEORDER PART_1994 KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + LINEORDER PART_1995 KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + LINEORDER PART_1996 KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + LINEORDER PART_1997 KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + LINEORDER PART_1998 KEEP ENABLED BASIC ENABLED NONE AUTO AUTO + + 9 rows selected. + + SQL> + ``` + + Note that tables enabled for inmemory will have the inmemory attribute of ENABLED. The default priority level is NONE which means that the object is not populated until it is first accessed. If the priority is set to any value other than NONE then the object will not be eligible for eviction. However, note that both the inmemory distribute and inmemory compression fields are set to AUTO. Recall that back in step 2 the parameter INMEMORY\_AUTOMATIC\_LEVEL was set to HIGH. This means that Automatic In-Memory was set to HIGH which enables all non-system objects to be eligible to be populated in the IM column store. We will talk more about this in the last lab. + +6. Let's populate the IM column store by accessing the tables that are enabled for inmemory with the following queries: + + Run the script *05\_im\_start_pop.sql* + + ``` + + @05_im_start_pop.sql + + ``` + + or run the query below: + + ``` + + select /*+ full(LINEORDER) noparallel(LINEORDER) */ count(*) from LINEORDER; + select /*+ full(PART) noparallel(PART) */ count(*) from PART; + select /*+ full(CUSTOMER) noparallel(CUSTOMER) */ count(*) from CUSTOMER; + select /*+ full(SUPPLIER) noparallel(SUPPLIER) */ count(*) from SUPPLIER; + select /*+ full(DATE_DIM) noparallel(DATE_DIM) */ count(*) from DATE_DIM; + + ``` + + Query result: + + ``` + SQL> @05_im_start_pop.sql + Connected. + SQL> + SQL> -- Access tables enabled for in-memory to start population + SQL> + SQL> select /*+ full(LINEORDER) noparallel(LINEORDER) */ count(*) from LINEORDER; + + COUNT(*) + -------------------- + 41760941 + + SQL> select /*+ full(PART) noparallel(PART) */ count(*) from PART; + + COUNT(*) + -------------------- + 800000 + + SQL> select /*+ full(CUSTOMER) noparallel(CUSTOMER) */ count(*) from CUSTOMER; + + COUNT(*) + -------------------- + 300000 + + SQL> select /*+ full(SUPPLIER) noparallel(SUPPLIER) */ count(*) from SUPPLIER; + + COUNT(*) + -------------------- + 20000 + + SQL> select /*+ full(DATE_DIM) noparallel(DATE_DIM) */ count(*) from DATE_DIM; + + COUNT(*) + -------------------- + 2556 + + SQL> + ``` + + Note the FULL and NOPARALLEL hints. These have been added to ensure that the table data is also read into the KEEP pool that was defined. This is only done for this Lab so that we can show you a true memory based comparison of the performance of the Database In-Memory columnar format versus the traditional row format fully cached in the buffer cache. This is not required to initiate Database In-Memory population. + +7. To identify which segments have been populated into the IM column store you can query the view V$IM\_SEGMENTS. Once the data population is complete, the BYTES\_NOT\_POPULATED attribute should be 0 for each segment. + + Run the script *06\_im\_populated.sql* + + ``` + + @06_im_populated.sql + + ``` + + or run the query below: + + ``` + + column owner format a10; + column segment_name format a20; + column partition_name format a15; + column populate_status format a15; + column bytes heading 'Disk Size' format 999,999,999,999 + column inmemory_size heading 'In-Memory|Size' format 999,999,999,999 + column bytes_not_populated heading 'Bytes|Not Populated' format 999,999,999,999 + select owner, segment_name, partition_name, populate_status, bytes, + inmemory_size, bytes_not_populated + from v$im_segments + order by owner, segment_name, partition_name; + + ``` + + Query result: + + ``` + SQL> @06_im_populated.sql + Connected. + SQL> + SQL> -- Query the view v$IM_SEGMENTS to shows what objects are in the column store + SQL> -- and how much of the objects were populated. When the BYTES_NOT_POPULATED is 0 + SQL> -- it indicates the entire table was populated. + SQL> + SQL> select owner, segment_name, partition_name, populate_status, bytes, + 2 inmemory_size, bytes_not_populated + 3 from v$im_segments + 4 order by owner, segment_name, partition_name; + + In-Memory Bytes + OWNER SEGMENT_NAME PARTITION_NAME POPULATE_STATUS Disk Size Size Not Populated + ---------- -------------------- --------------- --------------- ---------------- ---------------- ---------------- + SSB CUSTOMER COMPLETED 24,928,256 23,199,744 0 + SSB DATE_DIM COMPLETED 122,880 1,179,648 0 + SSB LINEORDER PART_1994 COMPLETED 565,338,112 479,330,304 0 + SSB LINEORDER PART_1995 COMPLETED 565,354,496 479,330,304 0 + SSB LINEORDER PART_1996 COMPLETED 567,484,416 480,378,880 0 + SSB LINEORDER PART_1997 COMPLETED 564,281,344 479,330,304 0 + SSB LINEORDER PART_1998 COMPLETED 330,407,936 279,642,112 0 + SSB PART COMPLETED 56,893,440 16,973,824 0 + SSB SUPPLIER COMPLETED 1,769,472 2,228,224 0 + + 9 rows selected. + + SQL> + ``` + +8. Now let's check the total space usage used in the IM column store. + + Run the script *07\_im\_usage.sql* + + ``` + + @07_im_usage.sql + + ``` + + or run the query below: + + ``` + + column pool format a16; + column alloc_bytes format 999,999,999,999,999 + column used_bytes format 999,999,999,999,999 + column populate_status format a15; + SELECT pool, alloc_bytes, used_bytes, populate_status, con_id + FROM v$inmemory_area; + + ``` + + Query result: + + ``` + SQL> @07_im_usage.sql + Connected. + SQL> column pool format a16; + SQL> column alloc_bytes format 999,999,999,999,999 + SQL> column used_bytes format 999,999,999,999,999 + SQL> column populate_status format a15; + SQL> + SQL> -- Show total column store usage + SQL> + SQL> SELECT pool, alloc_bytes, used_bytes, populate_status, con_id + 2 FROM v$inmemory_area; + + POOL ALLOC_BYTES USED_BYTES POPULATE_STATUS CON_ID + ---------------- -------------------- -------------------- --------------- ---------- + 1MB POOL 3,288,334,336 2,235,564,032 DONE 3 + 64KB POOL 167,772,160 6,029,312 DONE 3 + IM POOL METADATA 16,777,216 16,777,216 DONE 3 + + SQL> + ``` + +9. Lets also take a look at the current Heat Map statistics. Automatic In-Memory does not require that Heat Map be enabled, but under the covers it uses the same basic information. We will list that starting heat map statistics in this step and then we will take a look at the statistics in the last lab and compare that with how it affected AIM. + + Run the script *08\_hm\_stats.sql* + + ``` + + @08_hm_stats.sql + + ``` + + or run the query below: + + ``` + + col owner format a10; + col object_name format a20; + col subobject_name format a15; + col track_time format a16; + col segment_write heading 'SEG|WRITE' format a10; + col segment_read heading 'SEG|READ' format a10; + col full_scan heading 'FULL|SCAN' format a10; + col lookup_scan heading 'LOOKUP|SCAN' format a10; + col n_fts heading 'NUM FULL|SCAN' format 99999999; + col n_lookup heading 'NUM LOOKUP|SCAN' format 99999999; + col n_write heading 'NUM SEG|WRITE' format 99999999; + select + OWNER, + OBJECT_NAME, + SUBOBJECT_NAME, + to_char(TRACK_TIME,'MM/DD/YYYY HH24:MI') track_time, + SEGMENT_WRITE, + SEGMENT_READ, + FULL_SCAN, + LOOKUP_SCAN, + N_FTS, + N_LOOKUP, + N_WRITE + from + sys."_SYS_HEAT_MAP_SEG_HISTOGRAM" h, + dba_objects o + where + o.object_id = h.obj# + and track_time >= sysdate-1 + order by + track_time, + OWNER, + OBJECT_NAME, + SUBOBJECT_NAME; + + ``` + + Query result: + + ``` + SQL> @08_hm_stats.sql + Connected. + + SEG SEG FULL LOOKUP NUM FULL NUM LOOKUP NUM SEG + OWNER OBJECT_NAME SUBOBJECT_NAME TRACK_TIME WRITE READ SCAN SCAN SCAN SCAN WRITE + ---------- -------------------- --------------- ---------------- ---------- ---------- ---------- ---------- --------- ---------- --------- + SSB CUSTOMER 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB DATE_DIM 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB LINEORDER PART_1994 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB LINEORDER PART_1995 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB LINEORDER PART_1996 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB LINEORDER PART_1997 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB LINEORDER PART_1998 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB PART 06/12/2024 12:20 NO YES YES NO 1 0 0 + SSB SUPPLIER 06/12/2024 12:20 NO YES YES NO 1 0 0 + + 9 rows selected. + + SQL> + ``` + + Note that there has been 1 full table scan on each of the tables and no other activity. In Step 6 we ran a query against each table and that has been the only access so far. In the last lab we will explore heat map in more detail and what affect it has on AIM processing. + +10. Exit lab + + Type commands below: + + ``` + + exit + cd .. + + ``` + + Command results: + + ``` + SQL> exit + Disconnected from Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production + Version 23.4.0.24.05 + [CDB1:oracle@dbhol:~/labs/inmemory/setup]$ cd .. + [CDB1:oracle@dbhol:~/labs/inmemory]$ + ``` + +## Conclusion + +In this lab you saw that the IM column store is configured by setting the initialization parameter INMEMORY_SIZE. The IM column store is another pool in the SGA, and once allocated it can be increased in size dynamically. + +You also had an opportunity to populate and view objects in the IM column store and to see how much memory they use. In this lab we populated five tables into the IM column store, and the LINEORDER table is the largest of the tables populated with over 41 million rows. You may have noticed that it is also a partitioned table. We will be using that attribute in later labs. + +Remember that the population speed depends on the CPU capacity of the system as the in-memory population is a CPU intensive operation. The more CPU and processes you allocate the faster the populations will occur. + +Finally you got to see how to determine if the objects were fully populated and how much space was being consumed in the IM column store. + +You may now **proceed to the next lab**. + +## Acknowledgements + +- **Author** - Andy Rivenes, Product Manager, Database In-Memory +- **Contributors** - Maria Colgan, Rene Fontcha +- **Last Updated By/Date** - Andy Rivenes, Product Manager, Database In-Memory, June 2024 diff --git a/in-memory-23ai/workshops/ocw24-sandbox/index.html b/in-memory-23ai/workshops/ocw24-sandbox/index.html new file mode 100644 index 000000000..aaac634be --- /dev/null +++ b/in-memory-23ai/workshops/ocw24-sandbox/index.html @@ -0,0 +1,62 @@ + + + + + + + + + Oracle LiveLabs + + + + + + + + + + + + +
+
+
+
+
+
+
+
+ + + + + diff --git a/in-memory-23ai/workshops/ocw24-sandbox/manifest.json b/in-memory-23ai/workshops/ocw24-sandbox/manifest.json new file mode 100644 index 000000000..5cd75908d --- /dev/null +++ b/in-memory-23ai/workshops/ocw24-sandbox/manifest.json @@ -0,0 +1,50 @@ +{ + "workshoptitle": "Learn How Database In-Memory Enables Real-Time Analytics", + "help": "livelabs-help-db_us@oracle.com", + "tutorials": [ + { + "title": "Introduction", + "description": "About the In-Memory Workshop", + "filename": "../../intro/intro.md" + }, + { + "title": "Lab 1: Environment Setup", + "description": "Verify Setup of compute instance", + "filename": "https://oracle-livelabs.github.io/common/labs/verify-compute/verify-compute-ssh-and-novnc.md" + }, + { + "title": "Lab 2: Initialize Environment", + "description": "Initialize Environment", + "filename": "../../initialize-environment/initialize-environment.md" + }, + { + "title": "Lab 3: Setting up the In-Memory Column Store", + "description": "Populating and administering the In-Memory column store", + "filename": "../../setup/setup.md" + }, + { + "title": "Lab 4: Querying the In-Memory Column Store", + "description": "Queries on In-Memory Column Store", + "filename": "../../queries/queries.md" + }, + { + "title": "Lab 5: Joins and Aggregation", + "description": "In-Memory Joins and Aggregation", + "filename": "../../joins-aggregations/joins-aggregations.md" + }, + { + "title": "Lab 6: Automatic In-Memory in 23ai", + "description": "Using AIM in 23ai", + "filename": "../../aim-23ai/aim-high.md" + }, + { + "title": "Need Help?", + "description": "Solutions to Common Problems and Directions for Receiving Live Help", + "filename":"https://oracle-livelabs.github.io/common/labs/need-help/need-help-livelabs.md" + }, + { + "title": "Oracle CloudWorld 2024 - Get Help", + "filename": "https://oracle-livelabs.github.io/common/support/ocwsupportlab/ocwsupportlab.md" + } + ] +} diff --git a/in-memory-23ai/workshops/ocw24-tenancy/index.html b/in-memory-23ai/workshops/ocw24-tenancy/index.html new file mode 100644 index 000000000..aaac634be --- /dev/null +++ b/in-memory-23ai/workshops/ocw24-tenancy/index.html @@ -0,0 +1,62 @@ + + + + + + + + + Oracle LiveLabs + + + + + + + + + + + + +
+
+
+
+
+
+
+
+ + + + + diff --git a/in-memory-23ai/workshops/ocw24-tenancy/manifest.json b/in-memory-23ai/workshops/ocw24-tenancy/manifest.json new file mode 100644 index 000000000..e65a00094 --- /dev/null +++ b/in-memory-23ai/workshops/ocw24-tenancy/manifest.json @@ -0,0 +1,63 @@ +{ + "workshoptitle": "Learn How Database In-Memory Enables Real-Time Analytics", + "help": "livelabs-help-db_us@oracle.com", + "tutorials": [ + { + "title": "Introduction", + "description": "About the In-Memory Workshop", + "filename": "../../intro/intro.md" + }, + { + "title": "Get Started", + "description": "Login to Oracle Cloud", + "filename": "https://oracle-livelabs.github.io/common/labs/cloud-login/pre-register-free-tier-account.md" + }, + { + "title": "Lab 1: Prepare Setup", + "description": "How to download your ORM stack and update security rules for an existing VCN", + "publisheddate": "09/28/2020", + "filename": "../../prepare-setup/prepare-setup.md", + "type": "basics" + }, + { + "title": "Lab 2: Environment Setup", + "description": "How to provision the workshop environment and connect to it", + "publisheddate": "06/30/2020", + "filename": "https://oracle-livelabs.github.io/common/labs/setup-compute-generic/setup-compute-novnc-ssh.md" + }, + { + "title": "Lab 3: Initialize Environment", + "description": "Initialize Environment", + "filename": "../../initialize-environment/initialize-environment.md" + }, + { + "title": "Lab 4: Setting up the In-Memory Column Store", + "description": "Populating and administering the In-Memory column store", + "filename": "../../setup/setup.md" + }, + { + "title": "Lab 5: Querying the In-Memory Column Store", + "description": "Queries on In-Memory Column Store", + "filename": "../../queries/queries.md" + }, + { + "title": "Lab 6: Joins and Aggregation", + "description": "In-Memory Joins and Aggregation", + "filename": "../../joins-aggregations/joins-aggregations.md" + }, + { + "title": "Lab 7: Automatic In-Memory in 23ai", + "description": "Using AIM in 23ai", + "filename": "../../aim-23ai/aim-high.md" + }, + { + "title": "Environment Cleanup", + "description": "Cleanly dispose of all OCI resources created by ORM for the workshop, and delete the stack", + "filename": "https://oracle-livelabs.github.io/common/labs/cleanup-stack/cleanup-stack.md" + }, + { + "title": "Oracle CloudWorld 2024 - Get Help", + "filename": "https://oracle-livelabs.github.io/common/support/ocwsupportlab/ocwsupportlab.md" + } + ] +}