Another article in Ashok's series where he examines Oracle Data Integrator (ODI) Knowledge Modules (KM). For an basic introductory overview, check out 'What is an ODI Knowledge Module (KM)?'
Journalizing, commonly known as Change Data Capture (CDC), allows us to track changes that occur in data via inserts, updates or deletes. This enables a data transfomation tool, like Oracle Data Inetegrator (ODI), to load and process only changed or incremental data from a data source to a target. This eliminates processing of unchanged data during incremental loads which can significantly improve the performance of data extraction, transformation, and load processes.
Journalizing requires an infrastructure setup that can capture changes made to source records. Oracle Data Inetegrator (ODI) simplifies Change Data Capture (CDC) processes by providing pre-built functionality within "Knowledge Modules". These ODI Knowledge Modules leverage these Change Data Capture (CDC) mechanisms.
Unlike other Knowledge Modules which are used in Oracle Data Integrator (ODI) interfaces, Journalizing takes place within a model, sub model or in a datastore.
The limitation of Simple Jounalizing is that referential integrity is not checked across datastores. For example, when employee information is extracted from PeopleSoft and from ADP Payroll, there is no guarantee that the newly captured insert data from ADP Payroll will have an associated record for an Employee in PeopleSoft.
As an alternate example with Consistent Set Journalizing, changes from ADP Payroll will be captured only when the associated PeopleSoft employee changes have been captured (and vice versa). It guarantees the consistency of the changes.
The set of available of changes for which consistency is guaranteed is called the Consistency Window. Changes in this window should be processed in the correct order by designing and sequencing the integration interfaces into packages (employee followed by payroll).
Although Consistent Set Journalizing is more powerful, it is also more difficult to set up. Consistent Set Journalizing should be used when referential integrity constraints need to be ensured when capturing the data changes, as in our example scenario. Consistent Set Journalizing is also recommended is also recommended when a large number of records are required and performance is a concern.
NOTE: It is not possible to journalize a model or datastores within a model using both Consistent Set Journalizing and Simple Journalizing.
Journalizing Knowledge Module |
Description |
JKM Oracle to Oracle Consistent (OGG) |
Creates and manages the ODI CDC framework infrastructure when using Oracle GoldenGate for CDC |
JKM Oracle 10g Consistent (Streams) |
Creates the journalizing infrastructure for consistent set journalizing on Oracle 10g tables, using Oracle Streams. |
JKM Oracle 11g Consistent (Streams) |
Creates the journalizing infrastructure for consistent set journalizing on Oracle 11g tables, using Oracle Streams. |
JKM Oracle Consistent |
Creates the journalizing infrastructure for consistent set journalizing on Oracle tables using triggers. |
JKM Oracle Consistent (Update Date) |
Creates the journalizing infrastructure for consistent set journalizing on Oracle tables using triggers based on a Last Update Date column on the source tables. |
JKM Oracle Simple |
Creates the journalizing infrastructure for simple journalizing on Oracle tables using triggers. |
A Service Knowledge Module (SKM) is in charge of creating and deploying data manipulation web services to a Service Oriented Architecture (SOA) infrastructure. These web services provide access to data in a datastore via a web interface. Web services also allows access to changed captured data for these datastores.
Service Knowledge Modules (SKM) are similar to Journalizing Knowledge Modules (JKM). But unlike other Oracle Data Integrator Knowledge Modules, SKMs do not generate executable code. Instead, SKMs generate Web Services Deployment Archive Files. SKMs are designed to generate Java code using Oracle Data Integrator's framework for Web Services. The Java code is then compiled and then eventually deployed on the Application Server's containers.
Configuring the Web Service Containers
A web services container must be declared as a data server in the topology in order to let Oracle Data Integrator (ODI) deploy the web services upon iteslf. Configuration of a web services container depends on the type of web service container and the deployment mode.
ODI Supports Two Types of Web Service Container:
ODI Provides the Following Three Deployment Modes:
The Data Services generated by ODI do not contain connection information for sources and targets. Instead, it refers to data sources defined within the Web Services container or on the application server. These data sources contain connection properties required to access data, and must correspond to data servers already defined within the ODI topology.
A Data Source Can Be Configured Using One of the Following Options:
To configure Data Services, you must first create and populate a model. Information about application server, data source name, data service name and service knowledge modules are specified in a model.
After configuration, next step is to generate and deploy the data services. Generating the data services for a model generates model-level services when the model is enabled for consistent set CDC and datastore-level services. Each generated data service offers a range of operations depending on the SKMs used to generate it.
The Following Services are Available at Model-Level:
The Following Operations are Available at Datastore-Level:
SKM <Data server technology>
Data server technology is the technology into which the data to be accessed with web services is stored.
List of supported SKMs can be found in ODI Studio and also can be seen in installation directory <ODI Home>\oracledi\xml-reference.
Below are Examples of a Few SKMs:
Service Knowledge Module |
Description |
SKM Oracle |
Generates data access Web services for Oracle databases. |
SKM Informix |
Generates data access Web services for IBM Informix databases. |
SKM IBM UDB |
Generates data access Web services for IBM DB2 UDB databases. |
SKM HSQL |
Generates data access Web services for Hypersonic SQL databases. |
SKM SQL |
Generates data access Web services for ANSI SQL-92 compliant databases. Data access services include data manipulation operations such as adding, removing, updating or filtering records as well as changed data capture operations such as retrieving changed data. Data manipulation operations are subject to integrity check as defined by the constraints of source datastores. Consider using this SKM if you plan to generate and deploy data manipulation or changed data capture web services to your Service Oriented Architecture infrastructure. Use specific SKMs instead, if available for your database. |
Ashok Das is a Senior Consultant at KPI Partners and an Oracle Certified Specialist who specializes in OBIEE and the Oracle BI Applications. He has implemented BI Apps solutions for clients across various industries that include in Sales, Marketing, Supply Chain & Order Management, Financials, Projects, HR, and Procurement & Spend Analytics on source systems such as JDE, Salesforce, Siebel, Peoplesoft, and eBusiness. Check out Ashok's blog at KPIPartners.com. |