Thursday, February 11, 2016

When the Runtime Column Propagation (RCP) is trurned on in DataStage?

The Runtime Column Propagation (RCP) is OFF by default.

With the RCP is trurned on in DataStage:
  • Source data columns are read and flow through a stage or stages without explicitly defined in stage
  • Explicit mapping is not required from Source to the Target columns in a stage 
  • Also, DataStage does not enforce or validate any column mapping during design/compile time.
  • Input columns are mapped to un-mapped columns by name
  • No Modify operator inserted at compile time by DataStage 
  • Runtime error can occur if incoming column names do not match column names outgoing link (case sensitivity)
How RCP works in design time and runtime?

When working with Sequential files (at source level): 
  • We can use a schema file to read out underline schema details.
  • In Columns structure tab, we can define columns with appropriate datatypes
  • Then DataStage according propagates values at the 
When working with Database Stages (at source level):
  • Configure the SQL in Database stages (Oracle Enterprise/connector) as 
  • "Select * from <schema.><table_name>" 
  • Then, explicitly define as an output column in the Database stage column structure tab and accordingly what ever column structure / metadata provided (if exists in database table) DataStage run-time will propagates those column values
The RCP Enable/Disable done at?
  • Project level:  in Administrator project properties
  • Job level: Job properties General tab
  • Stage/s:  Link Output Column tab
The RCP as design preference:
  • It is good for the initial development and testing phase but once requirements, project plans and data models are finalized, must be enforced for source files structures/schemas 
  • Personally thinking while designing it should be better avoided rather than being surprised at real runtime or during the production schedule as we may not know what types of data may come  
  • It can involve frustrating production issues, fix patches, with additional development and testing time  

1 comment:

  1. Hi
    i have 4 sources(say 4 country oracle tables) which has to be loaded to single table and they have same schema for example say 1 country table has hashvalue which needs to defaulted to 0 or 1 ..ie minior changes while extracting data from source.
    please through some lights as i am new to this rcp and i have to build generic job

    Thanks in advance

    ReplyDelete