Search This Blog

Tuesday, December 27, 2011

PI tips: PI 7.3 processing of large files - teaser

In the past many companies used SAP PI not only for "real" integration scenarios that would require sender and receiver applications, mappings and complex routings but also for simple file transfers across folders or locations. Such simple transfers could be handled without SAP PI using different sorts of scripts but why use scripts if you have a nice middleware tool - SAP PI. Unfortunately SAP PI was not very strong with handling such files especially if they were very large. SAP PI would process the whole file at once and this caused many implications: 

- the file could not be larger then the memory size 
- handling of such big files caused other integration flows to run much slower 

As of PI 7.3 SAP has prepared a remedy for this. It's now possible push via SAP PI all of your big files if you use special options available in the file adapter and they won't cause any negative side effects as they will be processed in small chunks. How does the configuration look like then? 


Step 1

At first you need to make sure you're transfer will be EOIO (as only like that the chunks will not get mixed up). 

image

Step 2 

Make sure you've set the transfer up as Binary 

Step 3 

On the advanced tab select "advanced mode" and file splitting active and then choose the maximum size of a file part (in Mbs) 

image


Once you configure that your huge files will be processed in smaller chunks which will not impact your other flows and the file size can now be almost unlimited :)

File/FTP Adapter - Large File Transfer (Chunk Mode)


Background

Whenever large (binary) file needs to be transferred by using PI, there might be a chance for Out of Memory error which might results in server restart.  Technically, it often fails in the sender adapter.

image

The most popular work around is to split the file in to multiple parts in Sender Adapter and Combining them in Receiver Adapter (via OS Scripts)
From PI 7.30 onwards File/FTP adapter natively supports transferring large size files by splitting them in to smaller chunks based on the configured size. Each chunk will be processed as an individual XI Message in sender adapter and all the chunks are combined based on the sequence in receiver adapter. Since all the chunks have to be combined in the actual order, QoS EOIO has to be used to enable this feature.

image

Configuration Options

Select 'EOIO' as a QoS under Processing tab.

image

After that select 'Advanced Mode' under 'Advanced' Tab

Option 1

You can choose the required chunk size option from the given list of 1, 2, 5, 10, 20 and 50 MB.

image

Option 2

Custom values which are not possible in default options can also be configured in additional parameters.

Parameter Name
Possible Values
chunkMode  
Non emptry string  will be considered as true
chunkSizeKB
Positive Numeric  Value


image

Processing in Sender Adapter

  • Basic Mode will be enabled only if QoS is EOIO
  • Chunk Mode related Advanced Mode parameters will be ignored if QoS  not EOIO
  • If chunkSizeKB is not defined /incorrect value, default is 10KB
  • If both are configured, Basic mode will have higher precedence
Sender Adapter will create XI message with additional dynamic headers for each chunk and forwarded to Messaging System for IRD processing.

 Example

For this example, chunk size is configured as 2 MB and the below input file (~4.5 MB) is used.
 image

For demo purpose and to identify/analyze how the actual split happens, the input file is created with the data like below ( 1 to 600000)
 image

As per the configuration, three chunks should be created with the size of 2 MB, 2MB and ~500 KB.
image

Chunk 1 (2 MB)

Transferring first 2 MB
image

Chunk 2 (2 MB) 

Transferring next 2 MB (Total 4 MB)
image

Chunk 3 (~500 KB)

Transferring the remaining ~500 KB (Total ~4.5 MB)
image

Dynamic Headers

The entire chunk mode related dynamic headers will be set under the namespace http://sap.com/xi/XI/System/Generic
HeaderValue
ChunkStartStart of the Chunk (Position of bytes)
ChunkModeActive/End
ChunkKeyUnique Key (Same for all the chunks for a particular file)

 The corresponding screenshots for all 3 chunks are given below.

First Chunk

image

Next Chunk 

image

Last Chunk

image

If multiple files sent from different file sender channels, receiver adapter uses the chunk key to identify the proper output file(for combining). During the creation(and append) of output file, it uses the chunk key like below.

image

In Temp mode, the temp files won’t be deleted till the last chunk is received.

Overall Control Flow


image

Recovery Mode

If the input file is very large, it would normally take more time to be transferred completely.  If there is any server failure in the middle, the transfer can resumed from the chunk it stopped (Not from the beginning)

Individual Payloads after Split

Though this feature operates only on binary mode, text file is used to analyze how the split happens during chunk creation.

Payload of Chunk 1

image

Payload of Chunk 2

image

Payload of Chunk 3

image
As per the above screenshots, the split never cosiders the payload. It's just a binary split. So the following limitations would apply
  • Only for File Sender to File Receiver
  • No Mapping
  • No Content Based Routing
  • No Content Conversion
  • No Custom Modules

How to choose the chunk size? Smaller chunk size ensures more reliability?
Yes.  But significantly affects the performance. Smaller chunk size creates more chunks (more XI messages)

Thursday, December 15, 2011

Important URLs, Transaction Codes, ABAP Reports in Process Integration 7.1


Hi, I would like to share important URLs, Transaction Codes and ABAP Reports that are used in Process Integration PI 7.1 
I) Process Integration PI 7.1 URLs 
ADesign Time Tools URLs 
1http://<host>:5<sys#>00/dir/start/index.jspProcess Integration PI 7.1 tools
2http://<host>:5<sys#>00/sld/index.htmlSystem Landscape Directory (SLD)
3http://<host>:5<sys#>00/rep/start/repository.jnlpEnterprise Services Repository (ESR)
4http://<host>:5<sys#>00/dir/start/directory.jnlpIntegration Directory(ID)
5http://<host>:5<sys#>00/srService Registry(SR)
BRuntime Tools URLs 
1http://<host>:5<sys#>00/rwbRuntime Workbench(RWB)
2http://<host>:5<sys#>00/nwaLocal SAP Netweaver Administrator (NWA)
3http://<host>:5<sys#>00/nwa/quicklinksNWA Quicklinks
4http://<host>:5<sys#>00/nwapiCentral SAP Netweaver Administrator (Visual Admin in XI)
5http://<host>:5<sys#>00/index.htmlNetweaver Application Server Java
6http://<host>:5<sys#>00/CPACacheCPA Cache Monitoring
7http://<host>:5<sys#>00/CPACache/refresh?mode=deltaDelta CPA Cache refresh
8http://<host>:5<sys#>00/CPACache/refresh?mode=fullFull CPA Cache Refresh
9http://<host>:5<sys#>00/mdtMessage Display Tool (MDT)
10http://<host>:5<sys#>00/mdt/channelmonitorservletCommunication channel Monitoring
11http://<host>:5<sys#>00/mdt/amtServletAdapter Monitoring
12http://<host>:5<sys#>00/AdapterFramework/admin/channelstatus.jspCommunication channel Status
13http://<host>:5<sys#>00/MessagingSystemMessage Display Tool
14http://<host>:5<sys#>00/MessagingSystem/monitor/sequenceMonitor.jspEOIO Sequence monitor
15http://<host>:5<sys#>00/MessagingSystem/monitor/systemStatus.jspMessage Queue monitor
16http://<host>:5<sys#>00/rep/support/public/LockAdminServiceUnlock Enterprise Service objects
17http://<host>:5<sys#>00/TREX/MonitorTREX Application
18http://<host>:5<sys#>00/rwb/rtc?op=initRe-Initialize Runtime Workbench
II) Process Integration PI 7.1 Transaction Codes 
AAdministrative Transactions-ABAP 
1SXMB_IFRStart Enterprise Service Builder
2SXMB_ADMIntegration Engine Administration
3SMQRqRFC Monitor (QIN Scheduler / Registration)
4SMQ1qRFC Monitor (Outbound Queue)
5SMQ2qRFC Monitor (Inbound Queue)
6AL11SAP Directories Display
7SU01User Maintenance
8SICFHTTP Server Configuration
9AL08List Logged in users
10SM21System Error Logs
11ST22ABAP Runtime Errors
12SLDCHECKTest SLD Connection
13SXI_SUPPORTTest all Repository and Directory Objects
14SM58tRFC error Logs
15SLDAPICUSTSLD API Customizing
16SMICMICM Monitor
17STRUSTTrust Manager
18RZ20CCMS Monitoring
19RZ70SLD Administration
BWeb Service Transaction-ABAP 
1WSADMINWeb Service Administration
2WSCONFIGWeb Service Configuration
3SUDDIREGMaintaining UDDI Registry
4WSADMIN2Web Service Administration
5SOAMANAGERStart SOA Manager
6WSPUBLISHPublish Web Services
CIDOC Transactions-ABAP 
1IDX1Port Maintanenece in IDOC Adapter
2IDX2Metadata Overview for IDOC Adapter
3IDX5XML Messages in Adapter
4WE02IDOC Display
5WE05IDOC List
DBPM Transactions - ABAP 
1SXMB_MONI_BPEMonitoring Business Process Engine
2SWF_XI_PBUILDERDetailed BPM Process Builder
3SWF_XI_CUSTOMIZINGCheck Integration Process prerequisite
4SWF_XI_ADM_BPEStart and Stop Business Process Engine
5SWF_XI_ADM_BPE_DISPDisplay Status of Business Process Engine
6SWELSSwitch Event Trace On/Off
EMonitoring Transactions-ABAP 
1SXMB_MONIIntegration Engine Monitoring
2SXI_MONITORMonitor for Processed XML Messages
4SXI_CACHEPI Runtime Cache
5SM37Simple Job Selection
6SMGWGateway Monitor
FAlert Transactions-ABAP 
1ALRTCATDEFEditing Alert Category
2ALRTINBOXCall Alert Inbox
3ALRTDISPDisplay of Alerts
4ALRTPROCProcessing Alerts
GOther Transactions-ABAP 
1SE10Transport Organizer
2SE37Function Modules Builder
3SE38ABAP Editor
4SE11ABAP Dictionary
5SE80Object Navigator
6SE24OOABAP class Builder
7SM30Maintain Table View
8SXI_MAPPING_TESTTest ABAP Mapping
9SPROXYABAP Proxy Generation
10ST05Performance Analysis
 III) Process Integration PI 7.1 ABAP Reports

AABAP Reports 
1RSEOUT00Collects maximum no of IDOCs at a time and places in a file
2RSQIWKEXAutomatic qRfC failure recover report
3RSQOWKEXQout Scheduler-execution of registered outbound queues
4RSARFCEXAutomatic tRfC failure recover report
5RSPARAMDisplay Profile parameters
6RSBDCOSOExecute OS Command
7RSXMB_CANCEL_NO_COMMIT_MSGCancel message processing for messages without commit
8RSXMB_RESTART_MESSAGESAutomatic Mass Restart Scheduling Report
9RSXMB_SELECT_MESSAGESBackground to get a daily report of XI messages
10RSWF_XI_UNSENT_MSGSStatus of all the messages in XI server
11RSXMB_REMOTE_SERVICEStandard ABAP report for XI performance monitoring
12SPROX_CHECK_IFR_ADDRESSCheck if IFR address is maintained
13SRFCCONNRFC Test
14SXMS_MMF_MONITORDisplay bulk messages and individual messages
15SXMS_PF_LEVEL12_AGGDATAFunction module to get aggregated performance
16SXMS_REFRESH_ADAPTER_STATUSRefresh of outbound adapter status only for IDOC Adapter
17RSALERTPROCTrigger Alert
18RSALERTCATDEFMaintaining alert categories and alert server landscape
19SXMSALERT_LOGREADERdisplay log information for alerts report
20RSXMB _ ACTIVATE_ICF_SERVICESActivate HTTP Services for PMI
21RSXMB_CONFIG_PPEnable Principal Propagation of Messaging Components
22RSXMB_AGG_XPI_ADMINActivate Function for ABAP Runtime
23SXMSALERT_CONFIGTRANSTransport Alert Rules via CTS+
24IDX_NOALEConfigure IDoc Adapter Acknowledgements