Managed File Transfer Capabilities in the WebLogic Platform
Handling Very Large Files
Trading partners that send file-based orders could send very large files to the OMS of GlobalTx. These files could typically be more than 100 megabytes (MB) in size. Using traditional means to handle large document files sent in a single batch can create havoc for the system. This is because XML transformation entails the use of a lot of memory, which is rapidly exhausted due to the size of these very large files.
WebLogic Platform uses file control to easily read, write, or append to a file in a file system. In case of files of enormous size containing a lot of individual records, one approach is to use the file control to split the file into individual records using patterns. Look here for more information on how this design pattern can be employed.
Additionally using file controls for handling documents provides optimal performance. Following are the some of the features and advantages of using the WebLogic Platform file control to process very large files.
- The file control uses Java 1.4.1 NIO for increased performance.
- It supports read and write of large XML files.
- It supports readline and append operations for very large files.
- The file event generator supports pass-by-filename for extremely large files.
End-to-End File Management
File management is another key aspect for file transfer to be viable in a mission-critical, real-time environment as required by GlobalTx. As the usage in a real-life B2B application scenario extends beyond the boundaries of an enterprise, a single point of control with visibility of all transfer activity throughout the enterprise and beyond becomes an essential requirement. Figure 2 shows how GlobalTx can use WebLogic Portal's file system-based content repository to manage the files and associated metadata that the trading partner sends and receives. It provides high-level semantics such as versioning, folder management, check-in/check-out, and metadata management.
Metadata includes system attributes such as timestamp and content entity size that the virtual content repository uses to manage file-based content. Metadata also may include application attributes such as trading partner information (who sent the file), transformation applied on the received data, and the status of post-transfer processing. All these can easily be managed through a file system-based content repository. If the file transfer is used as high-volume ingestion to trigger a batch business process, then the file and associated metadata should be accessible from a business process. In the WebLogic Platform, any file managed by the content repository can be accessed using content management APIs from within a business process. Refer to javadoc of the CM API for more information.
Putting It All Together
We have seen how GlobalTx can build a comprehensive SOA-based managed file transfer solution with WebLogic Platform. The following table summarizes the requirements and the corresponding components that enable the solution.
| No | SOA Requirements | Components used for solution |
|---|---|---|
| 1 | Security | Custom Java control for secured FTP using AppGate's MindTerm |
| 2 | File handling automation | File event generator for event handling and timer service for scheduling |
| 3 | File transformation | Format Builder for generating transformation mappings from non-XML to XML and using WebLogic Integration message transformation capabilities |
| 4 | Handling very large files | File control and using patterns to split data into individual records |
| 5 | End-to-end file management | WebLogic Portal's file system-based content management repository |
Summary
Almost every company, like GlobalTx, uses file transfer to some extent today. Often, these transfers are an essential part of overall IT operations. Although message-based enterprise application integration (EAI) offers a flexible and granular solution for integration needs, it would be extremely ill-advised for a company to try to cut across to a 100-percent, message-based implementation in one big push. Instead, it is likely (and prudent) for a company to run in a "mixed" environment for a considerable number of months or even years. The lowest-risk approach will be to gradually move parts of the operation over to a message-based form of integration, replacing the file transfer activities piece by piece. In fact, it may be that some of the file transfer activity has to be retained indefinitely. There are situations where file transfer remains the most attractive way to achieve the desired results.
Therefore, a critical issue for integration strategies across the enterprise and beyond is how to interoperate between file-based and message-based activity. The answer is to have a capability to map files to messages and vice versa. By using such a facility, it will be possible for an application to become "message-enabled" but, at the same time, be invoked by applications that deal with a file interface. The mapping engine can split the file into individual messages (for example, corresponding to particular records in the file) and these can then be used to drive the message-enabled applications. In the reverse scenario, the mapping engine recombines the messages so that the file-based application can receive the response. We have seen how GlobalTx's requirements could be met successfully by employing the managed file transfer capabilities of BEA WebLogic Platform.
Resources
Senthil Kumar Krishnan is a member of the Platform Engineering team at BEA Systems. At BEA he has worked on WebLogic Platform Integration, end-to-end solution and customer use cases.
Ambarish Nagarajan is a senior software engineer on the Platform Engineering team.
Hari Prakash , a software engineer at BEA, has over 5 years experience in Enterprise Java Development and Performance Benchmarking. He is currently involved in the performance study of end to end product usage from the BEA suites of products.