Data File Abstract When technology complements business   Data Management Series

Copyright © 1987-2008  SimoTime Enterprises, LLC  All Rights Reserved http://www.simotime.com

Introduction
(Next) (Previous) (Table-of-Contents)

Ever since the second computer was introduced into the world the file management tasks of data file transfer, share, convert and compare (or data file validation) have been technically challenging. This document is an overview of the data file management aspects of an application migration between a Mainframe System and a Windows System running a Micro Focus sub-system such as Enterprise Server, Application Server or Net Express (referred to in this document as a Wintel System). This white paper provides a quick overview of the data file management process and data file structures (both file format and file content). Links to documents that provide detailed information for data file transfer, conversion and comparison are included.

The tasks performed within the file management process should be repeatable with audit or validation trails. A task should be executable on demand or as a scheduled, automated, unattended process. Requiring operator input during the execution of a task within the process introduces an exposure point for error.

A special "Thank you" to Larry Simmons of Micro Focus for providing much of the information that is presented in this series of white papers and sample programs.

Data File Management Tasks
(Next) (Previous) (Table-of-Contents)

There are seven tasks within a data file migration effort. The detail within the tasks is based upon the direction of migration and which system will be performing the individual tasks. The location and sequence of task execution may be predefined, required for both systems or optional to either system.

Task 1, Prepare the Sending and Receiving Systems
(Next) (Previous) (Table-of-Contents)

This should be a one-time process. However, consideration should be given for replicating the environment across development, test, production and backup systems.

The mainframe is the current production system. Mainframe resources are committed to production jobs and disk capacity may be a problem when attempting to copy and convert very large files. Therefore, most of the conversion jobs will be done on the Wintel platform where resources are not in contention with the production system.

Task 1, Prepare the Receiving Directory Structure
(Next) (Previous) (Table-of-Contents)

This may appear to be an insignificant task. However, the number and variety of files will grow during the download and conversion process because of the file transfer methodology (FTP or MFA) and the file format (sequential, Indexed, variable and fixed formats) and the file content requirements (ASCII or EBCDIC encoding schemes).

The following is a sample directory structure. The BaseDir1 is optional and is an umbrella directory for all the data.

Task 1, Install and Configure Software
(Next) (Previous) (Table-of-Contents)

The software required is as follows.

Task 1, Record Counts and Possible Currency Totals
(Next) (Previous) (Table-of-Contents)

Record counts are a minimum requirement to ensure that all the records sent are received. Currency totals are an option and provide an extra validation. When converting between EBCDIC and ASCII encoding formats the fields within records within files that contain currency values should be tracked with a batch total for comparison unless the parallel processing done during the testing cycle will produce reports that include this information.

Task 1, The Mainframe
(Next) (Previous) (Table-of-Contents)

In most situations record counts are not readily available on the mainframe. It is not unusual to create a program that reads through the file and counts the records. If it is necessary to do this then the same program should be used to accumulate currency totals.

Another item to consider is the use of third party software that provide alternative access methods that compress data and make record counts difficult to calculate. For example, Innovation Access Method (referred to as IAM and provided by Innovation Data Processing) is a high-performance, indexed access method for OS/390 and MVS/ESA operating systems. Existing as non-VSAM data sets provides IAM with capabilities to eliminate the 4.3-gigabyte file-size restriction in VSAM prior to DFSMS V1.3 and to choose a block size that optimizes space utilization on DASD.

Task 1, The Windows and Micro Focus Environment
(Next) (Previous) (Table-of-Contents)

If the record counts and currency totals are calculated during the data file conversion that is being done on the receiving system then it may be considered an optional or redundant task to count records or accumulate currency totals during the transfer process.

If the Micro Focus data file converter (DFCONV) or the SimoTime technologies are used for the file conversion then record counts are provided by both technologies. A definite advantage for accumulating currency totals is to use the SimoTime technologies since this approach does not do the conversion but creates COBOL source code that may be compiled and executed to do the conversion. These COBOL programs may be easily modified to accumulate currency totals during the file conversion process.

Task 1, Documentation and Reference Material
(Next) (Previous) (Table-of-Contents)

It is very important to document the process for preparing the sending and receiving systems so it may be repeated. This is especially true for any exception type of process.

For additional information refer to the Application Migration, File Management and Data Integrity document provided with this series of documents.

Task 2, Transfer the Data Files
(Next) (Previous) (Table-of-Contents)

Determine the transfer methodology. Mainframe Access (MFA) is recommended especially for Key Sequenced Data Sets (VSAM, KSDS) or sequential files with variable length records.

Task 2, File Transfer Protocol (FTP)
(Next) (Previous) (Table-of-Contents)

In many situations this is the protocol by default since it may be the only thing that is common to the Sending and Receiving systems.

If the FTP transfer technology is used then all non-sequential files will need to be converted to sequential files. Special handling will also be needed for sequential files that contain variable length records in order to retain the record descriptor word (RDW) that precedes each record and contains the record length.

Task 2, Mainframe Access (MFA)
(Next) (Previous) (Table-of-Contents)

This technology is available from Micro Focus and in most cases is the easiest way to transfer files from a mainframe to a Windows system running a Micro Focus sub system. For data files (i.e. non-source code) the received files will arrive in a Micro Focus format and the file content will be EBCDIC encoded. If Micro Focus Mainframe Access (MFA) technology is used then very little preparation will be required to transfer the data files.

Task 2, Machine Readable Media
(Next) (Previous) (Table-of-Contents)

The decision to use this media is determined by file sizes and availability of hardware that is accessible from both the sending and receiving system.

Task 2, Tape, Compact Disk (CD) or Shared Disk
(Next) (Previous) (Table-of-Contents)

This conversion may only be a file format conversion that addresses the media exchange issue and may require additional file format conversion and file content conversion once it is transferred to the Wintel system.

Task 2, Sub-Contract or Out Source
(Next) (Previous) (Table-of-Contents)

This is usually providing a sub-contractor or data conversion company with copies of the data files on tape in an agreed to format. The files are converted to a compatible format and delivered on a media that is accessible by the hardware available on the Wintel system. This conversion may only be a file format conversion that addresses the media exchange issue and may require additional file format conversion and file content conversion once it is transferred to the Wintel system.

Task 2, Prepare Scripts for File Transfer
(Next) (Previous) (Table-of-Contents)

On the mainframe batch jobs (JCL) will need to be created to prepare the files for transfer. Programs may need to be created to accumulate currency totals. On the Wintel system this is usually a suite of command files (previously known as batch files) that will be called from a mainline command file. Creating the scripts should be a one-time process.

Task 2, Execute Scripts to do the File Transfers
(Next) (Previous) (Table-of-Contents)

This needs to be a repeatable process that may be scheduled or executed on-demand and once started it should run as an automated, unattended process. Depending on the file transfer medium, the file types and the validation requirements the mainframe files may need to be pre-processed (or staged) along with the control information such as record counts and hash totals for critical currency fields. Once the files are staged then start the transfer process between the Mainframe system and the Wintel system. For additional information refer to the  Data File Transfer  document provided with this series of documents.

Task 3, Convert the Data Files
(Next) (Previous) (Table-of-Contents)

Since Net Express includes a Data File Converter (DFCONV) this may be used at no additional cost. However, this limits the data file conversion process to being executed on the Windows platform. Also, if the record structures of a file are multiple record types with complex structures then an alternate solution such as SimoTime technologies may be required.

Task 3, Prepare Specifications and Control Media
(Next) (Previous) (Table-of-Contents)

Create specification media for Data File Conversion Utility (DFCONV) provided by Micro Focus. This is a one-time task. If required, create specification media scripts (or control media) for SimoTime Technology to generate, compile and execute the data file conversion programs. This should be a one-time task. The scripts that set the environment and perform the execution of the conversion programs should be written to be an unattended and repeatable process.

Task 3, Prepare Scripts to Automate the Conversion Process
(Next) (Previous) (Table-of-Contents)

This is a one-time task. The number of scripts required depends on the number of files, the size of the files and the complexity of the record structures within the files. For the Wintel system a script is usually a command file that will convert a group of files or a single file. A mega script may then be created to call the individual scripts.

Task 3, Execute the Data File Conversion Process
(Next) (Previous) (Table-of-Contents)

This needs to be a repeatable process that may be scheduled or executed on-demand and once started it should run as an automated, unattended process. The SimoTime technology does not explicitly provide the capability of accumulating currency totals. However, since the SimoTime technology generates COBOL source code that is compiled and then used to do the data file comparison this COBOL source code may be easily modified to accumulate the batch totals. For additional information refer to the Data File Convert document provided with this series of documents.

Task 4, Validate Record Counts & Currency Totals
(Next) (Previous) (Table-of-Contents)

Validating record counts between the sending and receiving system is part of the basic requirements to ensure the files being transferred and converted will have an equal number of records.

Actual data file compares at the record level are not done at this point in the process. Only validation of record counts and possible subtotals of currency fields will be done at this point in the process. The goal at this point in the process is to validate that the number of records sent by the mainframe is equal to the number of records received and converted at the Wintel system.

The method for obtaining record counts on the mainframe or the Wintel platform varies widely depending on the software used to do the transfer or the conversion. The SimoTime convert technology provides record counts as part of the conversion process.

Task 5, Process the Data Files in Parallel Tasks
(Next) (Previous) (Table-of-Contents)

This requires taking a backup or transferring the data files of the mainframe or sending system prior to starting the processing (or execution of parallel tasks or jobs). Once the pre-process files are transferred the jobs that process the data may be executed on both systems. When the processing is completed on the mainframe the post-process files will then be transferred to the Wintel platform and then converted and compared with the post-process files on the Wintel system.

Task 5, Execute a Controlled, Parallel Test on the Mainframe System
(Next) (Previous) (Table-of-Contents)

This is a three-step process.

Task 5, Execute a Controlled, Parallel Test on the Windows System
(Next) (Previous) (Table-of-Contents)

This is a two-step process

Proceed with the task of comparing the files.

Task 6, Compare the Results
(Next) (Previous) (Table-of-Contents)

One of the early decisions is to determine the platform where the data files will be compared. Some of the questions or considerations are as follows.

For example, it may be necessary to sort some files (this is true if the files are in sequence by an alpha-numeric field). In addition to comparing the files at the record level there should be batch record count comparisons and possibly some subtotal comparison on critical currency fields.

The SimoTime compare technology has an option to read to "End-of-File" or to "Quit" after a certain number of differences have been identified.

If the "End-of-File" option is used then differences will be written to the log file and when the difference count exceeds the maximum then differences will not be written to the log file but reading of the two files being compared will continue until end-of-file is reached and a records count for the total number of records read from each file will be provided along with a non-zero return code.

If the "Quit" option is used and the maximum number of differences is exceeded then a count of the number of records read before the program is terminated with a non-zero return code will be provided.

It may be a requirement to accumulate batch totals for currency fields. The SimoTime technology does not explicitly provide this capability. However, since the SimoTime technology generates COBOL source code that is compiled and then used to do the data file comparison this COBOL source code may be easily modified to accumulate the batch totals.

Task 6, Prepare Specifications and Control Media
(Next) (Previous) (Table-of-Contents)

Create specifications media and scripts (or control media) for the SimoTime Technology to generate, compile and execute the data comparison programs. This should be a one-time task. The scripts that set the environment and perform the execution of the compare programs should be written to be an unattended and repeatable process.

Task 6, Prepare Scripts to Automate the Compare Process
(Next) (Previous) (Table-of-Contents)

This is a one-time task. The number of scripts required depends on the number of files and the positions within the records that will be compared. For the Wintel system a script is usually a command file that will compare a group of file sets or a single file set (a file set consist of two files). A mega script may then be created to call the individual scripts.

Task 6, Execute the Compare Process
(Next) (Previous) (Table-of-Contents)

This needs to be a repeatable process that may be scheduled or executed on-demand and once started it should run as an automated, unattended process. Review the log files for possible differences For additional information refer to the Data File Compare document provided with this series of documents.

Task 7, Correct and Repeat or Proceed Forward
(Next) (Previous) (Table-of-Contents)

If the compare process indicates the files from the mainframe process match with the files created by the Wintel process then it is time to move to the next level of testing or begin the process of completing the final task of moving to a production system on the Wintel system. If differences in the results are encountered then it will be necessary to remove the cause of the differences and repeat the testing prior to moving forward.

Summary
(Next) (Previous) (Table-of-Contents)

Migrating an application between systems of similar architecture (i.e. upgrade to existing mainframe) can be a challenging task. Migrating an application from a Mainframe, EBCDIC encoded system to a Wintel, ASCII encoded system requires an even higher level of planning and attention to detail. With careful planning and a documented process the application migration process can move forward to a successful conclusion.

Software Agreement and Disclaimer
(Next) (Previous) (Table-of-Contents)

Permission to use, copy, modify and distribute this software for any commercial purpose requires a fee to be paid to SimoTime Enterprises. Once the fee is received by SimoTime the latest version of the software will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

Permission to use, copy, modify and distribute this software for a non-commercial purpose and without fee is hereby granted, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

SimoTime Enterprises makes no warranty or representations about the suitability of the software for any purpose. It is provided "AS IS" without any express or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Enterprises shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software.

Downloads and Links to Similar Pages
(Next) (Previous) (Table-of-Contents)

You may download this example at http://www.simotime.com/sim4dzip.htm#COBOLCommaDelimitedFile or view the complete list of SimoTime Examples at http://www.simotime.com/sim4dzip.htm.

Note: You must be attached to the Internet to download a Z-Pack or view the list.

This document is part of the  Data File Management Series  of white papers that discuss the transferring, sharing, converting and comparing tasks required when moving or sharing data between different systems.

The hexadecimal dump of the parameter-buffer uses the same technique as describe in another SimoTime example that describes the dumping of a data string using COBOL. The name of the member that does the actual hexadecimal dump is called SimoDUMP. A copy file (PASSDUMP.CPY) is provided for defining the pass area.

The SimoZAPS Utility Program has the capability of generating a COBOL program that will do the conversion of sequential and VSAM (KSDS) files between EBCDIC and ASCII. SimoZAPS can also read a sequential file in EBCDIC format and create an ASCII/CRLF file or VSAM KSDS file in ASCII format. The conversion tables may be viewed or modified to meet unique requirements. The Hexcess/2 function provides the capability of viewing, finding or patching the contents of a file in hexadecimal.

This item will provide a link to  an ASCII or EBCDIC translation table. A column for decimal, hexadecimal and binary is also included.

The following table is a list of white papers that provides more detailed information about the four common numeric formats used on an IBM Mainframe.

Check out  The VSAM - QSAM Connection  for more examples of mainframe VSAM and QSAM accessing techniques and sample code.

This document provides a quick summary of the  File Status Key  for VSAM data sets and QSAM files.

Check out  The SimoTime Library  for a wide range of topics for Programmers, Project Managers and Software Developers.

To review all the information available on this site start at  The SimoTime Home Page .

Glossary of Terms
(Next) (Previous) (Table-of-Contents)

Check out  The SimoTime Glossary  for a list of terms and definitions used in the documents provided by SimoTime.

Comments or Suggestions
(Next) (Previous) (Table-of-Contents)

If you have any questions, suggestions or comments please call or send an e-mail to: helpdesk@simotime.com

We appreciate your comments and feedback.

About SimoTime Enterprises, LLC
(Next) (Previous) (Table-of-Contents)

Founded in 1987, SimoTime Enterprises is a privately owned, Limited Liability Corporation located in Novato, California. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. This includes the smallest thin client using the Internet and the very large mainframe systems. There is more to making the Internet work for your company's business than just having a nice looking WEB site. It is about combining the latest technologies and existing technologies with practical business experience. It's about the business of doing business and looking good in the process. Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complimentary manner with existing corporate mainframe systems. Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com