Data File Transfer When technology complements business   Data Management Series

Copyright © 1987-2008  SimoTime Enterprises, LLC  All Rights Reserved http://www.simotime.com

Introduction
(Next) (Previous) (Table-of-Contents)

This document is an introduction or overview of the data file transfer aspects of an application migration between a mainframe system and a Windows system running a Micro Focus sub-system such as Enterprise Server, Application Server or Net Express.

The data file transfer process should be a repeatable process with an audit or validation trail. The process should be executable as an automated, unattended process. Requiring operator input during the data file transfer process introduces an exposure point for error.

A special "Thank you" to Larry Simmons of Micro Focus for providing much of the information that is presented in this series of white papers and sample programs.

Application Migration, Data File Transfer
(Next) (Previous) (Table-of-Contents)

There are two categories of data file transfer requirements based upon the application migration requirements.

Determine the File Transfer Methodology
(Next) (Previous) (Table-of-Contents)

If you are not familiar with Micro Focus file formats you should refer to  http://www.SimoTime.com/mffile01.htm for more information. The following is used to determine the effort of transferring the data files.

File Transfer using Micro Focus Mainframe Access (MFA)
(Next) (Previous) (Table-of-Contents)

When transferring files from a Mainframe system to a Windows system running a Micro Focus sub-system this should be the preferred transfer method. Since this is a Micro Focus utility the file format conversion (Mainframe file format to Micro Focus file format) is done as part of the file transfer process. The transfer process may be performed on demand using the Graphical User Interface (GUI) with a point-and-click or drag-and-drop methodology. The transfer process may be scripted and automated using a Windows command file, a job scheduler and the MFA command line interface.

Additional consideration is required for very large files (greater than 2 gigabytes). The solution does require the MFA server component to be installed on a Mainframe system running z/OS and is not available for Mainframe systems running VSE.

The following is an example of the command line statement required to download a VSAM, KSDS from the Mainframe system to a Windows system running Micro Focus Mainframe Express. The file will be converted to a Micro Focus Indexed file as part of the transfer process. This is a single statement but is shown on two lines.

C:\> MFDAS IMPORT BINARY c:\SimoDEMO\DataMfa1\Simotime.data.itemmast.dat 
                    FROM ksds simotime.data.itemmast

Note-1: the file will be downloaded in its original EBCDIC-encoded format. The numeric fields will be maintained in their mainframe numeric formats (this includes packed, binary and floating point).
Note-2: If a conversion from EBCDIC to ASCII is required this will need to be done as a separate task.

If the file is larger than 2 gigabytes then extra configuration settings or environment variables will be required.

C:\> set MFA_INDEX_FORMAT=IDXFORMAT(8)
C:\> set extfh=c:\SimoDEMO\ProdLib1\ExtfhBig.CFG

The ExtfhBig.CFG file contains the following records.

[XFH-DEFAULT]
filemaxsize=8
idxformat=8
filepointersize=8

The preceding should allow for files that are greater than 2 gigabytes in size to be downloaded.

File Transfer using File Transfer Protocol (FTP)
(Next) (Previous) (Table-of-Contents)

This file transfer methodology is usually available on Mainframe systems (both z/OS and VSE), Windows systems, UNIX Systems and Linux systems. Because it is readily available it is usually the default transfer technology.

When transferring files between a Mainframe System and a Windows system the FTP technology works well with record sequential files of fixed-length records of printable text (i.e. no binary data such as numeric fields defined as PACKED or BINARY). For sequential files with variable length records additional FTP statements are required in order to preserve the Record Descriptor Word (the RDW is a binary value that precedes each record and defines the record length).

FTP works quite well when transferring source members that are stored in a Partitioned Data Set (PDS) as sequential files with fixed-length records of eighty characters of printable text. FTP in ASCII mode will do the file transfer, the file content (i.e. EBCDIC and ASCII) and the file format (record sequential to line sequential) conversions.

FTP, ASCII Mode
(Next) (Previous) (Table-of-Contents)

FTP, by default, will run in ASCII mode and when transferring files between a Mainframe system and a Windows, UNIX or Linux system it will attempt to convert the file content between EBCDIC encoding and ASCII encoding. Running in ASCII mode will also convert the file format from record sequential to line sequential. This will create a problem if the file content contains binary data.

FTP, BINARY Mode
(Next) (Previous) (Table-of-Contents)

FTP has the capability of transferring files in a BINARY mode without changing the content of the file. This mode is especially useful if the records in a file contain binary data (i.e. numeric fields that use the packed, binary or signed-zoned-decimal format).

For sequential files that contain EBCDIC-encoded text strings and binary text string and reside on the mainframe the BINARY mode will transfer the file to a Windows or UNIX platform as a record sequential file. The EBCDIC-encoded text strings and the binary text strings are not converted. In a Micro Focus environment it is possible to run in EBCDIC mode. However, if information is to be imported into a non-Micro Focus environment such as an Excel spreadsheet the text strings will need to be converted from EBCDIC to ASCII and the binary strings will need to be expanded (i.e. converted) to printable text.

For non-sequential files this approach usually requires a file format conversion to be done on the Mainframe system prior to the file transfer. For example, a VSAM, Key-Sequenced-Data-Set (KSDS) is usually converted to a flat, sequential file of fixed-length records (using the REPRO function of IDCAMS). The sequential file is then transferred to the Windows system. Once transferred the sequential file is then used to create a Micro Focus Indexed file.

File Transfer using Shared or Removable Machine Readable Media
(Next) (Previous) (Table-of-Contents)

Since the introduction of high-speed networks and shared Disk Access Storage Devices (or DASD) the use of machine-readable media (or MRM) for file sharing or transferring has steadily increased. The actual hardware, software and network configurations may vary based on the type and volumes of data. A typical configuration would be a Storage Area Network (SAN) in a clustered environment over a network. As the data volumes increase the use of Fibre Channel Protocol (FCP) may be required. The challenge is to provide access to storage area network (SAN) connectivity that meets both the budgetary and performance requirements for interconnecting primary data centers to remote locations.

For ongoing data sharing and transferring the use of removable media (tape or Compact Disk) that is available and compliant with both the Mainframe system and the Windows system is declining. The use of tape remains the popular media for backup on the mainframe.

The MRM approach usually requires file format conversions to be done on the Mainframe system prior to the file transfer. For example, a VSAM, Key-Sequenced-Data-Set (KSDS) is usually converted to a flat, sequential file of fixed-length records (using the REPRO function of IDCAMS). The sequential file is then transferred to the Windows system. Once transferred the sequential file is then used to create a Micro Focus Indexed file.

Estimate Scope and Validate Requirements
(Next) (Previous) (Table-of-Contents)

This section provides a list of questions that will aid in determining the scope of effort for creating the process to do data file transfers. The following questions should be answered at the beginning of the process to migrate an application and its associated data.

The following is used to determine the number of files and the file types and characteristics.

Possibilities & Considerations
(Next) (Previous) (Table-of-Contents)

This section provides additional detail about the process for the transfer of data files between a mainframe system and a Windows system running a Micro Focus sub-system. The file transfer technology and methodology that is used will determine the amount of file conversion (or preparation) that is required prior, during or after the file transfer.

Primary and Alternate Indices
(Next) (Previous) (Table-of-Contents)

Both MFA and FTP do not transfer the alternate indices. Alternate indices must be created after the file has been downloaded. Building the alternate index may be done using IDCAMS and JCL (Mainframe Express or the Batch Facility with ES/MTO) or with the Micro Focus BLDINDEX utility.

Note: If an EBCDIC to ASCII conversion is required the alternate index should not be created until after the file has been converted. Since we are changing the value (i.e. encoding format) of the alternate key (or index) then we should only create the alternate key after the data that makes up the alternate key has been converted.

Files with Variable Length Records
(Next) (Previous) (Table-of-Contents)

Both MFA and FTP will transfer files with variable length records. MFA will do the file format conversion from the Mainframe format (i.e. each record has a Record Descriptor Word or RDW) to a Micro Focus format (i.e. the file has a 128 byte header and each record is preceded by a Record Descriptor Word or RDW). It is important to note that the RDW used on the Mainframe is different structure than the RDW format used with Micro Focus.

FTP will transfer a file with variable length records. Because the RDW information is stored in a binary format it will be necessary to use BINARY mode with FTP. This is usually not a problem since most mainframe files contained packed or binary information anyway. In order to get the RDW information some additional commands (or statements) will be required in the FTP script or process. The following shows a sample FTP script from a client requesting to transfer a file from the Mainframe server to a Windows client.

userid
Password
CD ..
PWD
BINARY
LITERAL SITE RDW
GET  SIMOTIME.DATA.ZDDVBE01  C:\MFI01\FTPLIB01\ZDDVBE01.DAT
QUIT

If the Mainframe server is sending to a Windows client then the following changes are required to the preceding script. Change the LITERAL statement to the following.

LOCSITE RDW

Change the GET statement to a PUT statement.

Read a VSAM, KSDS & Reproduce a Sequential File using IDCAMS
(Next) (Previous) (Table-of-Contents)

When using FTP to transfer a VSAM, Key-Sequenced-Data-Set (KSDS) from a Mainframe System to a Windows System it will be necessary to read the KSDS and write to a sequential file and then transfer the sequential. The following member (CUSK2RJ1.JCL) is a two (2) step job. The first step will use IEFBR14 with a DD statement to delete a file that was created by a previous execution of this job. The second step will use IDCAMS and the REPRO function to read the Customer Master File and create a sequential file with the same record structure. and content.

//CUSK2RJ1 JOB SIMOTIME,ACCOUNT,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*                   THIS PROGRAM IS PROVIDED BY:                    *
//*                    SIMOTIME ENTERPRISES, LLC                      *
//*           (C) COPYRIGHT 1987-2008 ALL RIGHTS RESERVED             *
//*             WEB SITE URL:   HTTP://WWW.SIMOTIME.COM               *
//*                   E-MAIL:   HELPDESK@SIMOTIME.COM                 *
//* *******************************************************************
//*
//* TEXT   - COPY (OR REPRO) A KSDS TO A SEQUENTIAL FILE
//* AUTHOR - SIMOTIME ENTERPRISES
//* DATE   - JANUARY 01, 1989
//*
//* *******************************************************************
//* Step   1 of 2  Delete any previously created file...
//*
//QSAMDELT EXEC PGM=IEFBR14
//SEQ51201 DD  DSN=SIMOTIME.DATA.CUSR512,DISP=(MOD,DELETE,DELETE),
//             SPACE=(TRK,(10,1),RLSE),
//             DCB=(RECFM=FB,LRECL=512,DSORG=PS)
//*
//* *******************************************************************
//* Step   2 of 2  Create and populate a new QSAM file...
//*
//         EXEC PGM=IDCAMS
//SYSPRINT DD  SYSOUT=A
//KSD51201 DD  DSN=SIMOTIME.DATA.CUSTMAST,DISP=(SHR)
//SEQ51201 DD  DSN=SIMOTIME.DATA.CUSR512,
//             SPACE=(TRK,(10,1),RLSE),
//             DISP=(NEW,CATLG,DELETE),
//             DCB=(RECFM=FB,LRECL=512,DSORG=PS)
//SYSIN DD *
 REPRO -
  INFILE(KSD51201) -
  OUTFILE(SEQ51201)
/*

Record Counts
(Next) (Previous) (Table-of-Contents)

This is part of the basic requirements to ensure the files being transferred will have an equal number of records sent from the Mainframe system and received at the Windows system.

Audit Requirements of Management or Consulting Firm
(Next) (Previous) (Table-of-Contents)

Management and/or the auditors/consultants should be involved in the data migration (transfer, conversion and validation) process as early as possible. This will help ensure the process will meet the requirements and maintain the necessary level of data integrity at each step in the process.

When transferring data files a process for insuring the number of records sent from the mainframe system is received at the Windows system. This is usually accomplished by accumulating record totals on both systems and comparing these control totals. A secondary validation is to accumulate hash totals on both systems for critical currency fields and compare the totals. The totals for currency fields may already be available from standard reports created by the application.

The data file transfer process should be a repeatable process with an audit trail. The process should be executable as an automated, unattended process. Operator intervention during the data file transfer process should be considered as an exposure for introducing errors into the process.

Summary
(Next) (Previous) (Table-of-Contents)

Anyone considering a data migration should seek additional assistance from a consulting or programming services organization that has experience in this area.

Software Agreement and Disclaimer
(Next) (Previous) (Table-of-Contents)

Permission to use, copy, modify and distribute this software for any commercial purpose requires a fee to be paid to SimoTime Enterprises. Once the fee is received by SimoTime the latest version of the software will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

Permission to use, copy, modify and distribute this software for a non-commercial purpose and without fee is hereby granted, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

SimoTime Enterprises makes no warranty or representations about the suitability of the software for any purpose. It is provided "AS IS" without any express or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Enterprises shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software.

Downloads and Links to Similar Pages
(Next) (Previous) (Table-of-Contents)

Please view the complete list of SimoTime examples at  http://www.simotime.com/sim4dzip.htm .

Note: You must be attached to the Internet to download a Z-Pack or view the list.

This document is part of the  Data File Management Series  of white papers that discuss the transferring, sharing, converting and comparing tasks required when moving or sharing data between different systems.

Refer to the File Transfer Protocol  document for a quick reference of the FTP commands.

This item will provide a link to  an ASCII or EBCDIC translation table. A column for decimal, hexadecimal and binary is also included.

This suite of programs provides an example of how to identify and process embedded hexadecimal or non-printable characters in COBOL source code. Simply click on one of the following items to learn more or download this sample set of programs.

The  hexadecimal dump  of the parameter-buffer uses the same technique as described in another SimoTime example that describes the dumping of a data string using COBOL. The name of the member that does the actual hexadecimal dump is called SimoDUMP. A copy file (PASSDUMP.CPY) is provided for defining the pass area.

The SimoZAPS Utility Program has the capability of generating a COBOL program that will do the conversion of sequential and VSAM (KSDS) files between EBCDIC and ASCII. SimoZAPS can also read a record sequential file in EBCDIC format and create a line sequential (or ASCII/Text) file or VSAM KSDS file in ASCII format. The conversion tables may be viewed or modified to meet unique requirements. The Hexcess/2 function provides the capability of viewing, finding or patching the contents of a file in hexadecimal.

The following table is a list of white papers that provides more detailed information about the four common numeric formats used on an IBM Mainframe.

Check out  The VSAM - QSAM Connection  for more examples of mainframe VSAM and QSAM accessing techniques and sample code.

This document provides a quick summary of the  File Status Key  for VSAM data sets and QSAM files.

To review all the information available on this site start at  The SimoTime Home Page .

Glossary of Terms
(Next) (Previous) (Table-of-Contents)

Check out  The SimoTime Glossary  for a list of terms and definitions used in the documents provided by SimoTime.

Comments or Suggestions
(Next) (Previous) (Table-of-Contents)

If you have any questions, suggestions or comments please call or send an e-mail to: helpdesk@simotime.com

We appreciate your comments and feedback.

About SimoTime Enterprises, LLC
(Next) (Previous) (Table-of-Contents)

Founded in 1987, SimoTime Enterprises is a privately owned, Limited Liability Corporation located in Novato, California. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. This includes the smallest thin client using the Internet and the very large mainframe systems. There is more to making the Internet work for your company's business than just having a nice looking WEB site. It is about combining the latest technologies and existing technologies with practical business experience. It's about the business of doing business and looking good in the process. Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complimentary manner with existing corporate mainframe systems. Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com