Data File Convert  Data Management Series

Introduction
(Next) (Previous) (Table-of-Contents)

Ever since the second computer architecture was introduced the task of data conversion in preparation for data migration and data sharing has been a never-ending process. Data conversions may be driven by business requirements or system requirements such as changes in system architectures. This document is an introduction or overview of the data file conversion aspects of an application migration between a mainframe system and a Windows system running a Micro Focus sub-system such as Enterprise Server, Application Server or Net Express. The following discussion will divide the data file conversion tasks into two categories.

The document and links to other documents will cover the following items.

The data conversion process should be a repeatable process with an audit or validation trail. The process should be executable as an automated, unattended process. Requiring operator input during the conversion process introduces an exposure point for error.

A special "Thank you" to Larry Simmons of Micro Focus for providing much of the information that is presented in this series of white papers and sample programs.

Application Migration & Data Conversion
(Next) (Previous) (Table-of-Contents)

There are two categories of data conversion requirements based upon the target environment configuration.

Estimate Scope and Validate Requirements
(Next) (Previous) (Table-of-Contents)

This section provides a list of questions that will aid in determining the scope of effort for creating the process to do data file conversions. It is important to provide platform flexibility as to where the conversion is done. The following questions should be answered at the beginning of the process to migrate an application and its associated data.

Estimating, File Inventory and Types
(Next) (Previous) (Table-of-Contents)

The following is used to determine the number of files and the file types and characteristics.

Estimating, Identifying & Validating Conversion Processes
(Next) (Previous) (Table-of-Contents)

The following is used to determine the basic requirements for the data file conversion effort.

Estimating, File Transfer Effort
(Next) (Previous) (Table-of-Contents)

The following is used to determine the effort of transferring the data files.

Possibilities & Considerations
(Next) (Previous) (Table-of-Contents)

This section provides additional detail about the process for generating, compiling and executing a data file comparison program. This discussion is limited to the Windows environment. However, the process is very similar for the mainframe and UNIX environments once the COBOL source code has been generated on a Windows system.

Mainframe to Windows, Conversion Alternatives
(Next) (Previous) (Table-of-Contents)

If the requirement for data conversion is to do the conversion on a Windows client machine that is used for development and testing and the data is limited to sequential or indexed files then the Data File CONVerter included in Net Express (DFCONV) offers a cost effective and convenient solution. However, Micro Focus does not include the components for DFCONV into the various run time (or production) offering (Application Server and Enterprise Server) to allow full implementation in the production environments.

Note: Both DFCONV and the Data File Editor are delivered as technologies for a development environment running on a Windows System.

Note: Also, DFCONV is only available on Windows, it is not available on UNIX.

In many situations the requirement goes beyond just a simple file conversion that is performed on a Windows platform. The following may require more function than provided by DFCONV.

The following is a reasonable three-step guideline for approaching a data file conversion effort.

Using a Copy File to Map Fields for Content Conversion
(Next) (Previous) (Table-of-Contents)

With the exception of files with variable length records that are dynamically created at execution time a COBOL copy file that defines a record layout is usually available. If a copy file is not available a COBOL working storage definition can usually be "cut-and-pasted" to create a COBOL copy file.

With DFCONV the COBOL copy file or a working storage definition may be used as a record definition for the data file conversion.

With SimoTime technologies the COBOL copy file may be used to generate the COBOL source code that may be compiled and executed on a Mainframe system (MVS or VSE), a Wintel System with Micro Focus or a UNIX system with Micro Focus. Using a COBOL copy file is not an absolute requirement since an optional feature of the SimoTime Technologies provides for data conversion based on position within record.

Data File Format Conversion using IDCAMS
(Next) (Previous) (Table-of-Contents)

If the requirement is to only convert the file format in order to FTP between systems then the REPRO function of IDCAMS may be the solution. The following is an example of how to use IDCAMS to convert a VSAM, KSDS to a flat, sequential file.

//IDCAMSJ2 JOB SIMOTIME,ACCOUNT,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*        This program is provided by: SimoTime Enterprises          *
//*           (C) Copyright 1987-2012 All Rights Reserved             *
//*             Web Site URL:   http://www.simotime.com               *
//*                   e-mail:   helpdesk@simotime.com                 *
//* *******************************************************************
//*
//* TEXT   - COPY (OR REPRO) A KSDS TO A SEQUENTIAL FILE
//* AUTHOR - SIMOTIME ENTERPRISES
//* DATE   - JANUARY 01, 1989
//*
//         EXEC PGM=IDCAMS
//SYSPRINT DD  SYSOUT=A
//SEQGET01 DD  DSN=SIMOTIME.DATA.SEQGET01,DISP=(SHR)
//KSDPUT01 DD  DSN=SIMOTIME.DATA.KSDPUT01,
//             SPACE=(TRK,(10,1),RLSE),
//             DISP=(NEW,CATLG,DELETE),
//             LRECL=80,KEYOFF=0,KEYLEN=12,RECORG=KS
//SYSIN DD *
 REPRO -
  INFILE(SEQGET01) -
  OUTFILE(KSDPUT01)
/*

Differences in the ASCII & EBCDIC Collating Sequences
(Next) (Previous) (Table-of-Contents)

When converting a Sequential file that is downloaded from a Mainframe System (EBCDIC-encoded) to an Indexed file that is created on a Windows System (ASCII-encoded) it may be necessary to make adjustments for the differences in the ASCII and EBCDIC collating sequences. This is especially true if the field that determines the sequence of the file is alpha-numeric (if the key field is all numeric then sequencing should not be a problem).

If both the file format (Sequential to Indexed with an alpha-numeric key) and the file content (EBCDIC to ASCII) is being done in a single pass then an unordered load of the new Indexed file must be done to avoid getting an out-of-sequence error when adding new records into the new Indexed file.

Primary and Alternate Indices
(Next) (Previous) (Table-of-Contents)

An alternate index should not be created until after the file has been converted. Since we are changing the value (i.e. encoding format) of the alternate key (or index) then we should only create the alternate key after the data that makes up the alternate key has been converted.

Building the alternate index may be done using IDCAMS and JCL (Mainframe Express or ES/MTO with the Batch Facility) or with the Micro Focus BLDINDEX utility.

Multiple Record Types
(Next) (Previous) (Table-of-Contents)

Each record type within a file should be treated as if the record type were a separate file. If a file contains five (5) different record types then a process must be implemented that will determine the record type and then pass control to the appropriate conversion routine for the record type. The technique used by the SimoTime technology greatly simplifies this effort. Since the SimoTime technology generates a callable routine based on a COBOL copy file it will be necessary to create a callable routine for each record type. Once this is done the user logic that determines the record type may be "cut" from an existing COBOL program that accesses the file and pasted into the SimoTime generated COBOL source code that performs the file I/O. The file I/O program will then call the appropriate conversion routine based on the "cut-and-pasted" user logic.

Variable Length Records and Concatenated Segments
(Next) (Previous) (Table-of-Contents)

If a file contains variable length records created by using blank truncation techniques then the conversion process is similar to the process used to convert files with fixed length records.

If the variable length records are built dynamically by appending a variety of structured data segments to a base segment of a record then each segment must be treated as a separate entity and converted accordingly. In other words a separate routine should be generated for each segment of data and the user logic from an existing program should be used to determine the appended segment type, length and structure and do the segment conversion accordingly. Additional time should be allocated to convert this type of file content structure.

Maintain Mainframe Numeric Integrity
(Next) (Previous) (Table-of-Contents)

The IBM Mainframe has a variety of numeric formats or encoding schemes. This mix of numeric formats has been challenging for mainframe programmers since their inception. When migrating data the numeric formats require special consideration. The "BINARY" or "PACKED" numeric fields should not be converted (or translated) between EBCDIC to ASCII. These fields are supported on a Mainframe System and in a Micro Focus environment running on Windows or UNIX when using a mainframe dialect.

Possible Expansion of Numeric Fields
(Next) (Previous) (Table-of-Contents)

The following items should be considered when thinking about expanding packed or binary numeric fields.

Binary, Packed or Floating Point numeric fields may be used with "Record Sequential" files that are EBCDIC or ASCII encoded. The Binary, Packed or Floating Point numeric fields will cause a problem if attempting to include in a "Line Sequential" (or ASCII/Text) file.

Record Counts
(Next) (Previous) (Table-of-Contents)

This is part of the basic requirements to ensure the files being converted will have an equal number of records read from the input file and written to the output file. The SimoTime technology has an option to provide a read write count for an ordered load of an indexed file. When doing an unordered load it will provide a read, write and update count.

Audit Requirements of Management or Consulting Firm
(Next) (Previous) (Table-of-Contents)

Management and/or the auditors/consultants should be involved in the data migration (and data file content conversion) process as early as possible. This will help ensure the process will meet the requirements and maintain the necessary level of data integrity at each step in the process.

When converting data files an existing definition (such as a COBOL copy file) of the data structures (or record layouts) should be used. This will avoid introducing errors in a process that requires the data to be defined by a new or proprietary format.

The data conversion process should be a repeatable process with an audit trail. The process should be executable as an automated, unattended process. Operator intervention during the conversion process should be considered as an exposure to introducing errors into the process.

Summary
(Next) (Previous) (Table-of-Contents)

Here is what we have done over the years when faced with converting data files between ASCII and EBCDIC. In the world of programming there are many ways to solve a problem. This documents and the links to other documents are intended to provide a choice of alternatives.

Today, we take the following approach.

Anyone considering a data conversion should seek additional assistance from a consulting or programming services organization that has experience in this area.

Software Agreement and Disclaimer
(Next) (Previous) (Table-of-Contents)

Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Enterprises. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

SimoTime Enterprises makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Enterprises shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.

Downloads and Links to Similar Pages
(Next) (Previous) (Table-of-Contents)

This section includes links to documents with additional information that is beyond the scope and purpose of this document. The first sub-section requires an internet connection, the second sub-section references locally available documents.

Downloads and Links, Internet Access Required
(Next) (Previous) (Table-of-Contents)

The following links will require an internet connect.

For information about data file management in a diverse or mixed systems environment refer to the  Series of White Papers  for non-relational data files.

A good place to start is The SimoTime Home Page for access to white papers, program examples and product information.

Explore The Numbers Connection in the SimoTime Library for more examples of programs and documentation that describe and demonstrate techniques for understanding and processing the various numeric field formats used in a mainframe environment.

Explore The ASCII and EBCDIC translation tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.

Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and QSAM files.

Explore The Micro Focus Web Site for more information about products and services available from Micro Focus.

Downloads and Links, Local Access
(Next) (Previous) (Table-of-Contents)

The following links should be accessible without an internet connection.

Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and QSAM files.

Glossary of Terms
(Next) (Previous) (Table-of-Contents)

Check out  The SimoTime Glossary  for a list of terms and definitions used in the documents provided by SimoTime.

Comments, Suggestions or Feedback
(Next) (Previous) (Table-of-Contents)

This document was created and is maintained by SimoTime Enterprises.

If you have any questions, suggestions, comments or feedback please call or send an e-mail to: helpdesk@simotime.com

We appreciate hearing from you.

Company Overview
(Next) (Previous) (Table-of-Contents)

Founded in 1987, SimoTime Enterprises is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. This includes the smallest thin client using the Internet and the very large mainframe systems. There is more to making the Internet work for your company's business than just having a nice looking WEB site. It is about combining the latest technologies and existing technologies with practical business experience. It's about the business of doing business and looking good in the process. Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems.

Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com