Data File Management Transfer, Share, Convert and Confirm

Introduction

This document will describe and demonstrate techniques for maintaining data integrity during a file transfer and convert process. The primary task is to determine record counts and provide totals for numeric fields with a focus on validating currency amounts. The secondary task is to focus on the data file compare process.

We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.

Copyright © 1987-2024
SimoTime Technologies and Services
All Rights Reserved

Flowchart of Process

The following is an overview of the tasks involved in transferred and converting a data file along with a confirmation file with records counts and numeric totals.

Color Associations: The  light-green  boxes are unique to SIMOTIME Technologies using an IBM Mainframe System or Micro Focus Enterprise Developer. The  light-red  boxes are unique to the SIMOTIME Technologies using a Linux, UNIX or Windows System and COBOL Technologies such as Micro Focus. The  light-yellow  boxes are SIMOTIME Technologies, Third-party Technologies, decision points or program transitions in the processing logic or program generations. The  light-blue  boxes identify the input/output data structures such as Documents, Spreadsheets, Data Files, VSAM Data Sets, Partitioned Data Set Members (PDSM's) or Relational Tables. The  light-gray  boxes identify a system function or an informational item.

Record Structure for Test File

The following (RTNTOTB1.cpy) is the COBOL copy file that is used to define the record structure for the User Data Files that will be used by the Confirm and Compare processes.

      *****************************************************************
      *               RTNTOTB1.CPY - a COBOL Copy File                *
      *         A Record Structure Containing Numeric Fields          *
      *       Used for Calculating and Testing CheckSum Totals        *
      *         Copyright (C) 1987-2019 SimoTime Technologies         *
      *                     All Rights Reserved                       *
      *              Provided by SimoTime Technologies                *
      *        Our e-mail address is: helpdesk@simotime.com           *
      *     Also, visit our Web Site at http://www.simotime.com       *
      *****************************************************************
      *
       01  RTNTOTD1-RECORD.                                             01-80
           05  RTNTOT01-ALPHA-01 PIC X(32).                             01-32
           05  FILLER            PIC X.                                 33-33
           05  RTNTOT01-VALUE-01 PIC 9(10)V99.                          34-45
           05  FILLER            PIC X.                                 46-46
           05  RTNTOT01-VALUE-02 PIC 9(8)V999.                          47-57
           05  FILLER            PIC X.                                 58-58
           05  RTNTOT01-VALUE-03 PIC S9(5) SIGN TRAILING SEPARATE.      59-64
           05  FILLER            PIC X.                                 65-65
           05  FILLER            PIC X(15).                             66-80
      *
      ***  RTNTOTB1 - End-of-Copy File - - - - - - - - - - - RTNTOTB1 *
      *****************************************************************
      *

Program Generation

This section describes the process for generating programs that will do data conversion and comparison. A confirmation process is used to validate data integrity during the transfer and conversion processes. The SimoTime Utility Programs that do the COBOL source code generations based on user-defined specifications must be executed on a Windows System with Micro Focus Enterprise Developer technology. Once the COBOL source code is generated it may be compiled and executed on a variety of platforms that include the following.

Generate Convert Programs

This section and the following three sub-sections of this document we describe the generation process for the compare and checksum programs.

Generate CheckSum Programs

The following (RTNTOT02.pcf) is the Process Control File used to create the CheckSum Programs that are used by the Confirmation process.

***********************************************************************
*                RTNTOT02.pcf - a Process Control File                *
*               SimoTime Program Generation Technologies              *
*             (C) Copyright 1987-2019 All Rights Reserved             *
*               Web Site URL:   http://www.simotime.com               *
*                     e-mail:   helpdesk@simotime.com                 *
***********************************************************************
* SYSUT1 is a Line Sequential File with 80-byte records.
***********************************************************************
* This Process Control File will be used to generate a set of COBOL
* programs that will accumulate totals for user-defined numeric fields
* and provide a record count.
*
* Refer to http://www.simotime.com/utcomp01.htm for additional detail
* about the PCF statements for Data Validate, Dump and Compare.
*
* The following group of statements will define the high level
* functions and processes to be performed.
*
&SIMOPREP  call ..\Env1BASE
&USERPREP  call UserCOGI
&CONFORM   IBM
&COPYFILE  RTNTOTB1.cpy
&HTMLFILE  rtntotb1.htm
*
* The following group of statements will define the behavioral
* characteristics and environment variable for the file I/O Program
* to be generated.
*
*HEAD34    ....:....1....:....2....:....3....
&HEAD34    Accumulate Totals for Numeric Data
&PROGID    RTNTOTC2
&SYSUT1    name=RTNTOTD1 org=ASCII/Text recfm=FIXED    rmax=80
*
* The following group of statements will define the behavioral
* characteristics and environment variables for the record content
* conversion program to be generated.
* The following defines the name of the callable routine for special
* numeric processing.
*
&NUMCALL   RTNTOTR2
&NUMREC    RTNTOTD1-RECORD
*
* The following group of statements will define the type of processing
* and the name of the numeric fields.
*
&UT1TOTAL  RTNTOT01-VALUE-01
&UT1TOTAL  RTNTOT01-VALUE-02
&UT1TOTAL  RTNTOT01-VALUE-03
*
&END

Generate Compare Programs

The following described the compare programs used by this suite of programs.

Compare for the CheckSum Files

The following (RTNTOT03.pcf) is the Process Control File used to create the Compare Program that is used by the Confirmation process to compare the CheckSum Files.

***********************************************************************
*                RTNTOT03.pcf - a Process Control File                *
*                      A Data File Compare Program                    *
*              SimoTime Program Generation Technologies               *
*            (C) Copyright 1987-2019 All Rights Reserved              *
*              Web Site URL:   http://www.simotime.com                *
*                    e-mail:   helpdesk@simotime.com                  *
***********************************************************************
*
&SIMOPREP  call ../Env1BASE
&USERPREP  call USERCOMP
&COPYFILE  RTNTOTB1.CPY
&HTMLFILE  rtntotb1.htm
&FUNCTION  COMPARE
&CONFORM   MF
&USRMODEL  SYSCOMP1.txt
*HEAD34    ....:....1....:....2....:....3....
&HEAD34    Compare Totals and Record Counts
&progid    RTNTOTV3
&SYSUT1    name=RTNTOTX1 org=ASCII/Text recfm=FIXED rlen=121
&SYSUT2    name=RTNTOTX2 org=ASCII/Text recfm=FIXED rlen=121
*
&DELTAMAX  5 EOF
&UT1OMIT   if pos  12 len 1 EQ "*"
&UT2OMIT   if pos  12 len 1 EQ "*"
&COMPARE   SYSUT1 pos 12 len 110 SYSUT2 pos 12 len 110
*
&DELTAMAX  10 EOF
&DFORMAT   ASC NOHEX NOEBC
&DISPLAY   SYSOUT
&SYSLOG    ENABLED
*
&END

Compare for the User Data Files

WIP2

Transfer Programs, LUW to ZOS

WIP1

Compile & Execute, ZOS/EBCDIC

WIP1

Compile the CheckSum Programs

WIP2

Execute the CheckSum Programs

WIP2

//RTNTOTK1 JOB SIMOTIME,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*       RTNTOTK1.JCL - a JCL Member for Batch Job Processing        *
//*       This JCL Member is provided by SimoTime Technologies        *
//*           (C) Copyright 1987-2019 All Rights Reserved             *
//*             Web Site URL:   http://www.simotime.com               *
//*                   e-mail:   helpdesk@simotime.com                 *
//* *******************************************************************
//*
//* Text   - Read a file, calculate record counts and summary totals.
//* Author - SimoTime Technologies
//* Date   - January 24, 1996
//*
//* This is a single step job. It will execute a program that will
//* read a file and calculate record counts and summary totals for
//* a set of pre-defined numeric fields.
//*
//* This set of programs will run on a mainframe under MVS or on a
//* Personal Computer with Windows and Micro Focus Mainframe Express.
//*
//* *******************************************************************
//* Step 1 of 2, Delete any previously created file...
//*
//GETREADY EXEC PGM=IEFBR14
//WORKFILE DD  DSN=SIMOTIME.FTP1.RTNTOTX1,DISP=(MOD,DELETE,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=FB,LRECL=121,DSORG=PS)
//*
//* *******************************************************************
//* Step 2 of 2, Calculate record counts and summary totals...
//*
//NUMTOTAL EXEC PGM=RTNTOTV1
//RTNTOTD1 DD  DSN=SIMOTIME.DATA.RTNTOTD1,DISP=SHR
//SYSOUT   DD  DSN=SIMOTIME.FTP1.RTNTOTX1,DISP=(NEW,CATLG,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=FB,LRECL=121,DSORG=PS)
//

Transfer Files, ZOS to LUW

WIP1

The LUW-ASCII Environment

WIP1

Compile the Convert Programs

WIP2

Compile the CheckSum Programs

WIP2

Execute the Convert Programs

WIP2

Execute the CheckSum Programs

WIP2

//RTNTOTK2 JOB SIMOTIME,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*       RTNTOTK2.JCL - a JCL Member for Batch Job Processing        *
//*       This JCL Member is provided by SimoTime Technologies        *
//*           (C) Copyright 1987-2019 All Rights Reserved             *
//*             Web Site URL:   http://www.simotime.com               *
//*                   e-mail:   helpdesk@simotime.com                 *
//* *******************************************************************
//*
//* Text   - Read a file, calculate record counts and summary totals.
//* Author - SimoTime Technologies
//* Date   - January 24, 1996
//*
//* This is a single step job. It will execute a program that will
//* read a file and calculate record counts and summary totals for
//* a set of pre-defined numeric fields.
//*
//* This set of programs will run on a mainframe under MVS or on a
//* Personal Computer with Windows and Micro Focus Mainframe Express.
//*
//* *******************************************************************
//* Step 1 of 2, Delete any previously created file...
//*
//GETREADY EXEC PGM=IEFBR14
//WORKFILE DD  DSN=SIMOTIME.WRK1.RTNTOTX2,DISP=(MOD,DELETE,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=FB,LRECL=121,DSORG=PS)
//*
//* *******************************************************************
//* Step 2 of 2, Calculate record counts and summary totals...
//*
//NUMTOTAL EXEC PGM=RTNTOTV1
//RTNTOTD1 DD  DSN=SIMOTIME.DATA.RTNTOTD1,DISP=SHR
//SYSOUT   DD  DSN=SIMOTIME.WRK1.RTNTOTX2,DISP=(NEW,CATLG,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=FB,LRECL=121,DSORG=PS)
//

Compare the CheckSum Files

WIP2

@echo OFF
rem  * *******************************************************************
rem  *               RTNTOTW3.CMD - a Windows Command File               *
rem  *         This program is provided by SimoTime Technologies         *
rem  *           (C) Copyright 1987-2019 All Rights Reserved             *
rem  *             Web Site URL:   http://www.simotime.com               *
rem  *                   e-mail:   helpdesk@simotime.com                 *
rem  * *******************************************************************
rem  *
rem  * Text   - Compare two files using the omit function.
rem  * Author - SimoTime Technologies
rem  * Date   - January 24, 1996
rem  *
rem  * The job will compare the contents of an ASCII/Text file with the
rem  * content of a record sequential file with ASCII-encoded records.
rem  *
rem  * *******************************************************************
rem  * Step   1   Set the environment...
rem  *
     set CmdName=RTNTOTW3
     call ..\ENV1BASE %CmdName%
     if "%SYSLOG%" == "" set syslog=c:\SimoLIBR\LOGS\SimoTime.LOG
rem  *
     call SimoNOTE "*******************************************************%CmdName%"
     call SimoNOTE "Starting CmdName %CmdName%"
:DeleteQSAM
     call SimoNOTE "StepInfo Set environment variables
     set RTNTOTX1=%BaseLib1%\DATA\SPOOL\SYSOUT_RTNTOTW2.TXT
     set RTNTOTX2=%BaseLib1%\DATA\SPOOL\SYSOUT_RTNTOTW1.TXT
     set SYSLOG=%BASEAPP%\LOGS\SYSLOG_RTNTOTW3.dat
     set SYSOUT=%BASEAPP%\LOGS\SYSOUT_RTNTOTW3.txt
     if exist %SYSOUT% erase %SYSOUT%
rem  *
rem  * *******************************************************************
rem  * Step   2   Edit input, create a new output file...
rem  *
     call SimoNOTE "StepInfo Execute the Compare Program"
     call SimoNOTE "Expect.. %RTNTOTX1%"
     call SimoNOTE "Actual.. %RTNTOTX2%"
     run RTNTOTV3
     if not "%ERRORLEVEL%" == "0" set JobStatus=0010
     if not %JobStatus% == 0000 goto :EojNok
     start NotePAD %SYSOUT%
rem  *
     goto :EojAok
:EojAok
     call SimoNOTE "Finished JobName %CmdName%, Job Status is %JobStatus%"
     goto :End
:EojNok
     call SimoNOTE "ABENDING JobName %CmdName%, Job Status is %JobStatus%"
:End
     if not "%1" == "nopause" pause

A Sample User Data File

The following (RTNTOTJ8.jcl) is the JCL Member used to create the test file used in this example.

//RTNTOTJ8 JOB SIMOTIME,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*       This JCL Member is provided by SimoTime Technologies        *
//*           (C) Copyright 1987-2019 All Rights Reserved             *
//*             Web Site URL:   http://www.simotime.com               *
//*                   e-mail:   helpdesk@simotime.com                 *
//* *******************************************************************
//*
//* Text   - Create a file with numeric values for Numeric Testing.
//* Author - SimoTime Technologies
//* Date   - January 24, 1996
//*
//* The first job step (GETREADY) will delete any previously created
//* file. The second job step (BUILDNEW) will create a new file.
//*
//* This set of programs will run on a mainframe under MVS or on a
//* Personal Computer with Windows and Micro Focus Mainframe Express.
//*
//* *******************************************************************
//* Step 1 of 2, Delete any previously created file...
//*
//GETREADY EXEC PGM=IEFBR14
//RTNTOTD1 DD  DSN=SIMOTIME.DATA.RTNTOTD1,DISP=(MOD,DELETE,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=FB,LRECL=80,DSORG=PS)
//*
//* *******************************************************************
//* Step 2 of 2, Create and populate a new QSAM file...
//*
//BUILDNEW EXEC PGM=IEBGENER
//SYSPRINT DD  SYSOUT=*
//SYSIN    DD  DUMMY
//* :....1....:....2....:....3....:....4....:....5....:....6....:....7. ..:....8
//SYSUT1   DD  *
This is Record-001              =000000000123=00000001333=00765+
This is Record-002              =000000000456=00000012222=00021-
This is Record-003              =000000000789=00000123111=00044-
This is Record-004              =123456789012=00000001000=00145+
This is Record-005              =000000001200=00000001000=00100+
This is Record-006              =000123456789=00000001000=00200+
This is Record-007              =000000003995=00000001000=00300+
This is Record-008              =000000012345=00000001000=00875-
/*
//SYSUT2   DD  DSN=SIMOTIME.DATA.RTNTOTD1,
//             DISP=(NEW,CATLG,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=FB,LRECL=80,DSORG=PS)
//

A Sample CheckSum File

The following is the SYSOUT produced as a result of executing the job that calculates the Checksums.

* RTNTOTV1 Accumulate Totals for Numeric Data v12.01.11   helpdesk@simotime.com
* RTNTOTV1 A Program to Calculate Totals was generated by SimoTime Technologies
* RTNTOTV1 * Compiled for an ASCII environment...
* RTNTOTV1 RTNTOT01-VALUE-01                        1,235,802,647.09
* RTNTOTV1 RTNTOT01-VALUE-02                                  141.666
* RTNTOTV1 RTNTOT01-VALUE-03                                  570
* RTNTOTV1 Record count for RTNTOTD1                    000000008
* RTNTOTV1 Program to Calculate Totals generated by using SimoTime Technologies
* RTNTOTV1 Please send all inquires or suggestions to the helpdesk@simotime.com

Summary

This document will describe and demonstrate techniques for maintaining data integrity during a file transfer and convert process. This document may be used as a tutorial for new programmers or as a quick reference for experienced programmers.

In the world of programming there are many ways to solve a problem. This documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.

SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please contact us using the information in the  Contact or Feedback  section of this document.

Software Agreement and Disclaimer

Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.

SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.

Downloads and Links

This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.

Note: A SimoTime License is required for the items to be made available on a local system or server.

Current Server or Internet Access

The following links may be to the current server or to the Internet.

Note: The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.

    Explore the Principles of Data File Validation. This link includes guidelines for defining requirements and determining the scope of effort for a data validation effort. This effort includes a data file compare process, a technique for accumulating summary totals with a record count and a technique for reading a VSAM, KSDS and producing a hex-dump output based on a list of user-defined keys.

    Explore How to Generate a Data File Compare, Validate or Hex-Dump Program using simple specification statements in a Process Control File (PCF). This link to the User Guide includes the information necessary to create a Process Control File and generate the COBOL programs that will do a data file compare, accumulate summary totals with a record count or produce a Hex-Dump of records in a VSAM, KSDS based on a list of user-defined keys. The User Guide contains a list of the PCF statements that are used for the data file compare, validate or dump process.

    Explore the Principles of Data File Conversion. This link includes guidelines for defining requirements and determining the scope of effort for a data conversion effort.

    Explore How to Generate a Data File Convert Program using simple specification statements in a Process Control File (PCF). This link to the User Guide includes the information necessary to create a Process Control File and generate the COBOL programs that will do the actual data file conversion. The User Guide contains a list of the PCF statements that are used for the data file convert process.

    Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.

    Explore the COBOL Connection for more examples of COBOL programming techniques and sample code.

    Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.

    Explore The File Status Return Codes that are used to interpret the results of accessing VSAM data sets and/or QSAM files.

Internet Access Required

The following links will require an internet connect.

A good place to start is The SimoTime Home Page for access to white papers, program examples and product information. This link requires an Internet Connection

Explore The Micro Focus Web Site for more information about products (including Micro Focus COBOL) and services available from Micro Focus. This link requires an Internet Connection.

Glossary of Terms

    Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.

Contact or Feedback

This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.

We appreciate hearing from you.

Company Overview

SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.

Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.

Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.

Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com