Data File Compare When technology complements business   Data Management Series

Copyright © 1987-2008  SimoTime Enterprises, LLC  All Rights Reserved http://www.simotime.com

Introduction
(Next) (Previous) (Table-of-Contents)

This document is an introduction or overview of the data file comparison aspects of an application or data migration between a Mainframe System and a Windows System running a Micro Focus sub-system such as Enterprise Server, Application Server or Net Express. This white paper describes the technique for generating a COBOL program that will compare two data files in a Micro Focus or Mainframe environment. The format of the files being compared may be sequential (fixed or variable length records) or VSAM. When a difference occurs the content of each record is displayed in hexadecimal, ASCII and/or EBCDIC.

During the testing phase of an application migration between a Mainframe System and a Windows or UNIX System it is a requirement to compare the results of parallel test runs. Using a programmatic approach for data file comparison can be a quick and accurate way to compare large volumes of data.

For discussion purposes and to better understand the requirements and the scope of what is really needed for a "data file compare process" during a parallel test cycle we will make the following assumptions.

The objective is to define a process and use technology that will let us have the flexibility of doing data file compares on the mainframe, a Windows platform or a UNIX platform (Windows or UNIX may be a server or a client system).

A special "Thank you" to Larry Simmons of Micro Focus for providing much of the information that is presented in this series of white papers and sample programs.

Estimate Scope and Validate Requirements
(Next) (Previous) (Table-of-Contents)

This section provides a list of questions that will aid in determining the scope of effort for creating the process to compare files and identify differences.

Estimating, Identifying & Processing Differences
(Next) (Previous) (Table-of-Contents)

The following is used to determine the basic requirements for the data file comparison effort with a focus on the level of detail required when a difference occurs.

Estimating, File Inventory and Types
(Next) (Previous) (Table-of-Contents)

The following is used to determine the number of files and the file types and characteristics.

Software for Comparing Data Files
(Next) (Previous) (Table-of-Contents)

The file comparison technology that is available from SimoTime Enterprises executes on the Windows platform. However, this technology does not actually do the file compares but generates a COBOL program that performs the file compares. This COBOL program may be compiled and executed on an IBM Mainframe (z/OS or VSE), a Windows platform with Micro Focus or a UNIX platform with Micro Focus.

It is important to set up a directory structure to generate and compile the file comparison programs. The file comparison programs are generally treated as tools for use by the development and testing groups within an organization.

Therefore, the generated programs for file compares are kept separate from the mainstream source code. This is discussed in the "Possibilities and Considerations" section of this document.

Using the SimoTime technology for file compares is a two-step process. The generation and compilation of the COBOL source code is the first step and this is a one-time process. The second step is the repeatable task of defining the positions within the records within the files to be compared and then executing the compare program.

Define Positions to Compare
(Next) (Previous) (Table-of-Contents)

The SimoTime technology offers two alternates for defining the positions within the records that are going to be compared. The primary difference between the two alternatives is the point in the process where the positions to be compared are determined.

Define Positions at Execution Time
(Next) (Previous) (Table-of-Contents)

This type of file comparison program will read a control file (SYSUT3) containing /COMPARE statements at execution time and do positional comparisons within the records based on the /COMPARE statements submitted at execution time. The advantage of this approach is the compare positions are defined when the generated COBOL program is executed. The user may change the positions within the record to be compared without having to re-generate and compile the program. The disadvantage is that it is necessary to have a control file with compare statements at execution time.

Define Positions at Compile Time
(Next) (Previous) (Table-of-Contents)

This type of program will simply read two files and do the comparison of positions within the records base on hard-coded values in the generated program. This type of program has the /COMPARE statements in the same control file (SYSCNTL) that is used to generate the COBOL source code. The positions to be compared will be determined at compile time and become part of the generated source code. The advantage of using this approach is that fewer parameters are required at execution time. The disadvantage of using this approach is that it requires the program to be regenerated and compiled if a user wants to change the positions within the records to be compared.

Generating a File Compare Program
(Next) (Previous) (Table-of-Contents)

This step is usually a one-time process (this example uses the option to define positions to be compared at execution time). To generate a file comparison program requires a specification file to be provided to the SimoTime technology that will generate the file comparison source code. The following is an example of the records within an ASCII/Text file that will be required.

***********************************************************************
*    This is an example of the compare specifications to generate a   *
*        Data File Comparison Program. This is used by SimoZAPS       *
*                     SimoTime Enterprises, LLC                       *
*            (C) Copyright 1987-2007 All Rights Reserved              *
*              Web Site URL:   http://www.simotime.com                *
*                    e-mail:   helpdesk@simotime.com                  *
***********************************************************************
/Dialect  C2
/progid   ITUTL3C1
/SYSUT1   name=SYSUT1 org=Indexed recfm=variable rlen=512 klen=12 kpos=1
/SYSUT2   name=SYSUT2 org=Indexed recfm=variable rlen=512 klen=12 kpos=1
/SYSUT3   name=SYSUT3 org=ASCII/Text recfm=variable rlen=080
*
/DELTAMAX 250 EOF
/KEYFIELD SYSUT1 pos  1 len  12 SYSUT2 pos  1 len  12
*COMPARE  SYSUT1 pos  1 len 512 SYSUT2 pos  1 len 512
*COMPARE  uses SYSUT3 ...
*
/DFORMAT  ASC HEX EBC
*
/END

For detailed information about the statements in the preceding file refer to the SimoTime documentation for file compares at  http://www.simotime.com/simozaps.htm#Compare2 .

Executing a File Compare Program
(Next) (Previous) (Table-of-Contents)

The following is an example of a Windows command file (or batch file) that will set the environment and execute (or run) the program.

@echo OFF
rem  *
     set SYSLOG=c:\SimoDemo\TestLib1\DataWrk1\SYSLOGT1.TXT
     set SYSUT1=C:\SimoDEMO\TestLib1\DataAsc1\ITEMASC1.Dat
     set SYSUT2=C:\SimoDEMO\TestLib1\DataAsc1\ITEXPECT.DAT
     set SYSUT3=c:\SimoDemo\TestLib1\DataTxt1\ITUTL301.TXT
     if exist %SYSLOG% erase %SYSLOG%
rem  *
     run ITUTL3C1
rem  *
     if exist %SYSLOG% start Notepad %SYSLOG%
rem  *
     pause

For this example the information about the job execution and possible differences in the files being compared will be written to the SYSLOG file.

For this example the control file (SYSUT3) contains the following compare statement. Since the maximum record size for the file is 512 bytes this will compare the entire record content.

/COMPARE SYSUT1 pos 1 len 512 SYSUT2 pos 1 len 512

Seeing the Differences
(Next) (Previous) (Table-of-Contents)

What to do when a not equal condition occurs can be challenging on a single platform but in today's environment with multiple platforms and a mix of encoding schemes (such as EBCDIC and ASCII) and numeric formats (such as PACKED, BINARY and SIGNED-ZONE-DECIMAL) the task can become time consuming and difficult.

When a difference is found the type of display or logging information is defined by the use of a /DFORMAT statement in the control file used during program generation. When a not equal condition is encountered the following is displayed to the screen and written to a logging file. The following /DFORMAT statement was used in this example and will cause the compare program to dump the possible ASCII Text, the hexadecimal values and the possible EBCDIC text.

/DFORMAT  ASC HEX EBC

The RED shows the possible ASCII translation. The BLUE shows the possible EBCDIC translation. The BLACK shows the hexadecimal dump information on two lines (high nibble on line 1, low nibble on line 2). The GREEN shows reference information about each file. For example, the relative record number is displayed along with the position and length of the text string within the record that was compared. The MAROON row shows the positions that are equal (=) or not equal (#). The YELLOW vertical column highlights the differences. The following page shows sample output of the compare program when a difference occurs. Sample Output when a Difference Occurs.

*** 2005/04/01 08:57:37:40 Starting  - Data File Content Comparison by SimoTime Enterprises, LLC
*** 2005/04/01 08:57:37:41 SYSUT1.....Record Number(position:length) 000000001(00001:00094)
*** 2005/04/01 08:57:37:41 ....:....1....:....2....:....3....:....4....:....5....:....6....:....7....:....8....:....9....
*** 2005/04/01 08:57:37:42 000000000001Distributor Cap                                 ........Each            ...2....i.
*** 2005/04/01 08:57:37:42 3333333333334677766776724672222222222222222222222222222222220000000046662222222222220003900068
*** 2005/04/01 08:57:37:42 000000000001493429254F203100000000000000000000000000000000000000000051380000000000000002C0009C
*** 2005/04/01 08:57:37:42 .....................?.../.........................................../........................
*** 2005/04/01 08:57:37:43 SYSUT2.....Record Number(position:length) 000000001(00001:00094) 
*** 2005/04/01 08:57:37:43 ....:....1....:....2....:....3....:....4....:....5....:....6....:....7....:....8....:....9....
*** 2005/04/01 08:57:37:43 000000000001Distributor Cap                                 ...$....Each            ...2....i.
*** 2005/04/01 08:57:37:43 3333333333334677766776724672222222222222222222222222222222220002000046662222222222220003900068
*** 2005/04/01 08:57:37:44 000000000001493429254F203100000000000000000000000000000000000004000C51380000000000000002C0009C
*** 2005/04/01 08:57:37:44 .....................?.../.........................................../........................
*** 2005/04/01 08:57:37:44 ===============================================================#===#==========================
*** 2005/04/01 08:57:37:45 *
*** 2005/04/01 08:57:37:45 SYSUT1.....Record Number(position:length) 000000002(00001:00094)
*** 2005/04/01 08:57:37:45 ....:....1....:....2....:....3....:....4....:....5....:....6....:....7....:....8....:....9....
*** 2005/04/01 08:57:37:45 000000000002Rotor                                           ........Each            ........)\
*** 2005/04/01 08:57:37:46 3333333333335676722222222222222222222222222222222222222222220000000046662222222222220000800025
*** 2005/04/01 08:57:37:47 0000000000022F4F200000000000000000000000000000000000000000000000000051380000000000000009C0009C
*** 2005/04/01 08:57:37:47 .............?.?...................................................../.......................*
*** 2005/04/01 08:57:37:48 SYSUT2.....Record Number(position:length) 000000002(00001:00094)
*** 2005/04/01 08:57:37:48 ....:....1....:....2....:....3....:....4....:....5....:....6....:....7....:....8....:....9....
*** 2005/04/01 08:57:37:49 000000000002Rotor                                           ...$....Each            ........)\
*** 2005/04/01 08:57:37:49 3333333333335676722222222222222222222222222222222222222222220002000046662222222222220000800025
*** 2005/04/01 08:57:37:49 0000000000022F4F200000000000000000000000000000000000000000000004000C51380000000000000009C0009C
*** 2005/04/01 08:57:37:50 .............?.?...................................................../.......................*
*** 2005/04/01 08:57:37:51 ===============================================================#===#==========================
*** 2005/04/01 08:57:37:52 *
*** 2005/04/01 08:57:37:53 Summary   - Data File Content Comparison
*** 2005/04/01 08:57:37:53 000000005 - Record count for ITCOMPD1
*** 2005/04/01 08:57:37:54 000000005 - Record count for ITCOMPD2
*** 2005/04/01 08:57:37:55 000000002 - Unequal count
*** 2005/04/01 08:57:37:55 Finished  - Data File Content Comparison by SimoTime Enterprises, LLC

The summary information shows the record counts and the number of unequal compares by record count.

Possibilities & Considerations
(Next) (Previous) (Table-of-Contents)

This section provides additional detail about the process for generating, compiling and executing a data file comparison program. This discussion is limited to the Windows environment. However, the process is very similar for the mainframe and UNIX environments once the COBOL source code has been generated on a Windows system.

Directory Structure for Managing File Compare Assets
(Next) (Previous) (Table-of-Contents)

The following is a sample sub-directory structure for managing the assets used by the file compare process. The following directory structure is usually stored under a higher-level directory.

Record Counts
(Next) (Previous) (Table-of-Contents)

This is part of the basic requirements to ensure the files being compared will have an equal number of records. The SimoTime technology has an option to read to "End-of-File" or to "Quit" after a certain number of differences have been identified.

If the "End-of-File" option is used then differences will be written to the log file and when the difference count exceeds the maximum then differences will not be written to the log file but reading of the two files being compared will continue until end-of-file is reached and a records count for the total number of records read from each file will be provided along with a non-zero return code.

If the "Quit" option is used and the maximum number of differences is exceeded then a count of the number of records read before the program is terminated with a non-zero return code will be provided.

Tracking Record Inserts and Deletes
(Next) (Previous) (Table-of-Contents)

The SimoTime technology has the capability of tracking record inserts and deletes for Indexed files (or VSAM, Key-Sequenced-Data-Sets). This can also be done for sequential files if they are in sequence by an identified field that can be used as a key field.

Relating Record Positions to Copy File Field Names
(Next) (Previous) (Table-of-Contents)

The SimoTime technology has the capability of reading a COBOL copy file that defines a record layout and providing HTML documentation. The HTML documentation includes the field name, the relative position of the field within the data structure, the logical length of the fields (number of digits) and the physical field length (actual bytes of memory used).

When the comparison program is executed information is provided when a difference occurs by indicating the relative position within the record where a difference occurred. By referencing the HTML document it is a simple process to identify the field name where the difference occurred.

File Compare, Reports
(Next) (Previous) (Table-of-Contents)

Sequential files that contain print line images can present a challenge when attempting to compare. Reports usually contain multiple record types without a field that identifies the record type or print line. It is quite common that header information contains a date and possible time stamp and based on testing cycles these fields are rarely equal (especially the time field). Knowing when a record (print line) is a header, a footer, a detailed print line, a sub-total line or a total line can be difficult to determine. It is very important to determine up-front how many files to be compared are report-oriented and what the structure is for these files. Once the file format is understood it is usually a straight-forward process to add logic to the generated COBOL program to adjust to the multiple record types and to bypass the date and time fields. For example, if a header line contains three lines of information the COBOL compare program may be modified to look for the "skip to line 1" character in position 1 of the 133 byte print line and then bypass the compare for three records.

File Compare, ASCII & EBCDIC Collating Sequence
(Next) (Previous) (Table-of-Contents)

When comparing a sequential file that is downloaded from a Mainframe System (EBCDIC-encoded) with a file that is created on a Windows System (ASCII-encoded) it may be necessary to make adjustments for the difference in the ASCII and EBCDIC collating sequences. This is especially true if the field that determines the sequence of the file is alpha-numeric.

If the compare is being done on the Windows System then the file downloaded from the mainframe (via FTP in binary mode) will need to be converted to ASCII and then sorted prior to doing a compare.

Accumulate Hash Totals for Currency Fields
(Next) (Previous) (Table-of-Contents)

It may be a requirement to accumulate batch totals for currency fields. The SimoTime technology does not explicitly provide this capability. However, since the SimoTime technology generates COBOL source code that is compiled and then used to do the data file comparison this COBOL source code may be easily modified to accumulate the batch totals. Also, the batch totals for currency fields may already be available in reports created by the application.

Data Scrubbing
(Next) (Previous) (Table-of-Contents)

During an application migration it is always tempting to scrub the data (for example, replace leading spaces in numeric fields with zeroes). If the data is scrubbed this usually leads to a number of data file compare differences during the application testing or parallel testing phases of the migration effort. Extra time will be needed to manually process and validate this information.

Using the Compare Programs in an EBCDIC Environment
(Next) (Previous) (Table-of-Contents)

The programs generated to do data file comparison may be compiled and executed in an ASCII or EBCDIC-encoded environment. The generated compare programs will make calls to two additional programs. The additional programs are SIMOHEX4 and SIMOLOGS. SimoTime provides two DLL's that will work in the ASCII-encoded, Net Express environment. If the generated file compare programs are to be used on the mainframe or in an EBCDIC-encoded environment with Micro Focus technology the SIMOHEX4 and SIMOLOGS programs must be compiled to execute in these EBCDIC-encoded environments. For these environments the OS/390 or later dialect should be used. The next challenge will be to get the file compare program to write information to the SYSLOG file. The following JCL will show what needs to be done.

//A2E02KJ1 JOB SIMOTIME,ACCOUNT,CLASS=1,MSGCLASS=0,NOTIFY=CSIP1
//* *******************************************************************
//*                   THIS PROGRAM IS PROVIDED BY:                    *
//*                    SIMOTIME ENTERPRISES, LLC                      *
//*           (C) COPYRIGHT 1987-2007 ALL RIGHTS RESERVED             *
//*             WEB SITE URL:   HTTP://WWW.SIMOTIME.COM               *
//*                   E-MAIL:   HELPDESK@SIMOTIME.COM                 *
//* *******************************************************************
//*
//* TEXT   - Execute a program that compares two data files
//* Author - SimoTime Enterprises
//* Date   - September 20, 2007
//*
//* *******************************************************************
//* STEP 1, Delete the previously created SYSLOG file.
//*
//SYSLOGDT EXEC PGM=IEFBR14
//SYSLOG   DD  DSN=SIMOTIME.DATA.SYSLOG,DISP=(MOD,DELETE,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=V,LRECL=1055,DSORG=PS)
//*
//* *******************************************************************
//* STEP 2, Allocate a new SYSLOG file.
//* Note: The LRECL is four (4) bytes bigger than the logical record
//*       that is defined in the program that writes to SYSLOG. This
//*       allows for the four (4) byte Record Descriptor Word (RDW)
//*       that is appended to the front of each record. The program
//*       defines the records as varying in size from 64 to 1,051.
//* Note: It is necessary to pre-allocate the SYSLOG file. The program
//*       that does the actual file compare will do an "OPEN EXTEND".
//*       If the file is not pre-allocated, an open error will be 
//*       posted and the program will execute but will only display
//*       truncated information to SYSOUT.
//*
//SYSLOGCT EXEC PGM=IEFBR14
//SYSLOG   DD  DSN=SIMOTIME.DATA.SYSLOG,DISP=(NEW,CATLG,DELETE),
//             STORCLAS=MFI,
//             SPACE=(TRK,5),
//             DCB=(RECFM=V,LRECL=1055,DSORG=PS)
//*
//* *******************************************************************
//* STEP 3, Execute the compare program.
//* Note: when a difference occurs the information will be written to
//*       the SYSLOG file
//*
//STEP1CPR EXEC PGM=JGBT1
//COMPARE1 DD DSN=SIMOTIME.DATA.DATKS02K,DISP=SHR
//COMPARE2 DD DSN=SIMOTIME.DATA.DATKS02Z,DISP=SHR
//SYSLOG   DD DSN=SIMOTIME.DATA.SYSLOG,DISP=SHR
//SYSOUT   DD  SYSOUT=*
//

Summary
(Next) (Previous) (Table-of-Contents)

The file compare process can be automated through the use of Windows BAT or CMD files. The setup effort to do this is dependent on answers to the preceding questions and the application's sensitivity to the differences in the EBCDIC and ASCII collating sequences.

We are assuming the files will have the typical amount of date and time oriented fields that will have minor differences and that you may want to bypass or validity check these fields. Also, in some files there are filler items you may want to bypass in the compare process.

At one end of the spectrum the comparing of two sequential files that are in the same sequence and use the same encoding format is easy.

At the other end of the spectrum the comparing of two Key-Sequenced-Data-Sets that use different encoding formats with Alpha-Numeric keys and alternate indexes with variable length records and multiple records types with a mixture of packed and binary fields and wanting to track and identify differences with record inserts and deletes. This can be a real challenge.

The Data File Comparison section of the  SimoZAPS Reference Manual  is available on the Internet and provides additional information about data file comparison technology.

Software Agreement and Disclaimer
(Next) (Previous) (Table-of-Contents)

Permission to use, copy, modify and distribute this software for any commercial purpose requires a fee to be paid to SimoTime Enterprises. Once the fee is received by SimoTime the latest version of the software will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

Permission to use, copy, modify and distribute this software for a non-commercial purpose and without fee is hereby granted, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Enterprises.

SimoTime Enterprises makes no warranty or representations about the suitability of the software for any purpose. It is provided "AS IS" without any express or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Enterprises shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software.

Downloads and Links to Similar Pages
(Next) (Previous) (Table-of-Contents)

The  self-study session for comparing data files  uses a generated COBOL program to compare two data files. The session includes a hands-on exercise that does an actual data file comparison.

Note: You must be attached to the Internet to download a Z-Pack or view the list.

This document is part of the  Data File Management Series  of white papers that discuss the transferring, sharing, converting and comparing tasks required when moving or sharing data between different systems.

The hexadecimal dump of the parameter-buffer uses the same technique as describe in another SimoTime example that describes the dumping of a data string using COBOL. The name of the member that does the actual hexadecimal dump is called SimoDUMP. A copy file (PASSDUMP.CPY) is provided for defining the pass area.

The SimoZAPS Utility Program has the capability of generating a COBOL program that will do the conversion of sequential and VSAM (KSDS) files between EBCDIC and ASCII. SimoZAPS can also read a sequential file in EBCDIC format and create an ASCII/CRLF file or VSAM KSDS file in ASCII format. The conversion tables may be viewed or modified to meet unique requirements. The Hexcess/2 function provides the capability of viewing, finding or patching the contents of a file in hexadecimal.

This item will provide a link to  an ASCII or EBCDIC translation table. A column for decimal, hexadecimal and binary is also included.

The following table is a list of white papers that provides more detailed information about the four common numeric formats used on an IBM Mainframe.

Check out  The VSAM - QSAM Connection  for more examples of mainframe VSAM and QSAM accessing techniques and sample code.

This document provides a quick summary of the  File Status Key  for VSAM data sets and QSAM files.

Check out  The SimoTime Library  for a wide range of topics for Programmers, Project Managers and Software Developers.

To review all the information available on this site start at  The SimoTime Home Page .

Glossary of Terms
(Next) (Previous) (Table-of-Contents)

Check out  The SimoTime Glossary  for a list of terms and definitions used in the documents provided by SimoTime.

Comments or Suggestions
(Next) (Previous) (Table-of-Contents)

If you have any questions, suggestions or comments please call or send an e-mail to: helpdesk@simotime.com

We appreciate your comments and feedback.

About SimoTime Enterprises, LLC
(Next) (Previous) (Table-of-Contents)

Founded in 1987, SimoTime Enterprises is a privately owned, Limited Liability Corporation located in Novato, California. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. This includes the smallest thin client using the Internet and the very large mainframe systems. There is more to making the Internet work for your company's business than just having a nice looking WEB site. It is about combining the latest technologies and existing technologies with practical business experience. It's about the business of doing business and looking good in the process. Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complimentary manner with existing corporate mainframe systems. Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com