Data File Validation
Generate a Compare or Validate Program
|The SimoTime Home Page|
Many companies are looking for ways to transfer, convert and/or validate data files using an IBM Mainframe System and a Distributed System (i.e. Linux, UNIX and/or Windows or "LUW") running Micro Focus sub-systems. This utility program (UTCOMPRT) is intended to assist in the data file validation efforts.
The SimoTime Technologies used for Data Validation are divided into three groups. The first group provides a data file compare. The second group provides a capability for accumulating summary totals and record counts. The third group provides a hexadecimal dump capability to manually view and analyze the records in a VSAM, Key-Sequenced-Data-Set (KSDS).
The SimoTime Utility Programs for Data File Validation run on a Windows System with Micro Focus and will generate the COBOL source code for a program that will do a compare of two data files. The generation of the COBOL source code requires a Windows System with Micro Focus Enterprise Developer, version 2.2 or later.
The capabilities included in the generated program are determined by the content of a Process Control File (or PCF). The Process Control File is an ASCII/Text file. Each record in the control file contains a record identifier, a keyword and possible parameters. The generated COBOL programs that do the actual validation of the data files may be compiled and executed on a variety of systems.
We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.
Copyright © 1987-2019
SimoTime Technologies and Services
All Rights Reserved
The section describes the system requirements for running the "GENERATION and EXECUTION" phases of preparing for a data validation task and doing an actual data validation task of non-relational data structures.
To run the SimoTime Utility Program (UTCOMPRT) requires a Windows System (Windows/7, Windows/8 or Windows Server) with Micro Focus Enterprise Developer, version 2.2 or later. The utility will generate the COBOL source code for a program that will do a data validation task.
As a general rule and once the generation process has completed the resulting COBOL source code may be compiled and executed on an IBM Mainframe System or a Distributed System that is capable of running Micro Focus COBOL. The following table provides additional detail about system (or platform) requirements.
|System Possibilities for Generated Data File Comparison Programs|
The SimoTime Utility Program (UTCOMPRT) is dependent on other SimoTime modules for the generation and execution of COBOL programs that do the data file compare. The level of dependency is determined by the functions specified in the Process Control File and incorporated into the compare process. If the SIMOLIBR Enterprise License is used then all compare functions are supported. This is the recommended approach since it provides full functionality and additional utilities such as a Hex Dump utility for files.
The following is a flowchart of the job that executes the process to generate programs (COBOL Source Code) that do data file comparisons.
|Generate a COBOL Program that will Compare Two Data Files|
The main program (UTCOMPRT) will read the process control file and call additional SimoTime programs to generate the COBOL source code for a program that will compare two data files. The details of the comparison process will be determined by the definitions in the process control file and incorporated into the generated COBOL source code. The HTML documentation may be generated based on specifications in the process control file.
In today's environments there are a variety of data validation alternatives. This section will focus on three popular techniques. The 1st technique is Human Observation. The 2nd is a Data File Compare. The 3rd is the accumulation and review of Summary Totals.
One approach to application or data validation is human observation or analysis. This approach is very useful when evaluating the usability and business flow of an application.
Using this approach for data validation presents the following challenges.
|1.||Introduces additional risk of human error.|
|2.||May be useful for small amounts of data. As the volume of data increases the reliability of the "human observation" approach decreases.|
|3.||Data strings with binary content is not easy to interpret and can be presented in a misleading manner when using a text editor.|
|4.||The structure of some data strings may require specialized knowledge to properly interpret.|
|5.||A "hexadecimal dump" presentation of the data may be required.|
Because of these challenges the "human observation" approach is considered as a "reasonability check" or "diagnostic aid". Typically, a programmatic methodology that can be automated and run without human intervention will be a requirement.
The "Data File Compare" approach is very popular when doing regression testing. At the conclusion of a regression test a copy of a file that contains the "expected" results is compared with the file (actual results) created or updated during the regression test.
This approach first appeared when users began running applications that created sub-totals and final totals for numeric items. The reports and totals were then manually reviewed by the finance business group. Doing this for currency values quickly became a mandatory requirement.
The use of reports produced by a application remains a very effective validation approach. However, if a problem occurs the problem determination can be very time consuming based on the size and complexity of the application code and data structures.
Therefore, a technology that will focus on the processing of the numeric structures and produce summary totals with record counts could be very useful. SimoTime has the technology to do this by generating COBOL programs that will read a file and produce summary totals based on the user selecting the fields from a COBOL copy file definition of the record layout.
The ability to do a data file compare is a primary requirement for a data validation process. The following will describe and demonstrate the compare capabilities of the SimoTime Technologies.
A sample suite of programs will be used to describe the basic functionality provided by the SimoTime Data File Comparison processes and technology. This suite of programs will do a simple 80/80 Data File Compare of two sequential files that contain fixed length records of 80-byte text strings.
This suite of sample programs will be used to describe the functionality of moderate complexity that is provided by the SimoTime Data File Comparison processes and technology. This suite of programs will do a compare of two customer files that are keyed sequenced data sets (VSAM KSDS) with 512 byte records of fixed fields with some numeric fields that are packed and binary formats.
This suite of programs and documentation will describe and demonstrate additional capabilities and advanced data file compare functions for comparing report-oriented files that contain page headers with date and time stamps and user-defined line items. The ability to conditionally omit records from the compare process and to write the results of the compare process to a log file are included.
This suite of programs and documentation will describe and demonstrate an approach for reading a file, calculating record counts and producing summary totals for a set of pre-defined numeric fields. The approach uses two COBOL programs that are generated using SimoTime technology. The source code for the programs is generated on a Windows System with Micro Focus COBOL and may be compiled and executed on an IBM Mainframe or a Linux, UNIX or Windows System with Micro Focus COBOL.
This suite of programs and documentation will describe and demonstrate an approach for reading a VSAM, Key-Sequenced-Data-Set and producing hexadecimal dump output based on a list of user-defined keys. The approach uses a COBOL program that is generated using SimoTime technology. The program source code is generated on a Windows System with Micro Focus COBOL and may be compiled and executed on an IBM Mainframe or a Linux, UNIX or Windows System with Micro Focus COBOL.
The Process Control File contains the specifications that will be used to generate the COBOL source code that will be used to compare two data files on a record-by-record basis. The following describes the record layout for the Process Control File.
|The Process Control File, Record Layout|
The following describes the Keywords and Parameters for the statements in the Process Control File.
|The Process Control File for Data File Compare, Statement Keywords and Parameters|
This section provides additional detail about the secondary CMD files and programs that are used in this set of sample programs.
A command file (ENV1BASE.cmd) is called from other command files to set commonly used environment variables. This provides a single point of definition. The following is a listing of the contents of the command file.
@echo OFF rem * ******************************************************************* rem * ENV1BASE.cmd - a Windows Command File * rem * This program is provided by SimoTime Technologies * rem * (C) Copyright 1987-2019 All Rights Reserved * rem * Web Site URL: http://www.simotime.com * rem * e-mail: email@example.com * rem * ******************************************************************* rem * rem * Text - Provide a single point to set common environment variables. rem * Author - SimoTime Technologies rem * Date - January 24, 1996 rem * rem * Set the commonly used environment variables. This is used to provide rem * a single point for managing the commonly used environment variables. rem * set SimoLIBR=c:\SimoLIBR set BASELIB1=c:\SIMOSAM1\DEVL set BASELIB8=c:\SimoSAM8 set BaseWIP1=c:\SimoSAM1\WIP1 set DATAZERO=c:\SIMODATA\DEVL\DATA\ZERO set BASEAPP=%BaseLib1% set BASESYS=%BaseLib1%\SYS1 set BASECAT=%BaseLib1%\DATA set UMAPALIB=%BASECAT%\ASC1 set UMAPELIB=%BASECAT%\EBC1 set SYSLOG=%BASESYS%\LOGS\SYSLOG_USER.DAT set SYSOUT=%BASEAPP%\LOGS\SYSOUT_SIMSAM01.txt set SLZMSG=%BASEAPP%\LOGS\SLZMSG_USER.TXT set PostNOTE=%BASEAPP%\LOGS\JOBLOG_SIMONOTE.TXT set SimoNOTE=%BASEAPP%\LOGS\JOBLOG_SIMONOTE.txt call SIMONOTE "* SIMONOTE Job Log File is %SIMONOTE% " rem * set MQBASE=C:\Program Files\IBM\WebSphere MQ rem * rem * Set the location for the Apache-Tomcat Server... set CATALINA_HOME=C:\APACHETC\apache-tomcat-7.0.52 rem * rem * Set the Environment for the Java Environment... set JAVABASE=C:\Program Files (x86)\Java\jdk1.8.0_112 set JAVASDK="%JAVABASE%\bin" set JAVA_HOME=%JAVABASE% set JRE_HOME=%JAVABASE% rem * rem * Set the environment for the Micro Focus technology... set MIFOEDEV=C:\Program Files (x86)\Micro Focus\Enterprise Developer set MIFOVCBL=C:\Program Files (x86)\Micro Focus\Visual COBOL set MIFOESTU=C:\Program Files (x86)\Micro Focus\Studio Enterprise Edition 6.0 set MIFOEMFE="C:\Program Files (x86)\Micro Focus\Mainframe Express" rem * rem * Large file support, performance tuning and record locking of the File Handler set EXTFH=%BASESYS%\CONFIG\EXTFHBIG.CFG rem * rem * For IMS Support set ES_IMSLIB=%BASEAPP%\IMSLIB set ES_ACBLIB=%BASEAPP%\IMSLIB rem * rem * EZASOKETS Check EZASOKETS Enabled box or set ES_EZASOKET_SUPPORT=YES set EZACONFG=BASESYS1\CONFIG\EZACONFG.dat rem * rem * Resource Allocation and Performance for SORT and non-Relational Data rem set MFJSENGINE=SYNCSORT set SORTSCHEME=1 set SORTSPACE=750000000 set TMP=C:\SORTWORK rem * set ES_ALLOC_OVERRIDE=%BASESYS%\CONFIG\CATMAPA1.cfg rem * For CORE_ON_ERROR function, ABEND Dump rem * set COBCONFIG_=%BASESYS%\CONFIG\diagnose.cfg rem * rem * Consolidated Trace Facility (CTF) rem * set MFTRACE_CONFIG=%BASESYS%\CONFIG\ctf.cfg rem * set MFTRACE_LOGS=c:\ctflogs rem * rem * For Job Restart, ABEND Recovery set MF_UCC11=Y set ES_JES_RESTART=Y rem * rem * Set environment for MFBSI (Micro Focus Batch Scheduling Interface) set ES_EMP_EXIT_1=mfbsiemx set MFBSI_DIR=%BASESYS%\LOGS\%JESSERVERNAME% set MFBSIEOP_CMD=ENABLE set MFBSIEOP_CSV=ENABLE set MFBSIEOP_HTM=ENABLE set MFBSIEOP_XML=ENABLE rem * rem * Set Behavior and Trace Flags for GETJOBDD rem * Position=12345678/12345678 set JDDFLAGS=nnnWnnnn/YYnnnnnn rem * set MAINFRAME_FLOATING_POINT=true set COBIDY=%BASEAPP%\COBIDY set COBPATH=.;%BASEAPP%\LOADLIB;%BASESYS%\LOADLIB;%SimoLIBR% set LIBPATH=.;%BASEAPP%\LOADLIB;%BASESYS%\LOADLIB;%SimoLIBR% set TXDIR=%BASESYS%\LOADLIB;%MIFOBASE% set CobCpy=%BASEAPP%\CobCpy1;%BASEAPP%\CobCpy2;%BASEAPP%\CobCpy6;%SimoLIBR% rem * rem * If not already set then set the PATH for Micro Focus Directories if "%SIMOPATH%" == "Y" goto JUMPPATH if "%MIFOSYS1%" == "EDEV" goto JUMPEDEV if "%MIFOSYS1%" == "VCBL" goto JUMPVCBL if "%MIFOSYS1%" == "ESTU" goto JUMPESTU if "%MIFOSYS1%" == "EMFE" goto JUMPEMFE :JUMPEDEV set path=%BASESYS%\LOADLIB;%MIFOEDEV%\bin;%JAVASDK%;%BASEAPP%\JAVA;%PATH%; set CobCpy=%BASEAPP%\CobCpy1;%BASEAPP%\CobCpy2;%BASEAPP%\CobCpy6;%SimoLIBR%;%MIFOEDEV%\CPYLIB set MIFOBASE=%MIFOEDEV% goto JUMPPATH :JUMPVCBL set path=%MIFOVCBL%\bin;%JAVASDK%;%BASEAPP%\JAVA;%PATH%; set MIFOBASE=%MIFOVCBL% goto JUMPPATH :JUMPESTU set MIFOBASE=%MIFOESTU%\Base set MIFOBIN=%MIFOBASE%\bin set path=%BASESYS%\LOADLIB;%MIFOBASE%;%MIFOBIN%;%JAVASDK%;%BASEAPP%\JAVA;%PATH%; set CobCpy=%BASEAPP%\CobCpy1;%BASEAPP%\CobCpy2;%BASEAPP%\CobCpy6;%SimoLIBR%;%MIFOBASE%\SOURCE goto JUMPPATH :JUMPEMFE set MIFOBASE=%MIFOEMFE%\Base set MIFOBIN=%MIFOBASE%\bin set path=%BASESYS%\LOADLIB;%MIFOBASE%;%MIFOBIN%;%JAVASDK%;%BASEAPP%\JAVA;%PATH%; set CobCpy=%BASEAPP%\CobCpy1;%BASEAPP%\CobCpy2;%BASEAPP%\CobCpy6;%SimoLIBR%;%MIFOBASE%\SOURCE goto JUMPPATH rem * :JUMPPATH set SIMOPATH=Y rem * set USERCLASS=%BASELIB1%\LOADLIB set CLASSPATH=. set CLASSPATH=%CLASSPATH%;%JAVABASE% set CLASSPATH=%CLASSPATH%;%JAVABASE%\lib set CLASSPATH=%CLASSPATH%;\%USERCLASS%\simpacks set CLASSPATH=%CLASSPATH%;C:\APACHETC\apache-tomcat-7.0.52\webapps\simotcat\WEB-INF\classes set CLASSPATH=%CLASSPATH%;C:\APACHETC\apache-tomcat-7.0.52\webapps\simotcat\WEB-INF\classes\simpacks if "%MIFOSYS1%" == "ESTU" set CLASSPATH=%CLASSPATH%;%MIFOBIN% if "%MIFOSYS1%" == "EDEV" set CLASSPATH=%CLASSPATH%;%MIFOEDEV% if "%MIFOSYS1%" == "VCBL" set CLASSPATH=%CLASSPATH%;%MIFOVCBL% if "%MIFOSYS1%" == "VCBL" set CLASSPATH=%CLASSPATH%;%MIFOVCBL%\mfcobol.jar rem * set JobStatus=0000 call SimoNOTE "* Settings CmdName ENV1BASE.cmd, Version 18.09.15, %MIFOSYS1%" call SimoNOTE "* BaseAPP ..... %BASEAPP%" rem * call SimoNOTE "* MFBSIDIR .... %MFBSI_DIR% " call SimoNOTE "* MIFOSYS1 is %MIFOSYS1% " call SimoNOTE "* MIFOBASE is %MIFOBASE% " call SimoNOTE "* SIMOLIBR is %SIMOLIBR% "
The following Windows command file (SIMONOTE.cmd) is a listing of the contents of the command that displays information to the screen and writes to a log file with a date and time stamp.
@echo OFF rem * ******************************************************************* rem * SIMONOTE.cmd - a Windows Command File * rem * This program is provided by SimoTime Technologies * rem * (C) Copyright 1987-2019 All Rights Reserved * rem * Web Site URL: http://www.simotime.com * rem * e-mail: firstname.lastname@example.org * rem * ******************************************************************* rem * rem * Text - Display message on screen and write to a log file. rem * Author - SimoTime Technologies rem * rem * This script may be called from other scripts and expects a single rem * parameter enclosed in double quotes. The double quotes will be rem * removed. Before writing to the log file a date and time stamp rem * will be inserted in front of the message text. rem * rem * Note: The tilde (~) removes leading/trailing double-quotes. rem * if "%SimoNOTE%" == "" set SimoNOTE=c:\SimoLIBR\LOGS\SimoTime.LOG echo %date% %time% %~1>> %SimoNOTE% echo %~1
The data comparison or data validation process should be a repeatable process with an audit or validation trail. The process should be executable as an automated, unattended process. Requiring operator input during the data file validation process introduces an exposure point for error. The callable SIMONOTE command provides a consistent method for creating an audit or validation trail by writing information to a log file with a date and time stamp.
This utility program (UTCOMPRT) is intended to assist in the data file validation efforts. This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.
In the world of programming there are many ways to solve a problem. This document and the links to other documents are intended to provide a greater awareness of the Data Management and Application Processing alternatives.
The documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.
SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please send an e-mail to: email@example.com or call 415 883-6565. We appreciate hearing from you.
Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.
SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided "AS IS" without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.
This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.
Note: A SimoTime License is required for the items to be made available on a local system or server.
The following links may be to the current server or to the Internet.
Note: The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.
Explore the Principles of Data File Validation. This link includes guidelines for defining requirements and determining the scope of effort for a data validation effort. This effort includes a data file compare process, a technique for accumulating summary totals with a record count and a technique for reading a VSAM, KSDS and producing a hex-dump output based on a list of user-defined keys.
Explore the Principles of Data File Conversion. This link includes guidelines for defining requirements and determining the scope of effort for a data conversion effort.
Explore How to Generate a Data File Convert Program using simple specification statements in a Process Control File (PCF). This link to the User Guide includes the information necessary to create a Process Control File and generate the COBOL programs that will do the actual data file conversion. The User Guide contains a list of the PCF statements that are used for the data file convert process.
Explore the COBOL Connection for more examples of COBOL programming techniques and sample code.
Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.
Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and/or QSAM files.
The following links will require an internet connection.
A good place to start is The SimoTime Home Page for access to white papers, program examples and product information. This link requires an Internet Connection
Explore The Micro Focus Web Site for more information about products and services available from Micro Focus. This link requires an Internet Connection.
Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.
This document was created and is copyrighted and maintained by SimoTime Technologies.
If you have any questions, suggestions, comments or feedback please call at 1 415 827-7045 or send an e-mail to: firstname.lastname@example.org
We appreciate hearing from you.
SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.
Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.
Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform.
The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process.
SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.
Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call at 1 415 827-7045 or send an e-mail to: email@example.comAlso, check the web site at http://www.simotime.com
|Data Validation, How to Generate and Execute a Program|
|Copyright © 1987-2019
SimoTime Technologies and Services
All Rights Reserved
|When technology complements business|