Data Integration (Mapping) Guide

Data Integration (Mapping) Guide

View a printable version (pdf)

1.0 Document Purpose
The purpose of this guide is to document the process for preparing test data submission to N-DEx. This process includes the data mapping, integration development, and initial test data validation and verification.

2.0 Process
The mapping of agency data is the responsibility of the participating agency. An N-DEx liaison specialist will assist the agency during the integration process. The process begins with determining the type(s) of data that will be submitted to N-DEx and who will complete the work involved (see N-DEx Data Contribution Checklist). Currently, the N-DEx Program Office offers assistance in the mapping and integration process for data submission to N-DEx.

2.1 Data Types
Several N-DEx Information Exchange Packages Documentation (IEPD) have been posted to the www.it.ojp.gov website and reside in the Justice Standards Clearinghouse (this tab can be located at the left side of the www.it.ojp.gov webpage). The agency can contact the N-DEx liaison specialist for assistance through the process. The following is a high-level overview of the N-DEx IEPDs currently available:

N-DEx Incarceration/Booking/Probation/Parole IEPD version 2.1 is the submission standard for N-DEx Increment 3 for agencies submitting booking, incarceration, and probation and parole (correctional facilities, prisons, jails, presentence and pretrial, supervised release, and holding reports) data. This version conforms to National Information Exchange Model (NIEM) version 2.0 and the LEXS 3.1.4.

N-DEx Incident-Arrest IEPD version 2.1.1 is the submission standard for N-DEx Increment 3 for agencies submitting incident, arrest, service call, and missing person report information. This version conforms to the NIEM version 2.0 and the LEXS 3.1.4. This IEPD also enables N-DEx to provide the National Incident-Based Reporting System (NIBRS) extract to the Uniform Crime Reporting (UCR) program on behalf of NIBRS-certified agencies requesting this capability.

If a data item owner wants a NIBRS extract to be created from their submission, the incident report or the arrest report must include the NIBRS ORI element. This element indicates the identity of the NIBRS certified agency that requested the N-DEx system to provide an extract of the data item to the NIBRS. This extract is to satisfy reporting requirements. A data item owner must include the NIBRS Report Category Code element with corresponding value indicating what type of submission should be extracted to NIBRS. Finally, a data item owner must indicate which NIBRS entities to incorporate in the extract, including the Distribution Text element where the value is ‘NIBRS’ as part of the desired entities.

NIBRS participating agencies that do not wish to utilize the N-DEx extract services for NIBRS reporting or do not believe the contents of a data item represents those which are required to satisfy the NIBRS reporting requirements, should not provide NIBRS ORI, NIBRS Report Category Code, or Distribution Text elements. Additional details for creating a NIBRS extract will be provided upon completion of NIBRS extract documentation.

2.2 Data Integration Development
The Data Integration phase is comprised of two major tasks. The first task is mapping the agency data elements to the N-DEx IEPD. The second task is the development of the extraction application software. The mapping task determines where each element from the agency data migrates into the N-DEx schema. The extraction application development task, utilizing the mapping results, generates the means to extract the agency data, wrap it with N-DEx XML tags, and write it to an approved readable format of XML to be digested into N-DEx. The extraction application may be developed using XSLT, JAVA, C++, or .NET.

It is the agency’s responsibility to assess its capabilities in performing these tasks. The agency should contact the N-DEx liaison specialist to determine how the N-DEx Program Office can support the agency with data integration. If the agency chooses to perform the integration, the N-DEx Program Office can offer consultants to answer questions and guide the agency during the mapping/integration development process. Currently, the N-DEx Data Integration Team can perform the integration development task free of charge upon agency request.

2.3 Agency Test Data Validation 
After the agency data has been extracted into the N-DEx IEPD format, the next step is to conduct XML structural verification checks for NIEM, LEXS, and N-DEx IEPD conformance. This is accomplished by using the Conformance Testing Assistant (ConTesA). ConTesA is designed to validate an XML instance document against all current versions of the LEXS schema and the N-DEx IEPD. In addition to XML schema validation, ConTesA performs a number of N-DEx business rule validation checks. The tool also provides a style sheet for viewing the data. This may help to better visualize the LEXS and N-DEx schemas and the way associations work together. ConTesA is available to participating agencies in order to expedite the data integration process. It is highly recommended that data be tested and pass the ConTesA validation prior to submitting to the N-DEx Program Office for content verification. The ConTesA conformance tool is located at https://contesa.ittl.gtri.org/contesa. If a user does not have an account, one can be created. A sample of the account creation screen is provided below. A command line version of ConTesA is also available for local execution.

1. Log in to ConTesA. https://contesa.ittl.gtri.org/contesa/

a. Click File Validation
b. Enter username (e-mail address) and password

2. Browse your local machine to find the XML instance document you wish to validate

3. Select Upload

4. After validation has completed, select View

5. Click on the arrow next to the view you wish to render, e.g., LEXS 3.1 Detailed View

6. Click the View tab next to the package

2.4 Test Data File Specification
File names adhere to the following naming convention:

AAAAAAAAA_YYYYMMDD_HHMM_NNNN.xml, where AAAAAAAAA is the data submitter agency ORI, YYYYMMDD is the four-digit year, two-digit month, and two-digit day, HHMM is the two-digit hour (24-hour clock) and two-digit minute, and NNNN is a four-digit sequence number. The date and time in the file name should be the creation date and time of the XML file. The sequence number is used to guarantee unique file names. The agency system is responsible for making sure the files are named uniquely. A sample file name for second file created by a data submitter agency with ORI ‘ORIPD0001’ at 1:02 p.m. on March 9, 2007, is ‘ORIPD0001_20070309_1302_0002.xml’.

File Size:

Note: .xml file = XML document instance
XML document sizing recommendations

• 2,000 package maximum in a single XML document instance  i.e. WVNDX0000_20080130_1201_0001.xml)
• 70 MB maximum file size for a single XML document instance
• 12 total attachments inside a single XML document instance

Zip file sizes for batch ingest:

• see prerequisite XML document instance constraints above
• Maximum of 20,000 .xml files in a single .zip file. Each XML document instance (file) would have a single message and package
• 2 GB maximum file size for a single .zip file

2.5 CJIS Validation & Verification
Once the agency and/or the N-DEx Data Integration Team has generated error-free test data through ConTesA, the Validation & Verification submission task may begin.

The following are methods to submit test data:

• Compact disk (CD), digital video disk (DVD), hard drive, or other media
• Electronic submission, including e-mail and SFTP over the Internet.

The steps below describe the process to be followed for submission test data:

1. Submit no more than 100 incident/arrest submissions (ConTesA conformance tested and passed) to the N-DEx liaison specialist.

2. The N-DEx Program Office will assign a lead data content reviewer and a secondary reviewer for the data submission.

3. The data content reviewer will create and provide a report that includes the following:

• The data verification will include the N-DEx business rules found in the IEPD master documentation, which includes a review of the file name structure.
• The data verification will also include those elements found in the IEPD CMT spreadsheet regarding mandatory elements.
• Errors encountered will be noted in a report.
• The secondary reviewer will confirm the results of the report.

4. The N-DEx liaison specialist will provide the report to the agency.

5. The N-DEx liaison specialist and/or agency may request a meeting (face-to-face or teleconference) with agency representatives and N-DEx Program Office data content reviewer(s).

6. This process will continue until no errors are found during the test data verification process.

7. The agency will submit 10,000 incident/arrest records. Repeat steps 2-5. The process will continue until no errors are found during the process. At that time, the N-DEx liaison specialist will notify the agency of the next steps for legacy data submission.

N-DEx is gathering no more than five years of historical data from partnering agencies, unless the agency wishes to submit more historical data. During the test data process, the agency will work with the N-DEx liaison specialist on the procedure for submitted legacy data to N-DEx. Different options exist based on the amount of legacy data being submitted. The initial dataset can be loaded onto an external media device (CD, DVD, external hard drive, or thumb drive) for manual submission to N-DEx. The external media device should be sent via Federal Express to your N-DEx regional coordinator. The legacy data can also be submitted through the established electronic submission method (see Data Connectivity and Submission Guide).