Image-Based Matching Technology Offers Identification and Intelligence Prospects

December 28, 2012
Originally published in the December 2012 edition of the CJIS Link, Volume 14, Number 3

The FBI’s Integrated Automated Fingerprint Identification System (IAFIS)—the largest criminal biometric database in the world—is steadily making way for the Next Generation Identification (NGI), which will extend automated biometric identification capabilities beyond fingerprints and palmprints. Although law enforcement has used photographs of scars, marks, and tattoos (SMTs) for several years to help identify or eliminate suspects, the NGI will automate that process. In 2014, investigators will be able to query the NGI with descriptive data about tattoos to find images of potential matches of SMTs associated with individuals’ records. Equally notable, however, is that right now, the FBI’s Biometric Center of Excellence (BCOE) and its partners are advancing image-matching technology that will enable investigators to use a probe, or query image—with or without descriptive data such as key words or characters—and find similar images.

NGI will use image-based matching technology to help automate the process of using scars, marks, and tattoos to help identify or eliminate suspects. (From CJIS Link article)

The BCOE, headquartered at the Criminal Justice Information Services (CJIS) Division in Clarksburg, West Virginia, is the FBI’s focal point for biometrics and identity management. It is an initiative of the Science and Technology Branch, which includes the Laboratory Division, the Operational Technology Division, and the CJIS Division. Together, scientists, technicians, and biometric experts from these divisions are supporting the BCOE’s mission to “foster collaboration, improve information sharing, and advance the adoption of optimal biometric and identity management solutions within the FBI and across the law enforcement and national security communities.” In short, the BCOE is exploring new and existing biometric modalities, or types, and maximizing their potentials.

With image-based matching technology, the BCOE’s goal is to avoid the subjectivity inherent in text-based searches by using computer algorithms to search the features of other stored images and locate a potential match. An example of the subjectivity involved in labeling images, even when using the American National Standards Institute/National Institute of Standards and Technology (ANSI/NIST) standards for classifying an image’s class and subclass, follows.

When looking at the image shown above, some users might see the letter “D” in this image and label its ANSI/NIST class/sub-class as “Other/Wording.” Other users might consider the image a logo for the Detroit Tigers and label it as “Symbol/Miscellaneous Symbol,” based on what they think it represents. Still others, due to the image’s orientation, might interpret the image as a face and label it as “Human/Abstract Face.” Computerized labeling, or indexing, increases the likelihood of finding similar images.  

The BCOE’s academic partnership with Michigan State University facilitated the development of the TattooID prototype for the automated retrieval, indexing, and matching of SMTs. To use the TattooID prototype, an analyst enters a probe image, key words, or characters into the program that employs an image-matching algorithm. The algorithm runs against images stored in the database and provides any similar images, ranked with match scores. Such a search can be run for images of the same general design, or it can be limited to include only those images of tattoos on designated body areas, e.g., a person’s shoulder. Although search results do not provide positive identifications at this point, they could help narrow a suspect list.

The BCOE recently finished building a tattoo dataset to test and evaluate the image‑matching algorithms in the TattooID prototype. The testing will cover four basic scenarios: (1) tattoos from the same individual captured at different booking events; (2) similar tattoos on different individuals; (3) region of interest (matching small portions of an image to the larger image); and (4) matching across various media (drawings, stencils, photographs, etc.). To build the collection of images for testing, examiners analyzed tattoo images from the FBI Photo File and labeled the images according to ANSI/NIST classification standards. Furthermore, examiners ranked the similarities of images within the same class/sub-class, e.g., animal/bird. Once testing is completed, the BCOE will evaluate the prototype’s performance and assess strengths and weaknesses of the current capability.

Does Your Agency Have a Success Story Involving Tattoos, Symbols, or Graffiti?

If so, the BCOE would like to hear about it. Tell us your story by including a point of contact, agency name, name of database, and a brief description of how the tattoo, symbol, graffiti, or image of any of these helped to identify or eliminate a candidate, or how it provided information that aided an investigation. Send your e-mail to

While the value of image-to-image matching technology is obvious from an identification perspective, the benefits of knowing the symbolism and background behind tattoos and graffiti can be equally valuable. From an intelligence standpoint, certain symbols or graffiti may be used to help establish whether an individual is associated with a particular gang, terrorist organization, or extremist group. This may help determine the extent to which the individual or gang poses a threat to law enforcement or the community, and possibly to recognize and link crimes across the country. This helps secure the safety of officers during investigations and arrests, and it can even possibly help with crime intervention strategies.

Applying its research with image-based matching technology with a bend toward relating the meaning of symbols, the BCOE recently collaborated with the Cryptanalysis and Racketeering Records Unit (CRRU) of the FBI’s Laboratory Division on the development of the prototype image-comparison system, Tattoo and Graffiti Image-Matching and Graphic Evaluation (TAG-IMAGE). This investigative tool was designed to help the CRRU match images within its database and to determine the significance of tattoos, graffiti, or other cryptic symbols for FBI investigative programs dealing with foreign or domestic terrorism, violent crime, or gangs. Rather than using metadata or text searches, TAG-IMAGE uses advanced image-to-image technology to match symbols based on actual appearances. It may also provide raw information on a symbol’s actual use, rather than leaving speculation as to its meaning or origin.

With TAG-IMAGE, a contributor e-mails an image to the CRRU, where an analyst enters the probe image into the system, which compares it with images stored within the CRRU database. Once a search is completed, a CRRU analyst e-mails a response to the contributor that includes pictures of similar images as well as any associated details and contact information. The submitted image then becomes available for future comparisons by other agencies. The CRRU has begun a pilot program using the TAG-IMAGE prototype. After the pilot phase ends, TAG-IMAGE will become available to local, state, tribal, and federal law enforcement and correctional agencies.

The BCOE also plans to conduct a small operational pilot program with the National Gang Intelligence Center to assess the feasibility of image-based matching and to gain user feedback. Ultimately, the BCOE hopes to use this biometric research to enhance FBI operations.