Presentations from the International Symposium on Setting Quality Standards for the Forensic Community (Part 2; Forensic Science Communications, July 1999)
|
July 1999 - Volume 1 - Number 2 |
Presentations at the
International Symposium on Setting Quality Standards for the Forensic Community
San Antonio, Texas
May 3-7, 1999
Part 2
The following abstracts of the presentations are ordered alphabetically by authors’ last names.
Scientific Working Group on DNA Analysis Methods (SWGDAM)
B. Budowle
Federal Bureau of Investigation
Quantico, Virginia
DNA typing has been a boom to forensic science. It has enabled the resolution of cases where not previously possible, has been used to exonerate individuals falsely associated with evidence, and has provided numerous investigative leads. The full use of DNA typing technology (i.e., development of high quality, sensitive, and reliable typing methods and the establishment of a national DNA data bank) only could have come to fruition by coordinating the resources of the forensic laboratories in North America. The FBI Laboratory, in concert with several North American forensic science laboratories, recognized the need for setting peer-consensus standards of performance and creating an infrastructure of communication and compatible data, with the ultimate goal being the development of a national DNA data bank called CODIS (Combined DNA Index System). To achieve this end, the Technical Working Group on DNA Analysis Methods (TWGDAM) was created in 1988. TWGDAM was the first working group in the North American forensic science community and is hosted and supported by the FBI Laboratory. Recently, the working group has been renamed the Scientific Working Group on DNA Analysis Methods (SWGDAM) to better reflect its mission. More than 30 laboratories send members to participate in SWGDAM. Also, a community-wide meeting is held annually to foster communication and promote greater input.
The peer-consensus approach has established many guidelines and scientific studies regarding forensic DNA typing. Areas that have been addressed by SWGDAM include
- RFLP typing compatibility,
- chemiluminescent detection of RFLP profiles,
- quality control/quality assurance guidelines,
- ASCLD/LAB accreditation issues,
- audit guidelines,
- statistical interpretation issues,
- CODIS criteria,
- collection of population data,
- criteria for PCR-based typing, and
- criteria for mitochondrial DNA typing.
Many of the products of SWGDAM efforts are recognized worldwide. In fact, the DNA Identification Act of 1994 contained within the Omnibus Crime Control Act (The Violent Crime Control and Law Enforcement Act of 1994, Public Law 103-322, 108 statute 1796) recognized TWGDAM Quality Assurance Guidelines as de facto standards for forensic DNA typing laboratories. However, the Act also authorized the FBI Director to establish quality assurance standards for laboratories performing forensic DNA testing through recommendations of a DNA Advisory Board (DAB). Thus, SWGDAM took on a unique role for a working group and volunteered to support the DNA Advisory Board in its effort. In essence, SWGDAM became a working arm of the DAB. Once the mission of the DAB ends (March 2000), SWGDAM will be the entity that makes recommendations to the FBI Director for changes to the quality assurance standards for forensic DNA typing.
The mission of SWGDAM will continue to be to
- bring together organizations actively pursuing various forensic DNA analysis methods for the purpose of exchanging and disseminating information;
- discuss, share, and compare forensic DNA analysis methods, protocols, and research;
- establish forensic DNA analysis quality assurance guidelines and quality controls;
- cooperate with other national and international organizations in developing standards for forensic DNA analyses;
- serve as a mechanism for the review and revision of guidelines and standards for forensic DNA analyses; and
- disseminate to the forensic DNA community SWGDAM’s guidelines, forensic research results, and other work products of benefit to the forensic DNA community.
Current work of SWGDAM focuses on training guidelines, STR interpretation guidelines, and validation criteria.
Statistics
G. Carmody
Carleton University
Ottawa, Ontario, Canada
Statistics is often thought to be an unnecessary frill if a scientific experiment has been done correctly. This idea is a misconception. Analysis of experimental error is at the very core of the scientific method. In the following, I give examples of how, starting with asking what the problem or question is, statistics can be used to formulate an answer.
Once a question is framed, simple decisions about how to organize and display the data in a fashion that exposes its relevant structure can be made. In this presentation, I have explained the difference in the terms accuracy, validity, precision, reliability, bias, sensitivity, and specificity. I also have presented the standard Neyman-Pearson scheme of hypothesis testing and show how it applies to diagnostic testing, as well as accepting or rejecting an hypothesis.
I discuss two examples from the quality control literature where the absence of data can, paradoxically, be useful in uncovering an important source of error for quality control. I also describe the phenomenon called Benford’s Law, which allows us to use the surprisingly nonrandom, yet predictable, pattern of numbers to detect fraudulent tax returns.
Scientific Working Group for the Analysis of Drug Samples (SWGDRUG)
A. B. Clark
Drug Enforcement Administration
Dallas, Texas
The Scientific Working Group for the Analysis of Drug Samples (SWGDRUG) has met three times since 1997. The overall mission of SWGDRUG is to make recommendations for internationally accepted minimum standards for the forensic examination of seized drugs. The SWGDRUG Conference is an international endeavor that features representative managers and practitioners from forensic science organizations in the United Nations, United States, Canada, United Kingdom, Japan, Germany, and Australia. Organizationally, the SWGDRUG core committee is composed of four subcommittees having distinct missions, as follows:
- Communications: The mission of this subcommittee is to convey to the international forensic science committee the establishment of SWGDRUGCits goals and objectivesCand to promote the adoption of and adherence to the recommended standards.
- Methods and Reports: This subcommittee is to assess and evaluate available analytical methodologies and reporting methods.
- Quality Assurance: The mission of this subcommittee is to develop quality assurance guidelines for the operation of a forensic drug analysis laboratory.
- Education and Training: The mission of this subcommittee is to recommend minimum education and training requirements, competency standards, assessment protocols, and guidelines for continuing professional development for forensic drug examiners.
Representatives from the four subcommittees made reports and proposals that reflect their commitment to the process:
- The Communications Subcommittee assigned core committee members to present a report of the conference at regional, national, and international forensic science meetings.
- The Methods and Reports Subcommittee discussed with the core committee the survey results and their plans to formulate proposals based on the facts contained therein.
- The Quality Assurance Subcommittee made a number of proposals, which were published in the March 1999 issue of Microgram.
- The Education and Training Subcommittee made the first set of proposals, which were disseminated in the December 1998 issue of Microgram.
As more information becomes available, it will be disseminated on the SWGDRUG World Wide Web site (http://www.erols.com/scitechz) and in Microgram. Everyone is invited to participate in the process by sending their comments to SWGDRUG core committee members. The list of members, with contact information, is available on the SWGDRUG web site.
Qualifications of a Scientific Expert Witness
S. J. Cribari
University of Denver College of Law
Denver, Colorado
In discussing the law governing the qualifications of an expert scientific witness, I offer insights on the strategy and tactics of preparing yourself and your material for direct and cross-examination in court, as well as on scientific honesty and professional ethics in an adversarial system of justice.
Statistical Process Control or Graphical Quality Evaluation, Demonstration, and (Maybe) Control
D. L. Duewer
National Institute of Standards and Technology
Gaithersburg, Maryland
How do you convince yourself that results provided by some measurement system are trustworthy? How can you tell when the system is starting to act up before, you hope, it savages you? If the system is acting up, how do you identify what has changed? How can you convince yourself, your management, or both that you know what you’re talking about?
Although the term statistical quality control may suggest esoteric technology, the answers to the previously listed challenges can be accomplished graphically. Control chart techniques are sufficiently common sense to be interpreted by nonspecialists yet are powerful and rigorous. The basic tool is the simple scatterplot, plotting measurement results versus time. With a little attention paid to the scale of both axes, these simple charts simultaneously document how stable the measurement system has been, whether the system is changing, and where the system may be going. Superimposing various measurement quality indicators (whether set by regulation, policy, or statistical analysis) documents how good the measurement system is relative to how good it needs to be.
This presentation has been drawn from National Institute of Standards and Technology’s experience with forensic RFLP measurement systems. I have focused on what is appropriate for getting where you need to go:
- What is an appropriately quantitative measurement?
- What is an appropriate suite of blank, check, control, or standard samples?
- What are appropriate graphical tools?
The hard requirements—taking the data and getting it stored in a useful form—are probably already part of your routine protocol. If they’re not, they should be! The only added requirements for successful evaluation, demonstration, and control of measurement quality are management support, personal dedication, and a trained intuition. The latter is facilitated by always remembering to DTDP-ATLAI (Draw the Darned Plot, And Then Look At It!) and KISS (Keep It Short and Simple).
Many different computer programs, short courses, workshops, books, CD-ROM self-instruction kits, applications, and diatribes can be found on the World Wide Web using the search phase statistical process control. I recommend the advanced search at
http://www.altavista.com
The NASA Headquarter’s Library maintains a very useful list of statistical process control references at
http://www.hq.nasa.gov/office/hqlibrary/ppm/ppm31.htm.
DNA Advisory Board Standards
A. J. Eisenberg
University of North Texas Health Science Center
Fort Worth, Texas
The DNA Advisory Board (the Board or DAB) was established by the Director of the Federal Bureau of Investigation under the DNA Identification Act of 1994 (the Act) as a separate and distinct advisory board administered by the FBI. The Act provided funding for forensic laboratories to improve the quality and availability of DNA analyses and for the FBI to establish the national Combined DNA Index System (CODIS). The U.S. Congress authorized the Director of the FBI to develop standards to assure quality—including standards for testing the proficiency of forensic laboratories—and to provide guidance to forensic analysts performing DNA analyses. The legislation specified that access to federal funding provided under the DNA Identification Act is contingent upon a laboratory’s strict adherence to these standards. In accordance with the DNA Identification Act, the quality assurance guidelines adopted by the Technical Working Group on DNA Analysis Methods (TWGDAM) were deemed the FBI Director’s interim standards.
Members of the Board were appointed by the Director of the FBI from nominations proposed by the National Academy of Sciences and professional societies of crime laboratory officials. The Act specified that members of the Board include scientists from state, local, and private forensic laboratories; molecular geneticists and population geneticists not affiliated with a forensic laboratory; a representative from the National Institute of Standards and Technology (NIST); the chair of the Technical Working Group on DNA Analysis Methods (TWGDAM); and a judge. Initially, the Board consisted of 16 members, including the chair (Nobel prize winner Dr. Joshua Lederberg, a voting member); two voting members in a representational capacity (from NIST and TWGDAM); ten general voting members; and three nonvoting members. The three nonvoting members appointed by the FBI Director included an executive secretary, a quality control/quality assurance specialist, and a medical/legal ethicist. In addition, pursuant to the Federal Advisory Committee Act, the Director of the FBI appointed a designated federal employee to perform the functions set forth by law.
The Director of the FBI, in accordance with the DNA Identification Act of 1994, specified that the Board’s scope of activity was to
- develop, recommend, and if appropriate, periodically revise standards for quality assurance, including standards for testing the proficiency of forensic laboratories and forensic analysts, in conducting analysis of DNA;
- recommend standards that specify criteria for quality assurance and proficiency tests to be applied to the various types of DNA analyses used by forensic laboratories, including statistical and population genetics issues affecting the evaluation of the frequency or occurrence of DNA profiles calculated from pertinent population databases;
- recommend standards for acceptance of DNA profiles in the FBI’s Combined DNA Index System (CODIS) that take into account relevant privacy, law enforcement, and technical issues; and
- make recommendations for a system for grading proficiency-testing performance to determine whether a laboratory is performing acceptably.
Throughout its deliberation concerning these quality standards, the DNA Advisory Board recognized the need for a mechanism to ensure compliance with the standards. An underlying premise for these discussions was that accreditation would be required to demonstrate compliance with the standards and, therefore, assure quality control and a quality program. The Board recommended that forensic laboratories performing DNA analysis seek such accreditation with all deliberate speed. The Board strongly encouraged the accrediting bodies to begin positioning themselves to accommodate the increasing demand for accreditation. In accordance with the DNA Identification Act, the Board forwarded its recommendations for standards to the Director of the FBI. After careful review, the Director’s Quality Assurance Standards for Forensic DNA Testing Laboratories took effect on October 1, 1998. These national standards now have superseded the TWGDAM Guidelines for purposes of certifications required for the receipt of federal funding as well as participation in the National DNA Index System.
Inherent within the standards is the intent to upgrade the educational background and experience of forensic scientists. The Board spent a considerable amount of time discussing the qualifications for the technical manager/leader (or equivalent position or title as designated by the laboratory system). This is the individual who is accountable for the technical operations of the laboratory. The Board expressed strong belief that this position required an individual with a minimum of a master’s degree in biology, chemistry, or a forensic science-related area. In addition, the technical manager/leader must have successfully completed a minimum of 12 semester credit hours (or equivalent) in course work covering biochemistry, genetics, and molecular biology or other subjects that provide a basic understanding of the foundations of forensic DNA analysis as well as statistics, population genetics, or both as applied to forensic DNA analysis. A technical manager/leader of a laboratory must have a minimum of three years of forensic DNA laboratory experience. The Board recognized that there are individuals who have been serving in the capacity of technical manager or leader who do not meet the educational requirements specified by the standards. The standards include a provision for the waiver of the degree requirements for the technical manager/leader. The waiver is only for the educational requirement and does not apply to the minimum of three years of forensic DNA laboratory experience. ASCLD, in conjunction with ASCLD/LAB, has established a credentials review committee to grant waivers to current technical leaders/managers who do not fully meet the DAB educational standards for the DNA technical leader/manager position, but who qualify as technical leaders on the basis of their knowledge and experience. The waiver is available for a period of two years from the effective date of the standards (October 1, 1998). In addition, the wavier is permanent and portable.
Forensic DNA identification analysis currently involves forensic casework and convicted offender analyses for inclusion in CODIS. Separate standards have been drafted for laboratories performing the analysis of convicted offender samples. The separation was an acknowledgment of the differences in the nature or type of sample, the quantity and potential for reanalysis, and specialization that may exist in a laboratory. Standards for convicted offender laboratories, in some instances, are less stringent than for those laboratories performing forensic casework analyses, but in no case should the two documents be interpreted as conflicting. The Board forwarded its recommendations to the Director of the FBI. Following review, the Director issued his Quality Assurance Standards For Convicted Offender DNA Databasing Laboratories. These standards became effective April 1, 1999.
Pursuant to Section 210303 of the DNA Identification Act of 1994, the DNA Advisory Board was tasked with recommending standards for testing the proficiency of forensic laboratories that conduct analyses of DNA. The Act required that the National Institute of Justice (NIJ) review the availability and feasibility of blind external proficiency testing for forensic DNA analysis. A grant was awarded by NIJ, and after an extensive evaluation and review, the project’s participants concluded that by defining feasible as possible and practicable in terms of costs and logistics, a national blind proficiency-testing program employing blind proficiency tests administered by law enforcement agencies, conduit laboratories, blind analyst models, or a combination of these is not feasible at this time. Three recommendations were provided in their report:
“1. The accreditation system and associated quality assurance guidelines of the DNA Advisory Board need to be given the opportunity to take hold,
2. It is recommended that the DNA Advisory Board generate guidelines for more stringent external case audits for use by ASCLD/LAB, or another relevant accrediting body, as part of the accreditation process. The external case audits should be conducted regularly and serve as a measure of how well accreditation and its associated requirements are working in a quality assurance context.
3. In the extreme, blind proficiency testing is possible, but fraught with problems (including costs), and it is recommended that a blind proficiency-testing program, be deferred for now until it is more clear how well implementation of the first two recommendations are serving the same purposes as blind proficiency testing.”
After a review of the NIJ project, the Board issued the following resolution on blind external proficiency testing on April 23, 1999:
“The DNA Advisory Board accepts and adopts the finding of the NIJ Project that a blind DNA proficiency testing program is not feasible at this time. Although the DNA Advisory Board is supportive of the recommendations proposed in the Final Report, the Board suggests that ASCLD/LAB is in a better position to address the second recommendation.”
In compliance with the DNA Identification Act, the Board provided the Director of the FBI with a recommendation for a system for grading proficiency-testing performance to determine whether a laboratory is performing acceptably: “All proficiency tests are graded as satisfactory or unsatisfactory. A satisfactory grade is attained when there are no analytical errors for the DNA profile typing data. Administrative errors shall be documented and corrective actions taken to minimize the error in the future.”
According to the requirements of the DNA Identification Act, the Board will terminate on March 9, 2000, unless the Board’s term is extended by the Director of the FBI. The Board is currently evaluating and will make recommendations concerning the certification of forensic DNA personnel and statistical and population genetic issues related to calculations involving uniqueness, sample mixtures, parentage, and CODIS database searches.
Forensic Science Laboratory Accreditation Criteria and ISO 17025
J. Evans
National Association of Testing Authorities
Sidney, New South Wales, Australia
In Australia, all laboratories accredited by the National Association of Testing Authorities (NATA) including forensic science laboratories must comply with the international laboratory accreditation standard ISO Guide 25, soon to be renamed ISO 17025. ISO 17025 is a generic standard for laboratories and is applicable to all facilities performing tests and calibrations regardless of size or type of service. Although it can stand alone as a standard describing the elements of a laboratory’s operation, it makes specific provision for what it calls amplification or interpretation to ensure that the requirements of the standard are applied in a consistent manner.
The first ISO Guide 25 published in 1978 and the second ISO Guide 25 published in 1982 covered only technical requirements, that is the technical aspects of a laboratory’s operation. The third and current ISO Guide 25 was published in 1990. At that time, there was a recognition of the need for laboratories to have in place a quality management system. This was in line with the trend whereby organizations of all types were establishing quality management systems that complied with the ISO standard for quality management systems, ISO 9000. The 1990 ISO Guide 25 combines quality management system elements with technical elements. As a result, the 1990 ISO Guide 25 includes a considerably greater number of elements than the two previous versions.
Since 1990, the introduction of quality management systems by laboratories has increased greatly. In addition, the number of organizations that have sought certification to the quality management system standards ISO 9000 has also increased dramatically. With this focus internationally on the implementation of quality management systems, either at the broad organizational level or at the laboratory level, ISO felt that its Guide 25 needed revision to strengthen the quality management system elements and to harmonize or align them more closely with ISO 9000. This without diminishing in any way the technical requirements of the Guide. The publication of ISO 17025 is expected toward the end of 1999.
ISO 17025 will be divided into two sections: one covering quality management system elements and one covering technical requirements. The general format of the latest draft of ISO 17025 is as follows:
- Management Requirements
- Organization and management
- Quality system
- Document control
- General
- Document approval and issue
- Document changes
- Request, tender, and contract review
- Subcontracting of tests and calibrations
- Purchasing services and supplies
- Service to client
- Complaints
- Control of nonconforming work
- Corrective action
- General
- Cause analysis
- Monitoring of corrective action
- Special audits
- Preventive action
- Records
- General
- Technical records
- Internal audits
- Management reviews
- Technical Requirements
- General
- Personnel
- Accommodation and environmental conditions
- Testing and calibration methods including sampling
- Equipment
- General
- Selection of methods
- Laboratory-developed methods
- Nonstandardized methods
- Validation of methods
- Best measurement capability
- Estimation of uncertainty of measurement
- Control of data
- Equipment
- Measurement traceability
- General
- Specific requirements
- Calibration
- Testing
- Reference standards and materials
- Intermediate checks
- Transport and storage
- Sampling
- Handling and transporting items
- Assuring the quality of results
- Reporting results
- General
- Test reports and calibration certificates
- Opinions and interpretations
- Results obtained from subcontractors
- Electronic transmission of results
- Format of reports and certificates
- Amendments to reports and certificates
The accreditation criteria for NATA’s forensic science laboratory accreditation program are based on ISO 17025. For the additional interpretative criteria, NATA has drawn on a number of sources, including the forensic science laboratory accreditation programs of ASCLD/LAB, the United Kingdom Accreditation Service, and the Standards Council of Canada, in addition to developing our own.
In an effort to make the accreditation criteria as user-friendly as possible to as many laboratories as possible, the document has been structured in a way that would be logical to forensic science practitioners, and it includes a considerable amount of commentary or explanation.
The format of the NATA accreditation criteria is as follows:
- Organization and Management
- General
- Objectives
- Administrative practices
- Organizational structure
- Delegation of authority
- Supervision
- General
- Multisite laboratories
- Communication
- Quality Management System
- General
- Documentation
- Quality manual
- Document and information control
- Records
- General
- Case records
- Computers
- Internal audits
- Management review
- Complaints
- Corrective action
- Control of nonconforming work
- Corrective action
- Preventive action
- Personnel
- Qualifications
- Management
- Analysts/Examiners
- Controlled substances
- Toxicology
- Trace evidence
- Serology
- DNA
- Firearms/toolmarks
- Questioned documents
- Latent prints
- Crime scene investigation
- Technical support (nontestifying staff)
- Training and Development
- Evidence Management
- Evidence control
- Evidence retention and disposal
- Methods and Procedures
- General
- Documentation
- Method validation
- Reference materials
- Standards and reagents
- Subcontracting of technical work
- Quality Assurance
- Quality control
- Proficiency testing
- Case record review
- General
- Technical review
- Administrative review
- Court testimony monitoring
- Equipment
- General
- Calibration
- Reports
- Procurement of Services and Supplies
- Accommodation and Safety
- Security
- Environmental conditions
- Space
- Design
- Health and safety