Presentations from the International Symposium on Setting Quality Standards for the Forensic Community (Part 3; Forensic Science Communications, July 1999)
July 1999 - Volume 1 - Number 2
Presentations at the
International Symposium on Setting Quality Standards for the Forensic Community
San Antonio, Texas
May 3-7, 1999
The following abstracts of the presentations are ordered alphabetically by authors’ last names.
The heart of forensic science as distinct from forensic technology is scientific inference. Given that we can control the quality of all of the mechanisms for generating observations and data, it remains to establish the logic of how we use those results to reason within a legal framework.
The application of probability theory to forensic inference leads to basic principles of interpretation. These, in turn, show how to frame the right kind of questions to address in a given case. Although the principles have been established for many years, it is remarkable that failure to observe them is widespread. This issue strikes at the very heart of forensic science. If we are invoking terms such as probable and unlikely in our opinions without an understanding of the nature of probability theory, then we cannot claim to be logical. If we aren’t being logical, then we cannot claim to be scientific. The forensic catch-all phrase, consistent with, has no value for the purpose of inference. It is an obstacle to rational and balanced interpretation.
Within this presentation, I have discussed such issues from the perspective of a recent initiative within the Forensic Science Service that is known as the Case Assessment and Interpretation Project. Case examples have been given.
The development of standard protocols for analysis of fire debris began in 1989 when the American Society for Testing and Materials (ASTM) approached the forensic community about reestablishing the Committee on Forensic Science. Founded in 1898, ASTM is one of the largest providers of voluntary consensus standards related to technical information in the world. The Committee on Forensic Science, designated E30, has the mission of developing standard protocols for disciplines within forensic science. A criminalistics subcommittee, designated E30.01, first developed protocols for the sample preparation and analysis of fire debris standards. ASTM committees assign a task group composed of subject matter experts to develop standards, which are voted on by consensus before becoming published standards. Standards are reviewed at least every five years. The current standards are in Volume 14.02 of ASTM standards.
In 1997, the National Center for Forensic Science (NCFS), a joint project between the University of Central Florida and the National Institute of Justice, was created. The goal of the NCFS is to create a unique laboratory facility that is staffed and equipped to service the forensic and law enforcement communities in the areas of fire and explosion debris. Given this mandate, NCFS agreed to administer a technical working group to develop standards for fire debris and explosives analyses. In August 1997, NCFS hosted a National Needs Symposium to assess the forensic community’s need for standard protocols. Having identified that there were needs for standards in analysis protocols, training, and education requirements, TWGFEX (Technical Working Group for Fires and Explosions) was formed in August 1998. This group, composed of experts from federal, state, and local laboratories, is just beginning to address the needs of the forensic community. TWGFEX’s first task was to develop and distribute a survey identifying needs in the forensic community as a whole for standards in the area of fire debris and explosives analysis, training, and education. TWGFEX met again in February 1999 to review the results of the survey. Each subgroup within TWGFEX developed goals and tasks to address the needs identified.
Many factors enter into the conceptualization, planning, and execution of proficiency testing (PT) programs in forensic science laboratories. Among the first that must be considered is the purpose of PT, what role is it expected to play in the overall quality assurance (QA) program for the laboratory. A clear sense of the purpose and role of PT in the QA program helps form a framework for the discussion about whether PT should be open, blind, or both. In this presentation, I discussed the cases for, and against, blind PT as compared with open PT.
Other issues that must be considered include a definition, or guidelines for a definition, of acceptable performance on a PT. It is a general conclusion that laboratories are unlikely to want to participate in a PT program in which acceptable performance is not readily definable. This concept allows the issue of testing to the average versus testing to the margin to be raised and discussed.
Closely connected to any discussion of PT as a QA measure is the notion of laboratory error rates. Raised by the Supreme Court in Daubert v. Merrell Dow Pharmaceuticals, Inc., some have suggested that PT comprised a method or approach for the measurement of laboratory error rates. This notion has particularly been raised in connection with arguments for blind, as against open, PT.
At the University of Illinois in Chicago, we have completed a small-scale trial to determine the feasibility and estimate the costs of implementing a large-scale (national) blind PT program for the DNA analysis laboratories. In this presentation, I briefly discussed the results and findings of those studies.
The final phase of our work in blind DNA laboratory PT involves additional research on the extent to which cases in forensic laboratories are currently subject to some kind of review (internal, external, adversarial, or a combination of these). It may be that the objectives intended by a large-scale blind PT program are essentially being met by practices already being followed in many laboratories. In this phase, we also hope to determine something about the feasibility of blindly introducing somewhat more complex specimens into laboratories as PTs and to see what the performance of a series of reference laboratories on such specimens is like.
The provision of scientific support to the police and other parts of the justice system plays an important role in the effort to maintain law and order worldwide. Because of this, it is vital that society has the fullest confidence in the work carried out by forensic science practitioners. The vast majority of the work withstands the closest scrutiny. However, a few high-profile criminal cases have highlighted the damage that can be caused to the public’s confidence in the justice system when flawed scientific evidence contributes to the wrongful conviction of innocent people. The effective training, development, and above all the assured competence of staff working in the sector are of paramount importance. This was emphasized in the Royal Commission on Criminal Justice Report in 1994.
Many organizations in the forensic science sector in the United Kingdom have for several years developed quality management systems to underpin the quality of their work and to enable them to achieve accreditation to standards such as the ISO 9000 series for general management systems and the NAMAS M10 standards for scientific and technical work. The personal accreditation initiative that I describe in this presentation is now beginning to provide a complementary framework to assure the quality of the work of individuals in organizations. This was recognized several years ago when, under the leadership of the Forensic Science Service (FSS), the forensic science community first began to develop the necessary occupational standards of performance for those working in forensic science. In addition to the sector’s own initiatives, powerful support has come from two important directions in recent years.
First, the Royal Commission on Criminal Justice looked carefully into the training and qualifications of forensic scientists and, in their report to government, included the important recommendation: Continuing efforts should be made to develop qualifications related to the competence of forensic scientists.
In the government’s final response to the commission, it accepted this recommendation and said that it encourages the current progress being made towards developing national vocational qualifications for forensic scientists. In addition, it stated that these vocational qualifications will be the mechanism by which regular assessment of competences are carried out and the focus for training.
Second, the sector’s work was also strongly endorsed by the House of Lords Select Committee on Forensic Science.
The Chief Executive of the Forensic Science Service also endorsed the concept of workplace assessments and has encouraged the development of standards of performance based on the United Kingdom central government’s national vocational qualification initiative.
To set the scene, I refer to Paul Kirk who, in his article, “The Ontogeny of Criminalistics” (1) published in 1963, implied that “training does not guarantee competence” and went on to say that “some form of licensing, certification, or some other indication attesting a practitioner’s competence may need to be adopted ultimately.”
It is with the latter statement of Kirk, made all those years ago, that this presentation is concerned: “Some form of licensing, certification, or some other indication attesting a practitioner’s competence may need to be adopted ultimately.”
The goal is therefore clear: determine what practitioners are required to do, support them in some way to achieve that, and have in place a process to hallmark their achievements.
Competence is the ability to work to the standards set in employment. In its broadest sense this includes all of the activities required to do a job, not simply the technical activities, although in this presentation the discussion about competence will be confined, in the main, to technical activities.
As long ago as 1991 the FSS recognized that standards of performance could be the starting point for an effective process to assess the performance of individuals and encouraged the start of work to develop them for forensic science practitioners in the United Kingdom. Under the initiative of the Forensic Science Service, a meeting was called of individuals representing the entire forensic science community in the United Kingdom.
Often the word sector is used in place of community and refers to all those individuals and organizations that play a part in the provision of a scientific support service. The sector ranges from major public service laboratories to one-person organizations and large and small private companies. It includes police service employees working in the day-to-day examination of crime scenes and fingerprints and, in time, may include others. The support for establishing such a sector-representative body came from central government, which was at that time and still is encouraging the development of national vocational qualifications in all areas of work in the United Kingdom. The qualifications are based upon agreed standards of performance. It was, therefore, an initiative not unique to forensic science.
Occupational standards are developed by working groups of competent practitioners under the guidance of a consultant who specializes in standards development work.
The representation of the standards is important. They must be written in such a manner that they can be used to assess the performance of individuals. All occupational standards in whatever discipline are represented along similar lines.
First, a description of a main work activity is essential, which is sometimes called a unit of competence and it describes the area of work on which the candidate will be assessed. A subset of this work activity, often called an element of competence, describes the work activity that the candidate must complete for assessment purposes. A series of performance criteria that relate to each element of competence list what a candidate must do to carry out the activity to the agreed standard. In other words, it is a knowledge of the performance criteria that is vital to the assessment process. These performance criteria could equally be called measures of performance.
Each of the units of competence will have several elements of competence, and each of the elements of competence will have a series of performance criteria, all of which are determined by working groups of practitioners.
There are other nontechnical units of competence also, but I confine this presentation to the essential technical competences. Some of the nontechnical competences could be leading teams, facilitating learning in groups, and managing resources and people.
To illustrate this, take as an example the Examination of Marks and Impressions units of competence for this subject area, which have been determined as
1. Prepare to carry out examinations and comparisons,
2. Assess examination items,
3. Examine items,
4. Interpret findings,
5. Report findings, and
6. Undertake specialist scene examinations.
To continue with this example from Marks and Impressions, next look at the first unit of competence:
Prepare to carry out examinations and comparisons. I mentioned previously that each unit of competence comprised several elements of competence. For this unit they have been determined by working groups to be the following:
1.1 Determine case requirements and provide advice on investigation.
1.2 Prepare equipment and work area.
1.3 Establish the integrity of items and samples.
Now each of the elements of competence has several associated performance criteria against which a candidate is assessed. Consider the element of competence to determine case requirements and provide advice on investigation. These performance criteria have been established to be the following:
a. Details of circumstances are established and considered against exhibits received and examinations requested.
b. Operational constraints that could affect the examinations are identified and clarified with relevant personnel.
c. Items submitted are confirmed to be appropriate for the work to be undertaken and where the inappropriate client is advised of the decision.
d. Storage requirements of potential evidence are established and arrangements are made for safe, secure, and clean facilities.
e. A reliable estimate of resource requirements is provided to relevant personnel and proposed ways of working are modified to cope with resource constraints in accordance with the advice received.
f. The sequence of the various aspects of the work are agreed and prioritized with relevant personnel to enhance the effectiveness of the investigation.
g. Relevant information is accurately, legibly, and comprehensively recorded.
It is against these criteria, for each element of competence, that a candidate for that unit is assessed. To give you some idea of the amount of work carried by the working groups, keep in mind that each of the three elements of competence described here, in the example being discussed, will each have several associated performance criteria. This is repeated for each of the other units of competence and each of their elements of competence. There are other features such as range statements and a description of some of the evidence that may be required to prove competence. These are not discussed in this presentation.
There is also a description of the essential knowledge required to carry out the work activity competently.
During the standards development process the proposed standards, developed by the working groups, are exposed to sector-wide consultation throughout the forensic science community in the United Kingdom. This gives everyone the opportunity to comment on the standards and to make suggestions for their modification. The feedback from consultation is considered by the working group and acted upon where appropriate. This creates a feeling of ownership by the sector.
When all the units of competence, elements of competence, performance criteria knowledge, and other features are complete for a particular subject area, they are aggregated into qualifications with a series of units of competence constituting a complete qualification. However, each unit of competence is in itself a mini qualification, so it is possible to be assessed and certificated in specific units of competence, thereby offering a flexible modular pathway to competence. This modular flexibility may suit some organizational and individual needs better than a full vocational qualification.
The newly proposed qualification is now ready for national accreditation and is submitted to an organization independent of the forensic science community, which for England and Wales is the Qualifications and Curriculum Authority (QCA). The qualification is submitted through the National Training Organization and a chosen external accrediting body—which, in the case of the forensic science sector in the United Kingdom, is the Open University—that is independent of any influence by the forensic science sector but offers specialist accreditation and other services needed for the competence assessment program. Once accredited, the qualification becomes available on the open market.
Every five years the standards will be reviewed to test their currency, and a system is in place to ensure that any changes to working practices within the five-year period are identified and addressed immediately so there is little chance of out-of-date standards remaining in place.
Assessment of Competence
I now turn to the assessment of individual performance. In the early days of the standards development program there was a very formal way of writing standards. Now we are able to experiment and develop much more user-friendly formats so long as all the essential ingredients required for the assessment of competence are included. We are currently working on new methods of standards presentation that will be published shortly.
There is scope for many organizations, both national and international, to become involved and help to build on that which has already been achieved and thereby play a role in the process in these formative years.
The standards in their entirety reflect both the knowledge required to do the job and the ability to carry out the tasks associated with the job in a work environment. Both of these requirements demand a very versatile, but rigorous assessment process.
I have no intention of developing the next point in this paper but it must be said that qualifications are offered at a variety of levels from Level 1 to Level 5: Level 1 reflecting the more straightforward tasks with a requirement for the possession of very little underpinning knowledge, and Level 5 where we are looking at performance at the very highest professional level with the application of fundamental principles and knowledge to situations that may never have been previously encountered and may never be encountered again. I know of no other process that does this. If we are to do justice to our profession, we need to articulate these issues clearly.
The assessment of the essential underpinning knowledge, which is expressed in the standards developed by the working groups, can be through traditional written examinations. The Forensic Science Service, on behalf of the sector in the United Kingdom, is appraising the use of the American Board of Criminalistics (ABC) examinations, or some modified form of them, for the assessment of knowledge.
Other examinations by the University of Strathclyde, University of Durham with the National Training Centre for Scientific Support for Crime Investigation, the Metropolitan Police Service Scientific Support College in London with Kings College University of London, and the various Diplomas of the Forensic Science Society and others are also being appraised for their value in the assessment of knowledge.
As a trial for the feasibility of using the ABC examinations, an examination was recently held in the United Kingdom under the auspices of the American Board of Criminalistics and was successful. Further discussions on collaboration are continuing, and proponents hope to present a proposal from the ABC meeting in August this year for the implementation of the ABC examination program in the United Kingdom, perhaps under some form of license.
In time, these examinations may find a use in Europe through the European Network of Forensic Science Institutes (ENFSI). In our possible collaboration with ABC, we also see the ABC’s potential use in some form of a model based on the one used in the United Kingdom to assess workplace competence to complement the ABC written examinations.
Assessment of workplace ability is another matter and, at first sight, appears more difficult than the assessment of knowledge, which in itself does not necessarily say a great deal about the ability of an individual to carry out a work activity. The assessor will need to see evidence from the work of the candidate to prove competence against the agreed standards. It is this evidence of competence that is identified by the candidate and that is appraised by the assessor against the standards. An assessor will give one of two possible verdicts on the performance of a candidate: Either the candidate is competent now or not yet. There will be no degrees of competence such as 80% competent or 50% competent.
Assessors are competent practitioners in their own right, and in addition to this occupational competence, they are suitably qualified to carry out the assessment process. The occupational competence of assessors is a vital requirement.
You will have noticed that the standards are written in a very general manner. In the example we have been using (Marks and Impressions), the Element of Competence 1.1 is given as “determine case requirements and provide advice on investigation.” Neither this nor the associated performance criteria says anything specific about how this should be done. It will be the responsibility of the occupationally competent assessor to ensure that it is done to the standards required.
The quality management of the assessment process will address any shortfall in practices. In other words absolute standards, the details of how work is carried out, are not imposed by some external body, and that is one of the reasons why the process has the potential for universal application. In some sectors in the United Kingdom there are examples where the assessment process and its quality management have shown that organizations are not working to appropriate standards. The outcome has been to raise the standards of such organizations and individuals in a user-friendly manner, almost a form of self-regulation.
Assessors may be either internal or external to the candidates’ organization. The concept of an assessor being internal to the organization, a workplace colleague of the candidate, often attracts some criticism when first encountered. However, a knowledge of the quality management of the assessment process will go some way to putting this concern to rest.
Assessors collect evidence of competence from the candidates’ work. Part of this evidence comes from cases a candidate has worked on in the recent past or is currently working on. The source of the evidence of competence is documented by the candidate in what is known as a portfolio of evidence of competence. It may, in the most part, be documented as an audit trail for the assessor to follow, enabling the assessor to check back through the work presented as evidence against the achievement of a standard of performance. The assessor will look for evidence of competent performance over a period of time and not simply at a snapshot of one piece of evidence on one occasion.
It is permissible to use realistic simulations where evidence cannot be gained from real work situations. Simulations are, however, the exception and, even when used, must be in the work environment, and competent performance must be demonstrated over time, not simply once. This may still leave some room for concern in the use of internal assessors.
It has been said that if the assessor has played a role in the training of the candidate, it may be in the assessor’s interest to show that the individual is competent as a result of training. However, there is an assessment verification process both internal and external that addresses this.
The internal verification system will be carried out by someone from the sector who will visit assessment sites, which will be the workplaces of the candidates, on a spot-check basis to test whether the assessor has used real evidence of competence against the standards and not simply used a tick-box approach to assessment. This process will also help ensure that the rigour of assessments, on what could be geographically isolated sites, is consistent.
The rigour is supported by an external verification process where the external verifier is appointed by the awarding body, the Open University in the case of forensic science, and is independent of the sector. The external verifier will carry out a similar role to the internal verifier but with a more obvious measure of independence. Neither the external verifier nor the internal verifier will carry out assessments or reassessments of candidates but will be concerned with ensuring that the assessment process is adhered to and that the evidence of competence collected is valid proof of consistent competent performance.
The assessment of ongoing competence can be carried out through a similar process, always against the relevant standards and often through a system called assessment through prior experience and learning.
The process of assessment that has been described is well-established in many organizations other than forensic science, a sector that is a comparative newcomer to the process. It is encouraging to see how well the process works once it is given a fair trial and to see how, in time, it becomes comfortably established in the culture of the organization as part of the quality management process and not as an annoying addition to the daily burden. If the standards are right, the evidence required to prove competence will simply fall from the work activity. Assessors will never encourage candidates to fit the standards to the activity but rather to fit the activity to the standards.
Occupational Standards and Training
Another valuable use to which organizations and sectors have put standards is in the design of effective and efficient training programs. Training designers now, probably for the first time, have a meaningful description of what a sector requires of its employees in terms of performance, and some sectors that have been using the scheme for some time have found that their training budget has been dramatically reduced and the speed with which their staff achieves the defined and agreed competence levels has increased.
This is an important point. It rightly implies that training is a means to an end and not an end in itself. The end point of training is a competent individual. The outcome, not the process, of training is important. We are now able to measure the competence of practitioners and therefore we are able to check whether we have reached our end point. Sounds very much like what Kirk was suggesting we needed to address more than 35 years ago.
Progress to Date
In the United Kingdom the standards development program continues. To date the following standards have been developed or are in various stages of development:
- Marks and Impressions
- Blood and Body Fluids
- Questioned Documents
- Recovery of Evidential Material—Laboratory Based
- Investigating Scenes of Incident
- Providing Fingerprint Services
- Managing and Coordinating Scenes of Incidents
- Accident Investigation
- Fires and Explosions
- Firearms Examinations
- Trace Evidence (glass, paint, explosives residues, fibres)
Each qualification covers the complete forensic process from examination of the scene of the incident through laboratory examination, interpretation of findings, report writing, and the presentation of evidence in court.
National Training Organizations
An organization called the National Training Organization has been mentioned, and I would like to take a little time to elaborate on that and to bring into the discussion some other parts of an infrastructure that is now in place in the United Kingdom to support the ongoing development of standards and other associated processes.
Several years ago there was an unmanageable proliferation of bodies concerned nationally with training, education, standards development, and associated activities. I will not elaborate on these bodies here because they are now mostly redundant. Their valuable work, however, goes on in a more efficient manner, and I will take a little time to describe briefly how this is.
The amalgamation of over 200 bodies throughout the United Kingdom has led to a new and more streamlined structure of about 60 bodies for carrying out the tasks associated with training, education, standards development, and other work. In forensic science the key organization is the Science Technology and Mathematics Council National Training Organization. This organization looks after the training and education affairs of the forensic science sector. It embraces most of the specialist technical and professional sectors in the scientific domain and is a key body in the United Kingdom. It is chaired by a prominent academic and industrialist, Professor Sir Geoffrey Allen. The scope of the organization is wide, and it is concerned strategically with the whole spectrum of vocational education and training for the sectors that it represents, which includes the forensic science sector.
Through its alliance with the Science Technology and Mathematics Council National Training Organization, the forensic science sector has formed a Forensic Science Sector Committee. This committee is the decision-making forum for the sector. It comprises members with key employment interests and also those with a national interest in the provision of vocational education and training. The chairman of the Forensic Science Sector Committee is Dr. Bob Bramley who is the Chief Scientist and Head of Profession for the FSS, and through his position as chairman of the sector committee, he is also a member of the Science Technology and Mathematics Council Executive Committee. The Executive Committee is responsible for shaping and forming the policies, programs, and priorities of the council; reviewing program implementation, management, and quality control; and endorsing occupational standards and vocational qualifications within the sector.
The prime committee, or governing body, of the Science Technology and Mathematics National Training Organization is the Council. The tasks of this body are to oversee all of the activities of the National Training Organization and to approve budgets. However, its primary role is political rather than operational. It will be concerned with the wider view of science education and training, and it will provide an interface with the government at the highest levels. Representation here is at the highest level, and the Chief Executive of the FSS, Dr. Janet Thompson, is the Forensic Science Sector Council member.
Council for the Registration of Forensic Practitioners
I now turn to the work presently taking place in the United Kingdom to establish a Council for the Registration of Forensic Practitioners. The aim of the Registration Council is to engender and maintain public confidence in the use of forensic science by producing a register of forensic science practitioners who are recognized as competent in their areas of expertise and who work to the highest professional and ethical standards.
Under this project, trials are taking place in chosen subject areas to assess the competence of practitioners to be included on the register. This exercise is at present restricted to the subject areas of fingerprints, questioned documents, and DNA. Some of the processes I have described are being evaluated to assess their suitability for this purpose.
Another working group within the remit of the council is looking at courtroom competences. The latest working group to be established is considering the establishment of a code of ethics and disciplinary procedures.
We now see as a possible way toward international cooperation with the vision that one day we, in this demanding profession, are all supporting each other’s efforts to deliver best practice to support our various justice systems.
In the United Kingdom, increasingly we are becoming involved in collaboration with our European colleagues. In support of this, European funding has recently been secured to develop standards of performance and assessment protocols for those working in the management and coordination of the activities of scientific support personnel at the crime scene. This work will be based, in part, upon the work already completed in the United Kingdom and should set a precedent for increased collaboration and harmonization of standards of performance and the assessment of workplace competences throughout Europe and perhaps in other international directions.
I have offered an overview of the work we are currently engaged on in the United Kingdom to improve the service we offer through ensuring the continued personal competence of those working in forensic science. Before concluding let me take a moment to elaborate on how the process offers flexibility of use. Standards, as we have seen, are designed on a unit basis, each unit being in itself a mini qualification. One unit comprising several elements. Each element comprising several performance criteria with associated knowledge.
In some organizations, for example the FSS, the workforce operates on two levels: At one level are technical experts who carry out the work, and at the other level are those who take the responsibility for the management of the work, carry out interpretation, report findings, and present evidence to courts of law, in other words reporting officers. In such a scheme the reporting officer will not necessarily need to be competent in doing the work but will need to know how it is done. Hence our interest in knowledge-based examinations.
The nascent council for the Registration of Forensic Practitioners is concerned with the standards of performance of those who report cases. However, the flexibility of implementation is such that, if in an organization one person does the work and that same person reports on the work, then that person will need the competence to work in addition to the knowledge to satisfy the requirements for that organization. The process described will support this.
I mentioned national vocational qualifications. It now seems likely that the sector in the United Kingdom will not go for full national vocational qualifications but will use the principles on which they are based to design a more appropriate product. Work is well advanced along those lines. The principles we are using are based on the national vocational qualifications program and have been endorsed by the Sector Committee, that is to develop and agree on occupational standards and to assess individuals against the standards incorporating some form of external monitoring in the competence assessment process. The standards are also useful as a focus for training and as the basis of a route not only of continuing professional development but also to ensure the ongoing competence of practitioners.
I will leave you with yet another quote from Kirk: “Criminalistics is an occupation that has all the responsibility of medicine, the intricacy of the law, and the universality of science. Inasmuch as it carries higher penalties for error than other professions, it is not a matter to take lightly nor to trust to luck.”
I look forward to continuing this discussion.
Kirk, P. L. The ontogeny of criminalistics, Journal of Criminal Law, Criminology, and Police Science (1963) 54:235.