Table of Contents
Cover
Table of Contents
Series page
Title page
Copyright page
Dedication
Preface
ACKNOWLEDGMENTS
Part I: Introduction
Chapter 1: Introduction
1.1 OPENING
1.2 VVT SYSTEMS AND PROCESS
1.3 CANONICAL SYSTEMS VVT PARADIGM
1.4 METHODOLOGY APPLICATION
Part II: VVT Activities and Methods
Chapter 2: System VVT Activities: Development
2.1 STRUCTURE OF CHAPTER
2.2 VVT ACTIVITIES DURING DEFINITION
2.3 VVT ACTIVITIES DURING DESIGN
2.4 VVT ACTIVITIES DURING IMPLEMENTATION
2.5 VVT ACTIVITIES DURING INTEGRATION
2.6 VVT ACTIVITIES DURING QUALIFICATION
Chapter 3: Systems VVT Activities: Post-Development
3.1 STRUCTURE OF CHAPTER
3.2 VVT ACTIVITIES DURING PRODUCTION
3.3 VVT ACTIVITIES DURING USE/MAINTENANCE
3.4 VVT ACTIVITIES DURING DISPOSAL
Chapter 4: System VVT Methods: Non-Testing
4.1 INTRODUCTION
4.2 PREPARE VVT PRODUCTS
4.3 PERFORM VVT ACTIVITIES
4.4 PARTICIPATE IN REVIEWS
Chapter 5: Systems VVT Methods: Testing
5.1 INTRODUCTION
5.2 WHITE BOX TESTING
5.3 BLACK BOX—BASIC TESTING
5.4 BLACK BOX—HIGH-VOLUME TESTING
5.5 BLACK BOX—SPECIAL TESTING
5.6 BLACK BOX—ENVIRONMENT TESTING
5.7 BLACK BOX—PHASE TESTING
Part III: Modeling and Optimizing VVT Process
Chapter 6: Modeling Quality Cost, Time and Risk
6.1 PURPOSE AND BASIC CONCEPTS
6.2 VVT COST AND RISK MODELING
6.3 VVT TIME AND RISK MODELING
6.4 FUZZY VVT COST MODELING
Chapter 7: Obtaining Quality Data and Optimizing VVT Strategy
7.1 SYSTEMS’ QUALITY COSTS IN THE LITERATURE
7.2 OBTAINING SYSTEM QUALITY DATA
7.3 IAI/LAHAV QUALITY DATA—AN ILLUSTRATION
7.4 THE VVT-TOOL
7.5 VVT COST, TIME AND RISK OPTIMIZATION
Chapter 8: Methodology Validation and Examples
8.1 METHODOLOGY VALIDATION USING A PILOT PROJECT
8.2 OPTIMIZING THE VVT STRATEGY
8.3 IDENTIFYING AND AVOIDING SIGNIFICANT RISKS
8.4 IMPROVING SYSTEM QUALITY PROCESS
Appendix A: SysTest Project
A.1 ABOUT SYSTEST
A.2 SYSTEST KEY PRODUCTS
A.3 SYSTEST PILOT PROJECTS
A.4 SYSTEST TEAM
A.5 EC EVALUATION OF SYSTEST PROJECT
Appendix B: Proposed Guide: System Verification, Validation and Testing Master Plan
B.1 BACKGROUND
B.2 CREATING THE VVT-MP
B.3 CHAPTER 1: SYSTEM DESCRIPTION
B.4 CHAPTER 2: INTEGRATED VVT PROGRAM SUMMARY
B.5 CHAPTER 3: SYSTEM VVT
B.6 CHAPTER 4: VVT RESOURCE SUMMARY
Appendix C: List of Acronyms
Index
End User License Agreement
Copyright © 2010 by John Wiley & Sons, Inc. All rights reserved
Published by John Wiley & Sons, Inc., Hoboken, New Jersey
Published simultaneously in Canada
Editorial contribution—Dr. Peter Hahn
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data:
Engel, Avner.
Verification, validation, and testing of engineered systems/Avner Engel.
p. cm.—(Wiley series in systems engineering and management)
Includes bibliographical references and index.
ISBN 978-0-470-52751-1 (cloth)
1. Quality assurance. 2. Quality control. 3. Systems engineering. 4. System failures (Engineering)–Prevention. 5. Testing. I. Title.
TS156.6.E53 2010
658.5′62—dc22
2009045885
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1
To my parents:
Josef Engel, Lea Engel and Tova Engel
and my revered teachers:
Dr. Itzhak Frank, Professor Jerry Weinberg and Professor Miryam Barad
Systems testing is carried out one way or another in all development and manufacturing projects, but seldom is this done in a truly organized manner and no book currently available describes the process in a comprehensive and implementable form. Along the same line of thinking, virtually no systems Verification, Validation, and Testing (VVT) research is conducted throughout the academic world. This is especially odd, since some 50–60 percent of a systems development cost is expended on either performing VVT activities or correcting system defects during the development process or during the life of the developed product.
This book attempts to put together a comprehensive compendium of VVT activities and corresponding VVT methods for implementation throughout the entire lifecycle of systems (i.e. Definition, Design, Implementation, Integration, Qualification, Production, Use/Maintenance and Disposal). In addition, the book strives to alleviate the fundamental testing conundrum, namely: What should be tested? How should one test? When should one test? And, when should one stop testing? In other words, how should one select a VVT strategy and how should it be optimized? Although early quality pioneers (e.g., Juran in the 1950s) proposed a conceptual quality cost model, no one proposed a quantitative and credible model which can be used to answer the above questions. This book provides such a model, together with data from a real-life project, which show significant potential savings in either cost, time or both. The book is organized in three parts:
The first part (Chapter 1) provides introductory material about systems and VVT concepts. This part presents a comprehensive explanation of the role of VVT in the process of engineered systems throughout their lifetime and explains the essence of systems VVT and the linkage between VVT and systems development, manufacturing, use/maintenance and retirement.
The second part (Chapters 2–5) is essentially a reference guide, describing typical systems VVT activities which may be conducted during an engineered systems lifetime. A reciprocal and comprehensive set of methods for carrying out these VVT activities is also provided. More specifically, the second part describes 40 systems development VVT activities (Chapter 2) and 27 systems post-development activities (Chapter 3). Corresponding to these activities, this part also describes 17 non-testing systems VVT methods (Chapter 4) and 33 testing systems methods (Chapter-5). In-text citations are provided wherever needed, usually within theoretical sections of the book. In addition, subchapters contain a set of citations for further reading. Readers will undoubtedly be able to absorb and implement some or all of this information in their daily work-life as systems or test engineers.
The third part of the book (Chapters 6–8) describes ways to model systems quality cost, time and risk (Chapter 6), as well as ways to acquire quality data and optimize the VVT strategy in the face of funding, time and other resource limitations and in accordance with different business objectives (Chapter 7). Finally, this part describes the methodology used to validate the quality model along with examples describing a system’s quality improvements (Chapter 8). Readers will be able to learn how to collect and aggregate quality data within their organizations. In addition to becoming familiar with this significant information, readers will be introduced to four Cost, Time and Risk Models. Systems engineers are encouraged to use these models in order to optimize their VVT strategies, thereby realizing as much as ten percent reduction in engineering manpower or schedule in the development of engineered systems. The software is freely available to readers via an Internet site (http://www.adisw.com
) in binary form under Beta release conditions.
Fundamentally, this book is written with two categories of audience in mind. The first category is composed of VVT practitioners, including Systems, Test, Production and Maintenance engineers as well as first and second line managers. These people may be employed by development and manufacturing industries (e.g., Aerospace, Automobile, Communication, Healthcare equipment, etc.), by various civilian agencies (e.g. NASA, ESA, etc.) or with the military (e.g., Air force, Navy, Army, etc.). This book may also be used as a supplemental graduate level textbook in courses related to systems VVT. Typical academic readers may be graduate school students or members of Systems, Electrical, Aerospace, Mechanical, and Industrial Engineering faculties. This book may be fully covered in two to three semesters (although parts of the book may be covered in one semester). University instructors will most likely use the book to provide engineering students with knowledge about VVT, as well as to give students an introduction to formal modeling and optimization of VVT strategy.
ACKNOWLEDGMENTS
Many friends and colleagues have contributed generously to the writing of this book. To all of them, I would like to express my sincere gratitude and appreciation. In particular, I wish to thank Dr. Peter Hahn, who has been a tireless and devoted companion in the book-writing project from its inception. He edited the original manuscript and contributed numerous and valuable suggestions to improve the book.
The SysTest project, partially funded by the European Commission (see Appendix A), focused my attention onto systems verification, validation and testing. My appreciation goes to all the consortium members and in particular to professor Eduard Igenbergs of the Technical University of Munich, who provided both a philosophical foundation and ample encouragement, and to Professor Tyson Browning of the Texas Christian University, part of whose scientific writings and words of wisdom are embedded in this book. The Advanced System and Software Engineering Technology (ASSET) group at Israel Aerospace Industries (IAI) was a significant milieu for learning and expanding. My special gratitude goes to ASSET group leader, Dr. Michael Winokur. I am also grateful to Shalom Shachar of the IAI/Lahav Division, who conducted the SysTest pilot project at IAI, helped in collecting field data and became a sounding board and advisor regarding many aspects of the VVT quantitative model. In addition, I am beholden to Michael Garber of Adi Mainly Software (AMS), who developed the VVT-Tool software package which embodies the VVT model.
Several close friends were involved in creating this book. In particular, I would like to mention Avi Egozi and Arie Rokach, who suggested the book project in the first place and provided advice throughout the writing process. Also my sincere appreciation goes to Menachem Cahani (Pampam), who volunteered to illustrate several caricatures in the book. I also am genuinely indebted to Professor Miryam Barad of the Tel-Aviv University, an esteemed teacher who taught me how to conduct scientific research and write about it.
Most of all, my deepest thanks go to my wife, Rachel, and my children, Ofer, Amir, Jonathan and Michael, who encouraged my book efforts with advice, patience and love,
Avner Engel
Tel-Aviv, Israel