A Comparison of Software Development Methods in a Regulated and Commercial Environment

By Olivier, Daniel P

This article compares software development practices applied in regulated environments with those in commercial applications. Tradeoffs between these applications are identified and use of a common risk-based strategy is shown to have significant applicability for both. A caution regarding application of standards and lessons learned applicable to both environments are discussed.

Key words: FDA regulated software, medical device software development, regulated software development, risk-based development models, safely risk analysis, software validation

INTRODUCTION

Software development methods in the broadest sense refer to the techniques used for software development as well as methods to confirm the adequacy of software applications to meet their intended use. This, of course, applies to any environment, be it regulated or not. In a regulated environment, however, the specific practices required to support evidence of validation (artifact reviews and testing the application to confirm correctness) take on a more precise meaning. This article will highlight the author’s 20 years of experience in dealing with regulated and commercial software applications to compare the practices that are required in a medical device regulated environment (specifically, the medical device environment as regulated by the Food and Drug Administration (FDA), Title 21 of the Code of Federal Regulations Part 820) (FDA 1996) with the practices that may be used in a commercial environment. In conclusion, this article will identify recommendations for each of these environments based on the best practices of both.

TECHNOLOGY AND REGULATORY TRENDS

Significant trends are clear in the software industry as well as in the regulatory climate. These trends include escalating customer demands and evolving technological capabilities. To address the challenges that these trends present, successful software development companies will have to become increasingly more efficient in their software development processes, as these challenges place significant demands on improving quality and reducing time to market.

Customer demands for heightened product performance continue to escalate as new hardware technology becomes available. These demands translate into significant challenges for new software applications including:

* Increased functional capabilities

* Increased interoperability with other applications (ability to seamlessly transfer data between products)

* Increased ease of use

* Increased reliability

* Increased frequency of upgrades

* More cost-competitive products

In addition to technology trends, there are significant changes taking place in regulated requirements in the United States as well as in other countries in the global market. The following regulatory trends are discussed based on the perspective of medical device and pharmaceutical regulations as defined by the FDA and trends in the finance industry.

* Regulated bodies will continue to focus on the review and inspection of software systems as automation becomes a larger part of management and control systems. The FDA has recently published new guidance on how to address the concerns regarding security controls (FDA 2005a) for networked medical devices, and the finance industry has recently introduced regulations for corporate financial reporting (Sarbanes-Oxley Act 2002).

* Regulated bodies are hiring auditors with increasing levels of software expertise and providing training programs focused on increasing the software expertise of current investigators and reviewers.

* Regulated bodies are raising the threshold of what constitutes adequacy as they continue to emphasize product safety and integrity of data as crucial requirements for new products.

All of these demands point to the need to produce more complex software, with reduced schedules and increased levels of quality and reliability. The only reasonable approach to address these seemingly impossible demands is reliance on improved software development methods.

COMPARISON OF VALIDATION IN A REGULATED VERSUS NONREGULATED ENVIRONMENT

A question that is frequently raised with respect to documentation in a regulated environment is how much is required to ensure compliance with the regulations? Obviously, different levels of detail for documentation can dictate a significant difference in effort for development and testing. Although there is no absolute standard to define completeness of documentation, it is expected that as a minimum all requirements, as defined in a requirements specification, are formally tested (of course, this implies that in the absence of a requirements specification the completeness of the validation testing cannot be defended). Other factors are considered in the development process including customer satisfaction and time to market, but many times these objectives are overshadowed by the effort required to support regulatory compliance.

Figure 1 provides a classic “waterfall” representation of a development process that defines a software design and development model. The model illustrates different review and test activities (labeled as verification and validation) executed during the development phase. Similar review and test activities are defined in a commercial development model; however, the delineation of separate review and test phases and explicit documentation requirements are not as well delineated. In the commercial world the focus is more on the activity and less on the documentation. The regulatory review bodies (such as the FDA) tend to place more emphasis on the documentation.

In a nonregulated commercial development environment, the criteria for completeness of testing is not as clearly defined. The definition is prescribed by an external entity (regulated body) but a matter of what provides a high degree of confidence that the customer is satisfied balanced with meeting demands for reducing time to market. Tradeoffs are being made, but these tradeoffs are based on a business case. In the regulated scenario the decisions regarding the level of testing have an added dimension: the need to demonstrate test adequacy to support regulatory compliance. Figure 2 provides a depiction of a Spiral Model of software development as defined by Barry Boehm (1988). This is characterized as a risk- based model where evaluation of project risks is used to drive risk- reduction techniques such as prototyping throughout the software development process. The Spiral Model is more characteristic of the practices of software development for commercial software development companies.

Tradeoffs

Understanding the specific documents required to support compliance with FDA and ISO standards provides visibility to the increased costs that may be incurred in this environment. It would be inappropriate, however, to suggest that the regulated approach or the nonregulated approach to validation is superior without contrasting the advantages and disadvantages of each. This comparison shown in Table 1 identifies the tradeoffs associated with each approach.

Contrasting the FDA regulated industry and commercial software development fails to provide a clear winner in terms of which environment is better from all perspectives. The evaluation shows that there are more specific documentation demands associated with the higher risk inherent in medical devices. The regulatory model identified here demands adherence to more formal development methods and increased documentation for more critical applications.

FIGURE 1 Software development life cycle implied by regulatory model

TABLE 1 Tradeoffs betweeen regulated and commercial software development models

The level of risk may vary in criticality even within the medical device or pharmaceutical field. The FDA has recognized this and advocates a risk-based model for determining the level of formal documentation that is appropriate for a given function. The term used by the FDA for such an analysis to determine the level of documentation and testing required for an application is risk analysis.

It is interesting to note that this same assessment method (risk analysis) was identified by Boehm in the 1980s as the key strategy used by successful project managers to achieve project management success based on the previously referenced Spiral Model. Also of interest is that this basic technique of risk management is one that has been shown by many management gurus to also be a common practice for management excellence. Management practices that have been popularized that support a risk-based approach to management include:

1. Bottleneck theory

2. Cost drivers

3. Critical control points

4. Critical path

FIGURE 2 Spiral software development process

5. Critical to quality (CTQ) criteria

6. Management by exception

7. Pareto analysis

8. Top 10 list

9. Vital few versus the trivial many

10. Weakest link

Documentation Required

Although both the regulated and nonregulated processes have the same ultimate objective, to develop quality software in a timely manner to satisfy customer needs, the regulated environment also must satisfy a more strict documentation requirement. Table 2 shows a list of specific documents that may be used to address the requirements of the FDA QSR as well as the ISO 9001:2000 quality system requirements (ISO 2000). The need for explicit documents to satisfy specific regulatory requirements \is once again very clear in this matrix. In the commercial marketplace the definition of such explicit documents to be developed is not defined, it is left to the development group to identify specific documents and the formal nature of the documents to be produced (although it seems reasonable to require the content suggested by the categories of documents as listed in Table 2).

AN EXAMPLE OF APPLYING A RISK-BASED APPROACH

The FDA has identified “Levels of Concern” as a risk-ranking scheme for defining differing levels of documentation (design and test documentation). An increased level of documentation is required for applications that have a higher consequence of patient risk. The definition of safety risk levels as defined in FDA guidance documents is provided in Table 3 (FDA 2005b).

Using this approach, the FDA allows for the definition of varying levels of testing for each safety level. For functions that have a low level of risk (minor) the FDA only mandates a functional level of testing. For more critical risk levels (moderate and major) the FDA demands additional levels of testing (such as integration and unit-level testing and/or code-review techniques).

Application of such a model in the commercial marketplace could also have significant advantages, including providing more specific guidance on what areas of an application should be subject to more rigorous testing as a consequence of what areas of the application pose the greatest risk of defects (based on historical experience, customer preference, complexity, or other factors). A risk-based strategy for allocation of resources can also be used to define more efficient review and test practices and more quantitative decisions for product release decisions.

The need to prioritize activities to reduce time to market can be learned from the commercial market-place by regulated software developers. Many regulated companies tend to focus on the documentation as an end point and sometimes lose sight of the real intentto assure the quality of the produced software. Because of audit experiences, highly publicized compliance problems experienced by other manufacturers, and a motivation to adhere to latest industry standards, the default level of documentation expected to satisfy regulatory requirements seems to increasingly expand. There is a tendency to pursue quantitative criteria as the basis for adequacy unless there is a continued focus on being able to evaluate the qualitative benefits achieved from the documentation produced. Only when a regulated manufacturer becomes confident in their ability to defend the adequacy of their documents with regard to applicable regulations can a true qualitative focus be achieved.

Pitfalls

A major pitfall experienced by commercial software developers is the pressure to release software before adequate testing has been performed to assure adequate product quality. The consequence of this decision is a product in the marketplace that is unacceptable to the customers and results in a poor reputation. Although a manufacturer can never afford to wait until the software is perfect, failure to effectively balance the level of defects with the consequences on customer satisfaction will result in nonoptimal product release decisions.

TABLE 2 Regulator requirements for design control compliance and candidate documents

Although medical device manufacturers may also make a software release that has not been adequately tested, because of the regulated environment, the tendency is more often to extend development cycles due to requirements for the preparation of increasingly greater levels of documentation perceived as required to satisfy regulatory requirements. Once again the risk-based model can be used to support qualitative decisions for defining the levels of documentation/testing that are essential to assure compliance and also achieve target levels of product quality.

TABLE 3 FDA levels of concern

Are Standards the Answer?

In today’s market the need to constantly improve to stay competitive is no more prevalent than in the software development industry. To enhance product quality many companies (especially those in regulated industries) have relied on conformance to industry standards such as the Capability Maturity Model Integration (GMMI) (SEI 2002), IEEE standards, ISO standards such as 12207: Information technology-Software life-cycle processes (AAMI 2001), and other industry standards such as ANSI SW-68: Medical device software-Software life-cycle processes (ISO 1995). This exercise has become complicated by the fact that the standards are continually evolving and, in general, tend to maintain a steady pace of five to 10 years behind industry best practices. In addition, these standards tend to be a one-size-fits-all approach that may be optimum for some projects but is oftentimes ill-suited for others. Once again, use of a risk-based development model emerges as an excellent tool to tailor established standards and the practices to a specific application to yield the greatest value for a particular manufacturer. (In many cases, significant tailoring is not always allowed by such standards.)

There is a tendency for regulatory bodies (such as the FDA) to rely on adherence to industry standards as a substitute for being able to evaluate true product quality (this assumes that if an industry standard is followed then quality software applications will naturally result). As previously discussed, this assumption is flawed in that the quality of documents provided does not directly predict the quality of the software produced. History has clearly shown that use of structured methods has a higher likelihood of producing quality software than failure to follow an established method. However, using adherence to a published standard and quantity of documentation produced as the absolute measure of product quality is shortsighted and naive. A better predictor of software quality may be product maturity as measured by length of time in the market, customer reputation, and results of internal stress testing of relevant product functions.

LESSONS LEARNED

What can be learned from this comparison? Clearly the regulated medical and pharmaceutical companies cannot sacrifice compliance without regulatory consequences. As a result of this fear, however, there is a tendency to follow established regulations to an increasingly more detailed level, that is, to define increasingly greater levels of documentation in the name of compliance (almost as if expecting that greater levels of documentation can result in greater levels of compliance even though compliance is not a quantitative but rather a qualitative attribute). In the nonregulated industry there is tendency to avoid documentation often resulting in a nonoptimal level of documentation with respect to the benefits that can be gained as a result of having formalized certain steps of the development process.

Perhaps the most significant lesson learned is that the application of risk management principles (not just with regard to safety but with regard to any attribute that may be of importance to the development team) can be a key strategy to align engineering development objectives with business objectives. A risk management approach recognizes that there are always tradeoffs to be made. The tradeoffs can be between time to market and product quality, between resources and release schedules, between compliance and cost, or any other attributes a project manager can define. The key lesson learned is that management can select the most important attributes and structure the development process to focus on those areas.

CONCLUSION

This article has illustrated some of the tradeoffs that exist between a regulated software development environment (emphasis placed on the medical device and pharmaceutical markets as regulated by the FDA) and the commercial market. The comparison has shown that the regulated industries incur a more significant documentation burden as a result of being regulated; however, this increased documentation is a logical requirement in light of the increased criticality of these systems. In the commercial market, the level of documentation is not externally defined and therefore can be determined by the manufacturer.

A problem experienced in regulated industries is to take the documentation as defined by regulatory entities to an extreme, that is, using documentation as a surrogate measure for software quality. When this happens, the development process tends to shift focus from assuring the quality of the software to evaluation of documentation completeness (quantity) and quality (correctness) as the basis for making decisions on software quality. This is a concern, as this emphasis on documentation quantity tends to extend software development schedules without commensurate quality benefits. Use of a risk-based model can be used to balance the documentation produced with controls that assure the quality of the software produced.

It would be inappropriate to suggest that there is no added cost for being in a regulated industry. The conclusion offered by this article is that the increased overhead incurred for regulated industries, such as those regulated by the FDA, can be minimized through the use of techniques such as risk-based methods. Techniques that minimize the reliance on formal documentation, as often practiced by commercial software developers, have advantages in reduced time to market and reduced development costs. Each industry can learn from the other using risk-based assessment methods to determine the level of tradeoffs between formal documentation practices and more efficient practices (such as reduced development schedules) that can be optimum for software products produced.

SQP References

Variability in Software Development Process Families

vol. 3, issue 2

Ramkumar Ramaswamy

FDA Regulations and Auditing Practi\ces for Software Suppliers at a Pharmaceutical Manufacturer

vol. 6, issue 4

Timothy W. Theisen and Colin J. Neill

REFERENCES

AAMI. 2001. ANSI/AAMI SW-68:2991: Medical device software- Software life cycle processes. Arlington, Va.: Association for the Advancement of Medical Instrumentation.

Boehm, Barry W. 1988. A spiral model of software development and enhancement. New York: IEEE Computer.

FDA. 1996. 21 CFR Part 820-Quality System Regulation, U.S. Federal Register. Rockville, Md.: U. S. Food and Drug Administration (FDA).

FDA. 2005a. Guidance for industry cybersecurity for networked medical devices containing off-the-shelf (OTS) software. Rockville, Md.: U.S. Food and Drug Administration (FDA).

FDA. 2005b. Guidance for FDA reviewers and industry, guidance for the content of premarket submissions software contained in medical devices. Rockville, Md.: Department of Health and Human Services, U. S. Food and Drug Administration (FDA).

ISO. 2000. Quality systems – Management System – Requirements, ISO 9001:2000. Geneva, Switzerland: International Organization for Standardization (ISO).

ISO. 1995. ISO/IEC 12207 Information technology – Software life cycle processes. Geneva, Switzerland: International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC).

Sarbones-OxleyAct of 2002. 107th Congress, 22 January.

SEI. 2002. Capability Maturity Model Integration (CMMI), version 1.1. Pittsburgh: Software Engineering Institute, Carnegie Mellon University.

DANIEL P. OLIVIER

Certified Software Solutions, Inc.

BIOGRAPHY

Daniel P. Olivier is president of Certified Software Solutions, Inc. He is an acknowledged expert in the field of medical and pharmaceutical system validation and safety risk management. He has supported more than 200 medical device and pharmaceutical companies in addressing regulatory issues and process improvement. Olivier has been contracted by the FDA to provide training for field investigators and to prepare inputs for FDA validation guidance documents. He is an ISO 9001 RAB certified lead auditor and has been a reviewer for ISO WGlO defining software process assessment standards. A frequent speaker at professional conferences and author of several seminars, he has published more than a dozen articles on software validation, safety risk assessment, and design controls to meet FDA and ISO requirements. His company, Certified Software Solutions (CSS), Inc., provides validation consulting, safety risk analysis, audits, training, development, and testing services to meet the requirements of the FDA regulations, ISO 9001, and ISO 13485. He can be reached by e-mail at dolivier@certifiedsoftware.com.

CMM and CMMI are registered trademarks of the Software Engineering Institute, Carnegie Mellon University.

Copyright American Society for Quality Mar 2006