Utility

Mathematical models are important.

Engineers use mathematical models to describe the natural world and then rearrange the model equations to answer the question, “How do I create an environment that makes Nature behave the way I want it to?” The answer to the mathematical rearrangement of the model equations reveals how to design processes, products, and procedures. It also reveals how to operate, use, monitor, and control them. Modeling is a critical underpinning for engineering analysis, design, control, and system optimization.

Further, since mathematical models express our understanding of how Nature behaves, we use them to validate our understanding of the fundamentals about processes and products. We postulate a mechanism and then derive a model grounded in that mechanistic understanding. If the model does not fit the data, our understanding of the mechanism was wrong or incomplete. Alternately, if the model fits the data we can claim our understanding may be correct. Models help us develop knowledge.

These models usually have coefficients representing some property of Nature, which has an unknown value (e.g., the diffusivity of a new molecule in a new medium, drag coefficient on a new shape, curing time of a new concrete mix, a catalyst effective surface area per unit mass, a heat transfer fouling factor). Model coefficient values must be adjusted to make the model match the experimentally obtained data, and obtaining the value of the coefficient adds to knowledge.

The procedure for finding the model coefficient values that makes a model best fit the data is called regression.

Although regression is ages old, there seem to be many opportunities for improvements related to finding a global optimum; finding a universal, effective, simple, and single stopping criterion for nonlinear regression; validating the model; balancing model simplicity and sufficiency with perfection and complexity; discriminating between competing models; and distinguishing functional sufficiency from prediction accuracy.

I developed and used process and product models throughout my 13-year industrial career. However, my college preparation for the engineering career did not teach me what I needed to know about how to create and evaluate models. I recognized that my fellow engineers, regardless of their alma mater, were also underprepared. We had to self-learn as to what was needed. Recognizing the centrality of modeling to engineering analysis, I have continued to explore model development and use during my subsequent academic career.

This textbook addresses nonlinear regression from a perspective that balances engineering utility with scientific perfection, a view that is often missing in the classroom, wherein the focus is often on the mathematical analysis, which pretends that there are simple, first-attempt solutions. Mathematical analysis is intellectually stimulating and satisfying, and sometimes useful for the practitioner. Where I think it adds value, I included analysis in this book. However, development of a model, choosing appropriate regression features, and designing experiments to generate useful data are iterative procedures that are guided by insight from progressive experience. It would be a rare event to jump to the right answers on the first try. Accordingly, balancing theoretical analysis, this book provides guides for procedure improvement.

This work is a collection of what I consider to be best practices in nonlinear regression modeling, which necessarily includes guides to design experiments to generate the data and guides to interpret the models. Undoubtedly, my view of best has been shaped with my particular uses for the models within the context of process and product modeling. Accordingly, this textbook has a focus on models with continuous-valued variables (either deterministic, discretized, or probabilities) as opposed to rank or classification, nonlinear as opposed to linear, constrained as opposed to not, and of a modest number of variables as opposed to Big Data.

This textbook includes the material I wish I had known when starting my engineering career and now what I would like my students to know. I hope it is useful for you.

The examples and discussion presume basic understanding of engineering models, regression, statistics, optimization, and calculus. This textbook provides enough details, explicit equation derivations, and examples to be useful as an introductory learning device for an upper-level undergraduate or graduate. I have used much of this material in the undergraduate unit operations lab course, in my explorations of model-based control on pilot-scale units, and in modeling of diverse processes (including the financial aspects of my retirement and the use of academic performance in the first two college years to project upper-level success). A person with an engineering degree and some experience with regression should be able to follow the concepts, analysis, and discussion.

My objective is to help you answer these questions:

• How to choose model inputs (variables, delays)?
• How to choose model form (linear, quadratic, or higher order, or equivalent model structures or architectures such as dimension or number of neurons)?
• How to design experiments to obtain adequate data (in number, precision, and placement) for determining model coefficient values?
• What to use for the regression objective (vertical least squares, total least squares, or maximum likelihood)?
• How to define goodness of model (r-square, fitness for use, utility, simplicity, data-based validation, confidence interval for prediction)?
• How to choose the right model between two different models?
• What optimization algorithm should be used for the regression to be able to handle the confounding issues of hard or soft constraints, discontinuities, discrete and continuous variables, multiple optima, and so on?
• What convergence criteria should be used to stop the optimizer (to recognize when it is close enough to optimum)?
• Should you linearize and use linear regression or use nonlinear regression?
• How to recognize outliers?
• How can you claim that a model properly captures some natural phenomena?

The underlying techniques needed for the answers include propagation of uncertainty, probability and statistics, optimization, and experience and heuristics. The initial chapters review/develop the basics. Subsequent chapters provide the application techniques, description of the algorithms, and guides for application.

Access to Computer Code

Those interested can visit the author's web site, www.r3eda.com, for open access to Excel VBA macros to many of the procedures in this book.

Years back our college decided to standardize with Visual Basic for Applications (VBA) for the undergraduate computer programming course. As a result, routines supporting this text are written in VBA, which is convenient to me, and also a widely accessible platform. However, VBA is not the fastest, and some readers may not be familiar with that language. Therefore, this text also provides a VBA primer and access to the code so that a reader may convert the VBA code to some other personally preferred platform. If you understand any structured text procedures, you can understand the VBA code here.

Preview of the Recommendations

Some of the recommendations in this book are counter to traditional practice in regression and design of experiments (DoE), which seem to be substantially grounded in linear regression. As a preview, opinions offered in this textbook are:

1. If the equation is nonlinear in the coefficients, use nonlinear regression. Even if the equation can be log-transformed into a linear form, do not do it. Linearizing transformations distort the relative importance of data points within the data set. Unless data variance is relatively low and/or there are many data points, linearizing can cause significant error in the model coefficient values.
2. Use data pre-processing and post-processing to eliminate outliers.
3. Use direct search optimizers for nonlinear regression rather than gradient-based optimizers. Although gradient-based algorithms converge rapidly in the vicinity of the optimum, direct search optimizers are more robust to surface aberrations, can cope with hard constraints, and are faster for difficult problems. Leapfrogging is offered as a good optimizer choice.
4. Nonlinear regression may have multiple minima. No optimizer can guarantee finding the global minimum on a first trial. Therefore, run the optimizer for N trials, starting from random locations, and take the best of the N trials. N can be calculated to meet the user desire for the probability of finding an optimum within a user-defined best fraction. The equation is shown.
5. Pay as much attention to how constraints are defined and included in the optimization application as you do to deriving the model and objective function (OF) statement. Constraints can have a substantial influence on the regression solution.
6. The choice of stopping criteria is also influential to the solution. Conventional stopping criteria are based on thresholds on the adjustable model coefficient values (decision variables, DVs), and/or the regression target (usually the sum of squared deviations) that we are seeking to optimize (OF). Since the right choice for the thresholds requires a priori knowledge, is scale-dependent, and requires threshold values on each regression coefficient (DV) and/or optimization target (OF), determining right threshold values requires substantial user experience with the specific application. This work recommends using steady-state identification to declare convergence. It is a single criterion (only looking at one index – statistical improvement in OF relative to data variability from the model), which is not scale-dependent.
7. Design the experimental plan (sequence, range, input variables) to generate data that are useful for testing the validity of the nonlinear model. Do not follow conventional statistical DoE methods, which were devised for alternate outcomes – to minimize uncertainty on the coefficients in nonmechanistic models, in linear regression, within idealized conditions.
8. Design the experimental methods of gathering data (measurement protocol, number and location of data sets) so that uncertainty on the experimental measurements has a minimal impact on model coefficient values.
9. Use of the conventional least-squares measure of model quality, ∑(y_data − y_model)², is acceptable for most purposes. It can be defended by idealizing maximum likelihood conditions. Maximum likelihood is more compatible with reality and can provide better model coefficient values, but it presumes knowledge of the variance on both experimental inputs and output, and requires a nested optimization. Maximum likelihood can be justified where scientific precision is paramount, but adds complexity to the optimization.
10. Akaho's method is a computationally simple improvement for the total east-squares approximation to maximum likelihood.
11. Establish nonlinear model validity with statistical tests for bias and either autocorrelation or runs. Do not use r-square or ANOVA techniques, which were devised for linear regression under idealized conditions.
12. Eliminate redundant coefficients, inconsequential model terms, and inconsequential input variables.
13. Perform both logic-based and data-based tests to establish model validity.
14. Model utility (fitness for use) and model validity (representation of the truth about Nature) are different. Useful models often do not need to be true. Balance perfection with sufficiency, complexity with simplicity, rigor with utility.

Philosophy

I am writing to you, the reader, in a first-person personal voice, a contrast to most technical works. There are several aspects that led me to do so, but all are grounded in the view that humans will be implementing the material.

I am a believer in the Scientific Method. The outcomes claimed by a person should be verifiable by any investigator. The methodology and analysis that led to the outcomes should be grounded in the widely accepted best practices. In addition, the claims should be tempered and accepted by the body of experts. However, the Scientific Method wants decisions to be purely rational, logical, and fact based. There should be no personal opinion, human emotion, or human bias infecting decisions and acceptances about the truth of Nature. To preserve the image of no human involvement, most technical writing is in the third person. However, an author's choice of idealizations, acceptances, permissions, assumptions, givens, basis, considerations, suppositions, and such, are necessary to permit mathematical exactness, proofs, and the consequential absolute statements. However, the truth offered is implicitly infected by the human choices. If a human is thinking it, or if a human accepts it, it cannot be devoid of that human's perspective and values. I am not pretending that this book is separate from my experiences and interpretations so I am writing in the first person.

Additionally, consider the individuals applying techniques. They are not investigating a mathematical analysis underlying the technique, but need to use the technique to get an answer for some alternate purpose. Accordingly, utility with the techniques is probably as important as understanding the procedure basis. Further, the application situation is not an idealized simplification. Nature confounds simplicity with complexity. Therefore, as well as proficiency in use, a user must understand and interpret the situation and choose the right techniques. The human applies it and the human must choose the appropriate technique. Accordingly, to make a user functional, it is important for a textbook to understand the limits and appropriateness of techniques. The individual is the agent and primary target, the tool is just the tool. The technique is not the truth, so I am writing to the user.

It is also essential that a user truly understands the basis of a tool, to use it properly. Accordingly, in addition to discussing the application situations, this text develops the equations behind the methods, includes mathematical analysis, and reveals nuances through examples. The book also includes exercises so the user can develop skills and understanding.

In the 1950s Benjamin Bloom chaired a committee of educators that subsequently published a taxonomy of Learning Objectives, which has come to be known as Bloom's Taxonomy. One of the domains is termed the Cognitive, related to thinking/knowing. There are six levels in the Taxonomy. Here is my interpretation for engineering (Table 1).

Notably most of classroom instruction has the student working in the lower three levels, where there are no user-choices. There is only one way to spell “cat,” only one right answer to the calculation of the required orifice diameter using the ideal orifice equation and givens in the word problem, and so on. In school, the instructor analyzes the situation, synthesizes the exercise, and judges the correctness of the answer. By contrast, competency and success in professional and personal life requires the individual to mentally work in the upper levels where the situation must be interpreted, where the approach must be synthesized, and where the propriety of the approach and answer must be evaluated. When instruction prevents the student from working in the upper cognitive levels, it misrepresents the post-graduation environment, which does a disservice to the student and employers who have to redirect the graduate's perspective. Accordingly, my aim is to facilitate the reader's mental activity in the upper levels where human choices have to be made. I am therefore writing to the human, not just about the technology.

A final perspective, on the philosophy behind the style and contents of this book is grounded in a list of desired engineering attributes. The members of the Industrial Advisory Committee for our School helped the faculty develop a list of desired engineering attributes, which we use to shape what we teach and shape the student's perspectives. Engineering is an activity, not a body of knowledge. Engineering is performed by humans within a human environment; it is not the intellectual exercise about isolated mathematical analysis. There are opposing ideals in judging engineering and the list of Desired Engineering Attributes reveals them. The opposing ideals are highlighted in bold (Table 2).

Engineering is not just about technical competence. State-of-the-art commercial software beats novice humans in speed and completeness with technical calculations. Engineering is a decision-making process about technology within human enterprises, value systems, and aspirations, and I believe this list addresses a fundamental aspect of the essence of engineering. As a complement to fundamental knowledge and skill of the core science and technical topics, instructors need to understand the opposing ideals, the practice of application, so that they can integrate the issues into the student's experience and so that student exercises have students practice right perspectives as they train for technical competency.

A straight line is very long. Maybe the line goes between pure science on one end and pure follow-the-recipe and accept-the-computer-output on the other end. No matter where one stands, the line disappears into the horizons to the left and to the right. No matter where one stands, it feels like the middle, the point of right balance between the extremes. However, the person way to the left also thinks they are in the middle. If Higher Education is to prepare graduates for industrial careers, instructors need to understand the issues surrounding Desired Engineering Attributes from an industrial perspective, not their academic/science perspective. Therefore, I am writing to the human about how to balance those opposing ideals when using nonlinear regression techniques for applications.