Observations on commercial analytical instrumentation development

In his Heritage Lecture in 2012 at the Winter Conference on Plasma Spectrochemistry,¹ Professor deGalan defined four players in the analytical performance domain: the inventor, the instrument maker, the customer and the client. This presentation is about the instrument maker, and specifically about the management of technology developments and the technologists who bring these developments to market. It is about the environment within which successful development professionals bring new measurement instrumentation to life.

Having advised, mentored, managed, recruited, hired, fired, directed and led professional technologists for the majority of my professional career, my retrospective is some of what I have observed and learned from all those technology developers who have achieved success in their careers.

Introductory information

I'll begin by asking a question about my own career: how did an analytical chemist, who started out as a technology developer, end up spending most of his career as a manager of technologists and of technology developments? The trite answer is that analytical chemists are problem-solvers. The suggestion, then, is that instrument making is an endeavor of problem solving. My observation is that most technology development problems have at least as much to do with solving personnel and resources issues as they do with the science or technology involved.

The first corporate president under which I worked, and who became an important mentor, once explained to me, “Sometimes business has to make decisions in spite of your best science and engineering”.²

Consequently, learning how to accomplish projects within an organization is as important as getting the science and engineering right. My first two observations applicable to technology management need some background information on myself, information that isn't in the technical domain and which I will convey with a short story about an experience I had. It is a story about learning things, and knowing that being able to learn something may just be the best outcome for a technical development you will have available.

My participation on our high school football team resulted in an unexpected opportunity to play first string for most of two seasons. Since Dick Butkus, who became a National Football League Hall of Fame linebacker for the Chicago Bears and who is a year or two older than me, went to a high school in the same league as my high school, I eventually found myself playing in a football game against him. I was actually on the field playing a real football game against Dick Butkus who, even as a teenager, was ferociously competitive and overpowering. Not surprisingly, Mr Butkus and his team made short work of our team. Losing the game that day was not an enjoyable experience.

There are a couple of points to be made with this story. They are relevant to every endeavor, including scientific research and technology development.

First, there is always someone better than you are at what you do. Consequently, we should not be surprised when we encounter those losses in life in which someone better trumps our developments. It happens; learn to get over it quickly.

Second, losing is not the same as failing. Failure only happens when we are not wise enough to learn from our losses. In my case, what the loss to Mr Butkus and his team that day taught me was that my career track was not in athletics, but rather in the Chemistry I had started to pursue.

Additionally, this makes for a fun story based in fact that gives me a small tie to this great professional athlete. A few years later, Mr Butkus and I shared another interaction, we were both awarded varsity athletic letters at the University of Illinois, and mine was in wrestling. Clearly, Mr Butkus had substantially more career success with his athletic abilities than I would have had!

I need to provide a few additional details of my background as a lead in to another two observations more directly related to managing instrumentation development. Over my career, through conscious decisions, I accepted a variety of roles somewhat apart from direct development of technology. Among these were project management, program management, personnel management, technical input to business development, budget planning and management, and resource and facilities management. Through these roles, which were essentially facilitating technology development teams, I obtained three general insights.

The first of these insights is that scientists and technologists generally struggle to obtain and become accomplished with managerial skills and administrative activities. We might ask why should a scientist or technologist even care about managerial skills? The answer is that, whether one wants to or not, a technical developer needs to know how to manage projects and the people in them as their career proceeds, if for no other reason than one can't do all the work by themselves. Science curricula don't really teach these skills, and engineering curricula don't have very much of it. Consequently, it is up to the individual to obtain these skills. One has to enlist on-the-job training, or go to night school, or take a sabbatical for other degree work, etc., or find whatever other opportunities there are.

With respect to personnel management and program management skills, much of my training came through my 30-year career in the U.S. Naval Reserve. With that base and its continuing input, gravitating to industrial project and program management was not difficult.

The second insight I obtained from nontechnical roles is that for long-term success in an organization, it is key to have a diversity of skills that includes something orthogonal to your scientific or engineering expertise. Especially in this era of “lean” organizations, having more than technical skill makes one more valued.

The third insight is something we have all been told about success in our careers, which is that it is key to continue learning and broadening our skills throughout our careers. My version of this is the rate at which you are becoming adept at new skills is the rate of progress of your career.

Observations on technology development

I'll now address four (4) different topical areas for a series of observations on the technology development domain and the professionals within it. The four topics are the cost of technology development; innovation in a profit-oriented company; project management; and leadership in a technology development organization.

1. The cost of technology development

Most commentary on high tech development highlights its large expense. In the domain of chemistry and physics instrumentation development, multiple decades of experience in many organizations have shown that these costs are mostly quantifiable.

Cost = Labor + Materials + Overhead.

Labor = Salary + Benefits, with Benefits about equal to 40% of Salary.

Materials = 17%–25% of the labor cost of technologists.

Less risky project, 17%.

More risky project, 25%.

Overhead = 80%–95% (Labor + Materials).

When all these factors are reduced to a function of the technology staff salary, S, it is found that:

Hardware projects cost = 2.8S − 3.1S

Software projects cost = 2.5S − 2.8S, since software materials costs are lower.

As an example, assume an engineering department with a diverse staff of both new and experienced technologists, so the average salary is around US $135 000 per year. For even a small, 5-person project team the cost would be about US $3M for an average 18-months project development. This cost is independent of what exactly is being developed. Further, this calculation does not include manufacturing engineering, marketing or sales expenses. It does not include production costs, which can be as much as ×10 more than the development costs.

It is not hard to see that technology product development is extremely costly. It is even more costly to the company since the costs are not amortizable across multiple fiscal cycles. The costs come as raw cash flow. This is among the reasons that technology development managers are under acute pressure to contain costs of development projects. It is not surprising then, with project expenses so high, that technology development companies want to pursue only the most promising technologies.

2. Innovation

This brings us to the concept of innovation. The conventional wisdom is that the more innovative the development, the more likely it will be that the expected return on the investment (ROI) will be realized. However, this is a less-frequently attained objective than would be desired.

Innovation is precisely something that gains from uncertainty.³ Therefore, innovation is not often what “R&D” in the conventional, commercial, profit-oriented organization does. I'll address this from a couple of different views, the first of which is, “The Role of R in R&D”.

Development, not research, is what manufacturing organizations do best. As quoted in a recent study,⁴ “Even in small, start-up companies, the primary focus is on development rather than research”, “The technology push companies (i.e., those that do research) are the riskiest in terms of survival”. So, there's little to no real “R” in R&D in a commercial, profit-oriented organization. If what you want to do is publishable research, then don't expect to do it within a profit-oriented organization. It's not that it can't be done, but there won't be much, if any, encouragement, resources, or facilities for it.

Let's address aspects of the “D” component in “R&D”. New products get developed to meet customers' needs. Many instrumentation development organizations develop the same kind of products. For example, how many kinds of AA's or GC's or HPLC's or MS's are there? With each competitor's products performing the same functions, a development company seeks “market differentiation” with its products. But how can any company find the differentiation it desires when everyone is developing the same physics and chemistry technology?

Consequently, technology developers are told to “be innovative”. This is to be interpreted as, “come up with something different, yet make sure it's still the same, recognizable instrument,” i.e., make sure it is still an FTIR, or ICP-AE, or GC, or whatever. When technology developers and engineers accomplish this it is not innovation; it is very, very clever design and engineering. As a result, innovation is probably the wrong word for what a technology development organization does. We need to delve into what is called “innovation” as it is practiced in a technology development organization.

Innovation comes from uncertainty about how to move forward. Think, “Necessity is the mother of invention”. It comes from the determination to change things in order to make progress. However, engineering organizations drive out uncertainty, in everything they do. They get things done quickly and cost effectively. Therefore, an “innovative engineering organization” is an oxymoron. In such an environment, there is constant and intense stress between the changes to be made to support innovation, and the maintenance of the stability of design activity that makes for a cost-effective and efficient technology development organization.

Managing and directing a development group that is successful at producing innovative products is an exercise in continual frustration. One is regularly producing guidelines for the timely execution of development projects and, at the same time, encouraging the dismissal of those guidelines to reach breakthroughs. This is a substantially schizophrenic approach to doing anything.

Managing the professionals in such a balancing act of activities must be done very carefully and always individually. An optimum approach is to give them what they need to do their work, and then get out of their way. This leads to the next aspect of corporate innovation and what has been called the concept of corporate autoimmunity.⁵

Innovation is the same as saying “change things”. A dilemma facing companies is how to protect their successes that came from the innovations that got them successful, while at the same time striving for new innovations.

As companies seek to extend and protect their successes, they naturally develop immune systems designed to maintain the health of the organization. Business fitness depends on a “corporate immune system”.

These immune systems can be comprised of a complex set of management attitudes, decisions and controls that serve to protect, defend and perpetuate the value proposition of customers and corporate profits. They are established to minimize, if not eliminate, any uncertainty about the continued success of the commercial endeavor.

However, when corporate fitness becomes paramount in the minds of the managers of the organization, then the immune systems attack the corporation's own innovation efforts, mistaking these efforts as threats, or antigens. That is, well-intentioned and otherwise necessary corporate immune systems can turn on the host in a manner similar to autoimmunity.

“Corporate autoimmunity” happens when the organization's own need for order, sometimes in the form of predictable profitability, becomes more important than its purpose of providing new/innovative value to customers.

A classic example of such an autoimmune response is when a large corporation seeking to innovate disqualifies an effort based on market size. “Only looking for billion dollar businesses,” fails to acknowledge that the current billion-dollar business once started with relatively small and humble beginnings.

As a consequence, true innovation, which generates the revolutionary developments, is often repressed. What is generated instead is evolutionary change. The message received by the technology developers is: “Don't fix what isn't broken”. However, in true innovation, everything is always “broken”. A cynical way of conveying this is: “Success is its own worst enemy”.

Innovation in the corporate domain continues to be a much-debated topic. A recent conference on “Unleashing Innovation,”⁶ generated a number of lists of top agenda items for driving innovation in mostly large companies. Of the dozen or so “top agenda items” only one referred to any form of risk, and that was only an acknowledgement of the known risks that confront small companies entering a market. Nowhere was mentioned how to address the risks to current business that true innovation may present. The hidden assumptions are that business will proceed as usual, i.e., profits will be protected, and “innovative ideas” will be subsumed in this business model.

In the face of this type of corporate internal confusion with regard to the impacts of innovation, we can expect the many fine instrumentation development corporations to continue to produce excellent, cost-effective products for their customers. These systems will be well engineered and of high quality and lasting durability. They will contain within them numerous technical developments, which are the solution to numerous development problems of which the customer will be quite unaware. The instrument products will be, in essence, a bundle of very-well-solved technical and engineering problems. To name much, if any, of these fine assemblages of engineering as “innovative” stretches one's imagination. To put it another way, innovations always solve some sort of problem; but not all problem solutions are necessarily innovations.

As a final comment on innovation in technology development organizations, it is worth asking, “Where do the ideas for new products come from?” I suspect the following cliché is familiar to many:

“Market Rules:

1. The customer is always right.

2. When the customer is wrong reread Rule #1.”

The meaning of this is that one has to work hard to define the problem being presented as a new technology development activity. The customer may not present it correctly, so it is necessary to dig for the real issues. In general, it is a fallacy to think that someone knows what they want and will tell you if you ask them.

In many cases, defining an apparently obvious problem is where a lot of effort is expended in the early stages of a technology project. Personal interpretations of what the problem is, or “the world works my way not yours syndrome,” or versions of “we don't do it that way here” consumes more effort than it should, all compounding the efforts to initiate any new product development effort.

3. Project management practices

The third of the topics I'll address is project management. All organizations doing technology development follow some version of project management. It can be called almost anything; all the companies with which I was associated had their own names for it, such as concurrent engineering, teaming, PAR process, PDR, etc. At its core, project management is a disciplined decision-making process that allows for tracking and reporting metrics to gauge progress. To be cynical, project management approaches are invoked since “tech development costs a lot, so do something to control it”.

Project management practices are not guaranteed to produce anything, much less product success. Used well, these practices can help to avert expensive dead-ends; they cannot, though, ensure a successful outcome. They can only assist project teams and their managers make timely, cost effective decisions. Product development in a project management scenario is a complex and intricate endeavor. Numerous organizations, among which is the Project Management Institute, support the dissemination of the skills, tools and knowledge of best practices to assist project team members succeed with their development efforts. The area remains a stressful one, though. The often cited dilemma of the project team leader is: “Cost, Schedule, Performance: Pick Two”. If one lays on top of this scenario even more uncertainty, through pursuit of “innovation,” one is then left with an even more problematic situation.

Having been a product development project leader on a number of projects and, eventually, becoming an instructor of project management practices, I can offer a few suggestions and insights on pursuing high tech development projects.

The first of these is to fail quickly. Try things and uncover problems as fast as possible. Learn from the failures; then retry. Successful product development is based on numerous quick failures and not on methodical perfectionism.

Second, if a decision seems to need some sort of additional “study” to come up with justifications for it, then you're not at a decision point. Find out who is pushing for a decision and why. Obvious decisions require no more than a single reason, so if there is not concurrence on that reason then it's not the time to make a decision.

Third, beware of “Accelerated Product Development”. This is often invoked, though not acknowledged to be so, because of budget constraints and the high cost of technology development. The thought is to get the development done more quickly so spending on it can end sooner. The problem is, in analytical instrumentation development, it's rare that some part of the chemistry or physics can be deleted. Shortened development schedules leave the same totality of work to be done.

In my somewhat jaundiced view of accelerated project development, too much of it is demanded so that there is something to report more frequently. “Where's all the money being spent? What did we get for it this week, today?” Unfortunately, “busy-ness” of this sort is not an indicator of real productivity. This is an indication of a weak organization's way to overcome its lack of control over the technology it's trying to work with.

This brings us to the question of, “how long does it take to develop a new product?” As can be imagined, this is a complicated question to answer. There are a very large number of interdependent factors associated with technical development projects so that any generalizations I might make are tenuous. I'll proceed out onto this thin ice in spite of that.

Broadly speaking and assuming an accomplished development department that has a good set of project management practices in place, very little can be completed in less than 18 months when working from the proverbial “clean sheet of paper”. Then, unless the technical approach being attempted is totally new to everyone involved, which is not common, almost any project can be completed in about 24 months. Clearly, the definition of “completed” needs to be clarified.⁷

Some companies sometimes seem to put out new products every 3–6 months, which is a most impressive time frame. If one looks closely, though, what often is found is a “waterfall” product development pipeline with sequences of developments completing on a staggered time schedule. No single product development takes only 3–6 months; but products of different sorts and complexities emerge with that frequency.

A final comment on high tech product development projects, and something that's familiar to most successful project leaders, is what I call the “alligator in the swamp problem;” you don't see an alligator until it's too late to do much about it. No matter how well run a project; no matter how well the organization comes together to get the new product out the door cost effectively and on time, there will always be some sort of significant glitch in the process that comes close to upsetting everything.

This is why project leaders are always instructed to know what their fall back plans are, and what the fall back plan to the fall back plan is. The point is what Taleb calls “optionality”.⁸ It's not that one tries to know everything, but rather to provide for a surfeit of rational choices.

4. Leadership in technology development groups

My last major topical area is on leadership in the corporate environment, specifically among the people in technology development groups. In such a group, who is the leader? In such a group is the leader the one with the most prominent technical ideas? Is the leader the one who is obviously the genius of the group? Is the leader the one with the most product development successes? Is the leader the one who was appointed by the company management? The question, “Who is the leader” prompts the question, “What is meant by leadership?”

Volumes and volumes have been written on and about this question, and it's not my intention to add yet more to that discourse. However, I will touch on the dilemma of identifying leadership inside a high tech development organization. The problem is that there may be different kinds of leaders: administrative leaders, technology leaders, intellectual leaders, managerial leaders, etc. So who or what is the technology development organization supposed to take its lead from?

Without characterizing any particular kind of leadership, it is there to be taken if one is bold enough. Leadership in this environment requires the identification of situations that need decisions to be made and being courageous enough, and appropriately vocal, to call attention to the decisions that could be made. These decisions may be cast in any skill domain of one's choosing: for example, business, scientific, personnel, financial. What the organization requires is someone to convince it of a route to action, or at least to cause it to seriously consider the route proposed.

Too frequently, there is no one willing to risk challenging a developing direction of action that a project or program may be taking. This becomes a worse situation when the directive is mandated externally to the project or program. Leadership in a development organization requires challenging the internal status quo, to take risks, to show how to do things differently, to be impatient with “that's the way it's done around here,” or “that's how long things always take”.

Leaders are the ones who take care of the people. They trust people implicitly. They give them real responsibility early and often. They facilitate people so they can get their work done. Instead of directing anyone, they show them the way. A leader is one who knows how to and then does get the group to get things done, even if they are not the ostensible head of the group.

The successful product development groups are the ones who understand that all the different types of leaders have their roles to play in the combined efforts to drive the development to a cost effective, timely and properly performing outcome.

Final comments

Professional life inside a successful technology development organization is an intense existence, requiring a substantial amount more of an individual than just one's technical wherewithal. I offer a few comments from that perspective.

1. To the extent that you can, work with the best people. I particularly enjoy the recent comment, “Keeping one's distance from an ignorant person is equivalent to keeping company with a wise man”.⁹ Importantly, never stop learning from the wise ones.

2. Do not get comfortable. Comfortable situations are dangerous situations. As my first Navy ship captain always admonished us junior officers, “if everything seems quiet and in control, you've missed something!”¹⁰ Change (volatility) will always happen and one has to have a plan to work with it when it happens. Stay on your toes and have options.

3. Lean towards risk; have a bias toward risk. Go for challenging problems, the more challenging the better. The higher the risk, the higher the reward. Go for big hits and try to strike out early. Fail fast, so you learn faster.

4. Work within institutions. Unless one is independently wealthy, it will be necessary to work within someone's existing infrastructure. Learn the infrastructure within which you work, and then optimize its usefulness to your own productivity.¹¹

5. Leverage the resources of the organization. Don't threaten them with too much too new too quickly. That's not being ambitious and aggressive; that's being obstructionist. What one needs to do is challenge an organization to be as good as it's supposed to be with what it's got.

6. Trust your people and your coworkers implicitly. Give them responsibility early and often. Facilitate your good people. Provide challenging work and then get out of their way! Avoid directing anyone: lead them.

7. The rate at which you are becoming adept at new skills is the rate of progress of your career. A successful technology developer is someone who is always learning, building himself or herself into a multi-talented contributor.

8. You can often accomplish more if you don't work so hard. The point is to focus on what's important. “Busy-ness” is not often productive work. It's the quality of work, not hours of work that counts.

9. Nobody individually is indispensable to a corporation. Take your vacation time! Rarely is it heard from someone's deathbed, “I wish I had worked more”.

10. Knowing this, be sure to have a life outside your professional career. When recruiting, I always looked first at the bottom of someone's resume for what they did other than work. If I was looking at the resume at all then it meant the requisite skills to do the job were probably there. Then, what else was this person about? Whatever it is, have something besides your profession to go to. To paraphrase a comment made by General Colin Powell: “Find your train and drive it!”¹²

Our successes in technology development are not frequently the result of isolated, individual effort. It is always we. We are not alone as we strive for success. I am thankful for having had the opportunities to work with so many exceptionally talented technology developers who helped me find a successful career in analytical instrumentation product development.

References

1. L. deGalan, J. Anal. At. Spectrom., 2012, 27, 1173.
2. Private communication, R. D. Condon, President, SpectraMetrics, Inc., March 1983.
3. N. Taleb, Antifragile: Things That Gain from Disorder, Random House, 2012, p. 41: “How do you innovate? First, try to get into trouble. I mean serious, but not terminal, trouble”.
4. O. R. Butler and R. J. Anderson, Phys. Today, December 2012, 39–45.
5. L. Vincent, Vincent & Associates, Inc., Innovating Perspectives, Nov 2003, “The Corporate Autoimmunity Concept”.
6. J. Bussey, Wall Street Journal, February 26, 2013, page B11. “In Search of the Spark... and The Next Big Thing”.
7. Note: All of this generalization about time for product development is heavily dependent on whether the development is for something evolutionary or something revolutionary. Further, it's dependent on the ratio of hardware to software content involved. Other major factors can include new manufacturing processes needed and/or new sales structures to put in place.
8. N. Taleb, Antifragile: Things That Gain from Disorder, Random House, 2012, p. 261.
9. N. Taleb, Antifragile: Things That Gain from Disorder, Random House, 2012, p. 305.
10. Private communication: Anglim, D.F., Captain, USN, Commanding Officer, USS Johnston(DD821), 1970.
11. Note: There is one version of technology developer called a “maverick”. These are the seemingly, or probably, brilliant developers who don't fit into the infrastructure, but whose activities certainly keep its atmosphere exciting. While it is contended that mavericks have their place in technology development, it is my observation that they don't often show more developmental productivity than the organizational uncertainty, and consequent lack of productivity, they generate. They rattle the infrastructure enough to distract the productivity of others, and as a result lose their effectiveness.
12. C. Powell, It Worked for Me: In Life and Leadership, HarperCollins, 2012.

---