… or alternatively, five ways not to. Quite by chance, I recently came upon a paper by Neil H. Spencer and Lindsey Kevan de Lopez at the Statistical Services and Consultancy Unit of the University of Hertfordshire Business School entitled “Item-by-item sampling for promotional purposes”.

The abstract declares the aim of the paper.

In this paper we present a method for sampling items that are checked on a pass/fail basis, with a view to a claim being made about the success/failure rate for the purposes of promoting a company’s product/service.

The sort of statements the authors want to validate occur where all items outside some specification are classed as defective. I would hope that most organisations would want to protect the customer from defects like these but the authors of the paper seem to want to predicate their promotion on the defects’ escape. The statements are of the type:

There is a 95% probability that the true proportion of trains delayed by more than 5 minutes is less than 5%.

— or:

There is a 95% probability that the true proportion of widgets with diameter more than 1% from nominal is less than 5%.

I can see five reasons why you really shouldn’t try to promote your business with statements like this.

1. Telling your customers that your products are defective

Or to put it another way “Some of our products are defective. You might get one.” This might be a true statement at your current level of quality maturity but it is not something to shout at customers. All these statements do is to germinate doubt about your product in the customer’s mind. Customers want products and services that simply perform. Making customers think that they might not will be a turn off.

Customers will not think it okay to end up with a defective item or outcome. They will not put it down just to the “luck of the draw”. The products will come back but the customers won’t.

If you are lucky then your customer won’t even understand this type of promotional statement. There are just too many percentages. But they might remember the word “defect”.

2. Tolerating defects

Or to put it another way “Some of our products are defective and we don’t care.” Quoting the 5% defective with pride suggests that the producer thinks it okay to make and sell defects. In the 1980s Japanese motor manufacturers such as Toyota seized market share by producing reliable vehicles and using that as a basis for marketing and creating a reputation for quality.

Any competitive market is destined to go that way eventually. Paradoxically, what Toyota and others discovered is that the things you have to do to make things more reliable are the same things that reduce costs. Low price and high quality goods and services have an inbuilt advantage in penetrating markets.

3. Saying nothing about the product the customer is considering buying

The telling phrase is “true proportion of” trains/ widgets. As a matter of strict statistical technicality, Spencer and de Lopez don’t describe any “method for sampling” at all. They only describe a method of calculating sample size, worked out using Bayes’ theorem. Because they use the word “true”, it can only be that they were presuming what W Edwards Deming called an enumerative study, a characterisation of a particular sampling frame that yields information only about that frame. They took a particular slice of output and sampled that. Such a study is incapable of saying anything about future widgets or trains.

Put another way, “When we looked at a slice of our products we’re pretty sure that no more than 5% were defective. We don’t care. As to future products, we don’t know. Yours may be defective.”

I think we need a name for soi-disant Bayesians who chronically fail to address issues of exchangeability (stability and predictability).

4. Throwing away most of the valuable information on your product

Looking at the train example, “5% more than 5 minutes late” may mean:

• “5% were 6 minutes late, the rest were 4 minutes late”; or
• “4% were one hour late, 1% were cancelled and the rest were on time”; or
• …

These various scenarios will have wholly different psychological and practical impacts on customers. Customers care which will happen to them.

Further, where we actually measure delay in minutes or diameter in millimetres, that is data that can be used to improve the business process and, with diligence, achieve the sort of excellence where quality failures and defects simply do not happen. That then provides the sort of consumer experience of satisfaction that can be developed into a public reputation for quality, performance and cost. That in turn supports promotional statements that will chime with customer aspirations and build business. Simply checking on a pass/ fail basis is an inadequate foundation for such improvement.

5. Managing by specifications

This is the subtlest point to turn your attention towards once everything is within specification. In the train example, the customer wants the train to be on time. Every deviation either side of that results in customer dissatisfaction. It also results in practical operating and timetable problems and costs for the railway. In the 1960s, Japanese statistician Genechi Taguchi put forward the idea that such losses should be recognised and form the basis of measuring improvement. The Taguchi loss function captures the idea that losses start with every departure from nominal and then start to escalate.

That leads to the practical insight that the improvement objective of any business process is “on target, minimum variation”.

What the Spencer-de Lopez statements ultimately say is that the vendor is willing to tolerate any train being 5 minutes late and 5% of trains being delayed even longer than that, perhaps indefinitely. Whether even that depressing standard is achieved in the future, who knows? Perhaps the customer will be lucky.

I fear that such statements will not promote your business. What will promote your business is using measurement to establish, maintain and improve process capability. That will provide the sort of excellent customer experience that can be mapped, promoted and fed back into confident, data based marketing campaigns aimed at enhancing reputation. Reputation supports talent recruitment and fosters a virtuous circle of excellence. This is what reputation management is about.

I do note that Spencer and de Lopez protest that this is only a working paper but it has been on their website since mid-2012 so I presume they are now owning the contents.

Just as a final caveat I think I should point out that the capability indices C_p and C_pk, though useful, do not measure Taguchi loss. That is the topic for another blog.