{"id":800,"date":"2019-02-13T14:00:43","date_gmt":"2019-02-13T22:00:43","guid":{"rendered":"http:\/\/www.lightsondata.com\/?p=800"},"modified":"2019-02-18T13:13:52","modified_gmt":"2019-02-18T21:13:52","slug":"why-focus-reduce-variation","status":"publish","type":"post","link":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/","title":{"rendered":"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d"},"content":{"rendered":"

One of my favourite quality stories is when, just as W. Edwards Deming was about to begin his 4-day seminar on improving quality at a large enterprise, one of the executives scheduled to take the course came up to Deming and said that he was so busy that week that he could not possibly spend 4 days at the seminar. So, he asked Deming to distill the main message of the seminar to a few words so that the executive didn\u2019t have to \u2018waste\u2019 4 days. In the story, Deming smiled and said, \u201cYou should focus on reducing variation\u201d<\/a>.<\/p>\n

\"focus<\/p>\n

The amazing thing to me about this story is that those 2 words do such a great job of distilling the ideas behind what Deming called ‘profound knowledge<\/a>‘ and today that ‘reduce variation<\/em>‘ advice is still as applicable to the manufacture of cars on an assembly line as it is to the creation and management of policyholder records in an insurance company\u2019s database. Specifically, Deming would tell the car manufacturer to reduce variations in all the parts and assembly processes that go into making a car that customers would want to buy, just as he would tell a life insurance executive to measure, analyze, improve and control the data and processes that are involved in the creation of a life insurance policy record in a database.<\/p>\n

When I teach my 6-week course on Data Quality Improvement at BCIT<\/a>, I try to get the students thinking about how the processes involved in the production of data can be continuously improved. We\u2019ve only got 6 weeks so the focus is on teaching them how to use a simple data profiling<\/a> tool to quickly expose possible data quality issues and evolve the DQ issues into Data Quality Rules<\/a> (DQRs) using simple Excel charts such as Statistical Process Control<\/a> and Process Behaviour Charts<\/a> to show how the data quality issues<\/a> vary over time.<\/p>\n

\"mask
Patterns in credit card numbers<\/figcaption><\/figure>\n

The Data Profile of a data set from a Business Process provides the characteristics of each column in the data set, as well as the ‘behaviour’ of that column\u2019s data over time. For each column in the data set, the profile illustrates basic quality characteristics such as average, variance, patterns, masks, distributions, outliers, et cetera. Using these, the DQ Analyst can prepare a short list of \u2018interesting characteristics\u2019 about the data set and ask a subject matter expert, i.e. a person familiar with the business process and its outputs, to explain the significance, and priority, of each Poor Data Quality characteristic (PDQ). For example, this image shows that the Credit Card numbers, when entered, are consistently 16 digits except for 5 incorrect patterns.<\/p>\n

With the prioritization(s) in hand, the DQ Analyst can evolve the significant DQ issues into DQ Rules by analyzing the occurrence of the DQ issue over time to calculate the average number of occurrences per period and the variation of the occurrences over time. The average and variation define the upper and lower occurrence thresholds, and when the DQ Rule Violation counts are charted over time, the graph almost magically shows what Donald Wheeler calls the ‘Voice of the Process<\/a>‘.<\/p>\n

\"DQR
Statistical Profess Control – No Address DQ Violations<\/figcaption><\/figure>\n

Here, the DQR#1723 is run against the data at the end of each month and the number of ‘no address’ DQR Violations is recorded. In this chart you can see that there has been an average of 165 customer records without an address created each month and though there was a major variation in March of 2014, overall this DQ issue is stable.<\/p>\n

\"DQR
Statistical Process Control – Monthly average of no address records<\/figcaption><\/figure>\n

Since the past is the best predictor of the future, a person wanting to understand how many customer records would be entered into the system without addresses next month could safely estimate 165 plus or minus 15. Now that they are aware, the organization can decide that the \u2018no address\u2019 issue is significant and begin to continuously improve the upstream processes to ensure customer addresses are entered. In this chart of 2015-16 we can see that the number of \u2018no address\u2019 records on average has dropped to 140 per month, and there has been a steady improvement until the last month of each year.<\/p>\n

To quote Donald Wheeler:<\/p>\n

“The characterization of a process as either predictable or unpredictable is a fundamental dichotomy for data analysis”.<\/p><\/blockquote>\n

Simply put, if the process is not predictable, how can you manage it. Once you measure the baseline variability of the dataset generated by the business process, you can go further and measure the variability of the DQ characteristics that have a significant impact on the integrity and usability of the business process information. With the added depth of baseline measurements of the \u2018noise\u2019 of the DQ Rule violations, your organization\u2019s decision making will steadily improve.<\/p>\n","protected":false},"excerpt":{"rendered":"

One of my favourite quality stories is when, just as W. Edwards Deming was about to begin his 4-day seminar on improving quality at a large enterprise, one of the executives scheduled to take the course came up to Deming and said that he was so busy that week that he could not possibly spend […]<\/p>\n","protected":false},"author":6,"featured_media":806,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[4],"tags":[42],"class_list":["post-800","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-quality","tag-data-profile","post-wrapper","thrv_wrapper"],"yoast_head":"\nDeming\u2019s advice to busy executives: \u201cReduce variation\u201d | LightsOnData<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d | LightsOnData\" \/>\n<meta property=\"og:description\" content=\"One of my favourite quality stories is when, just as W. Edwards Deming was about to begin his 4-day seminar on improving quality at a large enterprise, one of the executives scheduled to take the course came up to Deming and said that he was so busy that week that he could not possibly spend […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/\" \/>\n<meta property=\"og:site_name\" content=\"LightsOnData\" \/>\n<meta property=\"article:published_time\" content=\"2019-02-13T22:00:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-02-18T21:13:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i1.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"700\" \/>\n\t<meta property=\"og:image:height\" content=\"394\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Gordon Hamilton\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DQStudent\" \/>\n<meta name=\"twitter:site\" content=\"@georgefirican\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Gordon Hamilton\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/\",\"url\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/\",\"name\":\"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d | LightsOnData\",\"isPartOf\":{\"@id\":\"https:\/\/www.lightsondata.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1\",\"datePublished\":\"2019-02-13T22:00:43+00:00\",\"dateModified\":\"2019-02-18T21:13:52+00:00\",\"author\":{\"@id\":\"https:\/\/www.lightsondata.com\/#\/schema\/person\/a1bb9a0e977c4f03699eb222c7f6380c\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1\",\"width\":700,\"height\":394},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.lightsondata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.lightsondata.com\/#website\",\"url\":\"https:\/\/www.lightsondata.com\/\",\"name\":\"LightsOnData\",\"description\":\"Practical resources, online courses, free articles and videos for data management, data governance, data quality, and business intelligence\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.lightsondata.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.lightsondata.com\/#\/schema\/person\/a1bb9a0e977c4f03699eb222c7f6380c\",\"name\":\"Gordon Hamilton\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.lightsondata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/76939d8ad0817daf78a0772372246851c03f3925ca16b0f5710b1c8d8fed22b9?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/76939d8ad0817daf78a0772372246851c03f3925ca16b0f5710b1c8d8fed22b9?s=96&d=retro&r=g\",\"caption\":\"Gordon Hamilton\"},\"description\":\"Gordon Hamilton is an unapologetic Data Enthusiast, with years of experience integrating data quality into the data warehouse development process, modeling dimensional and 3NF data models, leading data migrations, analyzing data and helping customers find the signal in the noise. In his downtime, Gordon teaches Data Quality Improvement at BCIT, helps his former students ease Data Governance into their organizations and supports the new DAMA Vancouver Chapter. GHamilton@DataQuality.ca\",\"sameAs\":[\"https:\/\/x.com\/DQStudent\"],\"url\":\"https:\/\/www.lightsondata.com\/author\/gordonwhamilton\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d | LightsOnData","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/","og_locale":"en_US","og_type":"article","og_title":"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d | LightsOnData","og_description":"One of my favourite quality stories is when, just as W. Edwards Deming was about to begin his 4-day seminar on improving quality at a large enterprise, one of the executives scheduled to take the course came up to Deming and said that he was so busy that week that he could not possibly spend […]","og_url":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/","og_site_name":"LightsOnData","article_published_time":"2019-02-13T22:00:43+00:00","article_modified_time":"2019-02-18T21:13:52+00:00","og_image":[{"width":700,"height":394,"url":"https:\/\/i1.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1","type":"image\/jpeg"}],"author":"Gordon Hamilton","twitter_card":"summary_large_image","twitter_creator":"@DQStudent","twitter_site":"@georgefirican","twitter_misc":{"Written by":"Gordon Hamilton","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/","url":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/","name":"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d | LightsOnData","isPartOf":{"@id":"https:\/\/www.lightsondata.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#primaryimage"},"image":{"@id":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1","datePublished":"2019-02-13T22:00:43+00:00","dateModified":"2019-02-18T21:13:52+00:00","author":{"@id":"https:\/\/www.lightsondata.com\/#\/schema\/person\/a1bb9a0e977c4f03699eb222c7f6380c"},"breadcrumb":{"@id":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#primaryimage","url":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1","width":700,"height":394},{"@type":"BreadcrumbList","@id":"https:\/\/www.lightsondata.com\/why-focus-reduce-variation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.lightsondata.com\/"},{"@type":"ListItem","position":2,"name":"Deming\u2019s advice to busy executives: \u201cReduce variation\u201d"}]},{"@type":"WebSite","@id":"https:\/\/www.lightsondata.com\/#website","url":"https:\/\/www.lightsondata.com\/","name":"LightsOnData","description":"Practical resources, online courses, free articles and videos for data management, data governance, data quality, and business intelligence","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lightsondata.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.lightsondata.com\/#\/schema\/person\/a1bb9a0e977c4f03699eb222c7f6380c","name":"Gordon Hamilton","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lightsondata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/76939d8ad0817daf78a0772372246851c03f3925ca16b0f5710b1c8d8fed22b9?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/76939d8ad0817daf78a0772372246851c03f3925ca16b0f5710b1c8d8fed22b9?s=96&d=retro&r=g","caption":"Gordon Hamilton"},"description":"Gordon Hamilton is an unapologetic Data Enthusiast, with years of experience integrating data quality into the data warehouse development process, modeling dimensional and 3NF data models, leading data migrations, analyzing data and helping customers find the signal in the noise. In his downtime, Gordon teaches Data Quality Improvement at BCIT, helps his former students ease Data Governance into their organizations and supports the new DAMA Vancouver Chapter. GHamilton@DataQuality.ca","sameAs":["https:\/\/x.com\/DQStudent"],"url":"https:\/\/www.lightsondata.com\/author\/gordonwhamilton\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2019\/01\/reduce-variation.jpg?fit=700%2C394&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p9BPV6-cU","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts\/800","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/comments?post=800"}],"version-history":[{"count":4,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts\/800\/revisions"}],"predecessor-version":[{"id":825,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts\/800\/revisions\/825"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/media\/806"}],"wp:attachment":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/media?parent=800"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/categories?post=800"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/tags?post=800"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}