{"id":3556,"date":"2021-11-10T07:51:00","date_gmt":"2021-11-10T15:51:00","guid":{"rendered":"https:\/\/www.lightsondata.com\/?p=3556"},"modified":"2021-11-09T02:19:21","modified_gmt":"2021-11-09T10:19:21","slug":"what-is-dark-data-a-clear-explanation","status":"publish","type":"post","link":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/","title":{"rendered":"What is dark data? (a clear explanation)"},"content":{"rendered":"<p>We&#8217;ve heard of big data or small data, but what is this concept of dark data?<\/p>\n<p>Being a Star Wars fan, my mind went straight into that lore to make an association between dark data and the dark side of the Force. Geek alert, right? Well, Star Wars is fictional, but there&#8217;s nothing fictional about dark data. So what is dark data?<\/p>\n<h2 id=\"t-1636448626038\">What is dark data?<\/h2>\n<p>There are actually a couple of views on what dark data is. Let&#8217;s go over the first:<\/p>\n<h3 style=\"text-align: center;\" id=\"t-1636448626039\">Dark data definition<\/h3>\n<p style=\"text-align: center;\">Dark data is data which an organization acquires through various processes and stores during regular business activities, but is not used for, in any manner, to derive insights or decisions or monetization.<\/p>\n<h2 id=\"t-1636448626040\">Examples of dark data<\/h2>\n<p>I don&#8217;t know about you, but I understand things a lot better when I&#8217;m being given examples. The way I see dark data is like all of the photos on your phone. Most of them will never be used or even viewed again, but they are there. According to <a href=\"https:\/\/gigaom.com\/2015\/01\/23\/personal-photos-videos-user-generated-content-statistics\" target=\"_blank\" rel=\"nofollow noopener\" style=\"outline: none;\">Gigaom<\/a>, the average person has 630 photos stored on their phone. And this was in 2015 so you can bet that number has increased considerably. Back in 2017, InfoTrends estimated that there were over 1.2 trillion photos being taken every day.What about some examples from the business side? Well, a good example of dark data could be data generated by sensors. <a href=\"https:\/\/siliconangle.com\/2015\/10\/30\/ibm-is-at-the-forefront-of-insight-economy-ibminsight\/\" target=\"_blank\" rel=\"nofollow noopener\" style=\"outline: none;\">IBM estimates<\/a> that roughly 90% of data produced by sensors and analog-to-digital conversions never get used.<\/p>\n<p>Let&#8217;s recall that dark data represents all the information companies collect in their regular business processes, don\u2019t use, have no plans to use, but will never throw out. This includes things like:<\/p>\n<ul>\n<li>Web logs\u200b<\/li>\n<li>Visitor tracking data\u200b<\/li>\n<li>Surveillance footage\u200b<\/li>\n<li>Email correspondences from past employees\u200b<\/li>\n<li>Old versions of documents<\/li>\n<li>Raw survey data&nbsp;<\/li>\n<li>Notes or presentations\u200b<\/li>\n<li>Maybe even transactional data<\/li>\n<\/ul>\n<h2 id=\"t-1636448626042\">Dark data explained in a video<\/h2>\n<h2 id=\"t-1636448626041\">Dark data: an alternative definition<\/h2>\n<p>This was the main definition of dark data, but as I mentioned, there re a couple of views of what dark data is.&nbsp;<\/p>\n<p><span><img data-recalc-dims=\"1\" decoding=\"async\" data-attachment-id=\"3561\" data-permalink=\"https:\/\/www.lightsondata.com\/there-is-another\/\" data-orig-file=\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/there-is-another.jpg?fit=1683%2C1501&amp;ssl=1\" data-orig-size=\"1683,1501\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"there is another\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/there-is-another.jpg?fit=300%2C268&amp;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/there-is-another.jpg?fit=1024%2C913&amp;ssl=1\" alt=\"there is another\" data-id=\"3561\" width=\"565\" data-init-width=\"1683\" height=\"504\" data-init-height=\"1501\" title=\"there is another\" loading=\"lazy\" src=\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/there-is-another.jpg?resize=565%2C504&#038;ssl=1\" data-width=\"565\" data-height=\"504\"><\/span><\/p>\n<p>In this second definition, dark data refers to:<\/p>\n<blockquote><p>Any data that is collected for a specific purpose, but not used for other suitable purposes as well.<\/p><\/blockquote>\n<p>Let&#8217;s take a healthcare example. There&#8217;s a lot of data being produced and collected from our smart devices like cell phones and tablets, thermostats and humidifiers, and virtual assistants like Google and Alexa.&nbsp;<\/p>\n<p>The data collected by these devices and services are not collected for healthcare purposes. Therefore, from a healthcare&#8217;s point of view, this is dark data.<\/p>\n<p>In that sense, we as individuals create a lot of dark data. Every time we:<\/p>\n<ul>\n<li>Make an online purchase<\/li>\n<li>Use our GPS<\/li>\n<li>Use the check-in function on Facebook<\/li>\n<li>Track our calories in a phone app<\/li>\n<li>Monitor our physical activities with our smart phone<\/li>\n<li>Have our smartwatch record our biometrics<\/li>\n<\/ul>\n<p>To a hospital or a healthcare professional, for example, all this data that you&#8217;re generating is dark data because it&#8217;s not collected, nor used for the purposes of healthcare. Nevertheless, it could be used for healthcare purposes and here lies the value of dark data.<\/p>\n<p>As you can imagine, in our healthcare example, medical staff could benefit greatly from having access to your dark data as it would provide them with a more holistic view of your lifestyle. This could result in a better treatment, one that would match your lifestyle, and it could also result in more applicable prevention medicine.<\/p>\n<p><a href=\"https:\/\/www.accenture.com\/_acnmedia\/pdf-85\/accenture-western-digital-value-of-data-dark-data-hyper-personalization-in-healthcare.pdf\" target=\"_blank\" rel=\"nofollow noopener\" style=\"outline: none;\">Research from Western Digital and Accenture<\/a> found that dark data similar to the one I described can save in the US 200 million work sick days and add 200 billion USD in value across the healthcare system by 2030. And these are just the benefits of the healthcare industry tapping into dark data.<\/p>\n<p>I&#8217;m not a big fan of this definition because then anything and everything could be considered dark data.<\/p>\n<h2 id=\"t-1636448626043\">Conclusion<\/h2>\n<p>Similar to dark matter in physics, dark data often comprises most organizations\u2019 universe of information assets. Thus, organizations often retain dark data for compliance purposes or record keeping. Some organizations believe that dark data could be useful to them in the future, once they have acquired better analytic and business intelligence technology to process the information. Because storage is inexpensive, storing data is easy. So why not, but storing and securing data typically incurs more expense (and sometimes greater risk) than value. More about that in a separate article.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Dark data is data which an organization acquires through various processes and stores during regular business activities, but is not used for, in any manner, to derive insights or decisions or monetization.<\/p>\n","protected":false},"author":1,"featured_media":3565,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[20],"tags":[],"class_list":["post-3556","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-master-data-management","post-wrapper","thrv_wrapper"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is dark data? (a clear explanation) | LightsOnData<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is dark data? (a clear explanation) | LightsOnData\" \/>\n<meta property=\"og:description\" content=\"Dark data is data which an organization acquires through various processes and stores during regular business activities, but is not used for, in any manner, to derive insights or decisions or monetization.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/\" \/>\n<meta property=\"og:site_name\" content=\"LightsOnData\" \/>\n<meta property=\"article:published_time\" content=\"2021-11-10T15:51:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"George Firican\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@georgefirican\" \/>\n<meta name=\"twitter:site\" content=\"@georgefirican\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"George Firican\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/\",\"url\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/\",\"name\":\"What is dark data? (a clear explanation) | LightsOnData\",\"isPartOf\":{\"@id\":\"https:\/\/www.lightsondata.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1\",\"datePublished\":\"2021-11-10T15:51:00+00:00\",\"author\":{\"@id\":\"https:\/\/www.lightsondata.com\/#\/schema\/person\/a6c554e2c0ae016f437ab91c59d65622\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1\",\"width\":1280,\"height\":720},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.lightsondata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is dark data? (a clear explanation)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.lightsondata.com\/#website\",\"url\":\"https:\/\/www.lightsondata.com\/\",\"name\":\"LightsOnData\",\"description\":\"Practical resources, online courses, free articles and videos for data management, data governance, data quality, and business intelligence\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.lightsondata.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.lightsondata.com\/#\/schema\/person\/a6c554e2c0ae016f437ab91c59d65622\",\"name\":\"George Firican\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.lightsondata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/3ded8f815b6acadce87408824c260bc65adf043dd0eb7accc6799e576d011254?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/3ded8f815b6acadce87408824c260bc65adf043dd0eb7accc6799e576d011254?s=96&d=retro&r=g\",\"caption\":\"George Firican\"},\"description\":\"George Firican is the Director of Data Governance and Business Intelligence at the University of British Columbia, which is ranked among the top 20 public universities in the world. His passion for data led him towards award-winning program implementations in the data governance, data quality, and business intelligence fields. Due to his desire for continuous improvement and knowledge sharing, he founded LightsOnData, a website which offers free templates, definitions, best practices, articles and other useful resources to help with data governance and data management questions and challenges. He also has over twelve years of project management and business\/technical analysis experience in the higher education, fundraising, software and web development, and e-commerce industries.\",\"sameAs\":[\"https:\/\/www.lightsondata.com\",\"https:\/\/www.linkedin.com\/in\/georgefirican\/\",\"https:\/\/x.com\/georgefirican\"],\"url\":\"https:\/\/www.lightsondata.com\/author\/juni_83yahoo-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is dark data? (a clear explanation) | LightsOnData","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/","og_locale":"en_US","og_type":"article","og_title":"What is dark data? (a clear explanation) | LightsOnData","og_description":"Dark data is data which an organization acquires through various processes and stores during regular business activities, but is not used for, in any manner, to derive insights or decisions or monetization.","og_url":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/","og_site_name":"LightsOnData","article_published_time":"2021-11-10T15:51:00+00:00","og_image":[{"width":1280,"height":720,"url":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1","type":"image\/jpeg"}],"author":"George Firican","twitter_card":"summary_large_image","twitter_creator":"@georgefirican","twitter_site":"@georgefirican","twitter_misc":{"Written by":"George Firican","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/","url":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/","name":"What is dark data? (a clear explanation) | LightsOnData","isPartOf":{"@id":"https:\/\/www.lightsondata.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#primaryimage"},"image":{"@id":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1","datePublished":"2021-11-10T15:51:00+00:00","author":{"@id":"https:\/\/www.lightsondata.com\/#\/schema\/person\/a6c554e2c0ae016f437ab91c59d65622"},"breadcrumb":{"@id":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#primaryimage","url":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1","width":1280,"height":720},{"@type":"BreadcrumbList","@id":"https:\/\/www.lightsondata.com\/what-is-dark-data-a-clear-explanation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.lightsondata.com\/"},{"@type":"ListItem","position":2,"name":"What is dark data? (a clear explanation)"}]},{"@type":"WebSite","@id":"https:\/\/www.lightsondata.com\/#website","url":"https:\/\/www.lightsondata.com\/","name":"LightsOnData","description":"Practical resources, online courses, free articles and videos for data management, data governance, data quality, and business intelligence","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lightsondata.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.lightsondata.com\/#\/schema\/person\/a6c554e2c0ae016f437ab91c59d65622","name":"George Firican","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lightsondata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/3ded8f815b6acadce87408824c260bc65adf043dd0eb7accc6799e576d011254?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3ded8f815b6acadce87408824c260bc65adf043dd0eb7accc6799e576d011254?s=96&d=retro&r=g","caption":"George Firican"},"description":"George Firican is the Director of Data Governance and Business Intelligence at the University of British Columbia, which is ranked among the top 20 public universities in the world. His passion for data led him towards award-winning program implementations in the data governance, data quality, and business intelligence fields. Due to his desire for continuous improvement and knowledge sharing, he founded LightsOnData, a website which offers free templates, definitions, best practices, articles and other useful resources to help with data governance and data management questions and challenges. He also has over twelve years of project management and business\/technical analysis experience in the higher education, fundraising, software and web development, and e-commerce industries.","sameAs":["https:\/\/www.lightsondata.com","https:\/\/www.linkedin.com\/in\/georgefirican\/","https:\/\/x.com\/georgefirican"],"url":"https:\/\/www.lightsondata.com\/author\/juni_83yahoo-com\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.lightsondata.com\/wp-content\/uploads\/2021\/11\/what-is-dark-data.jpg?fit=1280%2C720&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p9BPV6-Vm","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts\/3556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/comments?post=3556"}],"version-history":[{"count":8,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts\/3556\/revisions"}],"predecessor-version":[{"id":3566,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/posts\/3556\/revisions\/3566"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/media\/3565"}],"wp:attachment":[{"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/media?parent=3556"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/categories?post=3556"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lightsondata.com\/wp-json\/wp\/v2\/tags?post=3556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}