What Exactly is External Data?
My company has been talking a lot about our products because there has been some incredible progress on what were delivering to the organizations we work with. From tip to tail, were changing the way users on our platform access, enrich, manage, and monitor the data flowing through their environment. The way we see it, its not as easy to pin down as most assume. But theres a problem with that. The problem is, you dont use data as an organization  individuals or distinct business units within that organization use the data. Meaning, unless your company has perfect standards across the board, the way one person or department structures data wont be exactly the same as another. Theres a vast spectrum of differences in the way people and machines record labels, addresses, business names, dates, descriptions, and anything else you might find in a dataset. If not, that second dataset has to be treated like external data; it doesnt follow the same conventions, structures, and rules that the original dataset does. Theres work to be done to get the two to function harmoniously. Click to see larger. The farther you get from home-grown data, the less control you have and the greater the opportunity for variability. Taking all that into account, we say that any data created outside of your four walls is external (although, we came up with that before most people were working from home, so youll have to grant us a bit of leeway on that). If it doesnt follow the exact same standards and methods laid out by a direct mandate, it is external data. The call external data is coming from inside the house  Tying together an organizations data ecosystem is crucial to becoming data-driven, and for aligning your data strategies with your business strategies. Its also the data thats most predictable for your team, and, with a coordinated effort, it can be completely under the control of the company itself. Furthermore, by being able to analyze all the data under your own roof, you get the benefit of better process standards, deeper knowledge of your customer/client base, and hopefully a basis for connecting your whole team around data. Knowing where to start isnt easy. In fact, weve got a primer on the subject of conducting a data audit within your company, and its a process with a lot of moving pieces  you wont hear us say otherwise. But its critical to establish a baseline: it sets a precedent for new data; it reveals where there are gaps and overlaps in your data strategy. It also enforces getting mechanisms in place to see the use and movement of data between every department within your organization. Powerful stuff. Open, Public, and Premium Data  Even if you havent fully solved the puzzle that is internal external data, there is an entire universe of data available to anybody who wants it. Whats more, there are far better ways of getting it than using your highly-skilled team to set up and manage individual scripts for data gathering, not to mention the prep that goes into loading it into your system and transforming it to meet your required schema. All are required steps before you can even start using it. Governments, NGOs, private enterprise, and special interest groups release tremendous amounts of data daily. That data requires prep, cleansing, and transformation, but as the saying goes, theres gold in them thar hills! Weve known that since day one at our company. Our specialty is connecting our users to any data from any source. We handle the heavy lifting so that your data teams produce better insights faster. As we initially established, these third-party providers are sources of external data, and even though it may not be specific to your own internal data, its as or more valuable. Being Mindful of Data Tunnel Vision  Without getting into the semantics of known-unknowns, unknown-knows, and so on, there is a significant risk to using data thats too focused, segmented, or specific. Analyzing this kind of data produces skewed results, and its easy to get stuck with only your known-knowns. Its important to know these things, but its only a small corner of the whole picture. When COVID-19 hit, companies everywhere had to take a long, hard look at their supply chains to see where the weak links were, then try to pivot and adapt to maintain business continuity. Without a picture of the entire ecosystem, it was impossible to understand what else is out there. External data offers insights not just about where shortcomings are, but where opportunities lie. If the only thing your organization is looking at is whats in your immediate scope, youll never be able to get truly predictive. Youll never see the wider market shifts that are happening constantly. Why External Data Means Richer Insights  On the flip side of the argument above, tapping into broader signals will widen and enhance your insights, even when comparing against more granular data. By looking at the information ecosystem as a whole, analysts and data scientists are able to see more than outcomes. Theyre able to find trends and causes for shifts, gains, losses, and changes. This deeper, broader knowledge allows organizations to get predictive and unlocks a competitive advantage. Lets draw an example: we need the true value of a house. We can look at the historical sold price, but that might be years out of date. We can multiply that price by inflation and a national housing price increase factor to get something a little closer, but not perfect. Certainly, any renovations to the house and grounds should be accounted for. Now, lets start layering in external data:  crime statistics in the region will tell you about the area. demographics may indicate more houses in this area will be up for sale in less than 5 years. building permits within a small radius give you a sense of how the neighborhood is changing. historical weather patterns give a picture of the viability of solar panels, if renewable energy sources are a factor. new home developments in the area are indicators for growth in population and businesses in the next 10 years; and  business insolvencies in the region can tell you if the area is thriving or in decline. This is far more complete, even though its not information about that specific house. Its also all data that can be leveraged immediately if youve figured out a scalable way to get this external data into your ecosystem. If you havent, youll be limited to your known-knowns, and will be immediately outstripped by someone who can move faster and deliver better, fulsome insights. Taking a Ground-up Approach to External Data  If you want to get serious about bringing in external data, start the right way. Have a scalable method for bringing new data assets in. Find reliable partners and data marketplaces to discover new sources, and ensure your entire organization has a way to collaborate and rally around a single source of truth for data, even if different departments use it for different purposes. Centralizing teams around data assets is critical for success  becoming data-driven means your business revolves around the data, not the other way around.