Data being the heart of your digitalization – how does it unleash yours and your company's full potential?

• Innovation - Why and how?
  
• What does "Data" have to do with it?
  
• Data & Analytics (D&A) - What do I need to understand?
  
• What do I have to do to become truly data-driven?
• What should I aim for?
  
• What do I need to get right?
• Top management anchoring and sponsorship
• D&A Strategy clear part of your Business
• Organization primed for D&A leverage
• Platform primed for D&A leverage
• Process and mindset
  
• The main barriers of becoming truly data-driven
• Lack of top management sponsorship and involvement
• Organization not fit for data excellence - misaligned visions
• Organization not fit for data excellence - not working together
• Excel used as primary BI platform
• Low data quality
• Too much waste - spend too much time on non-value creation

Innovation - Why and how?

In the fierce competition for the customer’s attention, you as a service provider need not only be relevant but stay relevant over time. You need to be obsessed by the customer experience. You will always need to understand in what kind of state your customer is when you meet her to carve out what kind of services would be of value for her, given her situation right there and then. This is what we call innovation, and clearly the companies that manage to innovate at speed over time are primed for tremendous success.

However, there’s a catch – innovation is hard, and to do it over time is exponentially as hard. It is about trying to foresee the future, where the only thing we know for certain is that we do not have a clue about anything...more than that everything will change. You, your organization, your competitors, and the market, will all change. So, basically it comes down to making qualified guesses and simply trying them out. Knowing that, either you learn how to do things or how not to, but most importantly it will bring you, as a company, to a new state of knowledge and experience. From this new state, you can then find opportunities of new leaps that you didn’t even realize nor understand earlier. That is how innovation works, constant rejuvenation, constantly adding to what you have to evolve.

Given this, it is crucial to understand that it’s in the nature of innovation to actually fail, now and then. It is an important and inevitable part, but still, as caring people, failure might both hurt and mislead us. Too many times have I seen the pain of failure and the stress it might cause. How it paralyses an organization, a group of people and their culture, that are not standing strong in their values, missions and belief in that you’re stronger as a group than as individuals. Painful failures unfortunately tend to make us lose our confidence, as people and hence organizations. We become risk averse and less agile which easily introduces cultural plagues such as blame game and individualism. Gradually our company grows colder and darker. In attempts to get back on track we tend to do our biggest mistake of all, stop believing in our own people by introducing a top-driven hierarchy and micro management. This in turn takes out the last spark of hope - the motivation, passion and engagement of the co-workers fades and further speeds on the slippery slope towards death and averageness. So please beware!

Innovation at speed is about “doing the right things” and “do things right” ie. make the right priorities and deliver initiatives as fast and efficiently as possible. And when you fail, you fail fast, analyse what went wrong and go on with new experiences gained. Constant rejuvenation. I can calmly guarantee you that if you succeed with this over time and manage to scale it, you will be driving or disrupting whatever industry you are in.

 

What does “data” have to do with it?

Today companies are facing far more complex decisions than ever before and we are all quite certain the trend will continue going forward.

Competence – the ability to compete - has gone from pre industrialism being about Energy and Matter driving Generalism being Stationary and working in the Collective, to during the industrial evolution be about Knowledge and Function, driving Specialization through Planning and working Individually, to now, in the data revolution with the gathered knowledge of the world in your pocket, be about Perception and Situation driving Complexity being Agile and work in the Community. The boundaries between traditional industries has hence gradually erased and today you often find your products in several traditional industries, due to the fact that we’re now focusing on customers Perception and Situation, the full experience. A gas station doesn’t just sell gas anymore, it tries to help its customer in whatever situation she or he might be in.  

So, how would you propose making those more and more complex decisions? Would you like your organization to base them on biased opinions, feelings and guesses or would you like to base them on non-biased facts and insights?
Which is the more productive discussion, the one based on different beliefs and opinions or the one based on factful insights?

Putting out the question like this it is easily answered. But in fact,

a majority of companies today are still basing their decisions and priorities on opinions rather than on actual data insights

With this said it comes as no surprise that the companies that do manage to base decisions and priorities on factful data insights have a noticeably clear edge when it comes to innovation at speed, ie delivering better services faster and more efficiently, and hence are most likely having more fun. Where would you like to work?

In the pursuit of digitalization and becoming truly data-driven you must not limit yourself to making only strategic decisions based on data insights. There is such a competitive edge of becoming data-driven all the way, from your high-level strategic priorities, through your tactics and all the way into your operations and your products. How do I predict and stop fraudulent behavior at near-time? How do I optimize costly maintenance work to avoid expensive break-downs or tragic accidents? How do I optimally adjust the need of airport ground personnel to demand fluctuations? How do I engage my customers by identifying opportunities of up-sell or possibility to turn churn? Or, how do I get a clear overview of my full business based on aggregated logs and customer analytics?

These are all operational insights we could only dream of a couple of years ago but have now become reality for truly data-driven companies.

And given the rapid pace of digitalization and the exponential growth of data, today is the time to prepare for the future. It’s time to unleash your company’s and customer’s full potential.

Data and Analytics (D&A) - what do I need to understand?

Data and analytics (D&A) is the broad term we use to simply name the wide area of everything we do to gain value out of our data, ie turn raw data into useful business insights. Given the importance of D&A for more or less every company today, it is important to understand just how big and diverse the area really is. 

Competence Areas

There are four high-level key competences that need to work together to make your D&A Strategy come true, preferably in focused cross-functional product teams owning a specific business area end-to-end. 

Data Engineers – aims to implement and manage the infrastructure for data, D&A pipelines, and D&A services. 

Data Scientists- aims to make sense of all the messy raw data (unstructured, semi-structured or structured) from various data sources to make predictions about the future by using advanced data techniques and modeling, such as machine learning (ML) and artificial intelligence (AI).

Data Analysts - typically works with structured data to solve and visualize business problems by looking into the past and present, ie. business intelligence (BI).

Product Development - focuses on developing valuable products and services for the customers. 

D&A Platform

Your D&A platform is the technical infrastructure supporting the complete D&A domain. On the market there is a huge ecosystem of platforms, products and tooling as part of what is often referred as the “modern data stack”, the cloud-based way to support the modern D&A vision and needs. From a functional perspective the landscape or data tech stack is often broken down into the following tiers.

Data Movement – Retrieving and ingesting relevant data from any relevant source, may it be your own back-end or SaaS services, log files, documents, social media, IoT or sensor streams, videos, images or any other external data.

Data management infrastructure – the storage and management of data. Storage can be in any form from relational- or non-relational databases to Data Warehouses or Data Lakes. With the management part we refer to capabilities such as security and access control, cataloguing and data preparation (cleaning, transformation, augmentation and/or aggregation). 

Analytics – how to use and analyze your data.

Data visualization and customer engagement – your end user engagement in form of reports and dashboards or capabilities for exploratory self-service data querying, machine learning training and predictions to mention a few.

 

What do I have to do to become truly data-driven?

What should I aim for?

Get your organization “data literate” 

The meaning of “Data literacy” is to fluently be able to find, read, understand and utilize any data from any data source. I would say that getting the bigger part of your organization, not only your analysts and data scientists, “data literate” gives you the highest level of data-driven readiness possible. Since innovation is something everyone should be part of, everyone should be able to exploratory aggregate data to get valuable insight. 

In the organization with a strong culture of “data literacy” data will be a natural part of everything you do, say and think.

To gain the real leverage of data you must re-think and re-design your business models and products. Unfortunately, a face-lift is not enough. Your data analysis will then guide you in further development and innovation, instead of biased guesses and opinions. When you innovate, when you try out new products, features or playing with UI/UX, you do it in a data-driven manner. Within short your A/B tests will show you the result, in black and white, what improves your defined KPIs and what doesn't. Fail fast and re-iterate! Constant rejuvenation.

The more data you get, the more sophisticated you become in your analysis, the better your products will perform, given they are designed to use and leverage on data. Imagine the user experience you can provide when you truly understand your customer and its need for insights and guidance at the right moment. Imagine what you can achieve when you always have up-to-date models for making reliable predictions. Imagine the comfort of having a trustworthy and easily navigated monitoring, overviewing your complete operational landscape, in real time. 

What do I need to get right?

Defining a bold D&A vision, like becoming “data literate” and “let data become the driver of innovation and operational excellence”, is not particularly hard, but to realize it, make it come true, is everything but easy. Below please find a couple of key areas that need to be fulfilled for success. However, it should furthermore be implied that lack of fulfilling these areas are the most common reasons for failing to become truly data-driven, and hence failing to get any greater leverage of your data investments.

Top management anchoring and sponsorship

The vision of becoming truly data literate and data-driven must be deeply anchored and sponsored by the company management. D&A must be put on top of the agenda and must become a natural part of anything you say, think and do – throughout the company.

D&A Strategy clear part of your Business

Your D&A Strategy must be aligned with your overarching business strategy. It must be clear to all what your D&A vision aims to provide in terms of value proposition and stakeholder outcome. Hence, your D&A vision must be fully entangled with your product vision, they are one and the same. 

Organization primed for D&A leverage

Your organization must always mirror your business. In accordance with product vision and D&A vision being entangled, the same goes for your teams implementing the vision. Cross-functional product teams own the product vision and the aligned D&A vision. Logically, the product team owning the vision must also own the corresponding legacy in order to make those intricate trade-offs between taking bigger a leap towards the vision versus cleaning up technical debt, at every given point in time. Both areas, vision and legacy, are equally important and need to be managed in the pursuit of the final product vision. Separate them and you will find yourself with conflict of interest and a landscape diverging over time.

It hopefully comes without saying that the competence of the team, and hence team composition, needs to be based on the business area and vision owned by the team. What does the team need to become as successful as possible? Most likely, the key competence areas (“Product Development”, “Data Engineering” and “Data Science”) must all be represented somehow but the proportions depend on the nature of the business area owned.

Platform primed for D&A leverage

To get value out of your data you need a D&A platform, a data technology stack, that is fit for purpose. The technology stack provides the infrastructure, tools and products needed to support your D&A vision, in accordance with the functional tiers previously mentioned. Like any technology stack it is crucial to base it on architectural patterns that support your vision, support your urge for innovation at speed, as well as keeping it scalable, robust, secure and maintainable. All this at the lowest cost possible.

Given the bald vision of “data literacy” and “let data become the driver of innovation and operational excellence”, the base of your D&A technology stack should aim to support a couple of key capabilities. Here you need to keep in mind above all two things, 1) data - the value you can get out of your data will never become better than the underlying data itself, and 2) be agile - to become truly data-driven and innovation-driven you need a stable technical D&A base to let you efficiently and in an agile manner build, test and deploy new analytics project (in days instead of months).

• Avoid locking yourself in - Store data in open formats, avoid proprietary formats, to make the data available for any analytics engines of your choice (or several). Avoid vendor lock-in for data you want to be able to utilize elsewhere.

• Utilize cheaper storage and avoid unnecessary costly transformations - Initially store your raw data cheaply in a Data Lake, without any expensive transformations. For some data you will then want to invest in transformation (cleaning, curation, aggregation) to prepare the data for business use. To not end up with a data swamp you want to separate the quality of your data in “readiness” layers. Commonly used names are Bronz, Silver and Gold Layer, where Bronz is your raw data and Gold your well-structured datasets prepared and ready for data analysis.

• Unified and easy accessibility – aim to incapsulate all your relevant data within one global data catalogue layer. Here you typically want to include areas such as management of metadata, caching and indexing, schema validation, and transactions. By this you get the great benefit of one touchpoint to all of your data, may it be structured, semi-structured or unstructured. One touchpoint to find, query and join any data from any data source. Also you want to minimize the need of data duplication since your queries don’t need to copy nor move the actual data. Last but not least this can also provide crucial capabilities such as unified transactional management and fine-grained data access and governance, for all of your data. 

Until quite recently some of these capabilities were unheard of. But bold digital business visions like the one above have been the driver of a new architectural pattern when it comes to D&A - the “Open Lakehouse” pattern.

This pattern aims to unify and provide us the best out of the two worlds, a) the well-structured, Data Warehouses based on OLTP transactional processing for looking into history and present (BI) and b) the un-structured Data Lake, based on OLAP analytical processing for predicting the future (AI/ML).

Process and mindset

The growth of data is exponential, inside your organization as well as outside in the rest of the world. How do we make sure we get all data in a growing organization globally understandable and accessible, over time? (culture of “data literacy”)

A challenge I often experience is that central D&A teams are unaware of newly created data in product teams and vice versa product teams fail to see the global value of their data outside their own proximity.

A feasible strategy for this is of course a D&A platform that can scale with your business in terms of data growth and growth of number of data sources. But you also need an underlying mindset and process as part of you D&A strategy for the complete data lifecycle, from birth of data to customer engagement. We need to start thinking and managing data as we do our services. The vision of a business area, a product, must in parallel with its services include the data it provides and the value proposition and stakeholder outcome of this data. Data has to be seen as part of your product, part of your offering towards the rest of the company and towards customers to provide yet better services and value. It must be in the interest (and responsibility) of the product team to promote its data. By measuring the usage of data, you understand the value of it, like you do with your services. And what better way is it for a product team to promote and make its data accessible for analysis than through the D&A platform.

Hence, a product team should be responsible for its corresponding data end-to-end, from creation to correctly showing up in your D&A platform.

The main barriers of becoming truly data-driven

Below please find a list of the most common barriers for reaching your D&A vision and the harm they tend to cause, again based on my experience from companies I meet and people I talk to. 

Lack of top management sponsorship

From my own experience, but also backed up by studies, one of the most common hurdles of getting better leverage of your data is lack of top management sponsorship and involvement. With a D&A department level sponsorship only, you won’t get very far with your data vision unfortunately. You might run a few great initiatives, but you will lack any bigger impact. Sooner rather than later this will cause a lot of friction and frustration for your teams and organization, continuously ending up in conflicts about what is strategically important and hence priorities. And since the ROI of data leveraging investments fail to blow you away, in these cases, you will most likely do the mistake of decreasing rather than increasing further investments.

   

Organization not fit for data excellence – misaligned visions

Your D&A vision is misaligned with the overarching business. This will most likely be the case if you also lack top management sponsorship and involvement (see above) but can still be the case.

This is again a very frustrating anti-pattern for your organization. One part is that your teams and co-workers want to do “the right things” but due to misalignment of visions, the opinion of what “the right things” actually are might differ quite radically throughout the organization. Quite obviously this will lead to inefficiency as you will find yourself spending time on the wrong problems or solving the correct problems but in the wrong way, ie. not the optimal solution.

Organization not fit for data excellence – not working together

The other common challenge is, with or without aligned visions, that your organization is not optimized for innovation and solving problems efficiently together. Remember the four key competences (Data Engineers, Data Scientists, Data Analysts and Product Developers) that often need to work together to make sure all key perspectives are taken into consideration and are present when developing products. This to make sure you are “doing the right things” and “do things right”. If you fail to do so you end up again, in what I often see:

Engineering Teams making incorrect assumptions and spending time on made up and often wrong problems.

Data Scientists blame poor data quality or slow pipelines for not delivering according to plan.

Product Developers blame engineering teams for being too slow or for not understanding the business.

All this leads to dangerous gaps between the competences, which leads to further isolation, mistrust and frustration. And as we all know, frustration leads to further individualism and is hugely counterproductive for any company.

Excel used as primary BI platform

Excel isn’t a suitable BI platform and was never intended as one either. Excel doesn’t support crucial capabilities such as efficient collaboration (share or source control your solutions), scalability (support growing organizations and growing amount as well as complexity of data), data access & security management, or cloud data support to mention a few.

However sadly reports are stating that more than half of all companies are still using Excel as their primary BI platform.

With Excel widely spread and used as a BI platform locally by individuals or smaller groups of people, it implies that data models (business logic) and insights are also developed and managed locally and not easily shared. This will most likely result in misalignment when insights are based on different local sources and inefficiency due to lack of possible ways to collaborate. It also makes a big risk when crucial insights are dependent on one or at best a few key co-workers.
However, change is scary and resisted by many - working with your own local excel files that you have developed over the years is very efficient and comfortable for a few. It gives you a satisfying feeling to be deeply needed, in control and drives little incentive to abandon your files for a global collaborative solution.

Excel as the primary BI platform is obviously an antipattern towards innovation at speed since you can’t share nor contribute to each other's solutions in any efficient way nor can you globally find, understand and use data to grow your “data literacy”.

Low data quality

Another common challenge is poor data quality or the lack of relevant domain data altogether. Not being able to trust your data has proved an excessively big challenge for the organization that has realized the immense potential of data, for its innovation and growth. Data you can’t trust is obviously of very little use. When you need to base discussions and decisions on data you tend to end up questioning the trustworthiness of your data rather than building the future together. 

Too much waste – you spend more time on fixing problems than on creating actual business value 

All the time we spend on non-value creation is waste. Waste can be anything, such as repeatedly manual activities, cleaning up after mistakes, maintaining “low value stuff”, developing things that are not being used etc. etc.

Imagine how much time people in your organization spend on waste. Then imagine what you could achieve if more of their time would be spent on value creation - the potential is enormous.

I often hear the complaints of people within D&A in organizations that aren't fit for D&A leverage:

- "Whatever the task or feature you work on, you more often fail to finalize it than not"!

 

Typically this is because data turns out to be missing or not accessible, data quality is too low or data is in different silos or legacy systems and can’t be aggregated.

What do you think this does to your organization? I can tell you - it drives mistrust, frustration and drives away any joy and proudness. It makes creative people eventually give up and leave your organization.

Good luck with your own Data & Analytics journey!

 Discloser: this is not intended to be a fancy white paper but a rough transcript of my experience and thoughts out in the field – sharing is caring.