With a data technology landscape that is ever changing, it is at times a challenge even for technology firms to sort through the latest key terms and platforms that pervade the marketplace, each touting its advantage over the status quo. How much more a challenge, then, for organizations where technology is not their primary expertise? From a myriad of choices to build a modern data architecture for a competitive edge, what must companies buy and leverage to start or pivot their data journeys? 

This primer is intended to educate a growing number of clients we see in the Small and Medium-Size Business (SMB) segment who are looking for a good framework to consider as they think through their future data capabilities.

So what makes a business intelligence (BI) or data architecture? And what do we mean when we say less is more as it relates to this architecture? Regardless of the technology stack that eventually gets approved and implemented, our team at One Six likes to speak of tools and platforms in terms of four big buckets:

Data Acquisition

What is it? This bucket is the piece of the architecture where we consider the movement of data from source systems to a centralized location for holistic analytics.

Why less is more: There are products in this space today that frankly provide faster and robust capabilities for clients than connectors custom built by consulting firms or in-house developers. With a few clicks of a button, platforms such as Fivetran or Matillion’s Data Loader provide full data replication of a wide array of applications (e.g., Salesforce, Google Ads) and databases (e.g., SQL Server, MongoDB) into a cloud data warehouse. Fivetran, in particular, detects changes to the source system automatically and moves data over seamlessly. Subscribing to managed services by leaders in this space means less maintenance activities for your internal IT group and more time to focus on value-added tasks downstream.

Data Warehouse

What is it? Whether you hear terms like data lake, data warehouse, or data lakehouse, here we are discussing a part of the architecture that stores information from disparate systems for historical, current, or forecasted analyses.

Why less is more: Rather than trying to estimate the right-sized on-premise hardware and software that is required to house all current and future data, consider a cloud data warehouse (CDW) that scales storage and computing power up and down as you need within minutes. Unless you are an organization at the scale of Netflix or Airbnb, a complex architecture is not necessary. A simple yet powerful cloud data warehouse like Snowflake provides a fully managed, pay-as-you-go service that is secure and costs are based only on the storage and compute you use. Again, less burden on your internal team, and more flexibility to build the structures you need to analyze your data efficiently. Plus, as your company grows and data volume grows as a result, the CDW can grow with you.

Data Transformation

What is it? When it comes to data, there have always been and will still be a need to clean, wrangle, and structure data in a way that makes sense for reporting. This bucket relates to the architecture piece where this transformation takes place.

Why less is more: Putting this as the third bucket is intentional. In the old world, a separate staging server was required to process data transformations prior to loading into a data warehouse. In addition, not all data from the source moved to the data warehouse due to cost of storage and compute. Today, because of the cost effectiveness and power of the CDW, data transformations can happen after the data is fully replicated from the source to the warehouse on a repeatable schedule by leveraging tools like Matillion. This means the transformations happen in the data warehouse itself so you can remove the maintenance and costs of a staging server. Even more, the low cost of storage allows the data warehouse to have all data from the source system immediately available for any changes to reporting needs.

Data Analytics

What is it? This bucket is the most visible component to a BI or data architecture. Here we are looking at product capabilities for building standardized dashboards as well as ad-hoc analyses that will be consumed by a wide audience in an organization.

Why less is more: Whether it is Tableau, Power BI, Qlik Sense, or any number of tools in this space, cloud offerings provide a fully managed instance that provides a browser-based, single access point for dashboards and reports. Less management and increased accessibility. In general, while each tool has its advantages, we continue to see tools moving towards parity. Here, a data governance strategy is key is reducing the number of data sets and reports while promoting the reusability of well-structured, certified data sets. This reusability increases the trust of the data and also empowers business users to find answers to questions on their own not found in standardized dashboards. And with tools like Ki and DataRobot that can augment established BI tools with its artificial intelligence and machine learning capabilities, we see increasing ways for analysts to solve business problems that we can provide guidance on.

Final Thoughts

So there you have it – Data Acquisition, Data Warehouse, Data Transformation, Data Analytics – four big buckets to easily see what you need to consider in a modern BI architecture. We hope that this modern BI primer has been helpful as you take the next steps into your organization’s data journey. Please note that the above technology platforms listed were used as examples only. The team here at One Six Solutions works with a wide array of technologies in the data world. Our goal is to design and build an architecture that works best for you based on your industry and business needs. Let us know how we can help.