Data: The Queen Bee of the Digital Hive (Part One)
Part 1: Why data matters
In today's Information Age, data is the lifeblood of any business application, driving decision-making and operational efficiency. In this two-part series, our Solution Architect Lewis Veale explores the critical elements of data management in business applications, including data accuracy, security, scalability, and insightful reporting.
At PDMS, we pride ourselves on the solutions we build for our clients. We understand the pivotal role of data, focusing not just on the aesthetic appeal or functionality of our solutions but on the core purpose of any business application, which is to capture, store and utilise data effectively. That's it.
It's broadly accepted that we are now living in the Information Age, and data is the fundamental unit of currency. Or another way of looking at it is like bees making honey.
Why bees?
Imagine a beehive. At the heart of this intricate ecosystem is the queen bee, a symbol of central importance. All the worker bees, tirelessly gathering nectar and pollen, exist to support the hive’s purpose: to produce honey and ensure its survival. Every action, every flight, every return is part of an organised process designed to sustain the queen and the hive.
Now, think of data as the queen bee of the hive. Just like the queen, data sits at the core of any modern business or system. All other processes – data collection, storage, analysis, and sharing – are the worker bees. They are designed to bring in inputs (raw data) and refine it, just as bees transform nectar into honey. These processes ensure that the hive (your business or system) thrives, delivering insights, optimisations, and innovations.
Without the queen, the hive would collapse; without data, decisions would be aimless, and processes would lack direction. Just as the nectar fuels the hive, the flow of information in your organisation must be streamlined, efficient, and purposeful to produce the insights (or ‘honey’) that drive growth. So how do you do that?
In part one, we'll look at how to capture your data and what to do with it once you've got it. In part two, we'll see how data management works in the real world through our experiences and how we use data to build robust applications. So let's begin!
How to capture your data
A business software solution typically offers one or more means of capturing data. These include a user interface or 'UI', an application programming interface or 'API' (for machine-to-machine input), and imports (from files or other static sources). This is where the data's journey begins. Being able to reliably capture and accurately store data is fundamental to the purpose of the business application.
The first step in the process of data input is validation:
- Validation of individual data fields
- Validation between data fields on the same entity
- Validation between entities
For example, a date of birth must be supplied; start date must be after end date; and usernames must be unique across all users.
Once validated, the data must be stored in way that enforces the logical relationships between the entities. For example, child records must always have a parent record; orders must always have a customer and order lines; and a part must belong to a parts list.
Choosing the right tool to store your data (for example, in this case a relational database) is crucial to maintaining logical integrity and consistency.
How to store your data
It’s important to use appropriate data storage types for each field to accurately capture the properties of the data item. For example:
- Numbers with no decimal place should be stored in an integer field;
- Alphanumeric characters stored in a string field; and
- Points-in-time stored in a date-time field (including options for an offset from GMT).
Having validated the incoming data and stored it in a well-structured data store, you can have confidence in the quality of the reporting and insights generated from the data.
The three S's of data
Layering on top of the storage of your data, you need to consider the three S's: security, scalability and segregation.
Security and Segregation
Depending on the nature of your data, you may need to control who can see what. This generally occurs over two “axes”: the field axis, and the row axis. Some fields within a dataset may be restricted to users with a higher level of authorisation, such as the “medical information” field in a customer table.
Data can also be segregated by rows where different system users should only be able to see specific subsets of the overall rows in a table. For example, an account manager may only be permitted to see the accounts she is responsible for.
Scalability
Scalability is an important consideration for future growth, not only in data capture but also for data usage. As more people want to use your existing datasets, the storage system must be capable of scaling to handle the increased demand efficiently.
How to utilise your captured data
When it comes to making use of the data your application captures, there are many drivers for this and just as many options for doing so.
Management Reporting
A typical use for captured data, management reporting involves using aggregated and summarized data to help a company’s decision-makers monitor Key Performance Indicators (KPIs) and make informed business decisions.
These reports are often done “within application”, and are viewed through system dashboards, which ensure the outputs are easily accessible to those who need it. They are usually fixed in structure, with well-defined filters and output fields.
Exploratory and Analytical Reporting
Another usage is exploratory and analytical reporting, which often cannot be serviced with fixed reports because users don’t know what they want to report on, until they do! These reports are less structured and allow users to investigate data without predefined queries. For this type of reporting, it’s important to have a flexible and “human readable” dataset underpinning the report “builder”.
It’s easy for techies to forget that the typical user won’t understand the underlying database schema (and wouldn’t have reason to). For this reason, analytical reporting is often implemented using replicated, transformed, and read-only datasets. This may reflect the operational data “in real time”, or there may be an accepted delay until new operational data appears in the reports.
Integration with external data
The data held in a business application database may only be a small part of a bigger picture. Although the database holds an integrated and logically complete set of data, the real benefits may only be realised in conjunction with other data from internal or external datasets. For example, the application may hold reference numbers for staff or customers, but not their names or contact details. In this case, combining the data with a definitive source of “person” data would reveal more useful insights.
Equally, internal datasets may be further enhanced through combination with external sources of data. To integrate them, it’s important to establish a common “key” between datasets, so that they can be meaningfully joined up.
Business continuity and application replacement
Business applications are often central to the everyday running of a company, making any interruption to their usage unacceptable. This is a common barrier to replacing core business applications. However, the application and the data are two separate things: the application can be replaced without touching the data.
This allows for a gradual, piecemeal substitution of the old application with a new one, without interrupting the daily running of the business (check out our recent article about modernising legacy systems). The gradual modernisation approach allows a business to continue to benefit from the wealth of historical data that has been captured but via an evolving, modern interface.
In conclusion
In summary, effective data management is the cornerstone of any successful business application. From capturing and validating data to ensuring its integrity, security, and scalability, each step is crucial for generating reliable insights for your business. Understanding these fundamentals sets the stage for leveraging advanced data management techniques and technologies. In part two, we will delve into data management in the real world and demonstrate how we’ve applied these principles to build robust and scalable business applications over the last 30 years.
Topics
Author
