Data Quality in a Federated Organization

One of the I am finding in the world is that there is a need for data quality, especially in a federated organization. The problem being that there is not only redundant data, but also an inability to understand the data stored. This seems to compound itself when paired with the idea of big data. Not only do we store data that is unnecessary, it is often orphaned. Additionally there is the argument that storage is cheap, which is true but only proliferates poor practice and makes the meaningful data a lot less meaningful.

Federation is also one of the trends today that is shifting away from centralization. A problem that organizations on this journey face is system of record proliferation. This has major effects on data quality throughout the organization. In my experience, I have seen federation take hold to drive innovation by focusing on the core competencies of that federated entity. Consequently, the federated entity foregoes advantages like economies of scale and streamlined processes. Another competency that is foregone is data quality. Problems related to multiple system of records and data collision become paramount when trying to solve complex data integration problems. There could be tight rules associated with the data, but how would those be enforced without centralization.

One possible solution could be standing up a centralized business unit that is customer facing. This unit would be responsible for defining and disseminating information to the consumers of that data. The proposed “integration” unit would solve for the abstraction of those disparate systems of records and develop reporting structures in order to facilitate integration of the data into information. While this would work in smaller well managed companies, many organizational challenges of a bigger and more complex organization would present a challenge. One such challenge is rooted in the way the customer uses the systems. Each of these systems may capture the same data, but not with the same context. Master Data Management could help by providing metadata around the systems of record, but there are inherent challenges in it’s own right. Consumer and providers of the data not only contribute to the contextual ambiguities but also as the owners of the data change, so will the meaning of the data captured introducing a data lineage problem.

Federation being the current trend, I would gather that data architecture will grow and should organizations like the aforementioned idea, so will information architecture. The information architect would become more well defined around contextual data.