Whether it’s data warehousing, MDM or business intelligence, metadata is added to the project plan, is downgraded and eventually dropped from the project plan. The impacts of not including metadata and metadata management as part of the project have far-reaching and costly repercussions throughout the organization. Read on to learn more...
We are in the midst of a data architecture revolution! Whether it’s data warehousing, MDM, business intelligence or data modernization, there is some sort of related project, program or initiative.
However, as we all sit around the planning table, in work session after work session, the theme is always the same. That is, metadata is added to the project plan as an integral component but is downgraded and eventually dropped off the implementation part of the project plan. The reasons given in many instances are always related to time constraints, re-definition of scope or the metadata component is too large and should be treated as a separate project, which almost always never happens.
The impacts of not including metadata and metadata management as part of the project have far-reaching repercussions throughout the organization. These impacts are costly and affect the organization in a variety of ways. In my experience on many of these projects, not having metadata and/or metadata management rears its ugly head in the following ways:
Extended discovery phases of major data warehousing or MDM initiatives
During the early planning stages of any DW or MDM initiatives, it is almost a certainty that the time allocated to complete the discovery phase is not enough. Why? Because you almost always have to do the type of digging into the documentation and SME reviews/interviews that can easily be compared to an archeological dig at one of the pyramids in Egypt, in the desert heat! Let me walk you through one of my recent projects. This IT modernization project included building a centralized ODS from many silo-ed applications that were written back in the seventies, that is the 1970s! I was told that all applications were documented, kept in binders and were up to date. Well folks, not only was the documentation limited (only the name of the application and a description), but the binders had an inch of dust (remember Egypt and desert heat) and the “last updated date” was usually in the nineties that is the 1990s! I ended up identifying the SMEs, did some extensive interviews, recorded the information and cataloged the information gathered. I then built a series of data models and DB tables that cataloged the metadata and constructed queries to generate a series of reports. The original estimate for this discovery phase was three months, the actually time spent was SIX MONTHS! Now I know that this is an unusual case, however, I think the point is made that had the metadata been a part of application maintenance, we would not be having this conversation. I am just now recovering from the emotional scars!
Difficulty in completing business processes
In any organization, there is always the introduction of new products and services. These new offerings require the underlying IT infrastructure to make a number of modifications to the environment, including the data layer, application layer, etc. However, in the rush to get these products and services implemented (and I mean RUSH!), there is always the “cutting corners” to get it done sooner. And what do you think is cut first? DING! DING! That’s right! Tasks related to completing the metadata layer. So the products and services are deployed into production without the metadata. Everyone is happy! YEA! However, some time passes and management or an external agency makes a request for information related to the new products and services. The team that typically performs these functions does not have any of the metadata information to complete the request because it was a “corner” that was cut! Its almost like going on a road trip but the GPS was not updated with the new landmarks (that you want to visit) that were recently created. Therefore, you drive right past, get lost and spend hours trying to find it! How did I resolve this issue? I reversed engineered the database into a data model and worked with the business analysts to create definitions of the new entities and attributes and updated the metadata layer. I also documented the process for reference. The total time taken to complete the request is A LOT LONGER than the original estimate! All I am saying is that metadata is like the GPS that needs to be synchronized with new information on a very regular basis. It is NOT a “corner” to cut. Moreover, who do you know likes to go on a road trip with map that looks like a papyrus from ancient Egypt! Not me!
Cross training new members of the team
In the wonderful world of IT, we all know that there is a constant turnaround of staff on project and support teams. It’s a fact of life. Why does it have to be so difficult to get new members of a team ramped up with information needed to make an immediate positive impact and make valuable contributions sooner? Maybe because there is not enough of a library of information that is readily available as a reference to incoming teammates. How many folks out there can relate to that? I mean you are the person in the shoes of a new member of a team taking on massive projects related to data warehousing, business intelligence or MDM and you have like a paragraph of information to use as a reference to fulfill the role of the data architect, database administration or ETL architect! I see a lot of hands being raised. I have been in those shoes way too many times to count. However, I have also been on the other side where I have had to introduce new teammates to the project. The reaction I always have in this scenario is WOW! And a smile and look of relief. Why? Because I took the time to add the required metadata documentation and artifacts to the underlying architecture that can easily be generated into reports. I don’t know about you folks out there but I would rather have tons of information to reference as a new team member to add value to the project as quickly as possible.
Breaking the Cycle
A way to break from this painful cycle is perhaps placing a value on metadata and metadata management. I think as experienced professionals in IT that some responsibility lies with us to market the concept of metadata and metadata management. We should use our experiences and that of business areas that we support to dispel the notion that metadata is a “frill” and not a mandatory requirement to complete a project or initiative. Maybe we can market this idea by capturing “pain points” that business units have experienced or better yet not just putting a value on metadata but also actually placing a cost of NOT including metadata efforts as part of a project. It's just something to think about as we build scalable and sophisticated data architectures that includes SOA, HPC/cloud computing, data governance, and dashboards. There are many software solutions out there that allow us to break this painful cycle. Let’s put it in motion!
Full article...
