Clover Analytics

Enterprise Bus Matrix - Continued, Testing #backtobasics

Thu, 22 Mar 2012 15:30:00 -0700

I had the opportunity to follow up with some business analysts after the enterprise bus walk through, and we talked about testing. Having an EBM is a great advantage as a conceptual tool to work through the business needs and design tests that help development. Here is the overview I left them with:

#backtobasics

Enterprise Bus Matrix as a Testing Tool

Utilizing an enterprise bus matrix for testing purposes can help business analysts align test cases with functional business areas and produce meaningful test plans to improve the quality and speed of subsequent development.

One important mission of data warehouse development is to identify and align data needs with business processes. One efficient way to begin this process is through the use of an bus design, which isolates functional areas from business definitions in a way that is understandable to end users, and can also help shape the technical design of warehouse data structures. Because of the importance of the enterprise bus as an early data warehouse design tool, it also has significant utility when defining a test strategy; activities that can serve to refine and enhance later development efforts (or delay and hinder development if poorly defined).

Utilizing a bus matrix format can provide a significant advantage when designing test cases or beginning a strategy for test development. As a matrix of relationships it provides a base for identifying the reporting needs supported by downstream requirements. As an example, an X representing a relationship between Claim and Provider not only indicates a reporting requirement, but also a testing need. In order to support the business data need of “I need to report/see Claims by Provider” many related tests can be phrased using a standard template. Test cases must be written to validate the relationship between Claim and Provider, as well as the availability of Provider attributes that provide data definition and context. If there are calculations or aggregations defined in the Claim fact area, test cases must also be defined to independently validate these in addition to any relationships or aggregations through dimensional hierarchies.

As a standard format for the relationship Claim(s) by Provider, a suite of business needs are implied:

1. Claim by Provider

a. Relationship

i. Claim Fact has an appropriate foreign key relationship to Provider

b. Data Attribute Availability

i. Claim data attributes are available

ii. Provider data attributes are available for a GROUP BY clause

iii. Provider data attributes are available for filter criteria (WHERE)

c. Derived Data

i. Claim Fact attributes that are pre-calculated are accurate and consistent

d. Data Aggregation

i. Claim data can be aggregated by Provider Attributes (GROUP BY)

1. COUNT is implied by any DW Fact Table

A relationship defined via the bus matrix may have many specific attributes and calculations to test. Utilizing the bus as a baseline, this additional detail can fill in as specific tests based on the specific reporting needs. As an example, a sub-requirement for Claims is shown below to illustrate Average of Claim Paid Amount by ‘SEBMF’ Providers:

1. Claim By Provider

a. Average of Claim Paid Amount by ‘SEBMM’ Providers

i. Relationship

1. Test 1: Claim Fact has a foreign key relationship to Provider

2. Test 2: Claim record is correctly identified by Provider.Affiliate as SEBMF

ii. Data Attributes

1. Test 3: Claim attribute Paid Amount is available

2. Test 4: Claim attribute Paid Amount is not NULL

3. Test 5: Provider data attribute of Affiliate is available

4. Test 6: Provider data attribute of Affiliate is not NULL

5. Test 7: Returned data includes only SEBMF Affiliate values

iii. Derived Data

1. NA* – Paid Amount is carried through to the Claim Fact record (not derived)

iv. Data Aggregation (Average)

1. Test 7: AVG of Paid Date equals the SUM of Paid Date divided by the COUNT of Claim records in the data sample

*Even the addition of complex business logic can be fit into this format; with derived data tests created to ensure upstream calculations are consistent and accurate. The testing of derived attributes may be easier to track in a business rules section for more complex rules.

Adding refinement to reporting requirements has the additional benefit of helping to further define and enhance an enterprise bus design. As attributes and relationships are added to increase the complexity of reporting needs, the bus may require enhancement. As a general rule, an enterprise bus can and should remain abstracted from technical solutions, unless the discovery process identifies a new business process or business definition that is useful as an independent construct.

Enterprise Bus Matrix - Overview #backtobasics

Thu, 22 Mar 2012 13:30:00 -0700

I recently had the opportunity to lead a group through a Kimball-type implementation, and found the enterprise bus concept very helpful to get folks with a diverse set of experience aligned across business concepts and headed towards dimensional data modeling concepts. Here is an excerpt from my presentation:

Enterprise Bus Matrix

The enterprise bus diagram is a tool to visualize and map functional business areas (verbs; what we do) against relevant terms (nouns, business definitions) important in defining meaningful enterprise data for end users. As a matrix, this diagram can clearly illustrate important data relationships included in the SHEW data warehouse to a business and functional user without any prior database or data warehousing experience.

Rows of the enterprise bus diagram can be thought of as functional areas of the business; data collected as a result of a discrete business process. From a data warehouse perspective, these rows can be thought of as proxies for traditional fact tables. Each row in the ESB should identify a clear business function with little data overlap across other business functions. As an example, data captured from a claims process can be thought of as related, but independent of, data captured from an encounter. To bring these two areas together, relationships can be defined and included through common enterprise terms like patient, provider, or date.

Common enterprise terms define the columns of the enterprise bus diagram. These nouns may function as many things. In a mature data warehouse, these structures may even function as the single repository for data reported throughout the organization. In the role of enterprise repository, data like Provider and Place of Service are standardized and updated into formats useful across functional areas. In data warehouse terms, these columns act as proxies for traditional dimension tables. In an ESB design, these columns may also reflect common data needs captured through requirements gathering sessions about report groupings, filters, or parameters that would lead to dimensional tables.

Through relationships (marked as X) between verbs and nouns, the capability of an enterprise data warehouse can be quickly communicated to non-technical end users. This format also provides a framework to identify and define new requirements or relationships.

An established bus matrix can provide a useful and easy starting point for gap analysis and scope definition when developing new business requirements. Creating a new bus matrix should always lead a development effort, with the identification of nouns and verbs independent of technical designs. As development occurs, an bus matrix should be updated as appropriate to accurately reflect the data warehouse capability at an abstract, business level.

RT @NeilRaden: Here is a really overview of various decision theories and decision making approaches...

Thu, 19 May 2011 17:47:14 -0700

RT @NeilRaden: Here is a really overview of various decision theories and decision making approaches http://t.co/tOgE8Rj

The Seven Secrets of Successful Data Scientists

Wed, 18 May 2011 00:57:28 -0700

The Seven Secrets of Successful Data Scientists:

These may all sound like common sense, but I always appreciate a good list to bring things back to reality and put my ‘creativity’ in check. Sometimes is pays to stick to the tried and true. Picking the right tool for the job is a great choice for #1 and something I am struggling with on a job right now.

Don’t Bet Big. Little Bets Are The Ones That Turn Into Billion-Dollar Ideas

Wed, 18 May 2011 00:52:10 -0700

Don’t Bet Big. Little Bets Are The Ones That Turn Into Billion-Dollar Ideas:

The truth is, most entrepreneurs launch their companies without an brilliant idea and proceed to discover one, or if they do start with what they think is a superb idea, they quickly discover that it’s flawed and then rapidly adapt.

Data Science is catching on

Wed, 18 May 2011 00:48:51 -0700

Data Science is catching on:

A great overview of the data scientist role, something I aspire to be. There are some great links and talking points in the comments that follow the article as well. For those of us that don’t aspire to fit in the traditional silos and may be having a tough time understanding and selling the data scientist skill set, this is pretty inspirational.

Rise of the Data Scientist

Wed, 18 May 2011 00:41:41 -0700

Rise of the Data Scientist:

Some follow up thoughts from Mike Loukides, a really thoughtful and long read that has some great technical examples of a data scientist as well as some of the more intangible qualitative aspects of the role.

Analytic

Fri, 25 Jun 2010 22:44:50 -0700

Analytic:

A bit high level, but this article brings up several important issues when defining analytics. Specifically, the separation of the method and the result.

Agile Data Warehousing

Fri, 25 Jun 2010 22:29:28 -0700

Agile Data Warehousing:

Another article on ADW that gives a pretty good breakdown on some of the misconceptions and the expectations of this development process.

DePaul offers a degree in analytics

Fri, 25 Jun 2010 18:19:00 -0700

DePaul offers a degree in analytics:

Although not the first in this area, still one of the earliest in this growing academic area. It is interesting given the remote and professional programs DePaul offers. Hopefully something that will catch on beyond the MBA and other professional degrees.

10 Components of a Successful BI Strategy Plan

Sun, 06 Jun 2010 13:15:11 -0700

10 Components of a Successful BI Strategy Plan:

From Information Management.

Another great strategy article for the planning phase of a BI implementation.

Why Venn Diagrams Don't Work for Data -- BeyeNETWORK

Thu, 03 Jun 2010 23:11:00 -0700

Why Venn Diagrams Don't Work for Data -- BeyeNETWORK:

I am sure this is a very intuitive topic for all data professionals, but I think this article provides a very clear explanation of the complexities in dealing with more complicated data relationship and management issues.

Figure 4: Attempt at Venn Diagram for Records in Figure 3

Figure 4 is not a Venn diagram in any real sense. It violates the principles of a Venn diagram for two main reasons: the existence of duplicates, and the existence of nulls. Neither of these occur in reality – which is what the Venn diagram is based on – but they do occur in data

"Being" Agile vs. "Doing" Agile -- TDWI

Thu, 03 Jun 2010 17:43:00 -0700

"Being" Agile vs. "Doing" Agile -- TDWI:

A great overview of the practical differences of applying Agile without the core set of values needed to successfully incorporate it into an organization:

Individuals and interactions
Working software
Customer collaboration
Responding to change

Eight data prep lessons for advanced analytics -- Sascom Voices

Thu, 03 Jun 2010 17:02:00 -0700

Eight data prep lessons for advanced analytics -- Sascom Voices:

Communicating the logistical layout and complexity of #3

3. Distinguish between data warehouses, data marts, and analytic databases.

with the overall vision for #4

4. Design a data warehouse architecture that accommodates analytics.

are where most PMs have problems. In particular with #4, communicating the need to create something extensible and scalable ‘for the future’ clashes with the easy technical kludge to get something up and going immediately.

The Importance of Agile Business Intelligence

Wed, 02 Jun 2010 12:43:00 -0700

The Importance of Agile Business Intelligence:

From The Intelligent Enterprise Blog - An interesting topic in identifying and continuing to keep on top of BI issues. Reminds me that typical deployments are usually incomplete, and if allowed to stagnate, the full benefit of ‘Phase 2’ is never realized.

Welcome

Tue, 01 Jun 2010 16:53:05 -0700

Welcome to the new format! Going forward we will use tumblr to deliver BI and Analytics news commentary and interpretation. If you have not checked tumblr as a platform, be sure to give it a look. We will still continue with Twitter, and you will see our content here integrated with the @cloveranalytics tweets as time goes on. Thank you for checking us out at cloveranalytics.com!

Clover Analytics

Tue, 01 Jun 2010 14:28:00 -0700

Clover Analytics