KPI Dashboards
- At-a-glance comparisons of 23 leading products.
- Find out which solutions lead the market on 26 key criteria such as competitive win rate, goals achieved, proportion of employees using product, data volumes (log), product-related problems, product reliability and support quality.
- Discover which products are strongest for your particular requirements.
- The KPI Dashboards are based on a detailed survey of 2150 BI end-users.
- Results are presented in bullet-graph format so you can quickly digest the information presented.
- Find out more about how the KPI Dashboards are calculated.
Root KPIs
There are 26 ‘root’ KPIs included in these analyses. Of course, there cannot be as many as 26 key performance indicators, but different readers will have their own views of which of these KPIs are important to them. For example, some people will regard fast query performance as very important, whereas others may regard scalability or low unit costs as more important.
KPIs are only calculated if the samples (from The BI Survey) have at least 30 data points, so most of the products do not have a full set of root KPIs. It is important to exclude KPIs based on small (and therefore unreliable samples) to ensure that the graph scales are not distorted by outlier KPIs based on small data samples. In such cases, the product is still shown in the tables, but with a blank KPI value and no bar in the bullet graph.
Business Benefits Index
The BBI is possibly the most important KPI of all, as it focuses on the bottom line benefits of BI projects, rather than individual technical aspects. After all, if business intelligence doesn’t deliver business benefits, what is its purpose? Unlike core transaction systems, BI projects are optional, not mandatory, and so they must pay their way in terms of delivering business benefits.
Goal achievement, adjusted for age
Goal achievement may seem every bit important as the achievement of business benefits, but over the years, we have found a weaker link between products and goal achievements. A few years ago, we discovered that, unlike BBI, goal achievement rates rise with application maturity, so this year, we introduced an age-adjusted calculation for goal achievements.
Competitive win rate
When deciding which products to evaluate, it is useful to know which have fared well in other organizations’ product selections. This is an easy way to quickly eliminate ‘losers’ without having to waste too much effort on them.
Selection based on product factors
Our research has consistently found that BI projects are more successful if the products were chosen on product grounds, rather than because of vendor relationships or low prices. It is also a good sign if a product is mainly chosen on these grounds, rather than, for example, because it was bundled with another unrelated product from the same vendor. In fact, it is a warning sign if a product is typically chosen by organizations that did not actually evaluate it.
Prevalence rates in multi-product sites
Many respondents to this Survey work for organizations that have purchased more than one BI product. In such cases, we asked them to nominate one product on which to answer detailed questions, based on which they were most familiar with, or which was most widely used in their organizations. We regard it as a sign that a product is more important to its customers if it is chosen more often in such circumstances, and therefore calculate the ‘prevalence rate’, which is a measure of how often the product is nominated by respondents in multi-product sites.
Standardization preferences
As mentioned above, many organizations have purchased multiple BI products. We asked respondents in such organizations which of those products they would choose if forced to standardize on a single product.
Intention to buy more licenses
No one knows more about how a product performs in the real world than the customers already using it. All too often, they find that products don’t live up to expectations, or that the vendor does not support the product properly. Such customers will probably end up not even using all the licenses they have already bought, and certainly won’t be buying any more.
Discontinued usage rates
Another question that we ask organizations that have purchased multiple BI products is whether they have stopped using any of them. If they have, this might be a warning sign for potential future buyers of such products. One caveat with this measure is that it favors younger products; there is a natural life cycle with BI applications and products bought five, ten or fifteen years ago may have fallen into disuse simply because the applications they were used for are no longer required, possibly for reasons that have nothing to do with the product. Conversely, applications based on a very young product will not have had a chance to reach the end of their natural lives, and there may therefore be very few discontinuances. Even if applications have not been a roaring success, they may remain in use for a few years.
Proportion of employees using product
Many organizations try to avoid having multiple BI silos: small groups of users accessing different, specialist BI products. They prefer to buy products that can be deployed more widely, as widely-used products are easier to support and skills are more transferable. We therefore asked respondents what proportion of their employer’s employees used the product.
Range of applications deployed
Just as customers often prefer products that can be deployed more widely, they also favor products that can be used for multiple applications. This is, again, with a view to reducing the number of BI products in use. Obviously, this KPI favors multi-purpose, rather than specialist, products.
Number of departments served
Just as some buyers favor products that can be deployed more widely and used for more applications, they also are interested in the range of departments that can be served by BI products. For example, some BI products are mainly used in finance departments, meaning that different products need to be deployed for other departments. This measure obviously favors general-purpose, rather than specialist, products.
Deployed seats
One important measure of success, for both vendors and customers, is the number of seats actually deployed (as opposed to sold). An effective sales person may succeed in selling customers more licenses than they will ever use, so the real test of success is the number of seats actually deployed. It also allows us to include customers that have purchased server, site or enterprise licenses without a finite number of seats. However, useful as this is, it is also a question that many respondents are unable or unwilling to answer, so the response rate is low. We base this KPI on the median number of seats, rather than the mean, better to exclude outliers. Most customers have rather small BI deployments, with only customers of a few products having large numbers of users. This leads to a rather skewed chart, with the overall sample average far to the left on the scale.
Data volumes (log)
We have found that most BI projects handle surprisingly small data volumes – far less than popular myth would suggest. In fact, the typical application loads only 6.8GB of input data, and three quarters have less than 250GB. Most products can, at least in theory, deal perfectly well with any normal application, but they may still be optimized more for what their typical customers do, rather than the extreme cases. So this KPI will mainly be of interest to organizations that have a genuine need to analyze and report on at least a few hundred gigabytes of input data, which will be outside the range that most products are typically used for, even if they can, in theory, handle them.
Cost of Ownership Index
While no one can be certain that BI projects will deliver business benefits, it is certain that they will incur significant costs: software license and maintenance fees, implementation fees and hardware costs, plus whatever non-billed internal resources are used. We find that customers are apparently unconcerned about hardware costs, so we focused on license and implementation fees. Maintenance is usually a percentage of license fees (around 20-25 percent) and ongoing maintenance is likely to be related to implementation fees. Thus, license and implementation fees are a good proxy for all external costs. The headline figures are the totals paid, but we work out a more useful per capita figure in the Cost of Ownership Index.
Deployed seats per administrator head
The main internal cost is likely to be the number of people required to administer and run the BI applications. In The BI Survey, we measure this cost on a per capita basis as the ratio of admin heads to deployed users. This has the effect of making some apparently expensive products look good value, as they have much larger numbers of deployed seats than apparently low cost products.
Implemented within three months
There is plenty of evidence in The BI Surveys that faster implementations lead to deployments that are more successful. There are many probable reasons for this, including the likelihood that fast deployments respond better to changing user needs. They also minimize the risk of white elephant projects. Few projects are live after one month, and most are live within six months, so three months is a useful benchmark.
Product-related problems
We classify problems into three categories: people, data and product related. Only the last of these can be blamed on the products, and so this metric is a good choice for a KPI. Obviously, the smaller the number, the higher the KPI.
Product-related deterrents to wider deployment
Another measure of product-related problems is the extent to which they deter wider deployment. Interestingly, these results do not exactly parallel the technical problems reported in the current deployment, so it is worth tracking them separately. For example, a sophisticated but complex product may be regarded as successful when deployed to a small, skilled user community, but its complexity may be regarded as a deterrent to deployment to a wider community.
Product reliability
Product unreliability was the second-most often reported serious product-related problem, after poor query performance. If it occurs at a critical time, it can be very debilitating, and could even render an application unusable. It is therefore a suitable KPI, particularly for customers who need a very reliable application.
Product support quality
Related to product reliability is support quality: if a product is unreliable, or just hard to use, product support is called into action. This is yet another area where there are big differences between vendor performances. For example, 75.5 percent of customers related the highest-scoring vendor’s support performance as excellent — but only 4.9 percent said the same of one of the worst vendors.
Query performance complaints
This is not only the most frequent product-related problem, but in recent years, has been the most frequently reported problem of all types. Beyond that, poor query performance leads to reduced business benefits and goal achievements, so it is more than just a technical problem.
Query performance, adjusted for data volumes
Query performance is very important, and often disappointing, but it depends on more than just the product. We have found a clear link in the whole sample between query performance and input data volumes, so it seems reasonable to adjust reported median query performances for the median input data volumes for each product. There may well be other influences, such as hardware capacity, but we do not have the data to adjust for such influences. In any case, it seems unlikely that the majority of the customers of one product would under-specify the hardware. And if applications are hard to optimize for best performance, this is a fair reflection of user experiences with the product.
Data latency, adjusted for data volumes
Data latency — the delay between new data becoming available from feeder systems and its availability in BI queries — is much less of an issue for most users than query performance, but it is more important in the minority of near real-time BI applications. The time is required to load data and pre-calculate aggregates. Even more than with query performance, this factor is affected by data volumes, so we again adjust for them for this KPI. An important caveat is that this and the next KPI assess the performance of the underlying database technology, which may or may not be part of the BI tool being analyzed. For example, many of the products are reporting tools, which have no little or no effect on the data latency of the application as a whole. In such cases, the reported data latency figures may not be relevant. In particular, if the customer appreciates that any data latency issues are the result of inefficiencies in the underlying database, they would, rightly, not blame the frontend tool for the problem.
Data latency as a deterrent to wider deployment
In addition to analyzing the actual latency, we also ask respondents the extent to which it is a deterrent to wider deployment.
Web deployment (>50 percent Web)
Early BI solutions used either stand-alone or client/server architectures, but for the last decade or more, BI has been moving to the Web. To the surprise of many, however, a substantial number of users still access their BI applications via locally-installed Windows applications, not thin-client browser solutions, which are assumed to be easier to deploy and maintain. We found this year that only 48 percent of customers reported that their BI applications were at least 50 percent Web-deployed, meaning that a small majority of sites were still less than 50 percent Web deployed (consultants and vendors thought that Web-deployment rates were slightly higher).
Extranets deployed
A few years ago, virtual organizations linked by extranets were the “next big thing”. Like many such ideas, its impact was exaggerated, but BI extranets are reported to be in use by 16.4 percent of respondents. This varied from more than half the users of one product, down to less than five percent of the users of two products. Extranets are clearly not a key requirement for most BI customers, but for those who do need them, this KPI provides an indication of which products are most often used to deliver them.
Aggregated KPIs
The 26 root KPIs above provide a good selection from which readers can select the half dozen or so that they regard as key to their requirements. But another way of reducing the number is to combine groups of them to form a small number of aggregated KPIs, which is a familiar BI concept. The aggregates are geometric means (which means that the overall sample value remains 1.0), and the first level aggregated KPIs are only calculated if all the root KPIs are available. The overall KPI is calculated as long as at least 22 of the root KPIs are available.
Business achievement KPIs
This is based on the combination of the BBI and goal achievement KPIs. It is a measure of the extent to which the product delivers business value and helps organizations meet their goals.
Costs KPIs
This is a combination of the Cost of Ownership Index and Deployed seats per administrator head KPIs. It is an indication of the internal and external costs in acquiring and running solutions based on the product, on a per capita basis. Relatively few products had enough respondents answering all the underlying questions to be able to compute this KPI. Note that this aggregated KPI favors pure front-end tools, as they naturally incur fewer costs than all-in-one solutions.
Scalability KPIs
There are many definitions of scalability, and vendors naturally tend to choose whichever makes their product look best. Buyers should be more specific, choosing a definition that reflects their business needs. We have opted for a generalist definition by combining the KPIs for Proportion of employees using product, Range of applications deployed, Number of departments served, Deployed seats and Data volumes (log). You may want to choose a different set if this does not reflect your specific requirements. As this KPI is based on as many as five underlying root KPIs, a number of products do not have enough data available for the aggregate KPI to be computed reliably.
Quality and support KPIs
Product quality and product support are clearly related: if one is lacking, the other is needed more than ever. This aggregate KPI is therefore based on the aggregate of the Product reliability and Product support quality root KPIs.
Performance KPIs
Fast performance is more important than most people realize. You can work around missing features and even bugs, but nothing can disguise an application that is painfully slow. And few things can put users off from making the most of an application than irritation at its response times. There are two types of performance to consider, but in the interests of simplicity, we have combined both in this aggregated KPI, which is based on the Query performance complaints, Query performance adjusted for data volumes, Data latency adjusted for data volumes and Data latency as a deterrent to wider deployment root KPIs.
Loyalty KPIs
If existing customers are very loyal to a product, that is a useful hint to potential buyers that the product is worth considering. We combined the Standardization preferences, Intention to buy more licenses and Discontinued usage rates root KPIs to calculate this aggregated KPI.
Web KPIs
Buyers committed to an all-Web BI deployment strategy may want to favor products that are already heavily deployed over the Web, via both intranets and extranets. This aggregated KPI therefore combines the Web deployment (>50% users) and Extranets deployed root KPIs.
Overall KPI score
This aggregated KPI simply combines all the root KPIs. It does not attempt to select a sub-set or to weight some higher than others (though you may choose to do exactly that, with the root KPI data provided). It is only calculated for products that have at least 22 root KPIs available.