Jeremy Adamson - Minding the Machines

Procedural Implementation Will this fit in with existing processes? Will it require significant changes to current workflows? What are the risks associated with implementation? Will it introduce the potential for human error? Does it have dependencies on processes that are being sunset?

Interoperability Are there downstream processes depending on the results? What are the impacts of a disruption to these processes? Can it be shifted to another system? Does it create dependencies?

Extensibility Can the output be upgraded in the future? Does it require specialized skillsets? Is it generalized enough to be used for other purposes?

Scalability Would this approach work if the volume of data doubled? Tripled?

Stability Has it gone through thorough QA? Has it been tested at the boundary conditions? What happens if data are missing? What happens if it encounters unexpected inputs? How does it handle exceptions?

Interpretability Are the results clearly understandable? Does the process need to be transparent?

Ethics Is it legal? Does it have inherent bias?

Compliance Does it contain personally identifiable information? Does it comply with the laws of the countries in which it will be used? Does it use trusted cloud providers?

Without exception, effort is better spent in discussing and addressing the above considerations than in marginal improvements to model performance. Even a poorly designed model will work with strong phenomena, and a poorly performing model that is in use will outperform a sophisticated model that is sitting idle.

One of the most memorable scenes from the 1990s classic Office Space involves the protagonist being chastised repeatedly by his coworkers and managers about an incorrectly prepared TPS report. The reason this was so memorable is that it reflects the experience of every office worker at some point in their careers. Bureaucracy is a self-propagating danger, and for many larger legacy organizations, entire teams are dedicated to the preparation and dissemination of unread reports. There are several compounding factors that have led to an explosion in the number of reports recently.

With legacy reporting tools, the effort required to create reports created a natural limit on the number that could be produced. This provided a control on the spread of new reports. Unfortunately, the strength and ease of use of reporting and visualization tools that are available today make creating new reports trivial.

In addition, as tenure at the management level falls over time, new managers arrive and request views of the data that are most conducive to their way of thinking. This, combined with a bias to action, can lead to a flurry of new reports, which are then promptly abandoned at the next reorganization but continue to be maintained.

Finally, a common rationalization for poor decisions by leaders is that they are due to a lack of information. In response to a personal error, and as a cover to their rascalities, it is not unusual for a new report to be requested, ostensibly to prevent the incident from recurring, but almost certainly as presentable evidence of their proactivity.

Despite all of these drivers of new reports, the organization could conceivably reach an isotonic point of stability where new reports are balanced by discontinued reports. Unfortunately, the process to sunset a report in most organizations is more onerous than the process to create a new one, leaving zombie reports that are diligently maintained by an army of analysts but unused.

For decision makers who have a more tenuous understanding of the value offering of analytics, and a desire to have better-informed decision making, the one-off choice to request that the team create a report can quickly become a pattern of distracting asks. At the expense of large transformative projects, more reports are created and maintained. Over time, the value of the analytics team is questioned.

How can this be prevented? Clearly articulating the value proposition of the team and ensuring from the start that it has executive support is essential to standing up a successful team. Respectfully pushing back against reporting and other non-value-add requests, though politically sensitive, must be done. Every member of the team must be empowered and given the autonomy to question the work that is being requested.

Understanding the first-mover advantage associated with data and analytics, many organizations of means yet with limited analytical maturity have hired dozens of practitioners to rapidly scale up and advance their data capabilities. After 18 to 24 months, these large technocratic organizations review the cost/benefit of the new $10M division and question the value of analytics to the enterprise. Moving too quickly, without the cultural or technical infrastructure to support it, can put an end to analytical ambitions before the first hire has been made.

Alternatively, many organizations have moved too cautiously and hired one or two early-career employees and placed them under a line manager with conflicting priorities and limited analytical understanding. The maturity of the data science team in this case often peaks with gentle scripting and automation, and executives are again left questioning the value of analytics.

It can seem that a Goldilocks approach of creating a mid-sized team could work, but the issue in both extremes has not been the number of people involved, it has been with a lack of a defined strategy. To prevent this situation, organizations need to have a clear objective in mind for the team, an understanding of what the field of advanced analytics and AI is, and the support that is required to make it a success. Further, organizations cannot look within to build out this strategy, as they thoroughly lack the capabilities to define this strategy. It is only by consultation with others and thorough benchmarking to industry and practice standards that this strategy can be developed.

All organizations have some capacity for gathering data, and once the quantity of that data reaches a certain size, a data-centric team (whether business intelligence, data services, self-service insights, or general information technology) is naturally formed. The purview of these data teams in many cases has been confused with analytics and data science, to the point that the intersection of the two is very often considered the scope for both. Though there is certainly a strong mutually supportive relationship between these two functions, they exist in different spaces and with different incentives—one being a cost center and one being a profit center. Executives with medium-term ambitions to develop a next-generation analytical function will frequently prime the pump by engaging the data team and expanding their accountabilities.

The outcome of this is usually expensive, avant-garde data solutions such as data lakes, premature cloud migrations, and unnecessary latency reduction strategies. The organization, still in its analytical infancy, is unable to take advantage of these powerful technologies, the cost of which is allocated to the analytics team. Organizations need to learn to walk before they can run, and the burden of these data-centric initiatives, executed far too early, has weakened many emerging analytics teams who have had to justify their inheritance after the fact.

These issues are the result of a lack of strategic focus. Well-intentioned projects initiated from a mature data team, supported by an executive, and imposed on future analytics teams leads invariably to technical debt and a handicapped team. Analytical excellence needs to be the focus and the function to be optimized and not confused with the activities that enable it. Early holistic strategy development that considers the interactions between these different teams is essential.