Purpose of ML at Paystone

When evaluating new ideas for machine learning services, having a clear understanding of the role machine learning is meant to play is crucial.

A company that uses machine learning simply as an “automation tool” for tasks that don’t actually take very much time, or that don’t have a significant impact on the business, is wasting its efforts.

A company that uses machine learning to unlock capabilities for its users that would be impossible otherwise, or to leverage its own data to create benefits that are uniquely derivable by them, is maximizing its return on what is undoubtedly a hefty investment.

With that in mind, the goal of this document is to ensure that we align ourselves with the latter position; that we are maximizing the return on our investment in machine learning, by establishing principles for ourselves that guide us towards high-impact work, and away from trivialities.

To do this practically, we focus this document on establishing a set of questions that, if answered positively, ensure that the potential project being evaluated is one worth investing in. By asking ourselves these questions about a new (or indeed, existing) machine learning service, we can determine early on how much of a priority developing that service should be.

Each section following the initial set of questions will explore each one in more detail, defending its inclusion in the list.

Note that this document does not address work related to “MLOps”, the work of supporting the development and continual improvement of machine learning services. Prioritization of these tasks is an entirely different problem with different guiding principles. These are addressed in a separate document.

Questions to Ask of ML Services

When evaluating the need for a machine learning service, the following questions should be asked. If the answer to one or more is negative, serious consideration should be given as to whether machine learning is the best tool to create the benefit.

1. Is this creating a new capability for our clients that would not be possible, or would not be possible at scale, without machine learning?
2. Does the lift provided by a machine learning driven service over simple well-informed heuristics, deterministic algorithms, or off-the-shelf / third-party machine learning solutions justify the investment required to develop and maintain it?
3. Can this service take advantage of our own data to create unique value when compared to one using publicly available data?
4. Will the benefit given to the client by this service create benefits for that client’s customers?
5. Can this service be continually improved over time, given the right data?

Unlocking New Capabilities

As noted in the question, this could mean one of two things:

• This capability is not possible for a human to re-produce.
• This capability is possible for a human to re-produce, but would require a time commitment that would make it a negative value proposition, and so is something no human would choose to do themselves.

The justification in the first case is obvious. If we can create application functionality that gives our clients a brand new capability that is uniquely provided by our platform, that has such high value both to the client and to our business that we would be foolish not to pursue it.

Why is a feature that does something the user could do themselves something we can justify prioritizing? This is based on a core principle of the Growth Platform: that we want to give our clients their time back. Being a small business owner is incredibly time consuming, so anything that we can do to relieve our clients of necessary but unfulfilling tasks is highly valuable.

That said, there is almost always a way to represent a machine learning service as something that saves time. The key to this point is in the words “at scale”. This means that the number of times this service is invoked scales at least linearly with the size of the business. For example, the work involved in financial reporting at the end of each month is constant with respect to the growth of the business. In contrast, for an action that is required every time a new lead is generated, the number of times it occurs scales at least linearly with the growth of the business.

Many problems are easily solved manually when viewed at a small scale. Machine learning aims to tackle the problems that get out of hand as the client’s business scales beyond their reach.

The Lift is Worth the Cost

The number of machine learning operations that are offered “as a service” is growing by the day. An increasing number of startups are choosing to specialize in one particular machine learning capability, such as image classification or text generation. Beyond the startups, cloud providers offer a suite of services that expose models designed to solve these general problems for everyone. These services exist because they solve the given problem sufficiently for a majority of cases.

There are, in addition, many problems for which machine learning may seem on the surface to be applicable, but is actually overkill. It could be that the problem that machine learning is prescribed to solve could actually be solved deterministically, as either a basic rules engine or a set of heuristics. Now, often, the classification of a problem as either requiring machine learning or solvable with rules and heuristics is not trivial; it may require deep investigation into the problem and the available data. Sometimes, this investigation will be best handled by a machine learning engineer, because it requires statistical analysis and validation. This does not mean that once the problem is in a machine learning engineer’s hand, it becomes a machine learning problem. If the conclusion reached by that engineer is that the problem does not require machine learning, it can easily be handed back to the rest of the engineering team before any work is started. In this way, we avoid over-provisioning engineering resources and compute resources to simple problems.

In both of these cases we find an opportunity to take advantage of a simple solution to a complex problem, and we choose to forego in-house machine learning development. This is strictly a positive. We save time, we save engineering resources, and we get features into the hands of our clients faster.

We should also include in this evaluation any pre-existing services of our own. As an example, consider a content generation task such as writing reviews. Assume that we have a pre-existing service of our own that provides the capability of "content generation" broadly. In our initial investigation of this review writing task, we may explore third party solutions or simple heuristics that can write reviews sufficiently. During this time, we would also evaluate -- according to the same standards -- the ability of our "content generation service" to write reviews. If it works well, we can forego the creation of a new service, and instead work on optimizing our usage of this existing service for the problem. This optimization might include parameters such as the structure of the natural language prompt we provide to the content generator. If it does not meet our standards, we create a new service specifically for writing reviews.

Leveraging Our Data

We are in a unique position as an organization with regard to the customer journey. Many companies gather swaths of data regarding one piece of a customer journey: lead generation, client services, customer-company feedback, and customer-customer interaction.

Our Growth Platform collects data on the entire customer journey, and implements hooks into the journey that can potentially alter its outcome, and this is our unique value proposition.

We said at the outset that valuable machine learning is that which creates benefits uniquely derivable by the organization implementing the service. Therefore, any service which does not in some way extract value from the unique cross-section of customer journey data that we hold is one that is likely not worth building ourselves.

Conversely, any service which does extract value from this property of our platform is something brand new to the world.

Customer Benefits

We are a company guided by Customer-Driven Growth. Each software development task we undertake needs to justifiably be in service of that “North Star” goal. The importance of this only increases as the scope of the software under question increases, and surely machine learning services are among our largest and most general software artifacts. In less words: if you’re going to spend multiple months doing something, you had better be sure it will have an impact on Customer-Driven Growth.

There’s not much more to say than that given how much ink has been spilled on the concept company-wide. Instead we’ll lead with some examples, to drive the point home.

• A machine learning model to help developers auto-generate docstrings for functions in code is very low priority, because it is by the most generous of estimates 6 steps away from a customer benefit: the service helps the developer write a better docstring, which helps future developers understand their code better, which helps future developers make safer changes to the code, which decreases the likelihood of software breaking, which decreases the amount of unproductive time for clients, which increases the time and mental energy they have available to help customers.
• A machine learning model to automatically reply to customer reviews on behalf of the client is of medium priority, because it is only a couple of steps away from a customer benefit: it does the work of building a positive reputation for the client without their intervention, giving them more time to spend on human interactions with their customers.
• A machine learning model to recommend clients to customers is of very high priority, because it is an immediate customer benefit: the model suggests products and services to a customer that improve that customer’s life.

Continuous Improvement

A distinguishing feature of machine learning software is that it can always be improved, due to its nature as an approximation rather than a derivation.

Software which collects data, derives information from data, or presents information can be given an objective target state. While there is certainly subjectivity in determining what that target state is, both the determination and achievement of that target state are human processes. In other words: we decide what “done” means, and we are entirely responsible for getting ourselves there.

Machine learning software is different in both ways.

First, because machine learning is fundamentally about approximating that which cannot be derived with certainty, there is no objective target state. It is rather a process of continual and unbounded improvement, guided by metrics with no terminal value. We say “no terminal value” because any machine learning service whose error is zero is no machine learning service at all: it is a deterministic function being computed with gross inefficiency.

Second, these improvements do not necessarily require human intervention. The mature state of machine learning development is one where any service is able to improve autonomously at its assigned task, where improvement is defined by metrics that have been established by human processes.

With that background information, we ask ourselves: is there an objective target state that we can prescribe for the service? If not, is it possible to craft an environment in which the service can autonomously make progress on the metrics we assign to it?

It is a common mistake to aim for 100% “accuracy” for all machine learning services. This is, by definition, impossible. Consider the case of customer match, where the goal is to correctly identify the author of a given Review from a set of Persons. If this was possible to do 100% accurately, it would cease to be a machine learning problem: a 100% accurate algorithm implies a discrete set of rules that always work, no matter the circumstance. In the absence of such an algorithm, our objective is simply for the service to perform better today than it did yesterday. This is not an objective target state, it is a measurement for performance.

The second question is less about disqualifying a problem from being solved by machine learning, and more about potentially delaying the development of such a solution if we do not have the correct tools in place yet. A machine learning service that cannot be monitored for performance, and cannot be improved based on its interaction with the world, is ultimately of very limited utility, and hardly worth the required investment. Put simply: if we don’t have the right data to be able to collect the right metrics, or if our metrics cannot provide actionable feedback to the service, then we are not yet in a place to develop the service.

The answers to these two questions will either give us a green light to develop the service, stop us from doing so entirely, or cause us to postpone the development of the service until we are in a better position to help it succeed.