Skip to content

Experiments Log Template

A minor version iteration of a machine learning service is incomplete without updates to the experiment log for the service. The experiment log is a document describing each revision of a machine learning service, with the goal of explaining the rationale behind the solution.

For more details on the goal of the document and its overall tone, see our experimentation documentation advice.

Begin from the first sub-heading below. The sub-headings are fixed and should be directly copied. All content underneath a sub-heading is placeholder content that should be removed. No content underneath a sub-heading from this template is required to copy verbatim.

Solution Statement

  • This should be a relatively short section.
  • As concisely as possible, and without in-depth technical detail, describe the approach that is taken to solving the problem in this experiment.
  • Consider this the "elevator pitch" for the solution. This is how one would explain the approach to another person if they only had a couple of minutes.
  • Some leading questions to prompt this content:
    • Is this regression, binary classification, multi-class classification, or an unsupervised paradigm?
    • Is this a deep learning solution or a classical approach?
    • Are any public models used?

Data

  • The CDP data that is used to train a machine learning service is a key justification for the in-house development of the service; if the same functionality with the same performance could be achieved without Paystone's data, it would not be a service worth building ourselves.
  • Describe the data used from this perspective. Emphasize how each data source used adds to the unique value of the service.
  • If client or customer feedback was leveraged by means of CDP data, explain that here.
  • Some leading questions to prompt this content:
    • What CDP entities contributed to the training procedure?
    • What enrichments or predicted links contributed to the training procedure?
    • What combinations of entities unique to Paystone's CDP (e.g. having both payment and booking data) enhanced this service?
    • What auxiliary public data was used, if any?
    • What data sources were considered, but ultimately not used? Why?

Models

  • Describe each of the relevant training artifacts that are used in the service revision.
  • A technical explanation of how the artifact works (for example, explaining how transformers work) is not necessary. As highlighted in the experimentation documentation advice, the writer can assume base knowledge of machine learning concepts in all readers.
  • The focus in describing each artifact should be on its role in the service, and how it was chosen.
  • Some leading questions to prompt this content:
    • For each relevant artifact: what kind of model or serving data is it? How was it optimized?
    • For each relevant artifact: what is its role in the serving application?
    • Were other model architectures considered, but ultimately not used? ("Why" should be obvious, and likely doesn't need to be addressed)

Serving Logic

  • The serving application handlers likely wrap the artifacts from training with some amount of logic.
  • That logic should be explained here, as simply as possible.
  • Some leading questions to prompt this content:
    • How is the request transformed into inputs to the model artifacts?
    • How are the outputs of the model artifacts transformed into the prediction schema?

Potential Future Directions

  • There is always room for improvement. List any of the ideas that came up during experimentation that could potentially be worth exploring in the next iteration.
  • To the writer: remember that your audience here includes your future self. Write these ideas with enough clarity that you could theoretically act on them multiple years from now.