Overview

Organisations have been growingly adopting and integrating a non-trivial number of different frameworks at each stage of their machine learning lifecycle. Although this has helped reduce time-to-value for real-world AI use-cases, it has come at a cost of complexity and interoperability bottlenecks.

Each stage in the end-to-end lifecycle involves different stakeholders that make decisions and perform actions that can modify data and/or ML components with use-case-specific but ever compoinding risks, resulting in a growing need to ensure a minimum-level of metadata is collected, tracked and managed. This becomes growingly important due to the need to ensure relevant overarching compliance requirements, as well as architectural requirements on lineage, auditability, accountability and reproducibility.

In this session we will dive into the challenges present in the metadata layer of large-scale systems, as well as tooling, best practices and solutions that can be adopted to tackle these challenges. We will discuss the rise of the metadata management systems, the challenges they have been able solve, as well as critical shortcomings where ecosystem-wide collaboration will be key from tooling-level alignemnt to ensure long-term robustness of these heterogeneous end-to-end platform.

Benefits to the ecosystem

In recent years we have experienced how the evolution of the areas of DataOps and MLOps have introduced further complexities that involve concepts such as data-versioning, model-versioning, model registries, ML experiment tracking, ML model deployment, ML model promotion, monitoring, etc. These latter developments have raised new challenges that the ecosystem has been able to tackle through extending existing metadata management tools, as well as the creation of new tools.

This talk aims to help further the discussion on existing best practices where metadata schemas, protocols and tooling has been succesful in enabling interoperability across multiple systems in these end to end platforms. We hope that it brings benefits to the ecosystem that go beyond this current session into actionable discussions, collaborations and open source contributions towards continuing the momentum on improving interoperability across the MLops ecosystem.