MLOps – Fast-tracking AI Ethics?

MLOps – Fast-tracking AI Ethics?

MLOps, the big-bet for scaling Machine Learning, promises seamless development through to in-life use of models using automated, DevOps CI/CD workflows, but what does this mean for an AI Ethics discipline that has focused its fire-power on single-shot development projects?

In this series of posts I’m looking at some of the effort involved with implementation of AI Ethics, covering some key considerations for the board C-Suite, and some perhaps overlooked realities of applying AI Ethics principles to real-world adoption.

MLOps – DevOps for Machine Learning

The benefits of DevOps, a working approach that fuses IT development and operations into a seamless, highly-automated feedback loop of Continuous Integration, Delivery & Deployment, are coming to Machine Learning. It’s fair to note that as a nascent capability Machine Learning can struggle with linkage between the roles through data analyst, data scientist, and handover to systems implementation for packaging and release to real-world use. MLOps promises the same level of code-free integration, heavy on automation, as DevOps, allowing mature, traced and near real-time release of models in their packaged applications. That’s a far cry from the linear, single-shot development waterfall assumed by many of the AI Ethics frameworks and approaches.

MLOps = ML + Dev + Ops

MLOps = ML + Dev + Ops

Baking in ethical requirements

So how do we ensure that the highly relevant considerations of ethics and responsible-use remain represented throughout the MLOps cycle? Here’s some tips:

Harness development tracking. Good news, MLOps brings traceability goodness, if done right. Using an end-to-end workflow tool to manage the development, packaging and deployment of models in applications brings with it clear paper-trails of data sources, models used, training approach employed, etc. When something DOES go wrong, we can review the source of the problem quickly and adjust course. Mix-and-match and this transparency can be lost.

Use ethics metadata. Rich metadata on models in repositories is an investment that will repay handsomely. Ensure this includes information that will allow application developers to make informed decisions on use, including qualitative data, social reaction, social-acceptability testing, sensitivity (human) of prediction, etc as well as the usual suspects on technical specification.

Streamline testing of NFRs. Testing automation requires codification of requirements. Although some aspects of ethical considerations can be codified and automatically tested (for example, measures of bias in model predictions), many aspects are external Non-Functional-Requirements (external NFRs). NFRs are often tested manually, and some can be highly subjective. To get this right, ensure your CI environment is ready to test ethical NFRs. The sensitivity of the application, data or prediction should guide on the required threshold to trigger NFR review.

Ensure you monitor live-data models closely. Dynamic data and evolving models bring an extra dimension, requiring ongoing probing of intended mitigations for ethical risk during operations monitoring to ensure the model is adhering to the expected behaviour that got it to production. As the user may have spent some time customising, roll-back may have a significant impact on user experience. There’s a trade-off here on predicting unintended use and input vs. accepting risk of misbehaviour; where that balance sits is an informed decision to be made as part of specification of requirements.

Strengthen and empower the requirements/testing combo. The interworking between product owner, regulatory assurance and experience assurance (XA) roles is the lynchpin for assuring representation of ethics considerations. The product owner represents in their requirements not only the intended functionality for the customer, but also the voice of the societal license to operate, the policies of the organisation with respect to acceptable use, and the values the organisations espouses. The XA lead ensures these are fully represented in the final experience of the product. The regulatory lead ensures the representation of requirements with respect to human rights and other application regulations. These roles must be fully trained in understanding ethical concerns in the use of ML tech.

Ethical MLOps is only as good as its organisation

It’s important to note that a well-formed MLOps structure operating in an organisation that has not addressed the up and downstream drivers of ethical and responsible outcomes will fail. Most importantly, product owners who have no direction from leadership; no agreed norms of responsible use to reference; a lack of input sourced from the societal-context of the deployed product; and conflict between a company’s communicated values and the behaviours that bring success at the organisation will fail.

Ultimately, in addition to expanding the goals of the product, and so the roles of the team, MLOps makes demands that the business is positioned and conversant in understanding the concerns of responsible use to empower the decision makers in an MLOps project to make informed decisions.

TechInnocens provides C-Suite and board level advice on the implementation of AI Ethics frameworks using pragmatic, achievable interventions grounded in the reality of technology adoption.

Terms and Conditions: Emerging Technology Quick Scan Assessment

This assessment tool and all associated documentation has been prepared Maior Natu Pty Ltd ACN 649 020 039 as trustee for Sancus Trust ABN 46 507 741 242 trading as TechInnocens (TechInnocens) and is provided to you on the following conditions:

 

  • this tool and documentation is strictly confidential and is solely for your own use and that of your professional advisers. It must not be provided to any other party without the prior written consent of TechInnocens, which may be withheld in the absolute discretion of TechInnocens;
  • the content in this tool and documentation does not constitute advice (including tax, legal or ethical advice);
  • you should consider the appropriateness of the information contained in this tool and documentation and make your own decisions based on your individual and/or corporate objectives and needs. You should obtain independent legal, financial and/or other professional advice, as appropriate, relevant to your individual and/or corporate needs before making a decision based on this information.
  • you acknowledge that TechInnocens is the owner of the intellectual property owned or used in connection with this tool and documentation, including without limitation: any patent, know-how, trade mark, service mark, copyright, invention, design, trade secret or confidential information, and any other intellectual property or rights whether registered or not used in connection with or forming part of any business of TechInnocens (Intellectual Property);
  • you hereby disclaim any interest (implied or otherwise) that you may have or may be assumed to have in the Intellectual Property;
  • TechInnocens has the right to deal with the Intellectual Property in any way whatsoever, including to assign or licence the Intellectual Property to any third party; and
  • you agree not make any claim against TechInnocens in relation to the Intellectual Property.

 

Statements in this tool and documentation are made only as of the date of usage of the tool unless otherwise stated. TechInnocens is not responsible for providing updated information to you. Neither TechInnocens nor its officers make any representation or warranty as to, or take responsibility for, the accuracy, reliability or completeness of the information contained in this tool and documentation. Nothing contained in this tool and documentation , nor any other related information made available to you is, or shall be relied upon as, a promise, representation, warranty or guarantee, whether as to the past, present or the future. 

 

To the maximum extent permitted by law, TechInnocens and its officers disclaim all liability that may otherwise arise from reliance upon this tool and documentation or due to any information contained in this tool and documentation being inaccurate or due to information being omitted from this tool and documentation , whether by way of negligence or otherwise.  Neither TechInnocens, its officers nor any other person guarantees the performance of any proposed information referred to in this tool and documentation . You must accept sole responsibility associated with the use and/or reliance of the material in this tool and documentation , irrespective of the purpose for which such use or results are applied.

 

The copyright in all information contained in this tool and documentation is owned by or licensed to TechInnocens.  Except as expressly permitted, no information may be copied, reproduced, transmitted or re-distributed. All rights reserved.