Monitoring AI Systems

Carlos Phoenix is a CISO and Advisor with over 20 years' experience. Previously, he was the Product CISO at VMware and spent 15 years doing consulting/audit at Deloitte, KPMG, Cognizant, and Coalfire. He holds multiple security and compliance patents, published NIST publications, and has contributed to regulatory standards for PCI and NERC CIP. Carlos is certified as a Certified Information Systems Security Professional (CISSP), Certified Information Systems Auditor (CISA), and Project Management (PRINCE2).

Adapt or die. That adage applies to new technologies as surely as it applies to living organisms. Organizational and human behaviors that evolved in concert with the software development practices of the last decade will take time to adapt to the reality of new AI systems – but adapt they must.

Traditional system monitoring tracks uptime, detects vulnerabilities, helps manage capacity, or triggers contingencies when a system goes down. Institutional inertia often masquerades as progress in older organizations, and failure to ensure systems remain fit for purpose is common.

This aspect of software becomes more important over time because the decision makers who signed off may have departed, or the business may have prioritized new systems, rather than retiring older ones. Legacy systems often stand the test of time even though there are newer, more advanced systems, because decommissioning these systems can require a lot of effort. This aspect of system monitoring is crucial to understand.

As we enter the dawn of AI system development, organizations have a tremendous opportunity to rapidly experiment and iterate with new technology. The speed with which a project can go from idea into prototype is unparalleled. Identifying a baseline to compare the output of the AI systems is crucial. Next is training the AI based on relevant datasets and fine tuning the solution until it becomes fit for purpose. At this point, the system can be tested by the end user and rolled out with care.

For traditional software development this might feel like the job is complete, but for AI this should not be where the process ends since AI is dynamic, non-deterministic, and highly sensitive to inputs.

The initial requirements, business case, and process where AI is used will require careful oversight to ensure that fit for purpose remains appropriate. This will make monitoring of the system quite different to traditional software tools, especially those that were Commercial-Off-The-Shelf (COTS) products.

Monitoring of AI systems requires diligent oversight of the agent, model, and ecosystem where the system operates. With the addition of tremendous power to deliver value by producing extremely useful outcomes, AI’s focus on learning means data flow and usage can result in errors, policy violations, and security risks. Just setting global parameters at the corporate account level will not be enough because the complexity of AI can result in unintentional workarounds. The very nature of AI and its ability to generate content, collect data, and act, necessitates monitoring just as one would monitor a traditional employee’s work.

For example, scripts can often be written quickly but maintaining them can be a challenge. The inputs of the script will change, and these changes can lead to errors. The output of the script may need to be adjusted as the context around the problem evolves. New versions of software invoked by the script can render it broken. In this aspect, AI systems may share a lot in common with scripts. Both require knowledge of their existence, role, reliance on other systems, and the development to anticipate the inevitable breakage.

Monitoring AI systems will require an ongoing conversation between IT and the business. How often should AI system developers review the system’s fit for purpose? What kind of monitoring is appropriate? What are the risks that will need to be monitored?

IT tends to monitor systems on a range of frequencies:

●      Hourly

●      Daily

●      Weekly

●      Monthly

●      Quarterly

●      Yearly

The systems that have the largest impact on the business will need to be monitored with a higher frequency. However, more frequent monitoring also increases costs in human capital, digital resources to accommodate contingencies, or redundancy.

As businesses develop and deploy AI systems into production, they should determine and scope the level of monitoring in the requirements phase. The business should be aware of the monitoring required and its frequency to sign off as a known cost of using this powerful technology. Clear risks (known and unknown) around monitoring should be documented.

The AI path is not a “set it and forget it” panacea.

The dynamic nature of AI and its newness increase the risk. This requires additional monitoring and ongoing evaluation of its fit for purpose. Because this technology derives its power from its dynamic nature, hourly monitoring upon deployment may transition to daily checks once businesses have a better understanding of usage and other variables. Once confidence in the system and baseline data is confirmed, the monitoring frequency could be decreased from daily to weekly (and eventually from weekly to monthly, etc.).

The specifics of what to monitor and how to track risks should be part of the business plan for embarking on AI system development. This is a crucial control and should not be ignored. It goes without saying that yearly reviews (or even quarterly audits) will be inappropriate for most AI systems. In many cases, the use of AI is similar to embarking on the agile development method with short sprints (1-3 weeks) and incremental changes. In this paradigm, monitoring may become a crucial step in the software development lifecycle (SDLC) process. This presents an opportunity to validate security settings, catch any data leaks, evaluate the ecosystem of APIs and interconnected systems interacting with the AI system, as well as to ensure it remains fit for purpose.

Monitoring AI systems is a key control to AI system development and use. With tools this powerful, monitoring cannot be an afterthought. Monitoring should be included in plans and budgets throughout the development journey. The frequency of the control should be carefully evaluated and signed off by the business. Using monitoring to ensure clarity with AI systems will allow your business to adapt and thrive.