Video: Navigating AI safety and security: key considerations for enterprises | Duration: 2038s | Summary: Navigating AI safety and security: key considerations for enterprises | Chapters: Welcome and Introduction (9.498876631448105s), Security and Privacy Challenges (132.1938766314481s), Comprehensive AI Security (469.7438766314481s), AI Safety Approaches (653.1589166314482s), Secure AI Solutions (1248.168876631448s), Model Safety Assurance (1423.8188766314481s), Conclusion and Q&A (1651.328876631448s), Conclusion and Gratitude (1969.963876631448s)
Transcript for "Navigating AI safety and security: key considerations for enterprises":
Hi, everyone. Welcome to our, webinar today. This is Cohere's webinar on safety and security. We're so glad to have you here joining us, to talk about the the key considerations for enterprises in navigating AI security and safety. By way of intros, enterprises need to maintain pace of innovation as we know, but also to make sure that the AI deployments are secure and safe to avoid massive financial and reputational risks. We know one of the main challenges that enterprises face is that they're required to satisfy an ever growing list of standards and requirements. And at Cohere, what we hear a lot from our customers is that they consider LLM safer use not only when they're both safe and that they don't result in harmful outputs, the model's responding in ways that are are unsafe to users, but they also require the models to be secure and that systems and data can't be breached. We know that enterprise AI risks and all of the corresponding mitigation strategies are distinct from AI models that are integrated into consumer facing chatbots or smartphone AI systems that are more available to the general public, and, you know, con connected to personal or publicly available data sources. So today in this session, what we're really focused on is from the enterprise mindset, what is the state of Secure AI today, and how can you innovate to make sure that you are keeping safety and security front and center in a way that's very much appropriate for for enterprises. The latest introduction here, I am Diane Chang. I am the, AI safety product manager at Cohere, and I'll hand it over to Prutha. Yeah. Hey, everyone. Thanks for joining us. I'm Prutha Parekh. I lead security at Cohere. My team is responsible for functional areas such as identity and access management, security operations, product security, and governance risk and compliance. Looking forward to an insightful conversation on secure and safe AI. Fantastic. Maybe to start off, since, we're already talking about some of the areas that your team, really focuses on for our customers, Prutha. Would you mind sharing what do you see as the main security and privacy challenges for enterprises when working with AI? Yeah. So data privacy and compliance, is top of mind for enterprise customers. AI systems, generally interact with vast amounts of data, and this includes sensitive information. Right? So preserving the sanctity of that data is key from a privacy and security standpoint. We here at Cohere integrate privacy considerations into the development of AI systems, right from the outset. It includes designing systems, that ensure data protection and also provide enterprises with control over their data both to private deployments. And this is particularly beneficial, for industries with strict data residency requirements, such as health care or finance. And for these organizations, ensuring that their sensitive data is handled securely really also helps them comply, with with regulations and avoid legal consequences, as well as, reputational damage. Also wanna, now highlight, from a security standpoint, we imbibe security considerations into our product life cycle, from the get go. We are ISO 27,001 and SOC two, type two compliant. And then pivoting a little from data privacy and compliance, I think enterprises who are working with AI should also evolve their operational security stature. Right, with AI systems, identity is the new perimeter. The foundational concepts of traditional security are still very applicable, but with agents being a component of that identity perimeter, areas such as access control, logging and monitoring, they're all evolving, within the realms of, operational security. Right? It's a vanishing boundary, and that vanishing boundary also, makes AI specific security, considerations around model security, supply chain security, extremely, important and paramount for organizations, to evaluate, and and address. And then the last point I will add on this, Diane, is that, you know, enterprises also, care about, accuracy and explainability. Our models and products are optimized for retrieval augmented generation drag, which offers, that verifiable accuracy with inline citations, driving, you know, relevant responses. Nice. This is such an interesting dynamic, I think, that your and a set of feeds your team has to pull up to both be very strong in what's considered traditional security, but also be at the forefront of understanding all the unique and new risks, specific to AI models. Thank you. Thank you for sharing. I guess in that light then, what's Cohere's approach to mitigating the risks, in order to encourage successful AI deployment? Yeah. So, all our products are designed with security and privacy at the core, to protect customer data. We offer private deployments, and this includes both private cloud and on premise deployments, and that helps us securely deliver AI solutions to customers where their data remains. Right? So it's within their own control. It's within their secure and air gapped environments, and it ensures complete, control and ownership, of of their, data while allowing them to leverage the power of these AI applications, and not compromise on, data privacy, which, again, I I will emphasize that it is a differentiator for regulated industries like finance, health care, manufacturing, energy, critical infrastructure, and and the public, sector. And then when we talk about data security, LLMs in general and agentic AI applications are handling very sensitive information in the enterprise context. Right? So you have to think about safeguarding against vulnerabilities such as unauthorized access, data disclosure, data exfiltration, theft, and, you know, ensuring that we're safeguarding against, against those, vulnerabilities. We will not have access to an enterprise's data unless, they they want us to. I also wanna highlight, a resource for our audience here. We have an AI security, ebook that outlines how enterprises can mitigate cyber threats, and safely deploy AI, in their private air gapped environments. It has, technical information around OWASP, top 10 for LLMs, and then talks about best practice, guidance on, you know, steps that an enterprise can take, to safely integrate, these applications in their environments. Thank you for sharing. The link for our ebook is going to be provided or has been provided in the comments for for those of you following along. Definitely welcome you to to check it out. It's got a lot of really helpful information, based on what Prutha outlined here and more. Prutha, can you also then share a little bit more about Cohere's framework for ensuring comprehensive AI security? Yeah. So, we recently, published our secure AI frontier model framework. It describes our holistic approach, that enables enterprises to build safe and secure solutions for their customers. The framework, extends beyond traditional, security, and is applicable across the entire stack all the way from the application to the infrastructure, the models, the agentic layers, as well as the datasets. Right? So let's talk about model security. Within model development, we simulate adversarial attacks on our systems, to test for vulnerabilities and improve defenses. We also evaluate the models, against internal and external benchmarks, And we scan the model for a wide range of model specific vulnerabilities such as deserialization attacks, graph payloads, and model, backdoors. When it comes to data security, we carefully track the lineage of our training data to guard against data poisoning and malicious content. Then there's application security, and we conduct annual party audits, and pen test to identify and address these vulnerabilities, prior, to both our model as well as product releases. And then there are components of traditional security, you know, threat modeling and ensuring that, security architecture reviews, are are performed for features that are embedded into, our products. Infrastructure security is also an extremely important, component of this entire, life cycle. We can bring our models to your infrastructure, through virtual, private cloud and on premise deployments. So it allows you to customize your security configurations and controls, and we don't have access to either the computing infrastructure or the data as I had, previously, mentioned. And then there's a wide range of other things like access control, such as role based access control and the principle of least privilege, that is implemented across these various layers, truly making it, right, that holistic, holistic approach. I also wanna highlight that Cohere is one of the founding sponsors, for coalition of, Secure AI, and we are the workstream lead for supply chain security, one of the work stream leads, and we collaborate with industry peers on best practice guidance, around that. So our Secure AI frontier model framework offers a very comprehensive view both in security, and safety. Diane, I feel that this is, a good segue into talking about, AI safety. In a recent McKinsey report on AI security, the potential for significant model, output inaccuracies was considered as the highest threat. And then it was followed by cybersecurity, risks also. Can you share more about how we define safety, in an enterprise context, and how we build and maintain customer trust? Yeah. Absolutely. The report is such a good call out because it does echo a lot of what we do hear from customers as well. And what, we've learned broadly is that customers enterprise customers specifically, they'll consider LLMs safer use. You know, the definitions and the criteria are quite broad. But and foremost, not probably not surprisingly is that they wanna make sure that the models, the LLMs that that, that, Cohere provides are performing well. They're doing what what the user intends them to do. Right? That's, of course, about fundamentally model performance and accuracy in in following the instructions of the end user. And oftentimes, if they don't follow those those instructions, if they're not accurate, that could unintentionally lead to unsafe consequences. And so even though this is about core performance, customers think of that as part of what makes a model safe for use. And of all, this is what I think is part of our Cohere's unique offering to enterprises is that we hear that customers really want to have a strong degree of control over the level of safety guardrails that are, that they can set on a model to make sure that it's most suitable for their use cases. These are obviously very, powerful general purpose models that are meant to be highly performing across a range of use cases, whether the customer is in the financial services sector and is maybe, you know, a financial analyst or if they work in manufacturing and work on, maybe, like, workflow opera or operations process, optimization. These are very different types of jobs, and they have very different use cases and different needs. And so what we provide to our customers is a feature called safety modes, which essentially is two different levels of safety guardrails that the customer can set when they're deploying their model into either a brand new product that they're using, a user interface like a chatbot, or when they're integrating it into an existing model. So they could choose either a more strict mode, that really is follows very rigid, safety guardrails or what we call contextual mode, which is a little bit broader, a little bit more personasive, which is more appropriate for more creative use cases such as if, marketing team is using a model to help draft, marketing blog post as an example. And and we wanna also prevent part of what is important to safety is making sure that the model isn't so safe as to not be useful to users, and so this is why we offer that degree of control. And then lastly, of course, as you were sort of referencing, obviously, safety and security are so intertwined. This is something we also hear all of the time. So as an example, for a financial services customer, let's say, if the model responds with, private information of, one of their downstream customers, to their end user and the end user should not have had access to that kind of PII, that's a terrible, terrible situation that we wanna prevent. Right? That that represents actually a failure of data access controls. It's not actually an error with the model, but from the perception of the customer organization that's deployed that model, that's considered an unsafe behavior, and we're very mindful of that. And so that's why safety and security and getting the access control layer right, getting the models training, on safety, model training right are that's so important that the inter the interplay there, we we think about it, across the stack. And so that's why when we think about our mitigations to meet these customer needs and customer expectations, we really want to make sure that we're thinking about the model, the system, the infrastructure, through to the applications, as we begin to build applications as well, under really addressing those underlying safety and security risks, together kind of hand in hand. Yeah. You know, and I'm sure the audience is gonna, the fact that safety and security are so intertwined, this is gonna resonate, with the audience as well. We've heard it from our customers, and, there's definitely, you know, that that sentiment that you, it's impossible to sort of separate the two. And when most people are talking about that, they are talking about it from a wide range of things, and it includes, what has been traditionally defined as, security and, you know, safety with with the upcoming, yeah, models. Dan, can you share a bit more, about, Cohere approach to model safety when developing solutions for enterprise customers? Yeah. Absolutely. So when it comes to I think I've talked a little bit about this, but to go in greater detail, we obviously care a great deal about the kinds of risks that are most salient to enterprises. We wanna give customers assurance on on the areas that I've already mentioned. And in order to do deliver on that promise, we evaluate our models for safety throughout the model development process and iteratively improve to make sure that every new model that we deploy is safe for enterprise customers to use and also that each model is accurately adhering to the the guardrails that are set, whether it's the the strict mode or contextual mode. So those are sort of the two main criteria that we look at that we're optimizing towards to ensure that every newly launched model is, at least as good if not improving upon the safety of the prior model that we we launched before to meet customers' needs. And in terms of how we approach safety, how we define it, in in this approach, what's, I think, different, from our competitors, what makes us unique in our rigorous enterprise focus is that we don't look at the more speculative long term AI risks that do get a lot of flashy media coverage but are not realistic. We hear this from our customers all the time. Just really not relevant to their day to day focus when they're trying to run their business. We focus on what they the enterprise customers care about, which is the pragmatic near term risks that are more actually more likely to occur in their enterprise context. So an example here is, our customers care really deeply, for instance, about ensuring that if they use our models for a human resources related task like summarizing resumes, that our models are not gonna be biased, that they're gonna be able to summarize, resumes for a range of of, candidate types in a in a fair manner. Or if a customer is using our models to write code, we wanna make sure that the code that's generated won't create security vulnerabilities in their code bases. You know, these are very near term risks that we, can understand very clearly where we're also developing, the sort of leading edge, in the industry, evaluations and mitigations to really serve our our enterprise customers. And so secondly, in our approach, other than focusing on these pragmatic near term risks, I referenced this a little bit, but what we found and we've learned is very important to our customers is giving them the degree of control that they need over the level of safety guardrails to set when they're either integrating our, LLM into their products, or building a new product with our LLM. And this feature I mentioned is called safety modes, and it allows them to set the guardrail that's most appropriate for their use case. So I think I mentioned some examples. Other examples might be, we have customers that are building customer support chatbot as an example. And so when your customer support agent is using this chatbot to to be able to quickly query information to make sure they're getting the the best information to their downstream customers as possible, we don't want the customer support chatbot to accidentally, respond in ways that are inappropriate or toxic. So in that case, they might wanna select the more strict mode, to make sure that the the responses from the model always stick to the the most, kind of safe kind of, strictly safe ways of responding to the downstream end user. But on the other hand, let's say, a customer sometimes our customers like to use their own files to research topics such as medical topics, and maybe medical topics can be a little bit gory even in the in the real, you know, real context of of the subject matter that's very pertinent to your work. And so we don't want a model that's so overly cautious about kind of graphic medical subject matter that it refuses to provide helpful information. So in this case, maybe the customer there might choose to implement contextual mode, which will still prevent, you know, legitimately unsafe behavior that has nothing to do with the work use case, but won't be overly burdensome in in preventing the model from answering usefully to to the context, and the subject matter at hand. So these these are kind of the two main prongs of our approach to safety. One is the pragmatic near term, risks that occur for enterprises that enterprise customers care the most about. And secondly, giving them the control to set the the guardrails that make the most sense for their use cases. Yeah. Well, thanks thanks for sharing, all of that. maybe, Prutha, if we can return to you, would love to hear from you a little bit more on a con maybe a concrete example, like, a case study on how we've helped customers on security, especially in in a regulated industry. Could you share, an example here on on how we've worked with them? Yeah. For sure. I'm gonna share an example, from one of our customers in the financial, services sector. So with RPC, we're co developing North, for banking. It's a tailored AI solution, and it's designed to enhance workforce productivity, while meeting, no strict, security and privacy requirements, which is extremely critical, for, the regulated industry. So RBC has partnered with Cohere, for secure, and private talent deployment, and our lot for banking initiative, sort of, showcases how these financial institutions can harness AI while keeping full control, over their, data. Right? So we bring their models. We bring our models, to their, environments. With agents, RBC is really excited about unleashing the power of the technology, as it puts, the control into their hands and the hands of their subject matter experts, across, different, business lines. Right? So think about this from from the perspective, right, the ability to code create, share, and use agents as well as leverage the power of the technology, at the same time for their entire employee base. But doing so in a way that has the appropriate safeguards, as well as, observability, in place. North, is designed, to meet strict security and privacy standards, and our platform, the North platform is optimized, to run-in these private air gapped, environments, and organizations are able to safely integrate all of your sensitive data in one place. And so while this is, an example for RBC, this also holds true for, a lot of the other, customers, in this space. Right? So the same case, Diane, you mentioned for health care. It would be a similar scenario, as we're bringing, right, that intelligence, those models, over to, to their networks networks and then allowing customers to, have control, or over their data and then, putting in their security, consideration so that it embeds within their stack, and we don't, bring bring that any of the data back to, to our networks. Yeah. Yeah. Diane, you know, also wanted to hear, and, you know, an example of how we are supporting customers, with model safety. Now, would you mind sharing, an example, where, we've had, that with our customers? Yeah. Absolutely. I referenced an example, a little bit more, a little bit earlier, and I could share a little bit more about that maybe. Specifically, we've developed the hiring fairness evaluation, that applies to large language models. This is the in the industry, based on feedback we've gotten from a customer that's a leading software giant, that builds human resources software. The reason we you know, this is industry leading is that there's a wealth of AI fairness research out there that's applied to other forms of machine learning. But since large language models are so new that the kind of applying that to LLMs, this is still a new frontier. And given our focus on enterprises, we thought it was very important to sort of push the leading edge of this work. And so what our evaluation does is that it measures the fairness of our models in generating summaries from resumes based on gender or race. In other words, if the characteristics of gender or race on a set of resumes are changed, we'd look at whether the model, the summarization that the model provides, does it reduce in quality? And when we evaluated our model based on the evaluation we developed, we found that command a, which is our our latest, 111 b large language model, it's actually very strong and robust on fairness based on gender. Essentially, there was no change in quality when the characteristics of the candidate and resumes were were changed, and also quite strong on race with only 1% of failures when those demographic factors were changed on resumes. So this you know, we developed this, as I mentioned, to give customers that added assurance on our our model's fairness. And we're now applying, this research to other types of enterprise use cases in other sectors such as health care in order to really make sure that we could develop this assurance to our customers that we are mindful and understand, these concerns about bias and fairness in the course of their work, which is so deeply, already embedded in the practices, for instance, in the medical field or in the human resources field. And we're translating that into how we're training our models and evaluating our models and really being on the the leading edge on this. I believe our team will share, if they haven't already, linked to the the research paper, about this evaluation. And you can also read more about that in our command a technical report, which also I think there was a question about our overall our model, performance, metrics that they're all outlined in our tech technical report. I welcome you to to dive into that. Yeah. No. And what's that, Diane Chang? Taking security and safety, inherently into the design of our models and products, absolutely provides, our customers, with that assurance, that they could fully capture the value, of AI while balancing the risks, near term risks. More details on everything that we have discussed can be found in various reports that we've, we've shared. I wanted to highlight, our secure AI frontier model framework. It covers aspects of both, safety and security, and goes into much more details of the holistic view that we take, when we build our products and applications. And then there's the command a technical report, the AI security ebook. They all, go into additional technical details around how customers can deploy these, applications, safely. I wanna end by, saying that, right, in building up, AI security in general, you're gonna overall improve enterprise security readiness. And we're here to partner, with you and answer your questions, so that you can feel more confident in innovating, in the enterprise AI space in a more, secure way. Wanted to open it up to questions. I know we have some coming, up in the, in the chat. There Yeah. I could take for the I know there are a number of folks requesting, the link for the benchmark on fairness. Our team will share that shortly, and absolutely do some of the the callouts that fairness is broader than gender and race. It's also about age or other factors. Absolutely. This is so important. The gender and race, characteristics are are a starting point for us. This was how where we started in our research, and we're constantly looking to expand to a range of, demographic factors, you know, really following where existing science also, had has led us on on a in fairness and really thinking about how that applies to to the LLM field. That's very much a a big part of what we're continuing to research. So great call out there. Yeah. Prutha, are there any of these questions? Let's see. Take another look. I think some of the questions are around our model performance. We will also, reshare the link to our, technical report. You could see you can find that in the technical report there, including, details on how we approach model training, some of the techniques as well as results. Yeah. I'm going through through the list. There's a question around division of labor between, CISO SOC and, DevSecOps. So from our standpoint, we take a holistic view where various aspects of security are incorporated, within, within our development life cycle. There are areas, such as infrastructure security, application security, and a lot of these go, hand in hand. And then we also focus on AI specific, considerations. So, we are focused on model security, and we have various, card rails and, things that we do in terms of testing our models, prior to deployment and prior to making them available to our customers for, for deployments. Let me see. Someone asked if we have any fixed guardrails that cannot be removed and adjusted. We absolutely do. Yes. In our, I think, in our model training, we definitely account for basic safety measures. For instance, we will always wanna make sure that the model cannot generate child sexual abuse material. That's one of the most, kind of severe outcomes we wanna prevent, and so that is always part of of how we've trained the models inherently that can't be removed or adjusted. So we're very mindful of that. The areas, where we allow customers to control are are types of risks that, may vary to different degrees based on different legitimate business use cases. And, again, in the technical report, you can find how we've defined those different categories and and see our evaluation results based on on the different, types of guardrails. Right. Right. I'm going through the last. I wanna cover, one more question. There was a question around the use case, around the bank and, the chatbot, that someone would be developing and, providing advice how do I generate card rates, for the domain specific use case. So focusing on on the the use case in particular, we do have, the AI security ebook that talks about, OWASP top 10, for LLMs. And it goes into, technical details around, all of the top 10, vulnerabilities and what you could do, to implement that within your own setting. So definitely, I encourage you to, go through that, and, you know, gain insights, on what we're doing on on our platform and the things that can be done on your own private deployment. Alright. Awesome. Thank you all for, for joining us, today. Hoping, that, this was, useful. And, you know, thanks Diane for a very insightful conversation on, safe and secure AI, particularly in context of the enterprise. And thanks everyone for, joining us, today. And, you know, feel free to reach out, to us. Thanks. Thank you. Thanks all.