How we integrated AI in our product in one day

The story about our company-internal hack day, where we integrated AI in our product in one day

Von Lukas Heppler, Veröffentlicht am 19. September 2023 5 min Lesezeit
Hackday

In recent years, large language models have moved from academic curiosity to publicly accessible technologies that can be tapped into for a myriad of applications. Thanks to services like Azure OpenAI now available in Swiss datacenters, organizations like ours have access to powerful AI capabilities under conditions equivalent to our own control. But the promise of AI isn’t just in the models; it’s in the tools that allow us to use these models effectively. That’s where AI orchestration frameworks like Semantic Kernel and LangChain come in, simplifying the integration of AI services.

Picture this: an internal company hack day focused solely on integrating AI into our vulnerability collaboration platform. The challenge? To ideate, develop, and implement a use case that delivers immediate value – all within a single day.

Selecting the Use Case

The first step involved brainstorming and decision-making, balancing added value against implementation complexity. Given our one-day time frame, we chose to focus on summarizing vulnerability reports in a human-understandable way. These reports are filled with unstructured data containing the key details of vulnerabilities. Users typically need to read through the full text of the reports repeatedly during their lifecycle. Thus, generating concise summaries can significantly increase efficiency and facilitate interpretation at scale.

Sample Report

Example Report

What is Semantic Kernel?

But how do we achieve this? Enter Semantic Kernel, Microsoft’s open-source AI orchestration framework. This SDK allows developers to orchestrate series of prompts (semantic functions) and combine them with native code (native functions) to achieve a specific outcome. Microsoft employs Semantic Kernel in its AI copilots and assistants, making it a proven and developer-friendly open-source tool.

Implementation

Getting started with Semantic Kernel was easy and well-documented in this cookbook. After storing the model endpoints and API keys, we used the following code to instantiate Semantic Kernel:

var builder = new KernelBuilder();
builder.WithAzureChatCompletionService(
  configuration.GetAzureOpenAiDeploymentName(),
  configuration.GetAzureOpenAiUrl(),
  configuration.GetAzureOpenAiApiKey());
Kernel = builder.Build();

Preprocessing and Prompting

We preprocessed the data context, and stripped all formatting (Quill Delta in our case) from the full text reports:

We seeded synthetic test data in order to evaluate and refine the results of our prompt. Then we engineered our summary prompt based on inspiration we got from this resource. Prompt engineering took most of the time and required several iterations to get consistent results across our test set.

[SUMMARIZATION RULES]
DONT WASTE WORDS
USE SHORT, CLEAR SENTENCES
MAXIMIZE DETAIL
FOCUS ON THE CONTENT
FOCUS ON THE TYPE OF VULNERABILITY AND THE IMPACT
[...]

Summarize this report:
{{$input}}

Semantic Function

Next we created our first Semantic Function «FindingSummary» and registered it in Semantic Kernel:

var functionConfig = new SemanticFunctionConfig(promptConfig, promptTemplate);
var summaryFunction = kernel.RegisterSemanticFunction("SummarizeFindingSkill", "FindingSummary", functionConfig);

A Semantic Function consists of two main elements:

  • The prompt template containing the prompt and placeholders for input parameters as shown in the previous section
  • The prompt configuration controlling parameters like maximum number of tokens and temperature

Summary API

Finally, we wrapped up the day by implementing our finding summary API endpoint. It performs permission checks, fetches data from the database, and then invokes the semantic function:

var summary = await summaryFunction.InvokeAsync(input);

The function then returns the summarized finding, which might look like this:

An Insecure Direct Object Reference (IDOR) vulnerability has been discovered on www.example.com, specifically within the /users endpoint. This vulnerability allows unauthorized users to access or modify sensitive data belonging to other users.

Take Away Messages

  • Microsoft’s Azure OpenAI makes it feasible to leverage large language models in Switzerland even with sensitive customer data, given special arrangements as modified content filters & abuse monitoring.
  • Semantic Kernel simplifies AI orchestration to an extent that even a day-long hackathon can provide immediate value.
  • The primary challenge was engineering the prompt to obtain high-quality results.

The Road Ahead

Our hack day’s success marks just the beginning of an exciting journey. Upcoming more complex use cases include more mixing and matching of semantic functions, which query large language models, and native functions, which interact with the platform and its data. All this with the aim of providing a copilot experience, which can work with raw and unstructured data and turn it into human-understandable insights.

Responsible AI Usage

We’re committed to responsible AI practices. Rest assured:
  • Our models are never trained on sensitive customer data.
  • Customer data remains secure and separated within defined boundaries.
  • We uphold user privacy by ensuring that no prompts are logged during AI interactions.
With these principles at the forefront of our AI integration journey, we are ready to unlock the full potential of AI in the realm of cybersecurity.

Bleib auf dem Laufenden mit unserem Newsletter!

Du suchst Bug Bounty News, Hackerportraits und Erfolgsgeschichten aus dem Unternehmensalltag? Bleib auf dem Laufenden mit unserem Newsletter.

Hast Du Fragen?

Nimm mit uns Kontakt auf - wir freuen uns auf Dich!

Termin vereinbaren