In recent years, large language models have moved from academic curiosity to publicly accessible technologies that can be tapped into for a myriad of applications. Thanks to services like Azure OpenAI now available in Swiss datacenters, organizations like ours have access to powerful AI capabilities under conditions equivalent to our own control. But the promise of AI isn’t just in the models; it’s in the tools that allow us to use these models effectively. That’s where AI orchestration frameworks like Semantic Kernel and LangChain come in, simplifying the integration of AI services.
Picture this: an internal company hack day focused solely on integrating AI into our vulnerability collaboration platform. The challenge? To ideate, develop, and implement a use case that delivers immediate value – all within a single day.
Selecting the Use Case
The first step involved brainstorming and decision-making, balancing added value against implementation complexity. Given our one-day time frame, we chose to focus on summarizing vulnerability reports in a human-understandable way. These reports are filled with unstructured data containing the key details of vulnerabilities. Users typically need to read through the full text of the reports repeatedly during their lifecycle. Thus, generating concise summaries can significantly increase efficiency and facilitate interpretation at scale.
Sample Report
What is Semantic Kernel?
Implementation
var builder = new KernelBuilder();
builder.WithAzureChatCompletionService(
configuration.GetAzureOpenAiDeploymentName(),
configuration.GetAzureOpenAiUrl(),
configuration.GetAzureOpenAiApiKey());
Kernel = builder.Build();Preprocessing and Prompting
We preprocessed the data context, and stripped all formatting (Quill Delta in our case) from the full text reports:
We seeded synthetic test data in order to evaluate and refine the results of our prompt. Then we engineered our summary prompt based on inspiration we got from this resource. Prompt engineering took most of the time and required several iterations to get consistent results across our test set.
[SUMMARIZATION RULES]
DONT WASTE WORDS
USE SHORT, CLEAR SENTENCES
MAXIMIZE DETAIL
FOCUS ON THE CONTENT
FOCUS ON THE TYPE OF VULNERABILITY AND THE IMPACT
[...]
Summarize this report:
{{$input}}
Semantic Function
Next we created our first Semantic Function «FindingSummary» and registered it in Semantic Kernel:
var functionConfig = new SemanticFunctionConfig(promptConfig, promptTemplate);
var summaryFunction = kernel.RegisterSemanticFunction("SummarizeFindingSkill", "FindingSummary", functionConfig);
A Semantic Function consists of two main elements:
- The prompt template containing the prompt and placeholders for input parameters as shown in the previous section
- The prompt configuration controlling parameters like maximum number of tokens and temperature
Summary API
Finally, we wrapped up the day by implementing our finding summary API endpoint. It performs permission checks, fetches data from the database, and then invokes the semantic function:
var summary = await summaryFunction.InvokeAsync(input);
The function then returns the summarized finding, which might look like this:
An Insecure Direct Object Reference (IDOR) vulnerability has been discovered on www.example.com, specifically within the /users endpoint. This vulnerability allows unauthorized users to access or modify sensitive data belonging to other users.
Take Away Messages
- Microsoft’s Azure OpenAI makes it feasible to leverage large language models in Switzerland even with sensitive customer data, given special arrangements as modified content filters & abuse monitoring.
- Semantic Kernel simplifies AI orchestration to an extent that even a day-long hackathon can provide immediate value.
- The primary challenge was engineering the prompt to obtain high-quality results.
The Road Ahead
Responsible AI Usage
- Our models are never trained on sensitive customer data.
- Customer data remains secure and separated within defined boundaries.
- We uphold user privacy by ensuring that no prompts are logged during AI interactions.
