Using Azure's APIM orchestration provides a organizations with a powerful way to scale and manage their Azure Open AI service without deploying Azure Open AI endpoints everywhere. Administrators can issue subscription keys via APIM for accessing a single Azure Open AI service instead of having teams share Azure Open AI keys. APIM delivers usage metrics along with API monitoring to improve business intelligence. APIM policies control access, throttling, and a mechanism for chargeback cost models.
There are four solutions developed to meet the needs of the organization from a sandbox to model a production environment.
- Azure Commercial API Management to Azure Open AI
- Azure Commercial API Management to Azure Open AI with private endpoints
- Azure Government API Management to Azure Open AI
- Azure Government API Management to Azure Open AI with private endpoints
Once the service is deployed, use the following section to understand how to access your Azure Open AI service via APIM.
-
Contributor permissions to subscription or resource group
-
Resource Group (or ability to create)
-
Azure Open AI service deployed
-
Azure Open AI model deployed
-
Azure Open AI service URL
-
Azure Open AI key
-
Azure Open AI service public IP
- Azure Government w/ Private Endpoint Deploy Solution Only
- How-to: Get the public IP address of Azure Open AI service
Each solution provides a simple one-button deployment. Select the "Deploy to Azure" button which will open the Azure portal and provide a form for details.
To use the command line deployment method, fork the library and use Codespaces or clone the forked library to your local computer.
- How to install the Azure CLI | Microsoft Learn
- Connect to Azure Government with Azure CLI - Azure Government | Microsoft Learn
- How to install Azure PowerShell | Microsoft Learn
- Connect to Azure Government with PowerShell - Azure Government | Microsoft Learn
The following architectural solutions support two use-cases in the Azure Commercial and Azure Government environments. Determining which solution to implement requires understanding of your current utilization of Azure.
-
Azure Commercial API Management to Azure Open AI
- Azure Commercial is primary cloud environment used by the team or organization.
- Developing proof of concept or minimum viable production solution.
- Isolated from enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
-
Azure Commercial API Management to Azure Open AI with private endpoints
- Azure Commerical is primary cloud environment used by the team or organization
- Pilot or production solution.
- Connected to the enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
-
Azure Government API Management to Azure Open AI
- Azure Government is primary cloud environment used by the team or organization.
- Developing proof of concept or minimum viable production solution.
- Isolated from enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
-
Azure Government API Management to Azure Open AI with private endpoints
- Azure Government is primary cloud environment used by the team or organization
- Pilot or production solution.
- Connected to the enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
Use API management deployed to the Azure Commercial cloud using public IP addresses for accessing APIM and for APIM to access the Azure Open AI API. Access to the services is secured using keys and Defender for Cloud.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\public-apim.bicep --mode Incremental# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\public-apim.bicep -Verbose -mode Incremental- Now that APIM is deployed and automatically configured to work with your Azure Open AI service
Use API management deployed to the Azure Commercial cloud using private IP addresses for accessing APIM and for APIM to access the Azure Open AI API. Access to the services is secured using private network connectivity, keys and Defender for Cloud. Access to the private network is controlled by customer infrastructure and supports internal routing via Express Route or site-2-site VPN for broader enterprise network access like on-premises data centers or site-based users.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\private-apim.bicep --mode Incremental# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\private-apim.bicep -Verbose -mode Incremental- Now that APIM is deployed and automatically configured to work with your Azure Open AI service
Use API management deployed to the Azure Government cloud using public IP addresses for accessing APIM and for APIM to access the Azure Commercial-based Azure Open AI API. Access to the services is secured using keys and Defender for Cloud.
Network routing from the APIM to the Azure Open AI address uses Microsoft's backbone, eliminating public routing.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "usgovvirginia"
$subscriptionName = "MySubscription"
az cloud set --name AzureUSGovernment
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\public-apim.bicep --mode Incremental# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount -Environment AzureUSGovernment
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\public-apim.bicep -Verbose -mode Incremental- Now that APIM is deployed and automatically configured to work with your Azure Open AI service
Use API management deployed to the Azure Government cloud using private IP addresses for accessing APIM and for APIM to access the Azure Commercial-based Azure Open AI API.
Access to the services is secured using private network connectivity, keys and Defender for Cloud. Access to the private network is controlled by customer infrastructure and supports internal routing via Express Route or site-2-site VPN for broader enterprise network access like on-premises data centers or site-based users.
Network routing from the APIM to the Azure Open AI address uses Microsoft's backbone, eliminating public routing.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "usgovvirginia"
$subscriptionName = "MySubscription"
az cloud set --name AzureUSGovernment
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\private-apim-azure_government.bicep --mode Incremental# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount -Environment AzureUSGovernment
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\private-apim-azure_government.bicep -Verbose -mode Incremental- Now that APIM is deployed and automatically configured to work with your Azure Open AI service
Read through the following steps to setup interacting with APIM and how to use consoles or .net to programatically interact with Azure Open AI via APIM.
To determine if you have one or more models deployed, visit the AI Studio. Here you can determine if you need to create a model or use an existing model. You will use the model name when quering the Azure Open AI API via your APIM.
-
Navigate to your Azure Open AI resource in Azure
-
Select Model deployments
-
Select Manage Deployments
-
Review your models and copy the Deployment name of the model you want to use
The subscription key for APIM is collected at the Subscription section of the APIM resource, regardless if you are in Azure Commercial or Government.
You can use this key for testing or as an example on how to create subscriptions to provide access to you Azure Open AI service. Instead of sharing your Azure Open AI Key, you create subscriptions in APIM and share this key, then you can analyze and monitor usage, provide guardrails for usage, and manage access.
- Navigate to your new APIM
- Select Subscriptions from the menu
- Select ...
- Select Show/Hide keys
- Select copy icon
The URL for APIM is collected at the Overview section of the APIM resource, regardless if you are in Azure Commercial or Government.
Using your Azure Open AI model, API version, APIM URL, and APIM subscription key you can now execute Azure Open AI queries against your APIM URL instead of your Azure Open AI URL. This means you can create new subscription keys for anyone or any team who needs access to Azure Open AI instead of deploying new Azure Open AI services.
Copy and paste this script into a text editor or Visual Studio code.
Modify by including your values, then copy and paste all of it into PowerShell 7 terminal.
# Update these values to match your environment
$apimUrl = 'THE_HTTPS_URL_OF_YOUR_APIM_INSTANCE'
$modelName = 'GPT-3_5-Turbo' # Probaby what you named your model, but change if necessary
$apiVersion = '2023-03-15-preview' # Do not change this value, unless you are testing a different API version
$subscriptionKey = 'YOUR_APIM_SUBSCRIPTION_KEY'
# Do not touch these values
$url = $apimUrl + "/deployments/" + $modelName + "/chat/completions?api-version=" + $apiVersion
$key = "Ocp-Apim-Subscription-Key: " + $subscriptionKey
curl $url -k -H "Content-Type: application/json" -H $key -d '{
"messages": [
{
"role": "system",
"content": "You are an AI assistant that helps people find information."
},
{
"role": "user",
"content": "What are the differences between Azure Machine Learning and Azure AI services?"
}
]
}'Copy and paste this script into a text editor or Visual Studio code.
Modify by including your values, then copy and paste all of it into bash terminal or create a ".sh" file to run.
#!/bin/bash
apimUrl="THE_HTTPS_URL_OF_YOUR_APIM_INSTANCE"
modelName="GPT-3_5-Turbo" # Probaby what you named your model, but change if necessary
apiVersion="2023-03-15-preview" # Do not change this value, unless you are testing a different API version
subscriptionKey="YOUR_APIM_SUBSCRIPTION_KEY"
url="${apimUrl}"/deployments/"${modelName}"/chat/completions?api-version="${apiVersion}"
key="Ocp-Apim-Subscription-Key: ${subscriptionKey}"
curl $url -k -H "Content-Type: application/json" -H $key -d '{
"messages": [
{
"role": "system",
"content": "You are an AI assistant that helps people find information."
},
{
"role": "user",
"content": "What are the differences between Azure Machine Learning and Azure AI services?"
}
]
}'// Note: The Azure OpenAI client library for .NET is in preview.
// Install the .NET library via NuGet: dotnet add package Azure.AI.OpenAI --version 1.0.0-beta.5
using Azure;
using Azure.AI.OpenAI;
OpenAIClient client = new OpenAIClient(
new Uri("https://INSERT_APIM_URL_HERE/deployments/INSERT_MODEL_NAME_HERE/chat/completions?api-version=INSERT_API_VERSION_HERE"),
new AzureKeyCredential("INSERT_APIM_SUBSCRIPTION_KEY_HERE"));
// ### If streaming is not selected
Response<ChatCompletions> responseWithoutStream = await client.GetChatCompletionsAsync(
"INSERT_MODEL_NAME_HERE",
new ChatCompletionsOptions()
{
Messages =
{
new ChatMessage(ChatRole.System, @"You are an AI assistant that helps people find information."),
new ChatMessage(ChatRole.User, @"What are the differences between Azure Machine Learning and Azure AI services?"),
},
Temperature = (float)0,
MaxTokens = 800,
NucleusSamplingFactor = (float)1,
FrequencyPenalty = 0,
PresencePenalty = 0,
});
// The following code shows how to get to the content from Azure Open AI's response
ChatCompletions completions = responseWithoutStream.Value;
ChatChoice choice = completions.Choices[0];
Console.WriteLine(choice.Message.Content);When deploying to Azure Government with Private endpoints, the deployment process requires the public IP address of the Azure Open AI service. Use ping or nslookup with fqdn of your Azure Open AI url to determine it's public IP address.
- example:
- url is https://aoai.openai.azure.com
- fqdn is aoai.openai.azure.com
Then use nslookup in PowerShell terminal, or Linux console, along with the fqdn to find out the public IP address of your Azure Open AI service.
-
Deploy Solution into
AzureCloudorAzureUsGovernmentfrom the Azure Portal:Azure Commercial Azure Government
Update Portal Form to deploy Azure Open AI in the commercial version, assume deploy, check box to use existing AOAI, check box to enable High Availability using two AOAI endpoints in Different Regions. Use check box to deploy additional AOAI in another region. If that is selected make visible API retry policy. Make sure to show in things that are being deployed.