Generally, User submitted images (profile pictures, social media posts, and business documents) are needed to be moderated to protect business and quality. If a large number of images are submitted in an application then manual moderation is time consuming and is not instantaneous. This post explains how to automate image moderation using Azure functions and Cognitive Services. Automated moderation applies machine learning and AI to cost-effectively moderate at scale, without human involvement.
Introduction
Before we get started, let's understand some terms quickly if you are new in Azure world:
Azure Functions = AWS Lambda in Azure world = Serverless architecture (i.e. execute code without caring hosting)
Cognitive Services = provides APIs to transform your apps with artificial intelligence (AI).
Content Moderator = APIs to moderate Text, Image and Video to filter out offensive and unwanted content
Azure Blob storage = Amazon S3 = Object storage = to store unstructured data (document, media file, or application installer..etc.)
The hierarchy of blob service is
Account > Container > Blob (file)
i.e. Container = a grouping of a set of blobs.
Video
Watch the following video tutorial to automate image moderation:
What to Cook
Here is the flow we are going to implement:
1. An image is uploaded in Blob storage container (say Container1) by user or by any application.
2. This is configured to trigger an Azure function when a blob is added.
3. Azure function will call Face Detection API of Image Moderation in Microsoft Cognitive Services and get faces count. To ensure single person in photo, we validates faces count must be one.
4. After that, Image Moderation API’s Evaluate operation will be called to predict whether the image contains potential adult or racy content.
5. If the image is valid then some branding text is added as watermark and new image is saved in another container (say Container2) which is used by the application.
You can archive/delete the original image. So that, in case of manual moderation, only images will be present which are either automatic rejected or pending review.
Setup Content Moderator
It is assumed you have Azure subscribtion. On Azure portal, Click "+ New" and enter "Content Moderator" in search box.
Click on "Create" button to create. It is part of Cognitive Services.
Now Go to the resource and click Keys. You will get two keys- KEY 1 and KEY 2. We will use it in the Azure function.
Azure Functions App and Storage
On Azure portal, Click + New > Compute in Azure Marketplace > Function App. Enter Name and other information. For simplicity, we pick Create New option for Storage and enter name. We will use the same Storage for image upload. Click on Create button.
Now Go to All Resources, you will get Storage account and Azure function app created. Click on Storage account > Blobs > + Container.
Enter name container1 and set Container as public access level and click OK.
Similarly, create another Container container2. User submitted images will be saved on container1 and after moderation, the image is watermarked and saved on container2.
Triggers and Binding
Go to Azure Functions > Select the function app we created. Click on + icon of Function > click Custom Function in bottom > Blob Trigger > C#.
Enter Name: BlobTriggerCSharp1
Azure Blob Storage trigger
Path: container1/{name}
Storage account connection: select option which has connectionstring of the storage account.
Click Create.
Now this function is called when any blob is added in container1. By default, you will get following method:
public static void Run(Stream myBlob, string name, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}
Now, click on Integrate > + New Output in Outputs > Azure Blob Storage > Select.
Set following:
Blob parameter name: outputBlob
path: container2/{name}
Storage account connection: select option which has connectionstring of the storage account.
Now click on Advanced editor on top right, you will get following code:
{
"bindings": [
{
"name": "myBlob",
"type": "blobTrigger",
"direction": "in",
"path": "container1/{name}",
"connection": "AzureWebJobsDashboard"
},
{
"type": "blob",
"name": "outputBlob",
"path": "container2/{name}",
"connection": "AzureWebJobsDashboard",
"direction": "out"
}
],
"disabled": false
}
click on BlobTriggerCSharp1 to open run.csx file. Add one more argument "outputBlob" in the existing method
public static void Run(Stream myBlob, string name, Stream outputBlob, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}
Open Logs window and click on Save. You will get following message.
2018-01-14T03:45:07.061 Script for function 'BlobTriggerCSharp1' changed. Reloading.
2018-01-14T03:45:07.235 Compilation succeeded.On Save, the function is compiled. Any errors during the build are displayed in the Logs window. Now when any file is added on container1, this function will be called automatically. In the logs, you will be able to see the output logs.
Let's add some references and namespaces to be used
#r "Newtonsoft.Json"
#r "System.Drawing"
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using Newtonsoft.Json.Linq;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Drawing.Imaging;
Let's add following method for http request and get response as string. It will be used to call congnitive service APIs.
public static async Task<string> GetHttpResponseString(String url, String subscriptionKey, Stream image, string contentType)
{
using (var ms = new MemoryStream())
{
image.Position = 0;
image.CopyTo(ms);
ms.Position = 0;
using (var client = new HttpClient())
{
var content = new StreamContent(ms);
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
var httpResponse = await client.PostAsync(url, content);
if (httpResponse.StatusCode == HttpStatusCode.OK)
{
return await httpResponse.Content.ReadAsStringAsync();
}
}
}
return null;
}
Also, add following to get content type from file extension
public static string GetConentType(string fileName)
{
string name = fileName.ToLower();
string contentType = "image/jpeg";
if (name.EndsWith("png"))
{
contentType = "image/png";
}
else if (name.EndsWith("gif"))
{
contentType = "image/gif";
}
else if (name.EndsWith("bmp"))
{
contentType = "image/bmp";
}
return contentType;
}
Face Detection
Here, we are going to validate that there must be one person only in the image. So, we will get face count using Content Moderator’s Image Moderation API.
Before get started, first, copy the key of content moderator service which we subscribed. The best way is to use environment variable to use it in function. But for simplicity, we are assigning it to a variable directly.
public static string contentModerationKey = "zzzzzzzzzzzzzzzzzzzzzzzzzz";
add following method to validate faces count
public static bool IsGoodByFaceModerator(Stream image, string contentType, TraceWriter log)
{
try
{
var url = "https://southcentralus.api.cognitive.microsoft.com/contentmoderator/moderate/v1.0/ProcessImage/FindFaces";
Task<string> task = Task.Run<string>(async () => await GetHttpResponseString(url, contentModerationKey, image, contentType));
string result = task.Result;
if (String.IsNullOrEmpty(result))
{
return false;
}
else
{
dynamic json = JValue.Parse(result);
return ((bool)json.Result && json.Count == 1);
}
}
catch (Exception ex)
{
log.Error("Face API Error: " + ex.ToString());
return false;
}
}
Note: You need to update url according to your content moderator subscription.
Evaluating for Adult and Racy Content
Content Moderator’s Image Moderation Evaluate API is used to predict whether the image contains potential adult or racy content. Add following method to check it
//To filter adult or racy content
public static bool IsGoodByImageModerator(Stream image, string contentType, TraceWriter log)
{
try
{
var url = "https://southcentralus.api.cognitive.microsoft.com/contentmoderator/moderate/v1.0/ProcessImage/Evaluate";
Task<string> task = Task.Run<string>(async () => await GetHttpResponseString(url, contentModerationKey, image, contentType));
string result = task.Result;
if (String.IsNullOrEmpty(result))
{
return false;
}
else
{
dynamic json = JValue.Parse(result);
return (!((bool)json.IsImageAdultClassified || (bool)json.IsImageRacyClassified));
}
}
catch (Exception ex)
{
log.Error("Content API Error: " + ex.ToString());
return false;
}
}
First, we will validate faces count if it is okay then evaluate image for adult or racy content.
public static bool IsGoodByModerator(Stream image, string name, TraceWriter log)
{
string contentType = GetConentType(name);
if (IsGoodByFaceModerator(image, contentType, log))
{
log.Info("Face Moderation: Passed");
return IsGoodByImageModerator(image, contentType, log);
}
else
{
log.Info("Face Moderation: Failed");
return false;
}
}
Let's call it on Run function:
public static void Run(Stream myBlob, string name,Stream outputBlob, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
bool result = IsGoodByModerator(myBlob, name, log);
log.Info("Image Moderation " + (result ? "Passed" : "Failed"));
}
When you upload a valid image in container1 then you will get like following logs:
2018-01-14T05:56:16.199 Function started (Id=95f92c30-7cb4-4eea-b829-4681a8cd3edf)
2018-01-14T05:56:16.293 C# Blob trigger function Processed blob
Name:Sample.jpg
Size: 112764 Bytes
2018-01-14T05:56:19.422 Face Moderation: Passed
2018-01-14T05:56:20.530 Image Moderation Passed
2018-01-14T05:56:20.530 Function completed (Success, Id=95f92c30-7cb4-4eea-b829-4681a8cd3edf, Duration=4333ms)
In case of invalid image
2018-01-14T06:01:26.589 Function started (Id=0d67a5b6-da55-4bff-9d0e-4c19b15526c0)
2018-01-14T06:01:26.604 C# Blob trigger function Processed blob
Name:Sample.png
Size: 104049 Bytes
2018-01-14T06:01:27.354 Face Moderation: Failed
2018-01-14T06:01:27.354 Image Moderation Failed
2018-01-14T06:01:27.354 Function completed (Success, Id=0d67a5b6-da55-4bff-9d0e-4c19b15526c0, Duration=757ms)
Add Watermark
The next step is to add watermark on valid image and save to container2.
Add following method to add text as watermark on the image:
private static void WriteWatermark(string watermarkContent, Stream originalImage, Stream newImage)
{
originalImage.Position = 0;
using (Image inputImage = Image
.FromStream(originalImage, true))
{
using (Graphics graphic = Graphics
.FromImage(inputImage))
{
Font font = new Font("Georgia", 36, FontStyle.Bold);
SizeF textSize = graphic.MeasureString(watermarkContent, font);
float xCenterOfImg = (inputImage.Width / 2);
float yPosFromBottom = (int)(inputImage.Height * 0.90) - (textSize.Height / 2);
graphic.SmoothingMode = SmoothingMode.HighQuality;
graphic.InterpolationMode = InterpolationMode.HighQualityBicubic;
graphic.PixelOffsetMode = PixelOffsetMode.HighQuality;
graphic.TextRenderingHint = System.Drawing.Text.TextRenderingHint.AntiAliasGridFit;
StringFormat StrFormat = new StringFormat();
StrFormat.Alignment = StringAlignment.Center;
SolidBrush semiTransBrush2 = new SolidBrush(Color.FromArgb(153, 0, 0, 0));
graphic.DrawString(watermarkContent, font, semiTransBrush2, xCenterOfImg + 1, yPosFromBottom + 1, StrFormat);
SolidBrush semiTransBrush = new SolidBrush(Color.FromArgb(153, 255, 255, 255));
graphic.DrawString(watermarkContent, font, semiTransBrush, xCenterOfImg, yPosFromBottom, StrFormat);
graphic.Flush();
inputImage.Save(newImage, ImageFormat.Jpeg);
}
}
}
You can update font size as per your choice. Let's update Run method to call it.
public static void Run(Stream myBlob, string name,Stream outputBlob, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
bool result = IsGoodByModerator(myBlob, name, log);
log.Info("Image Moderation: " + (result ? "Passed" : "Failed"));
try{
string watermarkText = "TechBrij";
WriteWatermark(watermarkText,myBlob,outputBlob);
log.Info("Added watermark and copied successfully");
}
catch(Exception ex)
{
log.Info("Watermark Error: " + ex.ToString());
}
}
Now you can upload a valid image in container1 then you will get watermarked image in container2.
Logs:
2018-01-14T06:33:59.269 Function started (Id=9465ff87-c540-4cca-a0b1-de1e00068539)
2018-01-14T06:33:59.286 C# Blob trigger function Processed blob
Name:brij-250.png
Size: 105367 Bytes
2018-01-14T06:33:59.942 Face Moderation: Passed
2018-01-14T06:34:01.024 Image Moderation: Passed
2018-01-14T06:34:01.071 Added watermark and copied successfully
2018-01-14T06:34:01.102 Function completed (Success, Id=9465ff87-c540-4cca-a0b1-de1e00068539, Duration=1825ms)
Complete Code
#r "Newtonsoft.Json"
#r "System.Drawing"
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using Newtonsoft.Json.Linq;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Drawing.Imaging;
public static string contentModerationKey = "zzzzzzzzzzzzzzzzzzzzzzzzzz";
public static void Run(Stream myBlob, string name,Stream outputBlob, TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
bool result = IsGoodByModerator(myBlob, name, log);
log.Info("Image Moderation: " + (result ? "Passed" : "Failed"));
try{
string watermarkText = "TechBrij";
WriteWatermark(watermarkText,myBlob,outputBlob);
log.Info("Added watermark and copied successfully");
}
catch(Exception ex)
{
log.Info("Watermark Error: " + ex.ToString());
}
}
//To validate that there must be one person only in the image
public static bool IsGoodByFaceModerator(Stream image, string contentType, TraceWriter log)
{
try
{
var url = "https://southcentralus.api.cognitive.microsoft.com/contentmoderator/moderate/v1.0/ProcessImage/FindFaces";
Task<string> task = Task.Run<string>(async () => await GetHttpResponseString(url, contentModerationKey, image, contentType));
string result = task.Result;
if (String.IsNullOrEmpty(result))
{
return false;
}
else
{
dynamic json = JValue.Parse(result);
return ((bool)json.Result && json.Count == 1);
}
}
catch (Exception ex)
{
log.Error("Face API Error: " + ex.ToString());
return false;
}
}
//To filter adult or racy content
public static bool IsGoodByImageModerator(Stream image, string contentType, TraceWriter log)
{
try
{
var url = "https://southcentralus.api.cognitive.microsoft.com/contentmoderator/moderate/v1.0/ProcessImage/Evaluate";
Task<string> task = Task.Run<string>(async () => await GetHttpResponseString(url, contentModerationKey, image, contentType));
string result = task.Result;
if (String.IsNullOrEmpty(result))
{
return false;
}
else
{
dynamic json = JValue.Parse(result);
return (!((bool)json.IsImageAdultClassified || (bool)json.IsImageRacyClassified));
}
}
catch (Exception ex)
{
log.Error("Content API Error: " + ex.ToString());
return false;
}
}
//Image Validation
public static bool IsGoodByModerator(Stream image, string name, TraceWriter log)
{
string contentType = GetConentType(name);
if (IsGoodByFaceModerator(image, contentType, log))
{
log.Info("Face Moderation: Passed");
return IsGoodByImageModerator(image, contentType, log);
}
else
{
log.Info("Face Moderation: Failed");
return false;
}
}
//To add text as watermark
private static void WriteWatermark(string watermarkContent, Stream originalImage, Stream newImage)
{
originalImage.Position = 0;
using (Image inputImage = Image
.FromStream(originalImage, true))
{
using (Graphics graphic = Graphics
.FromImage(inputImage))
{
Font font = new Font("Georgia", 36, FontStyle.Bold);
SizeF textSize = graphic.MeasureString(watermarkContent, font);
float xCenterOfImg = (inputImage.Width / 2);
float yPosFromBottom = (int)(inputImage.Height * 0.90) - (textSize.Height / 2);
graphic.SmoothingMode = SmoothingMode.HighQuality;
graphic.InterpolationMode = InterpolationMode.HighQualityBicubic;
graphic.PixelOffsetMode = PixelOffsetMode.HighQuality;
graphic.TextRenderingHint = System.Drawing.Text.TextRenderingHint.AntiAliasGridFit;
StringFormat StrFormat = new StringFormat();
StrFormat.Alignment = StringAlignment.Center;
SolidBrush semiTransBrush2 = new SolidBrush(Color.FromArgb(153, 0, 0, 0));
graphic.DrawString(watermarkContent, font, semiTransBrush2, xCenterOfImg + 1, yPosFromBottom + 1, StrFormat);
SolidBrush semiTransBrush = new SolidBrush(Color.FromArgb(153, 255, 255, 255));
graphic.DrawString(watermarkContent, font, semiTransBrush, xCenterOfImg, yPosFromBottom, StrFormat);
graphic.Flush();
inputImage.Save(newImage, ImageFormat.Jpeg);
}
}
}
//get response as string for http request
public static async Task<string> GetHttpResponseString(String url, String subscriptionKey, Stream image, string contentType)
{
using (var ms = new MemoryStream())
{
image.Position = 0;
image.CopyTo(ms);
ms.Position = 0;
using (var client = new HttpClient())
{
var content = new StreamContent(ms);
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
var httpResponse = await client.PostAsync(url, content);
if (httpResponse.StatusCode == HttpStatusCode.OK)
{
return await httpResponse.Content.ReadAsStringAsync();
}
}
}
return null;
}
//To get content type from file extension
public static string GetConentType(string fileName)
{
string name = fileName.ToLower();
string contentType = "image/jpeg";
if (name.EndsWith("png"))
{
contentType = "image/png";
}
else if (name.EndsWith("gif"))
{
contentType = "image/gif";
}
else if (name.EndsWith("bmp"))
{
contentType = "image/bmp";
}
return contentType;
}
Conclusion
In this aricle, you learn how to
- Setup Content Moderator in Cognitive services
- Create an Azure functions app and blob storage account
- Create a container and set permissions
- Setup Azure function bindings and blob triggers
- Use Face Detection API of Image Moderation API
- Use Evaluation operation of Image Moderation API to filter out adult and racy images.
- Add watermark using C#
Enjoy Azure AI for making moderation easy !!