by Manav Sehgal | on APR 30 2019

So far we have been working with structured data in flat files as our data source. What if the source is images and unstructured text. AWS AI services provide vision, transcription, translation, personalization, and forecasting capabilities without the need for training and deploying machine learning models. AWS manages the machine learning complexity, you just focus on the problem at hand and send required inputs for analysis and receive output from these services within your applications.

Extending our open data analytics use case to New York Traffic let us use the AWS AI services to turn open data available in social media, Wikipedia, and other sources into structured datasets and insights.

We will start by importing dependencies for AWS SDK, Python Data Frames, file operations, handeling JSON data, and display formatting. We will initialize the Rekognition client for use in the rest of this notebook.

import boto3
import pandas as pd
import io
import json
from IPython.display import display, Markdown, Image

rekognition = boto3.client('rekognition','us-east-1')
image_bucket = 'open-data-analytics-taxi-trips'

We will work with a number of images so we need a way to show these images within this notebook. Our function creates a public image URL based on S3 bucket and key as input.

def show_image(bucket, key, img_width = 500):
    # [TODO] Load non-public images
    return Image(url='https://s3.amazonaws.com/' + bucket + '/' + key, width=img_width)

show_image(image_bucket, 'images/traffic-in-manhattan.jpg', 1024)

One of use cases for traffic analytics is processing traffic CCTV imagery or social media uploads. Let's consider a traffic location where depending on number of cars, trucks, and pedestrians we can identify if there is a traffic jam. This insight can be used to better manage flow of traffic around the location and plan ahead for future use of this route.

First step in this kind of analytics is to recognize that we are actually looking at an image which may represent a traffic jam. We create image_labels function which uses detect_lables Rekognition API to detect objects within an image. The function prints labels detected with confidence score.

In the given example notice somewhere in the middle of the labels listing at 73% confidence the Rekognition computer vision model has actually determined a traffic jam.

def image_labels(bucket, key):
    image_object = {'S3Object':{'Bucket': bucket,'Name': key}}

    response = rekognition.detect_labels(Image=image_object)
    for label in response['Labels']:
        print('{} ({:.0f}%)'.format(label['Name'], label['Confidence']))

image_labels(image_bucket, 'images/traffic-in-manhattan.jpg')

Vehicle (100%)
Automobile (100%)
Transportation (100%)
Car (100%)
Human (99%)
Person (99%)
Truck (98%)
Machine (96%)
Wheel (96%)
Clothing (87%)
Apparel (87%)
Footwear (87%)
Shoe (87%)
Road (75%)
Traffic Jam (73%)
City (73%)
Urban (73%)
Metropolis (73%)
Building (73%)
Town (73%)
Cab (71%)
Taxi (71%)
Traffic Light (68%)
Light (68%)
Neighborhood (62%)
People (62%)
Pedestrian (59%)

Now that we have a label detecting a traffic jam and some of the ingredients of a busy traffic location like pedestrians, trucks, cars, let us determine quantitative data for benchmarking different traffic locations. If we can count the number of cars, trucks, and persons in the image we can compare these numbers with other images. Our function does just that, it counts the number of instances of a matching label.

def image_label_count(bucket, key, match):    
    image_object = {'S3Object':{'Bucket': bucket,'Name': key}}

    response = rekognition.detect_labels(Image=image_object)
    count = 0
    for label in response['Labels']:
        if match in label['Name']:
            for instance in label['Instances']:
                count += 1
    print(f'Found {match} {count} times.')

image_label_count(image_bucket, 'images/traffic-in-manhattan.jpg', 'Car')

Found Car 9 times.

image_label_count(image_bucket, 'images/traffic-in-manhattan.jpg', 'Truck')

Found Truck 4 times.

image_label_count(image_bucket, 'images/traffic-in-manhattan.jpg', 'Person')

Found Person 8 times.

Another use case of traffic location analytics using social media content is to understand more about a traffic location and instance if there is an incident reported, like an accident, jam, or VIP movement. For a computer program to understand a random traffic location, it may help to capture any text within the image. The image_text function uses Amazon Rekognition service to detect text in an image.

You will notice that the text recognition is capable to read blurry text like "The Lion King", text which is at a perspective like the bus route, text which may be ignored by the human eye like the address below the shoes banner, and even the text representing the taxi number. Suddenly the image starts telling a story programmatically, about what time it may represent, what are the landmarks, which bus route, which taxi number was on streets, and so on.

def image_text(bucket, key, sort_column='', parents=True):
    response = rekognition.detect_text(Image={'S3Object':{'Bucket':bucket,'Name': key}})
    df = pd.read_json(io.StringIO(json.dumps(response['TextDetections'])))
    df['Width'] = df['Geometry'].apply(lambda x: x['BoundingBox']['Width'])
    df['Height'] = df['Geometry'].apply(lambda x: x['BoundingBox']['Height'])
    df['Left'] = df['Geometry'].apply(lambda x: x['BoundingBox']['Left'])
    df['Top'] = df['Geometry'].apply(lambda x: x['BoundingBox']['Top'])
    df = df.drop(columns=['Geometry'])
    if sort_column:
        df = df.sort_values([sort_column])
    if not parents:
        df = df[df['ParentId'] > 0]
    return df

show_image(image_bucket, 'images/nyc-taxi-signs.jpeg', 1024)

Sorting on Top column will keep the horizontal text together.

image_text(image_bucket, 'images/nyc-taxi-signs.jpeg', sort_column='Top', parents=False)

Traffic analytics may also involve detecting VIP movement to divert traffic or monitor security events. Detecting VIP in a scene starts with facial recognition. Our function detect_celebs works as well with political figures as it will with movie celebrities.

def detect_celebs(bucket, key, sort_column=''):
    image_object = {'S3Object':{'Bucket': bucket,'Name': key}}

    response = rekognition.recognize_celebrities(Image=image_object)
    df = pd.DataFrame(response['CelebrityFaces'])
    df['Width'] = df['Face'].apply(lambda x: x['BoundingBox']['Width'])
    df['Height'] = df['Face'].apply(lambda x: x['BoundingBox']['Height'])
    df['Left'] = df['Face'].apply(lambda x: x['BoundingBox']['Left'])
    df['Top'] = df['Face'].apply(lambda x: x['BoundingBox']['Top'])
    df = df.drop(columns=['Face'])
    if sort_column:
        df = df.sort_values([sort_column])
    return(df)

show_image(image_bucket, 'images/world-leaders.jpg', 1024)

detect_celebs(image_bucket, 'images/world-leaders.jpg', sort_column='Left')

It is possible that many data sources represent natural language and free text. Understand structure and semantics from this unstructured text can help further our open data analytics use cases.

Let us assume we are processing traffic updates for structured data so we can take appropriate actions. First step in understanding natural language is to break it up into grammaticaly syntax. Nouns like "today" can tell about a particular event like when is the event occuring. Adjectives like "snowing" and "windy" tell what is happening at that moment in time.

comprehend = boto3.client('comprehend', 'us-east-1')

traffic_update = """
It is snowing and windy today in New York. The temperature is 50 degrees Fahrenheit. 
The traffic is slow 10 mph with several jams along the I-86.
"""

def comprehend_syntax(text): 
    response = comprehend.detect_syntax(Text=text, LanguageCode='en')
    df = pd.read_json(io.StringIO(json.dumps(response['SyntaxTokens'])))
    df['Tag'] = df['PartOfSpeech'].apply(lambda x: x['Tag'])
    df['Score'] = df['PartOfSpeech'].apply(lambda x: x['Score'])
    df = df.drop(columns=['PartOfSpeech'])
    return df

comprehend_syntax(traffic_update)

More insights can be derived by doing entity extraction from the natural langauage. These entities can be date, location, quantity, among others. Just few of the entities can tell a structured story to a program.

def comprehend_entities(text):
    response = comprehend.detect_entities(Text=text, LanguageCode='en')
    df = pd.read_json(io.StringIO(json.dumps(response['Entities'])))
    return df

comprehend_entities(traffic_update)

Analysis of phrases within narutal language text complements the other two methods for a program to better route the actions based on derived structure of the event.

def comprehend_phrases(text):
    response = comprehend.detect_key_phrases(Text=text, LanguageCode='en')
    df = pd.read_json(io.StringIO(json.dumps(response['KeyPhrases'])))
    return df

comprehend_phrases(traffic_update)

Sentiment analysis is common for social media user generated content. Sentiment can give us signals on the users' mood when publishing such social data.

def comprehend_sentiment(text):
    response = comprehend.detect_sentiment(Text=text, LanguageCode='en')
    return response['SentimentScore']

comprehend_sentiment(traffic_update)

{'Positive': 0.04090394824743271,
 'Negative': 0.3745909333229065,
 'Neutral': 0.5641733407974243,
 'Mixed': 0.020331736654043198}

This section captures changes and updates to this notebook across releases.

Functions image_text and detect_celeb can now sort results based on a column name. Function image_text can optionally show results without parent-child relations.

Usability update for comprehend_syntax function to split part of speech dictionary value into separate Tag and Score columns.

This is the launch release which builds the AWS Open Data Analytics API for using AWS AI services to analyze public data.

by Manav Sehgal | on APR 30 2019