Evaluation: Amazon Kinesis

Amazon Kinesis didn’t require any installation of servers. The following steps, however, allowed it to run from my machine (see Install and Configure the AWS CLI)

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python get-pip.py
sudo pip install awscli

Configuring –

AWS Access Key ID [None]: ******
AWS Secret Access Key [None]: ******
Default region name [None]: us-east
Default output format [None]: json

Running through tutorial –

Launching AWS CloudFormation template:

image001

image001

kinesis3

kinesis4

kinesis5

Stack created:

kinesis6

Output from stack:

kinesis7

Data graph:

kinesis8

 

Stream details (at https://console.aws.amazon.com/kinesis/):

kinesis9

kinesis10

Now I’m ready to write my app.

Ideally, the way that Kinesis is set up, I would guess I’d have to create a stream for each airport I was tracking, since each is a different URL. But that may get expensive, so we’ll start with two streams, one for JFK and one for BWI, since those are the airports most likely to be affected by tomorrow’s storm. Also, a cursory look at JFK today showed that they have delays currently due to construction (sample record):

 

{“delay”:”true”,”IATA”:”JFK”,”state”:”New York”,”name”:”John F Kennedy International”,”weather”:{“visibility”:0.75,”weather”:”Fog/Mist”,”meta”:{“credit”:”NOAA’s National Weather Service”,”updated”:”6:51 PM Local”,”url”:”http://weather.gov/”},”temp”:”59.0 F (15.0 C)”,”wind”:”Southeast at 6.9mph”},”ICAO”:”Kjfk”,”city”:”New York”,”status”:{“reason”:”RWY-TAXI / CONSTRUCTION”,”closureBegin”:””,”endTime”:””,”minDelay”:””,”avgDelay”:”1 hour and 11 minutes”,”maxDelay”:””,”closureEnd”:””,”trend”:””,”type”:”Ground Delay”}}

I’m following the “Learning Amazon Kinesis Development” while creating my application. First, I create a stream:

kinesis11

Next, I create a security policy:

kinesis12

I had an issue with the policy because I inferred from the instructions that the ‘account’ it wanted was my email address. But they wanted the account number:

kinesis13

kinesis14

Next step is to create an IAM user:kinesis15

I then attach the policy to the created user.

 

Now I download the sample code provided to use as a skeleton for my own version of the code, based on the text code used for both Apache Spark and Apache Storm. Getting this to run was much easier, partly because I’d defined the logic earlier, but also because the sample code provided made it easier to create. Here’s a sample of the first output:


May 10, 2015 12:22:30 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BWI: false
May 10, 2015 12:22:32 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: MIA: false
May 10, 2015 12:22:33 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ORD: false
May 10, 2015 12:22:34 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ATL: false
May 10, 2015 12:22:35 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: LAX: false
May 10, 2015 12:22:37 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BOS: false
May 10, 2015 12:22:38 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: STL: false
May 10, 2015 12:22:39 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: DFW: true
May 10, 2015 12:22:39 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: WEATHER / THUNDERSTORMS
May 10, 2015 12:22:40 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: DEN: false
May 10, 2015 12:22:41 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: SEA: false
May 10, 2015 12:22:43 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: CLT: true
May 10, 2015 12:22:43 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: VOL:Multi-taxi
May 10, 2015 12:22:44 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: JFK: false
May 10, 2015 12:22:45 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: SFO: true
May 10, 2015 12:22:45 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: WX / LOW CIGS
May 10, 2015 12:22:46 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BWI: false
May 10, 2015 12:22:48 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: MIA: false
May 10, 2015 12:22:49 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ORD: false
May 10, 2015 12:22:50 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ATL: false
May 10, 2015 12:22:52 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: LAX: false
May 10, 2015 12:22:53 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BOS: false
May 10, 2015 12:22:55 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: STL: false
May 10, 2015 12:22:56 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: DFW: true
May 10, 2015 12:22:56 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: WEATHER / THUNDERSTORMS
May 11, 2015 8:59:21 PM edu.hu.faa.aws.processor.AirportStatusRecordProcessor checkpoint
INFO: Checkpointing shard shardId-000000000000
****** Shard shardId-000000000000 stats for last 1 minute ******
Airports Currently Delayed:
Count of Delays: ORD: 0 LAX: 0 MIA: 0 BOS: 0 STL: 0 CLT: 0 DEN: 0 SEA: 0 JFK: 0 DFW: 0 BWI: 0 ATL: 0 SFO: 0
Reasons for Delays:
****************************************************************

It’s not very exciting if there are no airport delays. I’ll check this periodically and, hopefully, have some to show for my video.

Here’s the current trending on the Kinesis stream dashboard:

kinesis16

 

Sample code for this demo is on gitlab: git@gitlab.com:amracel/FaaAws.git