This is the 4th lecture in a series introducing concepts from Data Science.
Tag Archives: Big Data
Evaluation: Amazon Kinesis
Amazon Kinesis didn’t require any installation of servers. The following steps, however, allowed it to run from my machine (see Install and Configure the AWS CLI)
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" sudo python get-pip.py sudo pip install awscli
Configuring –
AWS Access Key ID [None]: ****** AWS Secret Access Key [None]: ****** Default region name [None]: us-east Default output format [None]: json
Running through tutorial –
Launching AWS CloudFormation template:
Stack created:
Output from stack:
Data graph:
Stream details (at https://console.aws.amazon.com/kinesis/):
Now I’m ready to write my app.
Ideally, the way that Kinesis is set up, I would guess I’d have to create a stream for each airport I was tracking, since each is a different URL. But that may get expensive, so we’ll start with two streams, one for JFK and one for BWI, since those are the airports most likely to be affected by tomorrow’s storm. Also, a cursory look at JFK today showed that they have delays currently due to construction (sample record):
{“delay”:”true”,”IATA”:”JFK”,”state”:”New York”,”name”:”John F Kennedy International”,”weather”:{“visibility”:0.75,”weather”:”Fog/Mist”,”meta”:{“credit”:”NOAA’s National Weather Service”,”updated”:”6:51 PM Local”,”url”:”http://weather.gov/”},”temp”:”59.0 F (15.0 C)”,”wind”:”Southeast at 6.9mph”},”ICAO”:”Kjfk”,”city”:”New York”,”status”:{“reason”:”RWY-TAXI / CONSTRUCTION”,”closureBegin”:””,”endTime”:””,”minDelay”:””,”avgDelay”:”1 hour and 11 minutes”,”maxDelay”:””,”closureEnd”:””,”trend”:””,”type”:”Ground Delay”}}
I’m following the “Learning Amazon Kinesis Development” while creating my application. First, I create a stream:
Next, I create a security policy:
I had an issue with the policy because I inferred from the instructions that the ‘account’ it wanted was my email address. But they wanted the account number:
Next step is to create an IAM user:
I then attach the policy to the created user.
Now I download the sample code provided to use as a skeleton for my own version of the code, based on the text code used for both Apache Spark and Apache Storm. Getting this to run was much easier, partly because I’d defined the logic earlier, but also because the sample code provided made it easier to create. Here’s a sample of the first output:
May 10, 2015 12:22:30 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BWI: false
May 10, 2015 12:22:32 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: MIA: false
May 10, 2015 12:22:33 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ORD: false
May 10, 2015 12:22:34 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ATL: false
May 10, 2015 12:22:35 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: LAX: false
May 10, 2015 12:22:37 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BOS: false
May 10, 2015 12:22:38 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: STL: false
May 10, 2015 12:22:39 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: DFW: true
May 10, 2015 12:22:39 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: WEATHER / THUNDERSTORMS
May 10, 2015 12:22:40 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: DEN: false
May 10, 2015 12:22:41 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: SEA: false
May 10, 2015 12:22:43 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: CLT: true
May 10, 2015 12:22:43 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: VOL:Multi-taxi
May 10, 2015 12:22:44 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: JFK: false
May 10, 2015 12:22:45 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: SFO: true
May 10, 2015 12:22:45 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: WX / LOW CIGS
May 10, 2015 12:22:46 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BWI: false
May 10, 2015 12:22:48 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: MIA: false
May 10, 2015 12:22:49 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ORD: false
May 10, 2015 12:22:50 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: ATL: false
May 10, 2015 12:22:52 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: LAX: false
May 10, 2015 12:22:53 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: BOS: false
May 10, 2015 12:22:55 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: STL: false
May 10, 2015 12:22:56 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO: Putting trade: DFW: true
May 10, 2015 12:22:56 PM edu.hu.faa.aws.writer.AirportStatusWriter sendAirportStatus
INFO:
Reason for Delay: WEATHER / THUNDERSTORMS
May 11, 2015 8:59:21 PM edu.hu.faa.aws.processor.AirportStatusRecordProcessor checkpoint
INFO: Checkpointing shard shardId-000000000000
****** Shard shardId-000000000000 stats for last 1 minute ******
Airports Currently Delayed:
Count of Delays: ORD: 0 LAX: 0 MIA: 0 BOS: 0 STL: 0 CLT: 0 DEN: 0 SEA: 0 JFK: 0 DFW: 0 BWI: 0 ATL: 0 SFO: 0
Reasons for Delays:
****************************************************************
It’s not very exciting if there are no airport delays. I’ll check this periodically and, hopefully, have some to show for my video.
Here’s the current trending on the Kinesis stream dashboard:
Sample code for this demo is on gitlab: git@gitlab.com:amracel/FaaAws.git
Evaluation: Apache Storm
Here are the notes from the recent evaluation I did of this product. The sample code I created can be found here:
https://gitlab.com/amracel/FaaStorm
I used the instructions found on the Apache Storm site for installing a multi-node cluster. I figured that was a better way to compare the Amazon Kinesis application to the Storm.
- I set up 3 nodes, each running CentOS 7.
Here is the configuration file I came up with, based on the zookeeper site recommendations:
# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/var/lib/zookeeper # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 server.1=stormserver001:2888:3888 server.2=stormserver002:2888:3888 server.3=stormserver003:2888:3888
/usr/lib/apache-storm-0.9.4 to /usr/lib/storm.
########### These MUST be filled in for a storm configuration storm.zookeeper.servers: - "stormserver001" - "stormserver002" - "stormserver003" storm.local.dir: "/mnt/storm" # nimbus.host: "stormserver001" supervisor.slots.ports: - 6700 - 6701 - 6702 #
On each box:
java -cp /usr/lib/zookeeper/zookeeper-3.4.6.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/conf:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar org.apache.zookeeper.ZooKeeperMain -server 127.0.0.1:2181
On the master box:
/usr/lib/storm/bin/storm nimbus /usr/lib/storm/bin/storm ui
On the supervisor boxes:
/usr/lib/storm/bin/storm supervisor
The logs from the command line showed that the boxes couldn’t see each other. I realized I hadn’t opened up the ports between the boxes. After much trial and error, I ended up with the following ports opened up:
[root@StormServer001 conf]# firewall-cmd --list-ports 3888/tcp 2181/tcp 8080/tcp 2888/tcp 6627/tcp
8080 was only needed for the master server, to allow access to the web interface. At this point, I could check the web interface and see that everything was running:
I downloaded some sample code from:
https://github.com/nathanmarz/storm-starter
and ran one of the samples to confirm that the system was now working. The above shows 15 executions and 15 tasks continuously running from the sample task.
The last part of the set up process was to create systemd jobs that would run when the servers started. I created scripts similar to the following for all the working (zookeeper, plus storm) scripts:
[Unit] Description=Start monitoring service zookeeper. [Service] Type=simple ExecStart=/usr/bin/java -cp /usr/lib/zookeeper/zookeeper-3.4.6.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/conf org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/lib/zookeeper/conf/zoo.cfg [Install] WantedBy=multi-user.target
I ran the necessary commands to allow these to start upon boot:
systemctl start [filename] systemctl enable [filename]
I created the producer side of the sample code. It was pretty straightforward to get it to work. Here’s a sample of the output:
876891 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"STL","state":"Missouri","name":"Lambert-St Louis International","weather":{"visibility":10.00,"weather":"Mostly Cloudy","meta":{"credit":"NOAA's National Weather Service","updated":"9:51 AM Local","url":"http://weather.gov/"},"temp":"79.0 F (26.1 C)","wind":"South at 11.5mph"},"ICAO":"KSTL","city":"St Louis","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
876892 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@9067678]
878091 [Thread-11-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"DFW","state":"Texas","name":"Dallas/Ft Worth International","weather":{"visibility":7.00,"weather":"Light Rain","meta":{"credit":"NOAA's National Weather Service","updated":"9:53 AM Local","url":"http://weather.gov/"},"temp":"69.0 F (20.6 C)","wind":"East at 9.2mph"},"ICAO":"KDFW","city":"Dallas-Ft Worth","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
878092 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"DFW","state":"Texas","name":"Dallas/Ft Worth International","weather":{"visibility":7.00,"weather":"Light Rain","meta":{"credit":"NOAA's National Weather Service","updated":"9:53 AM Local","url":"http://weather.gov/"},"temp":"69.0 F (20.6 C)","wind":"East at 9.2mph"},"ICAO":"KDFW","city":"Dallas-Ft Worth","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
878092 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@a6629c6]
879322 [Thread-11-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"DEN","state":"Colorado","name":"Denver International","weather":{"visibility":10.00,"weather":"Overcast","meta":{"credit":"NOAA's National Weather Service","updated":"8:53 AM Local","url":"http://weather.gov/"},"temp":"45.0 F (7.2 C)","wind":"North at 3.5mph"},"ICAO":"KDEN","city":"Denver","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
879322 [Thread-11-read-status] INFO backtype.storm.daemon.executor - Processing received message source: __system:-1, stream: __tick, id: {}, [30]
879322 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"DEN","state":"Colorado","name":"Denver International","weather":{"visibility":10.00,"weather":"Overcast","meta":{"credit":"NOAA's National Weather Service","updated":"8:53 AM Local","url":"http://weather.gov/"},"temp":"45.0 F (7.2 C)","wind":"North at 3.5mph"},"ICAO":"KDEN","city":"Denver","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
879323 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@6b3e99f1]
880728 [Thread-11-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"SEA","state":"Washington","name":"Seattle-Tacoma International","weather":{"visibility":10.00,"weather":"A Few Clouds","meta":{"credit":"NOAA's National Weather Service","updated":"7:53 AM Local","url":"http://weather.gov/"},"temp":"50.0 F (10.0 C)","wind":"North at 0.0mph"},"ICAO":"KSEA","city":"Seattle","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
880729 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"SEA","state":"Washington","name":"Seattle-Tacoma International","weather":{"visibility":10.00,"weather":"A Few Clouds","meta":{"credit":"NOAA's National Weather Service","updated":"7:53 AM Local","url":"http://weather.gov/"},"temp":"50.0 F (10.0 C)","wind":"North at 0.0mph"},"ICAO":"KSEA","city":"Seattle","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
880729 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@731ca39]
I then added the consumer and produced the counts of times airport delays were found:
54584 [Thread-11-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:4, stream: default, id: {}, [{"delay":"true","IATA":"ORD","state":"Illinois","name":"Chicago OHare International","weather":{"visibility":0.50,"weather":"Fog","meta":{"credit":"NOAA's National Weather Service","updated":"8:51 PM Local","url":"http://weather.gov/"},"temp":"47.0 F (8.3 C)","wind":"Northeast at 8.1mph"},"ICAO":"KORD","city":"Chicago","status":{"reason":"VOL:Multi-taxi","closureBegin":"","endTime":"","minDelay":"16 minutes","avgDelay":"","maxDelay":"30 minutes","closureEnd":"","trend":"Increasing","type":"Departure"}} ]
54584 [Thread-11-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [ORD]
54585 [Thread-9-delays] INFO backtype.storm.daemon.executor - Processing received message source: process-json:3, stream: default, id: {}, [ORD]
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - ORD
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++ORD: 4+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++LAX: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++MIA: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++DFW: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++BWI: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++BOS: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++ATL: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++STL: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++CLT: 0+++
54586 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++DEN: 0+++
54586 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++SFO: 0+++
54586 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++SEA: 0+++
55760 [Thread-13-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"ATL","state":"Georgia","name":"The William B Hartsfield International","weather":{"visibility":10.00,"weather":"A Few Clouds","meta":{"credit":"NOAA's National Weather Service","updated":"10:52 PM Local","url":"http://weather.gov/"},"temp":"78.0 F (25.6 C)","wind":"North at 0.0mph"},"ICAO":"KATL","city":"Atlanta","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]