Here are the notes from the recent evaluation I did of this product. The sample code I created can be found here:
https://gitlab.com/amracel/FaaStorm
I used the instructions found on the Apache Storm site for installing a multi-node cluster. I figured that was a better way to compare the Amazon Kinesis application to the Storm.
- I set up 3 nodes, each running CentOS 7.
Here is the configuration file I came up with, based on the zookeeper site recommendations:
# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/var/lib/zookeeper # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 server.1=stormserver001:2888:3888 server.2=stormserver002:2888:3888 server.3=stormserver003:2888:3888
/usr/lib/apache-storm-0.9.4 to /usr/lib/storm.
########### These MUST be filled in for a storm configuration storm.zookeeper.servers: - "stormserver001" - "stormserver002" - "stormserver003" storm.local.dir: "/mnt/storm" # nimbus.host: "stormserver001" supervisor.slots.ports: - 6700 - 6701 - 6702 #
On each box:
java -cp /usr/lib/zookeeper/zookeeper-3.4.6.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/conf:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar org.apache.zookeeper.ZooKeeperMain -server 127.0.0.1:2181
On the master box:
/usr/lib/storm/bin/storm nimbus /usr/lib/storm/bin/storm ui
On the supervisor boxes:
/usr/lib/storm/bin/storm supervisor
The logs from the command line showed that the boxes couldn’t see each other. I realized I hadn’t opened up the ports between the boxes. After much trial and error, I ended up with the following ports opened up:
[root@StormServer001 conf]# firewall-cmd --list-ports 3888/tcp 2181/tcp 8080/tcp 2888/tcp 6627/tcp
8080 was only needed for the master server, to allow access to the web interface. At this point, I could check the web interface and see that everything was running:
I downloaded some sample code from:
https://github.com/nathanmarz/storm-starter
and ran one of the samples to confirm that the system was now working. The above shows 15 executions and 15 tasks continuously running from the sample task.
The last part of the set up process was to create systemd jobs that would run when the servers started. I created scripts similar to the following for all the working (zookeeper, plus storm) scripts:
[Unit] Description=Start monitoring service zookeeper. [Service] Type=simple ExecStart=/usr/bin/java -cp /usr/lib/zookeeper/zookeeper-3.4.6.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/zookeeper/conf org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/lib/zookeeper/conf/zoo.cfg [Install] WantedBy=multi-user.target
I ran the necessary commands to allow these to start upon boot:
systemctl start [filename] systemctl enable [filename]
I created the producer side of the sample code. It was pretty straightforward to get it to work. Here’s a sample of the output:
876891 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"STL","state":"Missouri","name":"Lambert-St Louis International","weather":{"visibility":10.00,"weather":"Mostly Cloudy","meta":{"credit":"NOAA's National Weather Service","updated":"9:51 AM Local","url":"http://weather.gov/"},"temp":"79.0 F (26.1 C)","wind":"South at 11.5mph"},"ICAO":"KSTL","city":"St Louis","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
876892 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@9067678]
878091 [Thread-11-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"DFW","state":"Texas","name":"Dallas/Ft Worth International","weather":{"visibility":7.00,"weather":"Light Rain","meta":{"credit":"NOAA's National Weather Service","updated":"9:53 AM Local","url":"http://weather.gov/"},"temp":"69.0 F (20.6 C)","wind":"East at 9.2mph"},"ICAO":"KDFW","city":"Dallas-Ft Worth","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
878092 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"DFW","state":"Texas","name":"Dallas/Ft Worth International","weather":{"visibility":7.00,"weather":"Light Rain","meta":{"credit":"NOAA's National Weather Service","updated":"9:53 AM Local","url":"http://weather.gov/"},"temp":"69.0 F (20.6 C)","wind":"East at 9.2mph"},"ICAO":"KDFW","city":"Dallas-Ft Worth","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
878092 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@a6629c6]
879322 [Thread-11-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"DEN","state":"Colorado","name":"Denver International","weather":{"visibility":10.00,"weather":"Overcast","meta":{"credit":"NOAA's National Weather Service","updated":"8:53 AM Local","url":"http://weather.gov/"},"temp":"45.0 F (7.2 C)","wind":"North at 3.5mph"},"ICAO":"KDEN","city":"Denver","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
879322 [Thread-11-read-status] INFO backtype.storm.daemon.executor - Processing received message source: __system:-1, stream: __tick, id: {}, [30]
879322 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"DEN","state":"Colorado","name":"Denver International","weather":{"visibility":10.00,"weather":"Overcast","meta":{"credit":"NOAA's National Weather Service","updated":"8:53 AM Local","url":"http://weather.gov/"},"temp":"45.0 F (7.2 C)","wind":"North at 3.5mph"},"ICAO":"KDEN","city":"Denver","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
879323 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@6b3e99f1]
880728 [Thread-11-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"SEA","state":"Washington","name":"Seattle-Tacoma International","weather":{"visibility":10.00,"weather":"A Few Clouds","meta":{"credit":"NOAA's National Weather Service","updated":"7:53 AM Local","url":"http://weather.gov/"},"temp":"50.0 F (10.0 C)","wind":"North at 0.0mph"},"ICAO":"KSEA","city":"Seattle","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
880729 [Thread-9-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:3, stream: default, id: {}, [{"delay":"false","IATA":"SEA","state":"Washington","name":"Seattle-Tacoma International","weather":{"visibility":10.00,"weather":"A Few Clouds","meta":{"credit":"NOAA's National Weather Service","updated":"7:53 AM Local","url":"http://weather.gov/"},"temp":"50.0 F (10.0 C)","wind":"North at 0.0mph"},"ICAO":"KSEA","city":"Seattle","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]
880729 [Thread-9-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [edu.hu.faa.storm.pojos.AirportStatus@731ca39]
I then added the consumer and produced the counts of times airport delays were found:
54584 [Thread-11-process-json] INFO backtype.storm.daemon.executor - Processing received message source: read-status:4, stream: default, id: {}, [{"delay":"true","IATA":"ORD","state":"Illinois","name":"Chicago OHare International","weather":{"visibility":0.50,"weather":"Fog","meta":{"credit":"NOAA's National Weather Service","updated":"8:51 PM Local","url":"http://weather.gov/"},"temp":"47.0 F (8.3 C)","wind":"Northeast at 8.1mph"},"ICAO":"KORD","city":"Chicago","status":{"reason":"VOL:Multi-taxi","closureBegin":"","endTime":"","minDelay":"16 minutes","avgDelay":"","maxDelay":"30 minutes","closureEnd":"","trend":"Increasing","type":"Departure"}} ]
54584 [Thread-11-process-json] INFO backtype.storm.daemon.task - Emitting: process-json default [ORD]
54585 [Thread-9-delays] INFO backtype.storm.daemon.executor - Processing received message source: process-json:3, stream: default, id: {}, [ORD]
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - ORD
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++ORD: 4+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++LAX: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++MIA: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++DFW: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++BWI: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++BOS: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++ATL: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++STL: 0+++
54585 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++CLT: 0+++
54586 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++DEN: 0+++
54586 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++SFO: 0+++
54586 [Thread-9-delays] INFO edu.hu.faa.storm.bolt.DelayCountBolt - +++SEA: 0+++
55760 [Thread-13-read-status] INFO backtype.storm.daemon.task - Emitting: read-status default [{"delay":"false","IATA":"ATL","state":"Georgia","name":"The William B Hartsfield International","weather":{"visibility":10.00,"weather":"A Few Clouds","meta":{"credit":"NOAA's National Weather Service","updated":"10:52 PM Local","url":"http://weather.gov/"},"temp":"78.0 F (25.6 C)","wind":"North at 0.0mph"},"ICAO":"KATL","city":"Atlanta","status":{"reason":"No known delays for this airport.","closureBegin":"","endTime":"","minDelay":"","avgDelay":"","maxDelay":"","closureEnd":"","trend":"","type":""}} ]