Friday, 24 February 2017

How to make Kafka run automatically on startup in Mac

Kafka Automatic startup in Mac Tutorial

Apache Kafka is a highly-scalable publish-subscribe messaging system that can serve as the data backbone in distributed applications. With Kafka’s Producer-Consumer model it becomes easy to implement multiple data consumers that do live monitoring as well persistent data storage for later analysis.

Note: You need zookeeper to setup kafka.

In Mac you can either install kafka via home brew using below command
brew install kafka
which will install it in /usr/local/Cellar/kafka directory.
*This will also install all dependencies, like zookeeper which is required to run kafka server.
You can also manually download it and extract it in /usr/local/kafka/ directory.

Now all you need is to add a plist file in order to have kafka running on startup and not manually running it everytime. Verified to be working in Mac-OS 10.12

Before moving ahead make sure you setup zookeeper to start automatically as below is dependent on zookeeper running. Read here.

1. Go to the kafka installation directory - usually /usr/local/kafka/kafka_2.11/
cd /usr/local/kafka/kafka_2.11/

2. Create a file named - homebrew.mxcl.kafka.plist
vi homebrew.mxcl.kafka.plist
sudo vi homebrew.mxcl.kafka.plist

3. Add below contents to the file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

Note that you need to create a custom kafka startup script which will wait for a minute before starting kafka so that on startup zookeeper is started.

vi /usr/local/kafka/kafka_2.11/bin/
Add below contents to that file
sleep 60
/usr/local/kafka/current/bin/ /usr/local/kafka/current/config/

4. Create a symbolic link to this file in ~/Library/LaunchAgents
ln -sfv /usr/local/kafka/kafka_2.11/homebrew.mxcl.kafka.plist ~/Library/LaunchAgents/

5. Now just use launchctl to load this plist file using below command
launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.kafka.plist

Now reboot your Mac and kafka should be running after startup :)

Additional Kafka Tips
Starting Kafka Server ../conf/

List Kafka Topics
bin/ --list --zookeeper localhost:2181

No comments :

Post a Comment