How to make Kafka run automatically on startup in Mac

Kafka Automatic startup in Mac Tutorial

Apache Kafka is a highly-scalable publish-subscribe messaging system that can serve as the data backbone in distributed applications. With Kafka’s Producer-Consumer model it becomes easy to implement multiple data consumers that do live monitoring as well persistent data storage for later analysis.


Note: You need zookeeper to setup kafka.

In Mac you can either install kafka via home brew using below command
brew install kafka
which will install it in /usr/local/Cellar/kafka directory.
*This will also install all dependencies, like zookeeper which is required to run kafka server.
or
You can also manually download it and extract it in /usr/local/kafka/ directory.

Now all you need is to add a plist file in order to have kafka running on startup and not manually running it everytime. Verified to be working in Mac-OS 10.12

Before moving ahead make sure you setup zookeeper to start automatically as below is dependent on zookeeper running. Read here.

1. Go to the kafka installation directory - usually /usr/local/kafka/kafka_2.11/
cd /usr/local/kafka/kafka_2.11/

2. Create a file named - homebrew.mxcl.kafka.plist
vi homebrew.mxcl.kafka.plist
or

sudo vi homebrew.mxcl.kafka.plist

3. Add below contents to the file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>KeepAlive</key>
    <dict>
        <key>OtherJobEnabled</key>
        <string>homebrew.mxcl.zookeeper</string>
    </dict>
    <key>Label</key>
    <string>homebrew.mxcl.kafka</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/kafka/kafka_2.11/bin/kafka-server-start-custom.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/usr/local/kafka/kafka_2.11</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/kafka/kafka_2.11/kafka.err</string>
    <key>StandardOutPath</key>
    <string>/usr/local/kafka/kafka_2.11/kafka.out</string>
  </dict>
</plist>

Note that you need to create a custom kafka startup script which will wait for a minute before starting kafka so that on startup zookeeper is started.

vi /usr/local/kafka/kafka_2.11/bin/kafka-server-start-custom.sh
Add below contents to that file

#!/bin/bash
sleep 60
/usr/local/kafka/current/bin/kafka-server-start.sh /usr/local/kafka/current/config/server.properties

4. Create a symbolic link to this file in ~/Library/LaunchAgents
ln -sfv /usr/local/kafka/kafka_2.11/homebrew.mxcl.kafka.plist ~/Library/LaunchAgents/

5. Now just use launchctl to load this plist file using below command

launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.kafka.plist

Now reboot your Mac and kafka should be running after startup :)

Additional Kafka Tips
Starting Kafka Server
kafka-server-start.sh ../conf/server.properties

List Kafka Topics
bin/kafka-topics.sh --list --zookeeper localhost:2181