Confluent develops and maintains a Go client for Apache Kafka® that offers a producer and a consumer.
Go Client installation¶
Installing Kafka on Mac with Homebrew. The process is straightforward here. Either way, ELK is a powerful analysis tool to have on your side in times of trouble. Apache Kafka: A Distributed Streaming Platform. Apache Kafka Quickstart. Interested in getting started with Kafka? Follow the instructions in this quickstart, or watch the video below.
The Go client, called confluent-kafka-go, is distributed via GitHuband gopkg.in to pin to specific versions. The Go client uses librdkafka, the C client,internally and exposes it as Go library using cgo.Starting with confluent-kafka-go v1.4.0, the librdkafka client is nowincluded in the Go client and no separate installation of librdkafka isrequired for the supported platforms (Linux (glibc and musl based), and Mac OSX).
For other platforms the following instructions still apply:To install the Go client, first installthe C client including its development package as well as a C build toolchain including
pkg-config
. On Red Hat-based Linux distributions install the following packages in addition to librdkafka:On Debian-based distributions, install the following in addition to librdkafka:
On macOS using Homebrew, install the following:
Next, use
goget
to install the library: Sensei mac os app.Your Go code can now import and use the client. You can also build and run a small command line utility,
go-kafkacat
,to ensure the installation was successful:See the clients documentation for code examples showing how to use the library.
The source code is also available in the ZIP and TAR archives under the directory
src/
.Go Client example code¶
For Hello World examples of Kafka clients in Go, see Go.All examples include a producer and consumer that can connect to any Kafka cluster running on-premises or in Confluent Cloud.They also include examples of how to produce and consume Avro data.
Itool For Mac
Kafka Producer¶
Initialization¶
The Go client uses a
ConfigMap
object to pass configuration to the producer:Asynchronous writes¶
In Go, you initiate a send by calling the
Produce()
method, passing a Message
object and anoptional chanEvent
that can be used to listen for the result of the send. The Message
object contains an opaque interface{}
field that can be used to pass arbitrary data with themessage to the subsequent event handler:To produce asynchronously, you can use a Goroutine to handle message delivery reports andpossibly other event types (errors, stats, etc) concurrently:
Synchronous writes¶
Making writes synchronous is typically a bad idea since it decreases throughput.Nevertheless, you can make writes synchronous by receiving from the delivery channelpassed to the
Produce()
method call as shown below:Alternatively, you can wait for all messages to be acknowledged and use the
Flush()
method:Important
Flush()
will only serve the producer’s Events()
channel, notapplication-specified delivery channels. If Flush()
is called and no goroutineis processing the delivery channel, its buffer may fill up and cause theflush to timeout.Kafka Consumer¶
Initialization¶
The Go client uses a
ConfigMap
object to pass configuration to the consumer:Basic usage¶
The Go client uses librdkafka internally so it followsa multi-threaded approach to Kafka consumption.The API returns only a single message or event at a time:
Go Client code examples¶
Basic poll loop¶
The consumer API is centered around the
Consumer.Poll()
method, which isused to retrieve records from the brokers. The Consumer.SubscribeTopics()
methodcontrols which topics will be fetched in poll. This consumer example shows typical usage,whichinvolves an initial call to Consumer.SubscribeTopics()
to setup the topicsof interest and then a loop which calls Consumer.Poll()
until theapplication is shut down.If no records are received, then
Consumer.Poll()
will returnan empty record set.Note that you should always call
Consumer.Close()
after you are finishedusing the consumer. Doing so will ensure that active sockets areclosed and internal state is cleaned up. It will also trigger a grouprebalance immediately which ensures that any partitions owned by theconsumer are re-assigned to another member in the group. If not closedproperly, the broker will trigger the rebalance only after the sessiontimeout has expired. Latch is added to this example to ensurethat the consumer has time to finish closing before finishingshut down.Synchronous commits¶
The Go client provides a synchronous
Commit()
method call. Other variants of commit methods also accept a list of offsets tocommit or a Message
in order to commit offsets relative to a consumed message.When using manual offset commit, be sure to disable the enable.auto.commit
configuration.In this example, a synchronous commit is triggered every
MIN_COMMIT_COUNT
messages. You could also trigger the commit on expiration of atimeout to ensure there the committed position is updated regularly.Delivery guarantees¶
In the previous example, you get “at least once”delivery since the commit follows the message processing. By changing the order,however, you can get “at most once” delivery, but you must be alittle careful with the commit failure.
For simplicity in this example,
Consumer.CommitMessage()
is usedprior to processing the message. Committing on every message wouldproduce a lot of overhead in practice. A better approach would be tocollect a batch of messages, execute the synchronous commit, and thenprocess the messages only if the commit succeeded.Asynchronous commits¶
To commit asynchronously, simply execute the commit in a goroutine to commit asynchronously:
Rebalance events are exposed as events returned by the
Poll()
method. To see these events you must create the consumer with thego.application.rebalance.enable
configuration and handleAssignedPartitions
and RevokedPartitions
events by explicitly callingAssign()
and Unassign()
for AssignedPartitions
andRevokedPartitions
respectively:API documentation¶
Click here to view the Go Client API documentation. Torrent dragon medical practice edition reviews.
Welcome to the Chocolatey Community Package Repository! The packages found in this section of the site are provided, maintained, and moderated by the community.
Moderation
Every version of each package undergoes a rigorous moderation process before it goes live that typically includes:
- Security, consistency, and quality checking
- Human moderators who give final review and sign off
More detail at Security and Moderation.
Kafka Tool For Mac
Organizational Use
If you are an organization using Chocolatey, we want your experience to be fully reliable. Due to the nature of this publicly offered repository, reliability cannot be guaranteed. Packages offered here are subject to distribution rights, which means they may need to reach out further to the internet to the official locations to download files at runtime.
Fortunately, distribution rights do not apply for internal use. With any edition of Chocolatey (including the free open source edition), you can host your own packages and cache or internalize existing community packages.
Disclaimer
Kafka Tool For Mac Download
Your use of the packages on this site means you understand they are not supported or guaranteed in any way. Learn more..