Skip to main content

Kafka Questions 面试问题

1. Do you use encryptions/security before sending data, decipher of message after get access to a topic?​

Kafka provides support for SSL/TLS encryption for securing data in transit between Kafka clients and Kafka brokers. This encryption ensures that the data is protected from eavesdropping and tampering during transmission.

Kafka also provides authorization mechanisms for controlling access to topics and partitions within a Kafka cluster. This enables administrators to ensure that only authorized users or applications can read or write data to specific topics or partitions.

In addition to SSL/TLS encryption and authorization mechanisms, Kafka also provides support for message-level encryption using the Kafka Streams API. This allows users to encrypt data at the message level before it is written to Kafka, and decrypt it after it is read from Kafka. This can provide an additional layer of security for sensitive data.

Kafka 安全配置

  • SSL/TLS加密配置：在服务器层面，需要为Kafka broker和Zookeeper节点生成SSL/TLS证书。在客户端代码层面，需要配置SSL/TLS协议和证书文件的路径。例如，在 Java客户端中，可以使用 ssl.truststore.location 和 ssl.keystore.location 属性来配置证书的路径，以及 security.protocol 属性来指定 SSL/TLS 协议。
  • 访问控制列表（ACL）配置：在服务器层面，可以使用Kafka提供的 kafka-acls.sh工具来配置 ACL 规则，以控制用户和应用程序对Kafka集群中特定主题和分区的访问权限。在客户端代码层面，需要使用配置文件或代码中的认证信息（例如用户名和密码）来进行身份验证。例如，在Java客户端中，可以使用security.protocol 属性指定SASL认证协议，并使用 sasl.mechanism 和 sasl.jaas.config 属性来指定认证机制和认证信息。
  • 消息级别的加密配置：在客户端代码层面，可以使用Kafka Streams API提供的加密和解密函数来对消息进行加密和解密。例如，可以使用 KStream#map 和 KStream#mapValues 函数来对消息进行加密，然后使用 KStream#map 和 KStream#mapValues 函数来对消息进行解密。在此过程中，需要使用加密算法和密钥来执行加密和解密操作。

2. What if not able to reach Kafka?​

If you are not able to reach Kafka, there could be several possible reasons:

  • Network connectivity issues: Check if there are any network connectivity issues between the Kafka client and the Kafka broker. This can be done by pinging the Kafka broker from the client machine or using network diagnostic tools like telnet.
  • Kafka broker is down: Check if the Kafka broker is running and reachable. This can be done by logging into the Kafka broker machine and checking the Kafka logs to see if there are any errors or exceptions.
  • Kafka topic/partition is unavailable: Check if the Kafka topic or partition that the client is trying to access is available. This can be done by using the kafka-topics.sh tool to list the available topics and partitions in the Kafka cluster.
  • Authentication and authorization issues: Check if there are any authentication or authorization issues that are preventing the client from accessing the Kafka cluster. This can be done by reviewing the Kafka security configuration and checking if the client has the necessary permissions to read or write to the Kafka topic or partition.
  • Client configuration issues: Check if there are any issues with the client configuration, such as incorrect Kafka broker IP addresses, incorrect port numbers, or incorrect Kafka client properties.

3. Best practices for setting the number of partitions in kafka​

  • Plan for scalability: When creating a topic, it's important to plan for future scalability. This means considering the expected growth in data volume and consumer load, and setting the number of partitions accordingly. A good rule of thumb is to have at least as many partitions as the number of consumer instances that need to consume from the topic.
  • Avoid over-partitioning: While having multiple partitions can help increase throughput, having too many partitions can lead to increased overhead and complexity in managing the partitions. 虽然拥有多个分区有助于提高吞吐量，但拥有太多分区会导致管理分区的开销和复杂性增加。
  • Consider ordering guarantees: If message ordering is important, it's important to keep in mind that increasing the number of partitions can impact the ordering guarantees. 如果消息排序很重要，请务必记住，增加分区数量会影响排序保证。
  • Test and optimize: It's important to test and optimize the number of partitions based on the actual data throughput and consumer load.
  • Avoid changing the number of partitions frequently: Changing the number of partitions on a topic can have implications for data retention and ordering guarantees, so it's important to plan and test these changes carefully before making them in a production environment. 更改主题的分区数可能会对数据保留和排序保证产生影响。