Streaming with Apache Kafka FAQ

General

What is OCI Streaming with Apache Kafka?

OCI Streaming with Apache Kafka is a managed service that enables you to build real-time data streaming applications using Apache Kafka on Oracle Cloud Infrastructure (OCI). It provides a fully managed Kafka environment, eliminating the need to manage underlying infrastructure.

What are the primary use cases for OCI Streaming with Apache Kafka?

Typical use cases include user change data capture (CDC), metrics and logs ingestion, behavior analysis, real-time demand forecasting, fraud detection, anomaly detection in trading, and healthcare analytics. This service is also ideal for scenarios that require high-throughput, low-latency data ingestion and processing.

What are the available cluster types in OCI Streaming with Apache Kafka?

There are two cluster types: starter and high availability (HA). A starter cluster can be set up in an availability domain–specific or regional subnet with between 1 and 24 brokers. HA clusters are created in regional subnets only and are recommended for all production workloads with high availability. HA clusters require a minimum of 3 broker nodes across availability domains (ADs) or fault domains (FDs) with a maximum of 24 brokers.

What versions of Apache Kafka are supported?

Our managed Kafka service supports the latest stable release of Apache Kafka 3.7, 3.6, and 3.5, as well as several previous versions. You can find the full list of supported versions in our documentation.

Does OCI Streaming with Apache Kafka support Kafka Raft (KRaft) mode?

Currently, OCI Streaming with Apache Kafka supports only ZooKeeper for managing Kafka metadata and cluster coordination. KRaft mode, which enables Kafka to manage metadata natively without ZooKeeper, is not yet supported. However, KRaft support is planned for a future release, enabling a simplified and more resilient Kafka architecture.

Are Apache Kafka APIs compatible with OCI Streaming with Apache Kafka?

Yes, all data plane and admin APIs are natively supported by OCI Streaming with Apache Kafka.

Can I choose different processor types for OCI Streaming with Apache Kafka?

Currently, the service supports ARM A1 shapes by default. If you need other processor families, such as AMD or Intel, please contact OCI Support to discuss additional options and configurations.

What is the default broker configuration for a cluster?

Unless otherwise specified, OCI Streaming with Apache Kafka uses the defaults specified by the open source version of Apache Kafka. Check the documented default settings for both cluster types.

Can I change the default broker configurations or upload a cluster configuration to OCI Streaming with Apache Kafka?

Yes, OCI Streaming with Apache Kafka allows you to create and apply custom configurations to both new and existing clusters. Currently, custom configurations can be created and versioned using the CLI, which can then be applied when creating a cluster. In the OCI Console, you can select either the default or a custom configuration during cluster setup. For more information on custom configurations, see the configuration documentation.

What configuration properties am I able to customize?

See the documentation on configuration properties you can customize.

Security

Is my data encrypted and protected?

Security is fundamental to OCI Streaming with Apache Kafka. The OCI Streaming with Apache Kafka service provides encryption in transit and at rest by default, secured with Oracle-managed encryption keys. Network access controls, user authentication, and authorization settings offer further layers of protection to safeguard your data and service access.

Can I restrict access to specific Kafka topics?

Yes, you can use Kafka Access Control Lists (ACLs) to restrict access to specific topics, consumer groups, and administrative operations. This helps ensure that only authorized clients can access Kafka resources.

What authentication methods are available for OCI Streaming with Apache Kafka?

OCI Streaming with Apache Kafka supports multiple authentication methods to secure access to your Kafka cluster. Users can authenticate via Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM) or Mutual TLS (mTLS) for secure communication and access control. These options provide robust security measures to help ensure that only authenticated users can interact with the Kafka clusters.

Does OCI Streaming with Apache Kafka support Lightweight Direct Access Protocol (LDAP) or Kerberos authentication?

At present, the service supports SASL/SCRAM and mTLS for authentication. In the future, OCI Streaming with Apache Kafka will support OCI Identity and Access Management (IAM), which will enable integration with Active Directory and LDAP through federated identity providers and identity synchronization for streamlined authentication and user management.

Does OCI Streaming support virtual cloud network (VCN) peering for secure, private access?

Yes, you can configure your Kafka clusters within private subnets and use VCN peering to securely connect from other OCI VCNs or your on-premises network, depending on your architecture requirements.

Performance and scalability

What are the throughput and scalability limits of OCI Streaming with Apache Kafka?

OCI Streaming with Apache Kafka supports scalable data ingestion and processing, with customizable broker configurations to meet high-throughput requirements. You can add up to 24 brokers in a cluster, depending on your selected configuration.

How does OCI Streaming handle load balancing and fault tolerance?

For high availability clusters, OCI Streaming with Kafka automatically distributes brokers across multiple ADs or FDs for resilience. The service also provides automated load balancing and partitioning across brokers.

Can I dynamically scale my Kafka cluster?

Yes, you can scale your Kafka cluster within the broker count limits (1–24 brokers). This allows you to adjust resources to match workload demands without disrupting service.

Connectivity

Can I access OCI Streaming with Apache Kafka from outside the Oracle Cloud?

Currently, public connectivity to OCI Streaming with Apache Kafka is not available. Access to Kafka clusters is restricted to OCI VCNs. To connect external producers and consumers, you can set up private connectivity solutions, such as OCI FastConnect or VPN, to securely extend your on-premises network or other cloud environments to OCI.

What configurations are needed to connect Kafka clients from my on-premises network?

For on-premises connectivity, you can set up an OCI FastConnect or VPN connection to securely access your Kafka cluster hosted on OCI.

Does OCI Streaming with Apache Kafka support replication between clusters (Active/Active and Active/Standby)?

Yes, OCI Streaming with Apache Kafka supports intercluster replication, which can be configured for Active/Active or Active/Standby setups using MirrorMaker 2. This allows for flexible data replication strategies across clusters, enabling disaster recovery and data synchronization. By default, data within an HA configuration is replicated within the same region for resiliency. You can use MirrorMaker 2 to customize additional cross-region or cross-cluster replication based on your specific requirements.

Can I migrate data within my existing Apache Kafka cluster to OCI Streaming with Apache Kafka?

Yes, you can use third-party tools or open source tools such as MirrorMaker 2 , supported by Apache Kafka, to replicate data from clusters into OCI Streaming with Apache Kafka.

Does OCI Streaming with Apache Kafka support change data capture (CDC) with Debezium?

Yes, OCI Streaming with Apache Kafka supports CDC using Debezium, along with any other Kafka connectors. Currently, users are responsible for managing and maintaining their own connectors for CDC. However, a fully managed Kafka Connect service, which will simplify connector management, is planned for a future release.

Operations and monitoring

How do I monitor my Kafka cluster on OCI?

Currently, direct integration with OCI Monitoring and JMX metrics is not supported for OCI Streaming with Apache Kafka. To monitor Kafka cluster performance and activity, you must use custom solutions or external monitoring tools. You can deploy monitoring tools within the same VCN as the Kafka cluster to collect metrics such as broker performance and topic activity.

Is logging available for OCI Streaming with Apache Kafka?

Currently, OCI Streaming with Apache Kafka does not support direct integration with OCI Logging for capturing operational logs. For troubleshooting and analysis, you may need to implement external logging solutions or monitoring tools within your environment.

How do I manage topic configurations and replication?

Topic configurations, partitions, and replication settings can only be managed using the Kafka CLI, SDKs, or Kafka APIs within the same network as the Kafka cluster. Currently, the OCI Console does not support direct management of these configurations. You can set the replication factor via these tools to meet your fault tolerance and availability needs.

Is there a UI for administering the Kafka cluster in OCI?

Currently, OCI Streaming with Apache Kafka does not provide a native UI for cluster administration. However, you can use your own third-party tools—such as Kafbat, AKHQ, and similar—to manage and monitor your Kafka clusters. These tools can be deployed within OCI, allowing you to view and administer the cluster seamlessly within your own environment.

Pricing and billing

How is OCI Streaming with Apache Kafka priced?

The pricing for OCI Streaming with Apache Kafka is based on the underlying infrastructure. You can configure the cluster by selecting the amount of Kafka brokers, cores, memory, and storage. The associated infrastructure costs are pass-through charges based on the OCI pricing list (see A1 flex shapes for OCPU pricing). Additionally, a service fee of $0.10 per OCPU per hour applies based on the number of OCPUs provisioned. For clusters using the Arm A1 shape, the service fee is discounted by 50%, resulting in a $0.05 charge per OCPU per hour.

Are there additional costs for data storage in OCI Streaming?

Yes, data storage costs are based on the Block Volume storage that you select during cluster provisioning. You can choose between 50 GB and 1 TB of storage per broker. The associated storage cost is a pass-through expense based on OCI Block Volume pricing. For instance, if you need 5 TB of storage, you will provision 5 brokers with 1 TB each, totaling 5 TB of storage for the cluster.

Integration and compatibility

Does OCI Streaming with Apache Kafka support Kafka Connect?

Currently, OCI Streaming with Apache Kafka does not include a managed Kafka Connect service, but you can deploy Kafka Connect on OCI Compute instances and integrate it with your Kafka clusters.

Can I use OCI Streaming with Apache Kafka with Oracle GoldenGate for change data capture?

Yes, you can integrate Oracle GoldenGate with OCI Streaming with Apache Kafka to capture and stream changes from databases to Kafka, supporting various data replication and analytics use cases.

How does OCI Streaming with Apache Kafka integrate with OCI Data Flow?

OCI Data Flow can be used to process data from Kafka topics directly, allowing you to build and execute Spark applications for data transformation, analytics, and storage.

Is a Schema Registry available with OCI Streaming with Apache Kafka?

Yes, the service currently supports integration with open source Schema Registry options, allowing you to manage and store schemas for your data streams. A fully managed Schema Registry offering is in development and expected in a future release, simplifying schema management within the OCI environment.