Software Development
Achieving Strong Consistency in Distributed Spring Boot Applications with Redpanda
Redpanda is a modern, Kafka-compatible streaming platform designed for high performance and low latency. As a drop-in replacement for Apache Kafka, it offers several advantages:
- Simplified architecture (single binary, no ZooKeeper dependency)
- Improved performance (lower latency, higher throughput)
- Full Kafka API compatibility (works with existing Kafka clients and tools)
- Strong consistency guarantees out of the box
1. Strong Consistency Patterns with Spring Boot and Redpanda
1. Transactional Messaging Pattern
public class KafkaConfig {
@Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "redpanda:9092");
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "tx-1");
configProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
return new DefaultKafkaProducerFactory<>(configProps);
}
@Bean
public KafkaTransactionManager<String, String> kafkaTransactionManager(
ProducerFactory<String, String> producerFactory) {
return new KafkaTransactionManager<>(producerFactory);
}
}
@Service
public class OrderService {
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
@Autowired
private OrderRepository orderRepository;
@Transactional("kafkaTransactionManager")
public void processOrder(Order order) {
// Database operation
orderRepository.save(order);
// Kafka operation in the same transaction
kafkaTemplate.send("orders", order.getId(), order.toJson());
}
}
2. Outbox Pattern for Strong Consistency
@Configuration
@Entity
public class OutboxEvent {
@Id
@GeneratedValue
private Long id;
private String aggregateType;
private String aggregateId;
private String eventType;
private String payload;
private LocalDateTime timestamp;
private boolean published;
}
@Service
public class OrderService {
@Transactional
public void createOrder(Order order) {
// 1. Save order to database
orderRepository.save(order);
// 2. Save event to outbox table in the same transaction
OutboxEvent event = new OutboxEvent();
event.setAggregateType("Order");
event.setAggregateId(order.getId());
event.setEventType("OrderCreated");
event.setPayload(order.toJson());
event.setTimestamp(LocalDateTime.now());
event.setPublished(false);
outboxRepository.save(event);
}
}
@Component
public class OutboxProcessor {
@Scheduled(fixedRate = 1000)
@Transactional
public void processOutbox() {
List<OutboxEvent> events = outboxRepository.findByPublishedFalse();
events.forEach(event -> {
kafkaTemplate.send("orders", event.getAggregateId(), event.getPayload());
event.setPublished(true);
outboxRepository.save(event);
});
}
}
2. Redpanda-Specific Optimizations
1. Leveraging Redpanda’s Linearizable Reads
@Configuration
public class ConsumerConfig {
@Bean
public ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "redpanda:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
// Redpanda specific optimization
props.put("enable.linearizable.reads", "true");
return new DefaultKafkaConsumerFactory<>(props);
}
}
2. High-Performance Consumer Configuration
@KafkaListener(topics = "orders", groupId = "order-group")
public void listen(Order order) {
// Process order with strong consistency guarantees
orderProcessingService.process(order);
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(4); // Tune based on Redpanda partition count
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
return factory;
}
3. Monitoring and Observability
@Configuration
public class MetricsConfig {
@Bean
public MicrometerProducerListener<String, String> micrometerProducerListener(MeterRegistry registry) {
return new MicrometerProducerListener<>(registry);
}
@Bean
public MicrometerConsumerListener<String, String> micrometerConsumerListener(MeterRegistry registry) {
return new MicrometerConsumerListener<>(registry);
}
}
// In application.properties:
management.endpoints.web.exposure.include=health,info,metrics
management.metrics.export.prometheus.enabled=true
4. Deployment Considerations
- Redpanda Cluster Sizing: Start with 3 nodes for production deployments
- Resource Allocation: Redpanda benefits from more CPU than Kafka
- Storage: Use direct-attached storage for best performance
- Networking: Low-latency networking is crucial for strong consistency
5. Comparison with Traditional Kafka
| Feature | Redpanda | Apache Kafka |
|---|---|---|
| Architecture | Single binary, no ZooKeeper | Requires ZooKeeper |
| Performance | Lower latency, higher throughput | Good performance with tuning |
| Consistency | Strong by default | Configurable |
| Operational Complexity | Simplified | More complex |
| Compatibility | Full Kafka API compatibility | N/A |
6. Conclusion
Redpanda provides an excellent alternative to Kafka for Spring Boot applications requiring strong consistency in distributed environments. Its simplified architecture and performance characteristics make it particularly suitable for high-throughput, low-latency systems while maintaining the familiar Kafka API that Spring developers are accustomed to.
