Kafka Processor
Event-driven processor that consumes messages from Apache Kafka topics and sends them to external ERP systems.
Purpose
-
Consumes messages from Kafka topics (optional, configurable)
-
Polls comerzzia database for pending documents
-
Validates and processes entity data
-
Sends data to ERP systems via HTTP
-
Supports concurrent processing with configurable consumers
-
Comprehensive retry logic with DLQ support
Prerequisites
This application requires an existing Apache Kafka cluster. It does not include Kafka server setup or topic creation.
Ensure: - Kafka cluster is accessible - Required topics are created (main topic and DLQ topic)
Configuration
Application Settings
See application.yml for full configuration.
Key settings: - Kafka bootstrap servers: Connection to Kafka cluster - Consumer configuration: Base group ID, concurrency, and offset management - Processing settings: Timeouts, retries, and acknowledgment - Retry configuration: Max attempts - Server configuration: Server port and context path - Actuator endpoints: Monitoring dashboard, metrics and health
Entity Configuration
Each entity is configured independently in entities-config.yml from Processor Common Library Module:
Per-entity settings: - topic: Independent topic for this entity - dlqTopic: Independent DLQ topic for failed messages - schedulerCron: Independent polling schedule (cron expression) - schedulerEnabled: Enable/disable database polling for this entity
Kafka Topic Structure
Architecture:
- Each entity type has its own main topic and DLQ topic
- Messages include entityType header for entity identification
- Failed messages (after max retries) are sent to their entity-specific DLQ topic
- Each entity uses an independent consumer group for offset management
