Kafka Processor
Event-driven processor that consumes messages from Apache Kafka topics and processes them to comerzzia database.
Purpose
-
Consumes messages from Kafka topics
-
Validates and processes entity data
-
Persists to comerzzia database via integration services
-
Supports concurrent processing with configurable consumers
Prerequisites
This application requires an existing Apache Kafka. It does not include Kafka server setup or topic creation.
Ensure:
-
Apache Kafka service is accessible
-
Required topics are created (main topic and error topic)
Processing Flow
flowchart TD
A[Kafka Main Topic] --> B[Kafka Consumer]
B --> C{Message<br/>Validation}
C -->|Invalid Entity| E1[Error Topic + ACK]
C -->|Size > Max| E1
C -->|Valid| D[Entity Processing]
D --> F[(comerzzia DB)]
F -->|Success| G[ACK Message]
F -->|Error| E2[Error Topic + ACK]
style A fill:none,stroke:#666,stroke-width:3px
style B fill:none,stroke:#666,stroke-width:3px
style C fill:none,stroke:#666,stroke-width:3px
style D fill:none,stroke:#666,stroke-width:3px
style E1 fill:none,stroke:#666,stroke-width:3px
style E2 fill:none,stroke:#666,stroke-width:3px
style F fill:none,stroke:#666,stroke-width:3px
style G fill:none,stroke:#666,stroke-width:3px
Kafka Topic Structure
Messages consumed from configured main topic with entity metadata headers. Failed messages published to error topic.
Configuration
Application Settings
Key settings:
| Spring Property | Environment var | Default value |
|---|---|---|
erp-to-czz.main-topic |
ERP_TO_CZZ_MAIN_TOPIC |
erp-to-czz-input |
erp-to-czz.error-topic |
ERP_TO_CZZ_ERROR_TOPIC |
erp-to-czz-input-errors |
erp-to-czz.consumer.group-id |
ERP_TO_CZZ_CONSUMER_GROUP_ID |
erp-processor-group |
spring.kafka.bootstrap-servers |
SPRING_KAFKA_BOOSTRAP_SERVERS |
kafka-server-svc:9092 |
Docker container execution
docker run -it --rm --name processor-kafka --env-file processor-dev.env --pull=always -p 8080:8080 europe-west3-docker.pkg.dev/czz-devops/comerzzia/erp-to-czz/processor-kafka-application:1.0
SPRING_KAFKA_BOOSTRAP_SERVERS=127.0.0.1:9092
ERP_TO_CZZ_MAIN_TOPIC=erp-to-czz-input
ERP_TO_CZZ_ERROR_TOPIC=erp-to-czz-input-errors
uidActividad=MYACTIVITY
uidInstancia=MYINSTANCE
SPRING_PROFILES_ACTIVE=jsonlogs
COMERZZIA_DB_CLASS=com.mysql.jdbc.Driver
COMERZZIA_DB_DRIVER=mysql
COMERZZIA_DB_USER=comerzzia
COMERZZIA_DB_PASS=comerzzia
COMERZZIA_DB_URL=jdbc:mysql://127.0.0.1:3306/comerzzia