Pub/Sub Processor
Event-driven processor that consumes messages from Google Cloud Pub/Sub topics and sends them to external ERP systems.
Purpose
-
Polls comerzzia database for pending documents
-
Transforms entities to ERP format (JSON/XML)
-
Publishes transformed messages to Google Cloud Pub/Sub topics
-
Consumes messages from Pub/Sub topics (optional, configurable)
-
Sends data to ERP systems via HTTP
-
Supports concurrent processing with configurable parallel pull
-
Comprehensive retry logic with DLQ support
Prerequisites
This application requires an existing Google Cloud Pub/Sub setup. It does not include topic or subscription creation.
Ensure:
-
Google Cloud project is configured
-
Required topics and subscriptions are created (main and DLQ)
-
Service account credentials are available
Configuration
Application Settings
See application.yml for full configuration.
Key settings:
- Project ID: Google Cloud project identifier
- Consumer activation: Optional - Enable/disable via czz-to-erp.consumer.enabled in application.yml
- Topic configuration: Main topic and DLQ topic
- Subscription configuration: Main subscription with credentials
- Processing settings: Timeouts, retries, flow control
- Retry configuration: Max attempts (default: 3)
- Server configuration: Application and actuator ports, context paths
- OpenAPI/Swagger UI: Interactive API documentation for testing endpoints
- Actuator endpoints: Health checks, Prometheus metrics, and monitoring dashboard
Important Notes
-
Entity types must match those defined in
entities-config.ymlfrom Processor Common Library Module -
Flow control settings prevent message overload
Credentials Configuration
Requires service account JSON keys in credentials/ directory for Pub/Sub authentication.
Error Handling and Retry Logic
Retry Mechanism
-
First failure: Error registered in database, message republished to main topic with
retry-count=1 -
Subsequent failures: Message republished with incremented
retry-count -
Max retries reached: Message sent to DLQ topic and ACKed
Processing Flow
flowchart TD
A[Pub/Sub Topic] --> B[Pub/Sub Subscriber<br/>Parallel pull: 2<br/>Max outstanding: 5]
B --> C{Message<br/>Validation}
C -->|Invalid Entity| Z1[ACK & Skip]
C -->|Size > 10MB| Z1
C -->|Valid| D[Entity Processing<br/>Timeout: 120s]
D -->|Success| E[Mark Success in DB]
E --> F[ACK Message]
F --> G[ERP System HTTP]
D -->|Error| H{Attempt < 3?}
H -->|First Error| I[Register Error in DB]
I --> J[Republish to Main Topic<br/>retry-count + 1]
J --> K[ACK Original]
K --> A
H -->|Retry 2,3| J
H -->|Max Retries<br/>Reached| L[Send to DLQ Topic]
L --> M[ACK Message]
style A fill:none,stroke:#666,stroke-width:3px
style B fill:none,stroke:#666,stroke-width:3px
style C fill:none,stroke:#666,stroke-width:3px
style D fill:none,stroke:#666,stroke-width:3px
style E fill:none,stroke:#666,stroke-width:3px
style G fill:none,stroke:#666,stroke-width:3px
style H fill:none,stroke:#666,stroke-width:3px
style I fill:none,stroke:#666,stroke-width:3px
style L fill:none,stroke:#666,stroke-width:3px