Pub/Sub Processor

Event-driven processor that consumes messages from Google Cloud Pub/Sub topics and sends them to external ERP systems.

Purpose

  • Polls comerzzia database for pending documents

  • Transforms entities to ERP format (JSON/XML)

  • Publishes transformed messages to Google Cloud Pub/Sub topics

  • Consumes messages from Pub/Sub topics (optional, configurable)

  • Sends data to ERP systems via HTTP

  • Supports concurrent processing with configurable parallel pull

  • Comprehensive retry logic with DLQ support

Prerequisites

This application requires an existing Google Cloud Pub/Sub setup. It does not include topic or subscription creation.

Ensure:

  • Google Cloud project is configured

  • Required topics and subscriptions are created (main and DLQ)

  • Service account credentials are available

Configuration

Application Settings

See application.yml for full configuration.

Key settings: - Project ID: Google Cloud project identifier - Consumer activation: Optional - Enable/disable via czz-to-erp.consumer.enabled in application.yml - Topic configuration: Main topic and DLQ topic - Subscription configuration: Main subscription with credentials - Processing settings: Timeouts, retries, flow control - Retry configuration: Max attempts (default: 3) - Server configuration: Application and actuator ports, context paths - OpenAPI/Swagger UI: Interactive API documentation for testing endpoints - Actuator endpoints: Health checks, Prometheus metrics, and monitoring dashboard

Important Notes

Credentials Configuration

Requires service account JSON keys in credentials/ directory for Pub/Sub authentication.

Database Configuration

Database connection is configured via Spring Boot datasource properties in application.yml or profile-specific files (e.g., application-dev.yml).

Pub/Sub Message Structure

Messages consumed from configured main subscription with entity metadata attributes. Failed messages (after max retries) published to DLQ topic.

Error Handling and Retry Logic

Retry Mechanism

  • First failure: Error registered in database, message republished to main topic with retry-count=1

  • Subsequent failures: Message republished with incremented retry-count

  • Max retries reached: Message sent to DLQ topic and ACKed

Message Validation

  • Entity type validation against registry

  • Message size validation (default: 10MB max)

  • Timeout protection (default: 120 seconds)

Flow Control

  • Parallel pull count: 2 concurrent message receivers

  • Max outstanding elements: 5 messages buffered

  • ACK extension period: 60 seconds max processing time per message

Processing Flow

flowchart TD
	A[Pub/Sub Topic] --> B[Pub/Sub Subscriber<br/>Parallel pull: 2<br/>Max outstanding: 5]
	B --> C{Message<br/>Validation}
	C -->|Invalid Entity| Z1[ACK & Skip]
	C -->|Size > 10MB| Z1
	C -->|Valid| D[Entity Processing<br/>Timeout: 120s]

	D -->|Success| E[Mark Success in DB]
	E --> F[ACK Message]
	F --> G[ERP System HTTP]

	D -->|Error| H{Attempt < 3?}
	H -->|First Error| I[Register Error in DB]
	I --> J[Republish to Main Topic<br/>retry-count + 1]
	J --> K[ACK Original]
	K --> A

	H -->|Retry 2,3| J
	H -->|Max Retries<br/>Reached| L[Send to DLQ Topic]
	L --> M[ACK Message]

	style A fill:none,stroke:#666,stroke-width:3px
	style B fill:none,stroke:#666,stroke-width:3px
	style C fill:none,stroke:#666,stroke-width:3px
	style D fill:none,stroke:#666,stroke-width:3px
	style E fill:none,stroke:#666,stroke-width:3px
	style G fill:none,stroke:#666,stroke-width:3px
	style H fill:none,stroke:#666,stroke-width:3px
	style I fill:none,stroke:#666,stroke-width:3px
	style L fill:none,stroke:#666,stroke-width:3px