Pub/Sub Processor
Cloud-native processor that consumes messages from Google Cloud Pub/Sub subscriptions and processes them to comerzzia database.
Purpose
-
Consumes messages from Google Cloud Pub/Sub subscriptions
-
Validates and processes entity data
-
Persists to comerzzia database via integration services
-
Supports concurrent processing with configurable parallel pull
Prerequisites
This application requires an existing Google Cloud Platform project with Pub/Sub. It does not include GCP project setup or topic creation.
Ensure:
-
GCP project with Pub/Sub API enabled
-
Service accounts with
roles/pubsub.subscriberandroles/pubsub.publisherroles -
Required topics and subscriptions are created
-
Service account JSON keys available
Processing Flow
flowchart TD
A[Pub/Sub Subscription] --> B[Parallel Pull Consumer]
B --> C{Message<br/>Validation}
C -->|Invalid Entity| E1[Error Topic + ACK]
C -->|Size > Max| E1
C -->|Valid| D[Entity Processing]
D --> F[(comerzzia DB)]
F -->|Success| G[ACK Message]
F -->|Error| E2[Error Topic + ACK]
style A fill:none,stroke:#666,stroke-width:3px
style B fill:none,stroke:#666,stroke-width:3px
style C fill:none,stroke:#666,stroke-width:3px
style D fill:none,stroke:#666,stroke-width:3px
style E1 fill:none,stroke:#666,stroke-width:3px
style E2 fill:none,stroke:#666,stroke-width:3px
style F fill:none,stroke:#666,stroke-width:3px
style G fill:none,stroke:#666,stroke-width:3px
Pub/Sub Subscription Structure
Messages consumed from configured subscription. Failed messages published to error topic.
Configuration
Application Settings
Key settings:
| Spring Property | Environment var | Default value |
|---|---|---|
erp-to-czz.project-id |
ERP_TO_CZZ_PROJECT_ID |
erp-to-czz |
erp-to-czz.main-topic.topic-id |
ERP_TO_CZZ_MAIN_TOPIC_TOPIC_ID |
erp-to-czz-input |
erp-to-czz.main-topic.subscription-id |
ERP_TO_CZZ_MAIN_TOPIC_SUBSCRIPTION_ID |
erp-to-czz-input-consumer |
erp-to-czz.main-topic.credentials-file |
ERP_TO_CZZ_MAIN_TOPIC_CREDENTIALS_FILE |
classpath:credentials/ep-to-czz-subcription-key.json |
erp-to-czz.error-topic.topic-id |
ERP_TO_CZZ_ERROR_TOPIC_TOPIC_ID |
erp-to-czz-input-errors |
erp-to-czz.error-topic.subscription-id |
ERP_TO_CZZ_ERROR_TOPIC_SUBSCRIPTION_ID |
erp-to-czz-input-error-consumer |
erp-to-czz.error-topic.credentials-file |
ERP_TO_CZZ_ERROR_TOPIC_CREDENTIALS_FILE |
classpath:credentials/erp-to-czz-publish-key.json |
Important Notes
-
Separate credentials allow fine-grained IAM permissions (subscribe vs publish)
-
Parallel pull count controls concurrent message processing
-
Credentials file paths must include a resource prefix (
file:orclasspath:) -
The error topic should be different from the main topic, otherwise failed messages are consumed again
Docker container execution
docker run -it --rm --name processor-pubsub $PWD:/apps/bin/lib --env-file processor-dev.env --pull=always -p 8080:8080 europe-west3-docker.pkg.dev/czz-devops/comerzzia/erp-to-czz/processor-pubsub-application:1.0
ERP_TO_CZZ_PROJECT_ID=erp-to-czz-test
ERP_TO_CZZ_MAIN_TOPIC_TOPIC_ID=erp-to-czz-input
ERP_TO_CZZ_MAIN_TOPIC_SUBSCRIPTION_ID=erp-to-czz-input-consumer-test
ERP_TO_CZZ_MAIN_TOPIC_CREDENTIALS_FILE=classpath:credentials/erp-to-czz-input-consumer-test-key.json
ERP_TO_CZZ_ERROR_TOPIC_TOPIC_ID=erp-to-czz-input-errors
ERP_TO_CZZ_ERROR_TOPIC_SUBSCRIPTION_ID=erp-to-czz-input-publish-test
ERP_TO_CZZ_ERROR_TOPIC_CREDENTIALS_FILE=classpath:credentials/erp-to-czz-input-publish-test-key.json
uidActividad=MYACTIVITY
uidInstancia=MYINSTANCE
SPRING_PROFILES_ACTIVE=jsonlogs
COMERZZIA_DB_CLASS=com.mysql.jdbc.Driver
COMERZZIA_DB_DRIVER=mysql
COMERZZIA_DB_USER=comerzzia
COMERZZIA_DB_PASS=comerzzia
COMERZZIA_DB_URL=jdbc:mysql://127.0.0.1:3306/comerzzia