Pub/Sub Processor

Cloud-native processor that consumes messages from Google Cloud Pub/Sub subscriptions and processes them to comerzzia database.

Purpose

  • Consumes messages from Google Cloud Pub/Sub subscriptions

  • Validates and processes entity data

  • Persists to comerzzia database via integration services

  • Supports concurrent processing with configurable parallel pull

Prerequisites

This application requires an existing Google Cloud Platform project with Pub/Sub. It does not include GCP project setup or topic creation.

Ensure:

  • GCP project with Pub/Sub API enabled

  • Service accounts with roles/pubsub.subscriber and roles/pubsub.publisher roles

  • Required topics and subscriptions are created

  • Service account JSON keys available

Processing Flow

flowchart TD
    A[Pub/Sub Subscription] --> B[Parallel Pull Consumer]
    B --> C{Message<br/>Validation}
    C -->|Invalid Entity| E1[Error Topic + ACK]
    C -->|Size > Max| E1
    C -->|Valid| D[Entity Processing]

    D --> F[(comerzzia DB)]
    F -->|Success| G[ACK Message]
    F -->|Error| E2[Error Topic + ACK]

    style A fill:none,stroke:#666,stroke-width:3px
    style B fill:none,stroke:#666,stroke-width:3px
    style C fill:none,stroke:#666,stroke-width:3px
    style D fill:none,stroke:#666,stroke-width:3px
    style E1 fill:none,stroke:#666,stroke-width:3px
    style E2 fill:none,stroke:#666,stroke-width:3px
    style F fill:none,stroke:#666,stroke-width:3px
    style G fill:none,stroke:#666,stroke-width:3px

Pub/Sub Subscription Structure

Messages consumed from configured subscription. Failed messages published to error topic.

Configuration

Application Settings

Key settings:

Spring Property Environment var Default value

erp-to-czz.project-id

ERP_TO_CZZ_PROJECT_ID

erp-to-czz

erp-to-czz.main-topic.topic-id

ERP_TO_CZZ_MAIN_TOPIC_TOPIC_ID

erp-to-czz-input

erp-to-czz.main-topic.subscription-id

ERP_TO_CZZ_MAIN_TOPIC_SUBSCRIPTION_ID

erp-to-czz-input-consumer

erp-to-czz.main-topic.credentials-file

ERP_TO_CZZ_MAIN_TOPIC_CREDENTIALS_FILE

classpath:credentials/ep-to-czz-subcription-key.json

erp-to-czz.error-topic.topic-id

ERP_TO_CZZ_ERROR_TOPIC_TOPIC_ID

erp-to-czz-input-errors

erp-to-czz.error-topic.subscription-id

ERP_TO_CZZ_ERROR_TOPIC_SUBSCRIPTION_ID

erp-to-czz-input-error-consumer

erp-to-czz.error-topic.credentials-file

ERP_TO_CZZ_ERROR_TOPIC_CREDENTIALS_FILE

classpath:credentials/erp-to-czz-publish-key.json

Important Notes

  • Separate credentials allow fine-grained IAM permissions (subscribe vs publish)

  • Parallel pull count controls concurrent message processing

  • Credentials file paths must include a resource prefix (file: or classpath:)

  • The error topic should be different from the main topic, otherwise failed messages are consumed again

Database Configuration

Database connection is configured via Spring Boot datasource properties in application.yml or profile-specific files (e.g., application-dev.yml).

Credentials Configuration

Requires service account JSON keys in credentials/ classpath directory for Pub/Sub authentication.

Docker container execution

docker run -it --rm --name processor-pubsub $PWD:/apps/bin/lib --env-file processor-dev.env --pull=always -p 8080:8080 europe-west3-docker.pkg.dev/czz-devops/comerzzia/erp-to-czz/processor-pubsub-application:1.0
processor-dev.env file example for environment variables
ERP_TO_CZZ_PROJECT_ID=erp-to-czz-test

ERP_TO_CZZ_MAIN_TOPIC_TOPIC_ID=erp-to-czz-input
ERP_TO_CZZ_MAIN_TOPIC_SUBSCRIPTION_ID=erp-to-czz-input-consumer-test
ERP_TO_CZZ_MAIN_TOPIC_CREDENTIALS_FILE=classpath:credentials/erp-to-czz-input-consumer-test-key.json

ERP_TO_CZZ_ERROR_TOPIC_TOPIC_ID=erp-to-czz-input-errors
ERP_TO_CZZ_ERROR_TOPIC_SUBSCRIPTION_ID=erp-to-czz-input-publish-test
ERP_TO_CZZ_ERROR_TOPIC_CREDENTIALS_FILE=classpath:credentials/erp-to-czz-input-publish-test-key.json

uidActividad=MYACTIVITY
uidInstancia=MYINSTANCE

SPRING_PROFILES_ACTIVE=jsonlogs

COMERZZIA_DB_CLASS=com.mysql.jdbc.Driver
COMERZZIA_DB_DRIVER=mysql
COMERZZIA_DB_USER=comerzzia
COMERZZIA_DB_PASS=comerzzia
COMERZZIA_DB_URL=jdbc:mysql://127.0.0.1:3306/comerzzia