1.0.0.RC1
Copyright © 2017-2018 Pivotal Software, Inc.
Table of Contents
The Spring Cloud GCP project aims at making the Spring Framework a first-class citizen of Google Cloud Platform (GCP).
Currently, Spring Cloud GCP lets you leverage the power and simplicity of the Spring framework to:
The Spring Cloud GCP Bill of Materials (BOM) contains the versions of all the dependencies it uses.
If you’re a Maven user, adding the following to your pom.xml file will allow you to not specify any Spring Cloud GCP dependency versions. Instead, the version of the BOM you’re using determines the versions of the used dependencies.
<dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-dependencies</artifactId> <version>1.0.0.RC1</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement>
In the following sections, it will be assumed you are using the Spring Cloud GCP BOM and the dependency snippets will not contain versions.
Gradle users can achieve the same kind of BOM experience using Spring’s dependency-management-plugin Gradle plugin. For simplicity, the Gradle dependency snippets in the remainder of this document will also omit their versions.
There are many available resources to get you up to speed with our libraries as quickly as possible.
There are three entries in Spring Initializr for Spring Cloud GCP:
The GCP Support entry contains auto-configuration support for every Spring Cloud GCP integration. Most of the autoconfiguration code is only enabled if other dependencies are added to the classpath.
Spring Cloud GCP Starter | Required dependencies |
---|---|
Logging | org.springframework.cloud:spring-cloud-gcp-starter-logging |
SQL - MySql | org.springframework.cloud:spring-cloud-gcp-starter-sql-mysql |
SQL - PostgreSQL | org.springframework.cloud:spring-cloud-gcp-starter-sql-postgres |
Trace | org.springframework.cloud:spring-cloud-gcp-starter-trace |
The GCP Messaging entry adds the GCP Support entry and all the required dependencies so that the Google Cloud Pub/Sub integrations work out of the box.
The GCP Storage entry adds the GCP Support entry and all the required dependencies so that the Google Cloud Storage integrations work out of the box.
There are code samples available that demonstrate the usage of all our integrations.
The Vision API sample shows how to use spring-cloud-gcp-starter
for authentication.
In a code challenge, you perform a task step by step, using one integration. There are a number of challenges available in the Google Developers Codelabs page.
A Spring Getting Started guide on messaging with Spring Integration Channel Adapters for Google Cloud Pub/Sub is available from Spring Guides.
At the center of every Spring Cloud GCP module are the concepts of GcpProjectIdProvider
and CredentialsProvider
.
Spring Cloud GCP provides a Spring Boot starter to auto-configure the core components.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter' }
GcpProjectIdProvider
is a functional interface that returns a GCP project ID string.
public interface GcpProjectIdProvider { String getProjectId(); }
The Spring Cloud GCP starter auto-configures a GcpProjectIdProvider
.
If a spring.cloud.gcp.project-id
property is specified, the provided GcpProjectIdProvider
returns that property value.
spring.cloud.gcp.project-id=my-gcp-project-id
Otherwise, the project ID is discovered based on a set of rules:
GOOGLE_CLOUD_PROJECT
environment variableGOOGLE_APPLICATION_CREDENTIALS
environment variableCredentialsProvider
is a functional interface that returns the credentials to authenticate and
authorize calls to Google Cloud Client Libraries.
public interface CredentialsProvider { Credentials getCredentials() throws IOException; }
The Spring Cloud GCP starter auto-configures a CredentialsProvider
.
It uses the spring.cloud.gcp.credentials.location
property to locate the OAuth2 private key of a Google service account.
Keep in mind this property is a Spring Resource, so the credentials file can be obtained from a number of different locations such as the file system, classpath, URL, etc.
The next example specifies the credentials location property in the file system.
spring.cloud.gcp.credentials.location=file:/usr/local/key.json
If that property isn’t specified, the starter tries to discover credentials from a number of places:
GOOGLE_APPLICATION_CREDENTIALS
environment variablegcloud auth application-default login
commandIf your app is running on Google App Engine or Google Compute Engine, in most cases, you should omit
the spring.cloud.gcp.credentials.location
property and, instead, let the Spring Cloud GCP
Starter get the correct credentials for those environments.
On App Engine Standard, the
App Identity service account credentials
are used, on App Engine Flexible, the
Flexible service account credential
are used and on Google Compute Engine, the
Compute Engine Default Service Account
is used.
By default, the credentials provided by the Spring Cloud GCP Starter contain scopes for every service supported by Spring Cloud GCP.
Service | Scope |
Pub/Sub | |
Storage (Read Only) | |
Storage (Write/Write) | |
Runtime Config | |
Trace (Append) | |
Cloud Platform |
The Spring Cloud GCP starter allows you to configure a custom scope list for the provided
credentials.
To do that, specify a comma-delimited list of Google OAuth2 scopes
in the spring.cloud.gcp.credentials.scopes
property.
spring.cloud.gcp.credentials.scopes
is a comma-delimited list of
Google OAuth2 scopes for Google
Cloud Platform services that the credentials returned by the provided CredentialsProvider
support.
spring.cloud.gcp.credentials.scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/sqlservice.admin
You can also use DEFAULT_SCOPES
placeholder as a scope to represent the starters default scopes,
and append the additional scopes you need to add.
spring.cloud.gcp.credentials.scopes=DEFAULT_SCOPES,https://www.googleapis.com/auth/cloud-vision
This starter is available from Spring Initializr through the GCP Support
entry.
Spring Cloud GCP provides an abstraction layer to publish to and subscribe from Google Cloud Pub/Sub topics and to create, list or delete Google Cloud Pub/Sub topics and subscriptions.
A Spring Boot starter is provided to auto-configure the various required Pub/Sub components.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter-pubsub</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-pubsub' }
This starter is also available from Spring Initializr through the GCP Messaging
entry.
A sample application is available.
PubSubOperations
is an abstraction that allows Spring users to use Google Cloud Pub/Sub without
depending on any Google Cloud Pub/Sub API semantics.
It provides the common set of operations needed to interact with Google Cloud Pub/Sub.
PubSubTemplate
is the default implementation of PubSubOperations
and it uses the
Google Cloud Java Client for Pub/Sub
to interact with Google Cloud Pub/Sub.
PubSubTemplate
depends on a PublisherFactory
and a SubscriberFactory
.
The PublisherFactory
provides a Google Cloud Java Client for Pub/Sub Publisher
.
The SubscriberFactory
provides the Subscriber
for asynchronous message pulling, as well as a SubscriberStub
for synchronous pulling and an Acknowledger
, for the cases where messages are automatically acknowledged.
The Spring Boot starter for GCP Pub/Sub auto-configures a PublisherFactory
and SubscriberFactory
with default settings and uses the GcpProjectIdProvider
and CredentialsProvider
auto-configured by the Spring Boot GCP starter.
The PublisherFactory
implementation provided by Spring Cloud GCP Pub/Sub, DefaultPublisherFactory
, caches Publisher
instances by topic name, in order to optimize resource utilization.
PubSubTemplate
provides asynchronous methods to publish messages to a Google Cloud Pub/Sub topic.
The publish()
method takes in a topic name to post the message to, a payload of a generic type and, optionally, a map with the message headers.
Here is an example of how to publish a message to a Google Cloud Pub/Sub topic:
public void publishMessage() { this.pubSubTemplate.publish("topic", "your message payload", ImmutableMap.of("key1", "val1")); }
By default, the SimplePubSubMessageConverter
is used to convert payloads of type byte[]
, ByteString
, ByteBuffer
, and String
to Pub/Sub messages.
For serialization and deserialization of POJOs using Jackson JSON, configure the PubSubTemplate
to use the JacksonPubSubMessageConverter
by calling the setMessageConverter()
method.
Alternatively, if you’re using the starter and have an instance of the Jackson ObjectMapper
in the application context, the JacksonPubSubMessageConverter
will be automatically configured for you.
Google Cloud Pub/Sub allows many subscriptions to be associated to the same topic.
PubSubTemplate
allows you to subscribe to subscriptions via the subscribe()
method.
It relies on a SubscriberFactory
object, whose only task is to generate Google Cloud Pub/Sub
Subscriber
objects.
When subscribing to a subscription, messages will be pulled from Google Cloud Pub/Sub
asynchronously, on a certain interval.
The Spring Boot starter for Google Cloud Pub/Sub auto-configures a SubscriberFactory
.
Google Cloud Pub/Sub supports synchronous pulling of messages from a subscription. This is different from subscribing to a subscription, in the sense that subscribing is an asynchronous task which polls the subscription on a set interval.
The pullNext()
method allows for a single message to be pulled and automatically acknowledged from a subscription.
The pull()
method pulls a number of messages from a subscription, allowing for the retry settings to be configured.
Any messages received by pull()
are not automatically acknowledged.
Instead, since they are of the kind AcknowledgeablePubsubMessage
, you can acknowledge them by calling the ack()
method, or negatively acknowledge them by calling the nack()
method.
The pullAndAck()
method does the same as the pull()
method and, additionally, acknowledges all received messages.
In order to acknowledge or negatively acknowledge the messages received from pull()
, you can use the acknowledger provided by PubSubTemplate.getAcknowledger()
.
The provided acknowledger allows for acknowledging or negatively acknowledging a set of acknowledge IDs, pertaining to a subscription.
The subscription name passed to the acknowledger must have the following format: project/[GCP_PROJECT_ID]/subscriptions/[SUBSCRIPTION_NAME]
.
PubSubTemplate
uses a special subscriber generated by its SubscriberFactory
to synchronously pull messages.
If the message payload contains a serialized POJO, it can be retrieved as a Class
compatible with that serialized payload:
this.pubSubTemplate.getMessageConverter().fromMessage(message, MyPojo.class);
PubSubAdmin
is the abstraction provided by Spring Cloud GCP to manage Google Cloud Pub/Sub
resources.
It allows for the creation, deletion and listing of topics and subscriptions.
PubSubAdmin
depends on GcpProjectIdProvider
and either a CredentialsProvider
or a
TopicAdminClient
and a SubscriptionAdminClient
.
If given a CredentialsProvider
, it creates a TopicAdminClient
and a SubscriptionAdminClient
with the Google Cloud Java Library for Pub/Sub default settings.
The Spring Boot starter for GCP Pub/Sub auto-configures a PubSubAdmin
object using the
GcpProjectIdProvider
and the CredentialsProvider
auto-configured by the Spring Boot GCP Core
starter.
PubSubAdmin
implements a method to create topics:
public Topic createTopic(String topicName)
Here is an example of how to create a Google Cloud Pub/Sub topic:
public void newTopic() { pubSubAdmin.createTopic("topicName"); }
PubSubAdmin
implements a method to delete topics:
public void deleteTopic(String topicName)
Here is an example of how to delete a Google Cloud Pub/Sub topic:
public void deleteTopic() { pubSubAdmin.deleteTopic("topicName"); }
PubSubAdmin
implements a method to list topics:
public List<Topic> listTopics
Here is an example of how to list every Google Cloud Pub/Sub topic name in a project:
public List<String> listTopics() { return pubSubAdmin .listTopics() .stream() .map(Topic::getNameAsTopicName) .map(TopicName::getTopic) .collect(Collectors.toList()); }
PubSubAdmin
implements a method to create subscriptions to existing topics:
public Subscription createSubscription(String subscriptionName, String topicName, Integer ackDeadline, String pushEndpoint)
Here is an example of how to create a Google Cloud Pub/Sub subscription:
public void newSubscription() { pubSubAdmin.createSubscription("subscriptionName", "topicName", 10, “http://my.endpoint/push”); }
Alternative methods with default settings are provided for ease of use.
The default value for ackDeadline
is 10 seconds.
If pushEndpoint
isn’t specified, the subscription uses message pulling, instead.
public Subscription createSubscription(String subscriptionName, String topicName)
public Subscription createSubscription(String subscriptionName, String topicName, Integer ackDeadline)
public Subscription createSubscription(String subscriptionName, String topicName, String pushEndpoint)
PubSubAdmin
implements a method to delete subscriptions:
public void deleteSubscription(String subscriptionName)
Here is an example of how to delete a Google Cloud Pub/Sub subscription:
public void deleteSubscription() { pubSubAdmin.deleteSubscription("subscriptionName"); }
PubSubAdmin
implements a method to list subscriptions:
public List<Subscription> listSubscriptions()
Here is an example of how to list every subscription name in a project:
public List<String> listSubscriptions() { return pubSubAdmin .listSubscriptions() .stream() .map(Subscription::getNameAsSubscriptionName) .map(SubscriptionName::getSubscription) .collect(Collectors.toList()); }
The Spring Boot starter for Google Cloud Pub/Sub provides the following configuration options:
Name | Description | Required | Default value |
| Enables or disables Pub/Sub auto-configuration | No |
|
| Number of threads used by | No | 4 |
| Number of threads used by | No | 4 |
| GCP project ID where the Google Cloud Pub/Sub API is hosted, if different from the one in the Spring Cloud GCP Core Module | No | |
| OAuth2 credentials for authenticating with the Google Cloud Pub/Sub API, if different from the ones in the Spring Cloud GCP Core Module | No | |
| OAuth2 scope for Spring Cloud GCP Pub/Sub credentials | No | |
| The number of pull workers | No | The available number of processors |
| The maximum period a message ack deadline will be extended, in seconds | No | 0 |
| The endpoint for synchronous pulling messages | No | pubsub.googleapis.com:443 |
| TotalTimeout has ultimate control over how long the logic should keep trying the remote call until it gives up completely. The higher the total timeout, the more retries can be attempted. | No | 0 |
| InitialRetryDelay controls the delay before the first retry. Subsequent retries will use this value adjusted according to the RetryDelayMultiplier. | No | 0 |
| RetryDelayMultiplier controls the change in retry delay. The retry delay of the previous call is multiplied by the RetryDelayMultiplier to calculate the retry delay for the next call. | No | 1 |
| MaxRetryDelay puts a limit on the value of the retry delay, so that the RetryDelayMultiplier can’t increase the retry delay higher than this amount. | No | 0 |
| MaxAttempts defines the maximum number of attempts to perform. If this value is greater than 0, and the number of attempts reaches this limit, the logic will give up retrying even if the total retry time is still lower than TotalTimeout. | No | 0 |
| Jitter determines if the delay time should be randomized. | No | true |
| InitialRpcTimeout controls the timeout for the initial RPC. Subsequent calls will use this value adjusted according to the RpcTimeoutMultiplier. | No | 0 |
| RpcTimeoutMultiplier controls the change in RPC timeout. The timeout of the previous call is multiplied by the RpcTimeoutMultiplier to calculate the timeout for the next call. | No | 1 |
| MaxRpcTimeout puts a limit on the value of the RPC timeout, so that the RpcTimeoutMultiplier can’t increase the RPC timeout higher than this amount. | No | 0 |
| Maximum number of outstanding elements to keep in memory before enforcing flow control. | No | unlimited |
| Maximum number of outstanding bytes to keep in memory before enforcing flow control. | No | unlimited |
| The behavior when the specified limits are exceeded. | No | Block |
| The element count threshold to use for batching. | No | unset (threshold does not apply) |
| The request byte threshold to use for batching. | No | unset (threshold does not apply) |
| The delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. | No | unset (threshold does not apply) |
| Enables batching. | No | false |
Spring Resources are an abstraction for a number of low-level resources, such as file system files, classpath files, servlet context-relative files, etc. Spring Cloud GCP adds a new resource type: a Google Cloud Storage (GCS) object.
A Spring Boot starter is provided to auto-configure the various Storage components.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter-storage</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-storage' }
This starter is also available from Spring Initializr through the GCP Storage
entry.
A sample application is available.
The Spring Resource Abstraction for Google Cloud Storage allows GCS objects to be accessed by their
GCS URL using the @Value
annotation
@Value("gs://[YOUR_GCS_BUCKET]/[GCS_FILE_NAME]") private Resource gcsResource;
or the Spring application context
SpringApplication.run(...).getResource("gs://[YOUR_GCS_BUCKET]/[GCS_FILE_NAME]");
This creates a Resource
object that can be used to read the object, among
other possible operations.
It is also possible to write to a Resource
, although a WriteableResource
is required.
@Value("gs://[YOUR_GCS_BUCKET]/[GCS_FILE_NAME]") private Resource gcsResource; ... try (OutputStream os = ((WritableResource) gcsResource).getOutputStream()) { os.write("foo".getBytes()); }
If the resource path refers to an object on Google Cloud Storage (as opposed to a bucket), then the resource
can be cast as a GoogleStorageResourceObject
and the getGoogleStorageObject
method can be called
to obtain a Blob
.
This type represents a GCS file, which has associated metadata, such as content-type, that can be set.
The createSignedUrl
method can also be used to obtain signed URLs for GCS objects.
However, creating signed URLs requires that the resource was created using service account credentials.
The Spring Boot Starter for Google Cloud Storage auto-configures the Storage
bean required by the
spring-cloud-gcp-storage
module, based on the CredentialsProvider
provided by the Spring Boot
GCP starter.
The Spring Boot Starter for Google Cloud Storage provides the following configuration options:
Name | Description | Required | Default value |
| Creates files and buckets on Google Cloud Storage when writes are made to non-existent files | No |
|
| OAuth2 credentials for authenticating with the Google Cloud Storage API, if different from the ones in the Spring Cloud GCP Core Module | No | |
| OAuth2 scope for Spring Cloud GCP Storage credentials | No |
Spring Cloud GCP adds integrations with Spring JDBC so you can run your MySQL or PostgreSQL databases in Google Cloud SQL using Spring JDBC, or other libraries that depend on it like Spring Data JPA.
The Cloud SQL support is provided by Spring Cloud GCP in the form of two Spring Boot starters, one for MySQL and another one for PostgreSQL. The role of the starters is to read configuration from properties and assume default settings so that user experience connecting to MySQL and PostgreSQL is as simple as possible.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter-sql-mysql</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter-sql-postgresql</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-sql-mysql' compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-sql-postgresql' }
In order to use the Spring Boot Starters for Google Cloud SQL, the Google Cloud SQL API must be enabled in your GCP project.
To do that, go to the API library page of the Google Cloud Console, search for "Cloud SQL API", click the first result and enable the API.
![]() | Note |
---|---|
There are several similar "Cloud SQL" results. You must access the "Google Cloud SQL API" one and enable the API from there. |
Available sample applications:
The Spring Boot Starters for Google Cloud SQL provide an auto-configured
DataSource
object.
Coupled with Spring JDBC, it provides a
JdbcTemplate
object bean that allows for operations such as querying and modifying a database.
public List<Map<String, Object>> listUsers() { return jdbcTemplate.queryForList("SELECT * FROM user;"); }
You can rely on
Spring
Boot data source auto-configuration to configure a DataSource
bean.
In other words, properties like the SQL username, spring.datasource.username
, and password,
spring.datasource.password
can be used.
There is also some configuration specific to Google Cloud SQL:
Property name | Description | Default value | Unused if specified property(ies) |
| Enables or disables Cloud SQL auto configuration |
| |
| Name of the database to connect to. |
| |
| A string containing a Google Cloud SQL
instance’s project ID, region and name, each separated by a colon. For example,
|
| |
| File system path to the Google OAuth2 credentials private key file. Used to authenticate and authorize new connections to a Google Cloud SQL instance. | Default credentials provided by the Spring GCP Boot starter |
Based on the previous properties, the Spring Boot starter for Google Cloud SQL creates a
CloudSqlJdbcInfoProvider
object which is used to obtain an instance’s JDBC URL and driver class
name.
If you provide your own CloudSqlJdbcInfoProvider
bean, it is used instead and the properties
related to building the JDBC URL or driver class are ignored.
The DataSourceProperties
object provided by Spring Boot Autoconfigure is mutated in order to use
the JDBC URL and driver class names provided by CloudSqlJdbcInfoProvider
, unless those values were
provided in the properties.
It is in the DataSourceProperties
mutation step that the credentials factory is registered
in a system property to be SqlCredentialFactory
.
DataSource
creation is delegated to
Spring Boot.
You can select the type of connection pool (e.g., Tomcat, HikariCP, etc.) by
adding
their dependency to the classpath.
Using the created DataSource
in conjunction with Spring JDBC provides you with a fully configured
and operational JdbcTemplate
object that you can use to interact with your SQL database.
You can connect to your database with as little as a database and instance names.
If you’re not able to connect to a database and see an endless loop of
Connecting to Cloud SQL instance […] on IP […]
, it’s likely that exceptions are being thrown
and logged at a level lower than your logger’s level. This may be the case with HikariCP, if your
logger is set to INFO or higher level.
To see what’s going on in the background, you should add a logback.xml
file to your application
resources folder, that looks like this:
<?xml version="1.0" encoding="UTF-8"?> <configuration> <include resource="org/springframework/boot/logging/logback/base.xml"/> <logger name="com.zaxxer.hikari.pool" level="DEBUG"/> </configuration>
If you see a lot of errors like this in a loop and can’t connect to your database, this is usually a symptom that something isn’t right with the permissions of your credentials or the Google Cloud SQL API is not enabled. Verify that the Google Cloud SQL API is enabled in the Cloud Console and that your service account has the necessary IAM roles.
To find out what’s causing the issue, you can enable DEBUG logging level as mentioned above.
We found this exception to be common if your Maven project’s parent is spring-boot
version
1.5.x
, or in any other circumstance that would cause the version of the
org.postgresql:postgresql
dependency to be an older one (e.g., 9.4.1212.jre7
).
To fix this, re-declare the dependency in its correct version. For example, in Maven:
<dependency> <groupId>org.postgresql</groupId> <artifactId>postgresql</artifactId> <version>42.1.1</version> </dependency>
Spring Cloud GCP provides Spring Integration adapters that allow your applications to use Enterprise Integration Patterns backed up by Google Cloud Platform services.
The channel adapters for Google Cloud Pub/Sub connect your Spring
MessageChannels
to Google Cloud Pub/Sub topics and subscriptions.
This enables messaging between different processes, applications or micro-services backed up by
Google Cloud Pub/Sub.
The Spring Integration Channel Adapters for Google Cloud Pub/Sub are included in the
spring-cloud-gcp-pubsub
module.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-pubsub</artifactId> </dependency> <dependency> <groupId>org.springframework.integration</groupId> <artifactId>spring-integration-core</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-pubsub' compile group: 'org.springframework.integration', name: 'spring-integration-core' }
A sample application is available.
PubSubInboundChannelAdapter
is the inbound channel adapter for GCP Pub/Sub that listens to a GCP
Pub/Sub subscription for new messages.
It converts new messages to an internal Spring
Message
and then sends it to the bound output channel.
Google Pub/Sub treats message payloads as byte arrays.
So, by default, the inbound channel adapter will construct the Spring Message
with byte[]
as the payload.
However, you can change the desired payload type by setting the payloadType
property of the PubSubInboundChannelAdapter
.
The PubSubInboundChannelAdapter
delegates to conversion to the desired payload type to the PubSubMessageConverter
configured in the PubSubTemplate
.
To use the inbound channel adapter, a PubSubInboundChannelAdapter
must be provided and configured
on the user application side.
@Bean public MessageChannel pubsubInputChannel() { return new PublishSubscribeChannel(); } @Bean public PubSubInboundChannelAdapter messageChannelAdapter( @Qualifier("pubsubInputChannel") MessageChannel inputChannel, SubscriberFactory subscriberFactory) { PubSubInboundChannelAdapter adapter = new PubSubInboundChannelAdapter(subscriberFactory, "subscriptionName"); adapter.setOutputChannel(inputChannel); adapter.setAckMode(AckMode.MANUAL); return adapter; }
In the example, we first specify the MessageChannel
where the adapter is going to write incoming
messages to.
The MessageChannel
implementation isn’t important here.
Depending on your use case, you might want to use a MessageChannel
other than
PublishSubscribeChannel
.
Then, we declare a PubSubInboundChannelAdapter
bean.
It requires the channel we just created and a SubscriberFactory
, which creates Subscriber
objects from the Google Cloud Java Client for Pub/Sub.
The Spring Boot starter for GCP Pub/Sub provides a configured SubscriberFactory
.
It is also possible to set the message acknowledgement mode on the adapter, which is automatic by
default.
On automatic acking, a message is acked with GCP Pub/Sub if the adapter sent it to the channel and
no exceptions were thrown.
If a RuntimeException
is thrown while the message is processed, then the message is nacked.
On manual acking, the adapter attaches an AckReplyConsumer
object to the Message
headers, which
users can extract using the GcpPubSubHeaders.ACKNOWLEDGEMENT
key and use to (n)ack a message.
@Bean @ServiceActivator(inputChannel = "pubsubInputChannel") public MessageHandler messageReceiver() { return message -> { LOGGER.info("Message arrived! Payload: " + new String((byte[]) message.getPayload())); AckReplyConsumer consumer = message.getHeaders().get(GcpPubSubHeaders.ACKNOWLEDGEMENT, AckReplyConsumer.class); consumer.ack(); }; }
PubSubMessageHandler
is the outbound channel adapter for GCP Pub/Sub that listens for new messages
on a Spring MessageChannel
.
It uses PubSubTemplate
to post them to a GCP Pub/Sub topic.
To construct a Pub/Sub representation of the message, the outbound channel adapter needs to convert the Spring Message
payload to a byte array representation expected by Pub/Sub.
It delegates this conversion to the PubSubTemplate
.
To customize the conversion, you can specify a PubSubMessageConverter
in the PubSubTemplate
that should convert the Object
payload and headers of the Spring Message
to a PubsubMessage
.
To use the outbound channel adapter, a PubSubMessageHandler
bean must be provided and configured
on the user application side.
@Bean @ServiceActivator(inputChannel = "pubsubOutputChannel") public MessageHandler messageSender(PubSubTemplate pubsubTemplate) { return new PubSubMessageHandler(pubsubTemplate, "topicName"); }
The provided PubSubTemplate
contains all the necessary configuration to publish messages to a
GCP Pub/Sub topic.
PubSubMessageHandler
publishes messages asynchronously by default.
A publish timeout can be configured for synchronous publishing. If none is provided, the adapter
waits indefinitely for a response.
It is possible to set user-defined callbacks for the publish()
call in PubSubMessageHandler
through the setPublishFutureCallback()
method.
These are useful to process the message ID, in case of success, or the error if any was thrown.
To override the default destination you can use the GcpPubSubHeaders.DESTINATION
header.
@Autowired private MessageChannel pubsubOutputChannel; public void handleMessage(Message<?> msg) throws MessagingException { final Message<?> message = MessageBuilder .withPayload(msg.getPayload()) .setHeader(GcpPubSubHeaders.TOPIC, "customTopic").build(); pubsubOutputChannel.send(message); }
It is also possible to set an SpEL expression for the topic with the setTopicExpression()
or setTopicExpressionString()
methods.
These channel adapters contain header mappers that allow you to map, or filter out, headers from Spring to Google Cloud Pub/Sub messages, and vice-versa.
By default, the inbound channel adapter maps every header on the Google Cloud Pub/Sub messages to the Spring messages produced by the adapter.
The outbound channel adapter maps every header from Spring messages into Google Cloud Pub/Sub ones, except the ones added by Spring, like headers with key "id"
, "timestamp"
and "gcp_pubsub_acknowledgement"
.
In the process, the outbound mapper also converts the value of the headers into string.
Each adapter declares a setHeaderMapper()
method to let you further customize which headers you want to map from Spring to Google Cloud Pub/Sub, and vice-versa.
For example, to filter out headers "foo"
, "bar"
and all headers starting with the prefix "prefix_", you can use setHeaderMapper()
along with the PubSubHeaderMapper
implementation provided by this module.
PubSubMessageHandler adapter = ... ... PubSubHeaderMapper headerMapper = new PubSubHeaderMapper(); headerMapper.setOutboundHeaderPatterns("!foo", "!bar", "!prefix_*", "*"); adapter.setHeaderMapper(headerMapper);
![]() | Note |
---|---|
The order in which the patterns are declared in |
In the previous example, the "*"
pattern means every header is mapped.
However, because it comes last in the list, the previous patterns take precedence.
The channel adapters for Google Cloud Storage allow you to read and write files to Google Cloud
Storage through MessageChannels
.
Spring Cloud GCP provides two inbound adapters, GcsInboundFileSynchronizingMessageSource
and
GcsStreamingMessageSource
, and one outbound adapter, GcsMessageHandler
.
The Spring Integration Channel Adapters for Google Cloud Storage are included in the
spring-cloud-gcp-storage
module.
To use the Storage portion of Spring Integration for Spring Cloud GCP, you must also provide the
spring-integration-file
dependency, since they aren’t pulled transitively.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-storage</artifactId> </dependency> <dependency> <groupId>org.springframework.integration</groupId> <artifactId>spring-integration-file</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-storage' compile group: 'org.springframework.integration', name: 'spring-integration-file' }
A sample application is available.
The Google Cloud Storage inbound channel adapter polls a Google Cloud Storage bucket for new files
and sends each of them in a Message
payload to the MessageChannel
specified in the
@InboundChannelAdapter
annotation.
The files are temporarily stored in a folder in the local file system.
Here is an example of how to configure a Google Cloud Storage inbound channel adapter.
@Bean @InboundChannelAdapter(channel = "new-file-channel", poller = @Poller(fixedDelay = "5000")) public MessageSource<File> synchronizerAdapter(Storage gcs) { GcsInboundFileSynchronizer synchronizer = new GcsInboundFileSynchronizer(gcs); synchronizer.setRemoteDirectory("your-gcs-bucket"); GcsInboundFileSynchronizingMessageSource synchAdapter = new GcsInboundFileSynchronizingMessageSource(synchronizer); synchAdapter.setLocalDirectory(new File("local-directory")); return synchAdapter; }
The inbound streaming channel adapter is similar to the normal inbound channel adapter, except it does not require files to be stored in the file system.
Here is an example of how to configure a Google Cloud Storage inbound streaming channel adapter.
@Bean @InboundChannelAdapter(channel = "streaming-channel", poller = @Poller(fixedDelay = "5000")) public MessageSource<InputStream> streamingAdapter(Storage gcs) { GcsStreamingMessageSource adapter = new GcsStreamingMessageSource(new GcsRemoteFileTemplate(new GcsSessionFactory(gcs))); adapter.setRemoteDirectory("your-gcs-bucket"); return adapter; }
The outbound channel adapter allows files to be written to Google Cloud Storage.
When it receives a Message
containing a payload of type File
, it writes that file to the Google
Cloud Storage bucket specified in the adapter.
Here is an example of how to configure a Google Cloud Storage outbound channel adapter.
@Bean @ServiceActivator(inputChannel = "writeFiles") public MessageHandler outboundChannelAdapter(Storage gcs) { GcsMessageHandler outboundChannelAdapter = new GcsMessageHandler(new GcsSessionFactory(gcs)); outboundChannelAdapter.setRemoteDirectoryExpression(new ValueExpression<>("your-gcs-bucket")); return outboundChannelAdapter; }
Spring Cloud GCP provides a Spring Cloud Stream binder to Google Cloud Pub/Sub.
The provided binder relies on the Spring Integration Channel Adapters for Google Cloud Pub/Sub.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-pubsub-stream-binder</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-pubsub-stream-binder' }
A sample application is available.
This binder binds producers to Google Cloud Pub/Sub topics and consumers to subscriptions.
![]() | Note |
---|---|
Partitioning and consumer groups are not currently supported by this binder. |
You can configure the Spring Cloud Stream Binder for Google Cloud Pub/Sub to automatically generate the underlying resources, like the Google Cloud Pub/Sub subscriptions for the consumers.
For that, you can use the spring.cloud.stream.gcp.pubsub.bindings.[CHANNEL-NAME].consumer.auto-create-resources
property, which is turned ON by default.
If automatic resource creation is turned ON and the subscription and the topic do not exist for a consumer, a subscription and a topic will be created with the same name.
For example, for the following configuration, a topic and a subscription called myConsumer
would be created.
application.properties.
spring.cloud.stream.bindings.output.destination=myConsumer spring.cloud.stream.gcp.pubsub.bindings.output.consumer.auto-create-resources=true
If you are using Pub/Sub auto-configuration from the Spring Cloud GCP Pub/Sub Starter, you should refer to the configuration section for other Pub/Sub parameters.
![]() | Note |
---|---|
To use this binder with a running emulator, configure its host and port via
|
Spring Cloud Sleuth is an instrumentation framework for Spring Boot applications. It captures trace informations and can forward traces to services like Zipkin for storage and analysis.
Google Cloud Platform provides its own managed distributed tracing service called Stackdriver Trace. Instead of running and maintaining your own Zipkin instance and storage, you can use Stackdriver Trace to store traces, view trace details, generate latency distributions graphs, and generate performance regression reports.
This Spring Cloud GCP starter can forward Spring Cloud Sleuth traces to Stackdriver Trace without an intermediary Zipkin server.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter-trace</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-trace' }
You must enable Stackdriver Trace API from the Google Cloud Console in order to capture traces. Navigate to the Stackdriver Trace API for your project and make sure it’s enabled.
A sample application is available.
![]() | Note |
---|---|
If you are already using a Zipkin server capturing trace information from multiple platform/frameworks, you also use a Stackdriver Zipkin proxy to forward those traces to Stackdriver Trace without modifying existing applications. |
Spring Cloud Sleuth uses the Brave tracer to generate traces.
This integration enables Brave to use the StackdriverTracePropagation
propagation.
A propagation is responsible for extracting trace context from an entity (e.g., an HTTP servlet request) and for injecting trace context into an entity.
A canonical example of the propagation usage is a web server that receives an HTTP request, which triggers other HTTP requests from the server before returning an HTTP response to the original caller.
In the case of StackdriverTracePropagation
, first it looks for trace context in the x-cloud-trace-context
key (e.g., an HTTP request header).
The value of the x-cloud-trace-context
key can be formatted in three different ways:
x-cloud-trace-context: TRACE_ID
x-cloud-trace-context: TRACE_ID/SPAN_ID
x-cloud-trace-context: TRACE_ID/SPAN_ID;o=TRACE_TRUE
TRACE_ID
is a 32-character hexadecimal value that encodes a 128-bit number.
SPAN_ID
is an unsigned long.
Since Stackdriver Trace doesn’t support span joins, a new span ID is always generated, regardless of the one specified in x-cloud-trace-context
.
TRACE_TRUE
can either be 0
if the entity should be untraced, or 1
if it should be traced.
However, at the moment, if TRACE_TRUE
is set to 1
, the entity isn’t necessarily traced.
Currently, to make sure a request is traced, the Sleuth property spring.sleuth.sampler.probability=1
should be used, to trace every entity.
If a x-cloud-trace-context
key isn’t found, StackdriverTracePropagation
falls back to tracing with the X-B3 headers.
Spring Boot Starter for Stackdriver Trace uses Spring Cloud Sleuth and auto-configures a StackdriverSender that sends the Sleuth’s trace information to Stackdriver Trace.
All configurations are optional:
Name | Description | Required | Default value |
| Auto-configure Spring Cloud Sleuth to send traces to Stackdriver Trace. | No |
|
| Overrides the project ID from the Spring Cloud GCP Module | No | |
| Overrides the credentials location from the Spring Cloud GCP Module | No | |
| Overrides the credentials scopes from the Spring Cloud GCP Module | No | |
| Number of threads used by the Trace executor | No | 4 |
| HTTP/2 authority the channel claims to be connecting to. | No | |
| Name of the compression to use in Trace calls | No | |
| Call deadline in milliseconds | No | |
| Maximum size for inbound messages | No | |
| Maximum size for outbound messages | No | |
| Waits for the channel to be ready in case of a transient failure | No |
|
You can use core Spring Cloud Sleuth properties to control Sleuth’s sampling rate, etc. Read Sleuth documentation for more information on Sleuth configurations.
For example, when you are testing to see the traces are going through, you can set the sampling rate to 100%.
spring.sleuth.sampler.probability=1 # Send 100% of the request traces to Stackdriver. spring.sleuth.web.skipPattern=(^cleanup.*|.+favicon.*) # Ignore some URL paths.
Spring Cloud GCP Trace does override some Sleuth configurations:
StackdriverHttpClientParser
and StackdriverHttpServerParser
by default to populate Stackdriver related fields.Integration with Stackdriver Logging is available through the Stackdriver Logging Support.
If the Trace integration is used together with the Logging one, the request logs will be associated to the corresponding traces.
The trace logs can be viewed by going to the Google Cloud Console Trace List, selecting a trace and pressing the Logs → View
link in the Details
section.
Maven coordinates, using Spring Cloud GCP BOM:
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gcp-starter-logging</artifactId> </dependency>
Gradle coordinates:
dependencies { compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-logging' }
Stackdriver Logging is the managed logging service provided by Google Cloud Platform.
This module provides support for associating a web request trace ID with the corresponding log entries.
It does so by retrieving the X-B3-TraceId
value from the Mapped Diagnostic Context (MDC), which is set by Spring Cloud Sleuth.
If Spring Cloud Sleuth isn’t used, the configured TraceIdExtractor
extracts the desired header value and sets it as the log entry’s trace ID.
This allows grouping of log messages by request, for example, in the Google Cloud Console Logs viewer.
![]() | Note |
---|---|
Due to the way logging is set up, the GCP project ID and credentials defined in |
A sample application is available.
For use in Web MVC-based applications, TraceIdLoggingWebMvcInterceptor
is provided that extracts the request trace ID from an HTTP request using a TraceIdExtractor
and stores it in a thread-local, which can then be used in a logging appender to add the trace ID metadata to log messages.
![]() | Warning |
---|---|
If Spring Cloud GCP Trace is enabled, the logging module disables itself and delegates log correlation to Spring Cloud Sleuth. |
LoggingWebMvcConfigurer
configuration class is also provided to help register the TraceIdLoggingWebMvcInterceptor
in Spring MVC applications.
Applications hosted on the Google Cloud Platform include trace IDs under the x-cloud-trace-context
header, which will be included in log entries.
However, if Sleuth is used the trace ID will be picked up from the MDC.
Currently, only Logback is supported and there are 2 possibilities to log to Stackdriver via this library with Logback: via direct API calls and through JSON-formatted console logs.
A Stackdriver appender is available using org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml
.
This appender builds a Stackdriver Logging log entry from a JUL or Logback log entry, adds a trace ID to it and sends it to Stackdriver Logging.
STACKDRIVER_LOG_NAME
and STACKDRIVER_LOG_FLUSH_LEVEL
environment variables can be used to customize the STACKDRIVER
appender.
For Logback, a org/springframework/cloud/gcp/autoconfigure/logging/logback-json-appender.xml
file is made available for import to make it easier to configure the JSON Logback appender.
Your configuration may then look something like this:
<configuration> <include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-json-appender.xml" /> <root level="INFO"> <appender-ref ref="CONSOLE_JSON" /> </root> </configuration>
If your application is running on Google Container Engine, Google Compute Engine or Google App Engine Flexible, your console logging is automatically saved to Google Stackdriver Logging.
Therefore, you can just include org/springframework/cloud/gcp/autoconfigure/logging/logback-json-appender.xml
in your logging configuration, which logs JSON entries to the console.
The trace id will be set correctly.
Your Logback configuration may then look something like this:
<configuration> <include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml" /> <root level="INFO"> <appender-ref ref="CONSOLE_JSON"/> </root> </configuration>
If you want to have more control over the log output, you can also configure the ConsoleAppender
yourself.
The following properties are available:
Property | Default Value | Description |
---|---|---|
| If not set, default value is determined in the following order:
| This is used to generate fully qualified Stackdriver Trace ID format: This format is required to correlate trace between Stackdriver Trace and Stackdriver Logging. If |
|
| Should the |
|
| Should the |
|
| Should the severity be included |
|
| Should the thread name be included |
|
| Should all MDC properties be included. The MDC properties |
|
| Should the name of the logger be included |
|
| Should the formatted log message be included. |
|
| Should the stacktrace be appended to the formatted log message. This setting is only evaluated if |
|
| Should the logging context be included |
|
| Should the log message with blank placeholders be included |
|
| Should the stacktrace be included as a own field |
This is an example of such an Logback configuration:
<configuration > <property name="projectId" value="${projectId:-${GOOGLE_CLOUD_PROJECT}}"/> <appender name="CONSOLE_JSON" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder"> <layout class="org.springframework.cloud.gcp.logging.StackdriverJsonLayout"> <projectId>${projectId}</projectId> <!--<includeTraceId>true</includeTraceId>--> <!--<includeSpanId>true</includeSpanId>--> <!--<includeLevel>true</includeLevel>--> <!--<includeThreadName>true</includeThreadName>--> <!--<includeMDC>true</includeMDC>--> <!--<includeLoggerName>true</includeLoggerName>--> <!--<includeFormattedMessage>true</includeFormattedMessage>--> <!--<includeExceptionInMessage>true</includeExceptionInMessage>--> <!--<includeContextName>true</includeContextName>--> <!--<includeMessage>false</includeMessage>--> <!--<includeException>false</includeException>--> </layout> </encoder> </appender> </configuration>
Spring Cloud GCP provides support for Cloud Foundry’s GCP Service Broker. Our Pub/Sub, Storage, Stackdriver Trace and Cloud SQL MySQL and PostgreSQL starters are Cloud Foundry aware and retrieve properties like project ID, credentials, etc., that are used in auto configuration from the Cloud Foundry environment.
In cases like Pub/Sub’s topic and subscription, or Storage’s bucket name, where those parameters are not used in auto configuration, you can fetch them using the VCAP mapping provided by Spring Boot.
For example, to retrieve the provisioned Pub/Sub topic, you can use the vcap.services.mypubsub.credentials.topic_name
property from the application environment.
![]() | Note |
---|---|
If the same service is bound to the same application more than once, the auto configuration will not be able to choose among bindings and will not be activated for that service. This includes both MySQL and PostgreSQL bindings to the same app. |
![]() | Warning |
---|---|
In order for the Cloud SQL integration to work in Cloud Foundry, auto-reconfiguration must be disabled.
You can do so using the |