Google Cloud Pipeline with Dart and FaaS

post-thumb


In today’s data-driven world, building efficient pipelines for processing and storing data is crucial.

In this article, which is Part 1 of a series, we’ll explore how to leverage Dart, a powerful and expressive programming language, along with the Function as a Service (FaaS) framework, to build a scalable and portable Google Cloud pipeline.

1 - Introduction

We’ll demonstrate how to leverage Dart to build a scalable and portable Google Cloud pipeline, through a demo application that fetches NASDAQ stock data from the IEXCloud API, stores it in Google Cloud Storage. Additionally, we’ll cover the requirements for developers who want to try the code and provide their own API credentials.

2 - Prerequisites

Before we get started, ensure that you have a Google Cloud Platform account set up and have the Dart SDK installed on your local machine. Familiarity with Dart basics will also be helpful.

Requirements:

To get started, if you want to try the application, you will need to follow these steps:

Obtain IEXCloud API Credentials:

  1. Sign up for an account on the IEXCloud platform (https://iexcloud.io/) and obtain your API token.

Replace the placeholder in the assets/sa-key-iexcloud file with your actual IEXCloud API token. This file is used to store the API key securely. Provide Service Account Credentials:

  1. You must also provide your own service account credentials for authentication with Google Cloud Platform services.

Replace the assets/sa-key.json file with the service account credentials JSON file corresponding to your Google Cloud Platform project. This file contains the necessary credentials for authentication.

3- Creating a Cloud Run Application with FaaS:

We’ll utilize the FaaS framework, which is an open-source project from the Google Dart and Cloud Functions teams. This framework allows us to write portable Dart functions easily and deploy them on various cloud platforms, including Google Cloud Run.

Once our development environment is ready, we can create a Cloud Run application using the FaaS framework.

Cloud Run is a fully managed serverless platform that enables us to deploy containerized applications.

functions.dart
 1@CloudFunction()
 2Future<void> function(StockRequest request, RequestContext context) async {
 3  try {
 4    List<StockConfig> stocks = await service.loadStockConfiguration();
 5
 6    List<Future<void>> apiCalls = stocks.map((element) async {
 7      StockResponse response = await service.fetchStock(element.symbol);
 8      await service.writeToStorage(
 9          element.symbol, jsonEncode(response.toJson()));
10      context.logger.info(
11          'Context - symbol : ${element.symbol} - Response: ${response.toJson()}');
12    }).toList();
13
14    await Future.wait(apiCalls);
15  } catch (error, stackTrace) {
16    context.logger.error('An error occurred: $error\n$stackTrace');
17    // Handle or report the error accordingly.
18  }
19}

4- Fetching Data from IEXCloud API:

Our demo application retrieves stock data from the IEXCloud API. Using the FaaS framework, we can write a Dart function that makes HTTP requests to the API, retrieves the desired stock information, and parses the response. With Dart’s built-in libraries and powerful ecosystem, handling JSON parsing and error handling becomes straightforward.

service.dart
 1Future<StockResponse> fetchStock(String symbol) async {
 2  final apiKey = utils.getApiKey();
 3  final api = 'https://cloud.iexapis.com';
 4
 5  final urls = [
 6    '$api/stable/stock/$symbol/quote/latestPrice?token=$apiKey',
 7    '$api/stable/data-points/$symbol/LAST-DIVIDEND-AMOUNT?token=$apiKey',
 8    '$api/stable/data-points/$symbol/COMPANYNAME?token=$apiKey',
 9    '$api/stable/data-points/$symbol/SECURITYNAME?token=$apiKey',
10    '$api/stable/data-points/$symbol/EXCHANGE?token=$apiKey',
11    '$api/stable/data-points/$symbol/INDUSTRY?token=$apiKey',
12    '$api/stable/data-points/$symbol/SECTOR?token=$apiKey',
13    '$api/stable/data-points/$symbol/ISSUETYPE?token=$apiKey',
14  ];
15
16  final client = http.Client();
17  final responses = await Future.wait(urls.map((url) => client.get(Uri.parse(url))));
18
19  client.close();
20
21  return StockResponse.fromJson({
22    'symbol': symbol,
23    'lastPrice': responses[0].body.cleanValue(),
24    'dividend': responses[1].body.cleanValue(),
25    'companyName': responses[2].body.cleanValue(),
26    'securityName': responses[4].body.cleanValue(),
27    'industry': responses[5].body.cleanValue(),
28    'sector': responses[6].body.cleanValue(),
29    'issueType': responses[7].body.cleanValue(),
30  });
31}

5- Storing Data in Cloud Storage:

After fetching the stock data, we need a reliable storage solution. Google Cloud Storage offers durability, scalability, and easy access to our data. We can use the gcloud package, which provides an idiomatic Dart interface to Google Cloud Platform services, including Cloud Storage. Our Dart function will upload the fetched data to a designated bucket in Cloud Storage, allowing for easy retrieval and further processing.

service.dart
 1Future<void> writeToStorage(String symbol, String bodyAsString) async {
 2  final accountCredentials = auth.ServiceAccountCredentials.fromJson(utils.getSAKey());
 3
 4  final client = await auth.clientViaServiceAccount(accountCredentials, Storage.SCOPES);
 5  final storage = Storage(client, 'PROJECT-NAME');
 6  final bucket = storage.bucket('YOUR-BUCKET'');
 7
 8  final now = DateTime.now();
 9  final formattedDate = DateFormat('yyyy-MM-dd-hhmmss').format(now);
10
11  final filePath = 'stocks/${now.year}/${now.month}/${now.day}/$symbol/$symbol-$formattedDate.json';
12  final fileBytes = utf8.encode(bodyAsString);
13
14  await bucket.writeBytes(filePath, fileBytes);
15
16  client.close();
17}

6- Configuration Management with Cloud Datastore:

To manage the configuration of our pipeline, we’ll utilize Google Cloud Datastore. Cloud Datastore is a fully managed NoSQL database service, offering a robust and scalable solution for storing structured data. The gcloud package allows us to interact with Cloud Datastore using Dart. We can load the pipeline configuration from Cloud Datastore, such as the stocks to fetch or any other settings, and adjust our pipeline accordingly.

7- Building the Pipeline:

With our Dart functions in place, Cloud Storage for data storage, and Cloud Datastore for configuration management, we can now build our Google Cloud pipeline. The pipeline workflow starts by fetching stock data from the IEXCloud API using the FaaS function. The fetched data is then stored in Cloud Storage for future processing or analysis. The configuration data from Cloud Datastore ensures that the pipeline is flexible and adaptable. By combining these services, we have created a scalable and portable pipeline to handle our stock data processing needs.

Conclusion:

In this article, we explored how to leverage Dart and the FaaS framework to build a Google Cloud pipeline.

Code Source

Github Repository