Cloud Native and DevSecOps at Scale with Capgemini Agile Innovation Platform and Oracle Cloud

When a leading financial services organization was faced with a pipeline of business change requirements that their complex and aging IT infrastructure had to handle, Capgemini and Oracle Cloud offered them a paradigm shift for innovation. In addition to delivering all the new requirements, we reimagined their operating model, eradicated technical silos, and opened up a world of new possibilities to drive business value.

The Problem

The Revised Payment Services Directive (PSD2) is a directive by the European Union for electronic payments and is relevant for payment services and open banking. Our customer had to meet the directive’s deadlines for payment services and payment service providers, which meant delivering functionality and services by deadlines in March and September 2019. Furthermore, the launch of a National Credit Register the same year imposed another requirement for access to information by an external agency.

At the same time that PSD2 aimed to increase competition in the financial services market, our customer was wrestling with a legacy IT problem. Their business goals focused on agility, but their enterprise architecture was dated, with a monolithic middleware application at its heart. The multiplicity of stakeholders and the volume of demand on the legacy IT estate had led to the gradual accrual of technical debt. Performance and stability issues had started to surface with increased frequency, and the complex monolithic architecture had begun to seriously impact time to market. Underpinning the IT estate was a third-party hosting solution with long lead times and none of the advantages of hyperscale public cloud such as compelling commercial price points and consumption models, scalability and PaaS solutions.

The Solution Requirements

Capgemini had already done a successful proof of concept migration to Oracle Cloud Infrastructure for a subset of the customer’s workloads, and the customer had an existing Oracle-centric estate. These were two reasons to look at Oracle Cloud Infrastructure as a target platform, but the salient reason was the customer’s requirement to create a largely cloud-agnostic solution, and one which embraced cloud native technologies and tools.

Some of the key nonfunctional requirements for the platform were speed to market, performance and stability, scalability, security, and a modular and decoupled architecture.

Figure 1: Key Nonfunctional Requirements

Some key factors that influenced the customer to move to Oracle Cloud Infrastructure included Oracle's support of the vibrant open source, cloud native ecosystem, contributions to the Cloud Native Computing Foundation, and commitment to open source ecosystems such as Java and MySQL.

Figure 2: Oracle's Commitment to Open Source and Cloud Native Computing

The Solution

Capgemini delivered a rapidly deployed, cloud native platform with our Agile Innovation Platform. Agile Innovation Platform uses simple architectural principles and modern open source tools such as Terraform and Ansible. It enables many proven patterns for integration with legacy systems and facilitates the delivery of event-driven, microservices-based solutions. With the ability to accommodate proprietary and open source technology, our approach meant that in just a few weeks, we went from initial design to initial service development in a provisioned environment.

The following graphic provides a high-level overview of the solution.

Figure 3: High-Level Solution Overview

API First

PSD2 and the debt register demand that APIs be exposed. In fact, our solution is a platform that enables the adoption of the “API first” principle for modern service design. We chose Oracle API Management because it offers third-generation API capabilities: it’s lightweight, subscription-based, and centrally managed, and can be used to gain business insight through intelligent use of operational data.

Cluster Management

At the heart of the solution is Oracle Cloud Infrastructure Container Engine for Kubernetes, where we implemented Java Spring Boot services, supported by Oracle Cloud Infrastructure Registry. The microservices use MySQL CE for persistence and read events from the Kafka topics.

Identity and Access Management

At the top, a third-party Identity and Access Management (IDAM) solution handles client authentication and authorization. It has some out-of-the-box capabilities for regional European banking services that perform digital identity verification. It also has a plug-in for encryption of OAuth tokens. Identity and authorization for the Oracle Cloud Infrastructure stack was implemented in Oracle Identity Cloud Service.

Data

The lifeblood of this solution is the customer data itself. Accessing customer data is the point of PSD2 and the debt register. So, engineering a secure, robust, and resilient transit for on-premises data into the Kafka hub was a crucial aspect of the solution. Our customer already used Oracle Databases and Oracle GoldenGate to replicate and transform data internally.

We configured new GoldenGate Data capture processes on-premises to write encrypted trail files. The GoldenGate pump process then communicates with an Oracle Data Integration Platform Cloud Remote Agent running on an Oracle Cloud virtual machine (VM), which pulls the trail files over a private network. The Data Integration Platform Cloud agent then uses the Oracle GoldenGate for Big Data Kafka Handler as a Kafka producer that writes change data capture from the trail to the Kafka topic.

A key aspect of the solution was to ensure that Kafka didn’t become a bottleneck under load. Kafka topics are configured into partitions—in this case, 30 per topic—so that read and write operations can be parallelized. We needed to ensure that the data would be consumed sequentially, even with multiple partitions and subscribers. The GoldenGate Kafka Connect Handler can be configured to use hashing to pin changes to the same source database record into the same target Kafka partition. By using the table primary key, the GoldenGate producer process puts changes relating to the same record sequentially in the same partition in a multipartition Kafka topic. This enabled multiple consumers to read multiple partitions of a topic, with the sequencing of a record guaranteed.

CI/CD for Infrastructure

A CI/CD solution that uses Jenkins on Oracle Cloud with Container Engine for Kubernetes in a master/agent architecture supports the entire platform.

Figure 4: Infrastructure Pipelines for the Platform

This configuration uses Jenkins agents running in a dedicated Container Engine cluster to run jobs orchestrated by the Jenkins master. As with the cluster for the microservices, Container Engine automatically provisions the cluster across three availability domains. This redundancy makes the solution inherently resilient.

Although CI/CD pipelines for applications had been used for a considerable time, this was the first time that the customer had been able to deliver infrastructure by using the CI/CD paradigm. The Terraform state is stored centrally in Oracle Cloud Infrastructure Object Storage.

The Results

Because Oracle Cloud Infrastructure services can be provisioned so quickly, Capgemini took our customer from design to launch in six months. When our bootstrapping phase hit throughput issues in the microstorage layer, we swapped out the standard VM shapes for DenseIO shapes, and the bottleneck was solved. When we needed to reconfigure networks, we recoded in Terraform and pushed changes through the pipeline.

This kind of agility allowed us to try new ideas and commission proofs of concept quickly. In the old on-premises world, such agility simply wasn’t possible. But we delivered the PSD2 portal on time, went launched the debt register API a week ahead of schedule, and delivered the PSD2 APIs on time.

On the day that the first APIs launched, the platform served 250,000 API requests. Some months later, that figure is now over 300,000 API requests per day. Our original target API response time was sub 500 ms for predicted user concurrency. Performance testing yielded sub 50-ms response times. Scaling our testing to 200% of expected maximum user concurrency, we still measured response times averaging 77 ms, and this result came with modestly sized infrastructure nodes.

Closing Thoughts

Providing the customer with a scalable and future-proofed microservices platform has enabled them to break traditional technology silos. Rather than continuing with a rigid business-to-IT interface, where project teams are mobilized to tackle discrete packages of work, the enterprise can now adopt product-centric thinking, where small teams have the responsibility to deliver a particular business capability over the duration of its life.

Customer teams can now apply domain-driven design thinking to the business capabilities delivered by their monolithic middleware applications, which enables new processes and innovative ways to do business. The process of decomposing the legacy monolith into modern microservices is still in progress and will gradually replace the legacy application. Critically, time to market for new business processes is dramatically reduced, and although difficult to quantify exactly, it has exceeded the original target of a 50% improvement.

The platform is not just a route to compliance, nor simply a deconstruction of the monolith. It has become an enabler for automation, new business opportunities, and even new business models.

See how Capgemini Cloud Solutions for Oracle, Oracle Cloud Infrastructure, and Oracle Cloud Native Services can help your business, too.