14th December 2020

Introducing Payments IQ

Continuous integration blogpost

Adrian Cuschieri

Head of Site Operations

You know that a payments company moves money from one place to another, enabling customers to seamlessly make and manage payments. But you probably don’t know the technical in’s and out’s of how it's done — and it’s a lot more interesting than you might think!

We’re proud to have a highly skilled and seasoned technical team here at Ixaris who has pioneered building robust, innovative payment technologies for almost 20 years. Their pooled payments knowledge is extensive, and they are ready to share it with the world.

To help them do just that, we are launching a new Payments IQ series on our blog to share payments insights penned by our talented team. In the first instalment of our Payments IQ series, our Head of Site Operations Adrian Cuschieri explains how his team revamped our continuous integration practices to raise the bar for building better code quickly — without compromising quality.

Scaling up for success

Code is the foundation of our products, and Ixaris’ engineering team’s main objective is to produce top-quality code while responding to the fintech industry’s accelerating pace of development.

Imagine a factory where different teams build different parts of an engine at the same time. Those teams would need to come together frequently to exchange parts, test that each of their pieces fit together and works as intended, etc. Software development teams work in much the same way.

At Ixaris, multiple development teams work on different projects using the same code repository. It is extremely important to combine their work frequently to reduce integration problems and allow each team to develop cohesive software faster. To do that, development teams like ours use a software development practice called continuous integration (CI).

In continuous integration, after a developer commits code changes, the software is immediately built and tested using specific tools. However, as Ixaris grew, so did our code size, the size of our test suite, and the frequency of our code commits. This presented real scalability challenges, which is where our Site Operations team enters the picture. Here’s how we revamped Ixaris’ continuous integration process to deliver ultimate scalability that supports our software engineers’ ability to quickly deliver features — without compromising quality.

An open-source solution

Historically, we used Jenkins as our primary continuous integration tool, which orchestrates a chain of automated actions. However, scalability became a challenge with Jenkins. Jenkins has a master node and several worker nodes that jobs can run on. Each of its worker nodes was a virtual machine, which had to be provisioned in advance by one of our System Administrators. The number of worker nodes was fixed, so jobs would queue if all the available workers were busy, eventually delaying the feedback that teams needed after committing a new change in code. We needed a flexible and efficient continuous integration platform that would be:

  • Automatically configurable, to handle upgrades easier and avoid manual work when new instances were needed
  • Easy to replicate, and extendable to run on third party cloud providers
  • Able to consolidate configuration across all worker nodes using templates
  • Elastic and efficient in resource usages, finding a balance between having various idle machines (waste) and few loaded machines with a long queue to process (underpower)
  • A single source of version-controlled truth for jobs and their steps

Our solution was to leverage several open-source tools (such as Kubernetes, Helm, Jenkins, SonarQube, Selenium Grid and Prometheus) — most of which were already part of Ixaris’ infrastructure. At Ixaris, we are advocates for open source because it allows us to focus on building better software. In fact, we believe so strongly in the value of open source tools, that we are now contributing to them by sharing some of our code publicly!

Continuously improving continuous integration

We started revamping our continuous integration platform by containerising all our infrastructure using a tool called Docker. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: the code, runtime, system tools, system libraries and settings. Rather than having a virtual machine for each worker node, we created a few powerful virtual machines to host all the containers on top of them, making our use of resources much more efficient.

To automate the deployment, scaling and management of these containerised applications, we chose to use a tool called Kubernetes. Kubernetes allows us to take advantage of rollouts and rollbacks, service discovery and load balancing, service topology, storage orchestration, secret and configuration management, batch execution, self-healing and horizontal scaling. To set up the cluster, we used Kubespray: a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks.

After we set up our Kubernetes cluster, we configured and deployed a newer version of the Jenkins application on the Kubernetes Engine using a packaged manager called Helm, which allowed us to deploy from the Charts repository.

Our upgraded Jenkins brought with it the functionality to create pipelines as code, which means they can be easily versioned and split in common steps to be shared between teams. We can re-use these steps in different pipelines to enhance our scalability.

We set up several Jenkins jobs using pipelines to automatically trigger builds whenever a new change is committed on our code repository. Each build performs a static analysis using a tool called SonarQube, and tests new changes from low-end tests (like unit testing) to end-to-end tests using technologies like Junit and Selenium Grid.

To scale up Jenkins while running it on Kubernetes, we used a Jenkins plugin that runs dynamic workers in our Kubernetes Cluster. This plugin creates a Kubernetes Pod for each worker, defined by a specific template that specifies which docker images to use, the name of the pod, the volumes to mount and other configurations related to the worker node. Once the build is finished, the pod is deleted, and any resources that were previously used are released. In other words, this plugin allowed us to achieve elasticity to scale on a needs-basis.

Kudos for Kubernetes

Kubernetes makes it much easier to scale up things since it was designed on the same principles that allow Google to run billions of containers a week. Kubernetes can also run anywhere, so it provides the freedom to take advantage of on-premise, hybrid or public cloud infrastructure.

Kubernetes is also highly available and resilient and can function despite failures of system components. For example, our newly created cluster is made up of 3 master nodes and 6 worker nodes. If any worker node is down (either for maintenance or due to a failure), Kubernetes would automatically move all the applications running on that worker node to another one.

Days to minutes

At Ixaris, maintaining the highest code quality is really important to us. Our role is to advocate DevOps practices and create tools that make the lives of our Software Engineers easier, so they can focus on creating and enhancing our products.

Using our revamped CI platform, we can spin up a new test environment — including the ~40 microservices that make up the Ixaris Payments platform — and all the required infrastructure needed to run these services. And the coolest part? This process used to take days. Today, we can do it all in less than 10 minutes.

The result? Our software engineers can quickly deliver features without compromising quality. As it should be.

About the author

Adrian Cuschieri is Ixaris’ Head of Site Operations.

At Ixaris, we are advocates for open source because it allows us to focus on building better software. In fact, we believe so strongly in the value of open source tools, that we are now contributing to them by sharing some of our code publicly.

Adrian Cuschieri, Head of Site Operations

Copy link to clipboard