Integrating with external APIs should be a simple task for any software engineer, almost all companies will rely on external parties or integrations which means that writing an integration should be a simple task.
SOAP API vs REST API
In most scenarios, the third party uses some REST API where the integration is a service that makes calls out to the API to store or retrieve data. In our use case, we were using a third-party API to store and retrieve pension details but using a rather outdated means of communication called a SOAP API.
From the first inspection of an example of both types of responses, we can see that the SOAP example has a lot more boilerplate:
<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
<soap:Header>
<MessageId>57156b82-ad22-4809-9811-c3eaf142cc12</MessageId>
</soap:Header>
<soap:Body>
<m:GetUser>
<m:UserId>123</m:UserId>
</m:GetUser>
</soap:Body>
</soap:Envelope>There’s quite a lot of heavy parsing required to have to dig through this response to find the UserId and for HTML elements we usually have to parse the entire document to find the value we’re looking for, though shortcuts can be used to just pull out the value, to make a reliable client we need to parse the whole response. When compared to an equivalent REST API that uses JSON responses we can see the difference in parsing complexity:
{
"GetUser": {
"UserId": "123"
}
}Parsing a Response
In our Node.JS TypeScript environment, I was using the XMLDOM and XPath to parse and pull out values from the responses we received. XMLDOM is used to parse text into an XML Document and XPath can be applied to retrieve any value from it. For example, when retrieving the UserId in the previous example, we would parse the response as so:
import * as xpath from "xpath";
import { DOMParser } from "xmldom";
const xmlResponse = `
<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
<soap:Header>
<MessageId>57156b82-ad22-4809-9811-c3eaf142cc12</MessageId>
</soap:Header>
<soap:Body>
<m:GetUser>
<m:UserId>123</m:UserId>
</m:GetUser>
</soap:Body>
</soap:Envelope>
`;
const domParser = new DOMParser();
const namespacedSelect = xpath.useNamespaces({
soap: "http://www.w3.org/2003/05/soap-envelope",
});
const doc = this.domParser.parseFromString(xmlResponse, "application/xml");
const userId = this.namespacedSelect(
"string(//m:UserId)",
doc,
true
)?.toString();
console.log(`User ID: ${userId}`);Our XPath library supports nested namespaces and then selecting an element from within any defined namespace that returns, this means we can just look for any field in the response document called m:UserId to retrieve our text value.
Serverless Functions with Event Sourcing
We used the CQRS pattern to split out our read queries and write modifiers to the database, this means that when calling third-party APIs we queued up a modifier to put the result of the API call into the database. The way we read that data changes as we can store data as a stream of events that occur over time and then read them based on a purpose and collate all that data into a more distinctive view. Using a stream of events means that we can track each individual request sent to an API rather than just the current state of the user records through a conventional database model.
Because we used serverless functions in Google Cloud Platform, a requirement is that our function has to finish running within the 9-minute lifespan or it will be killed off. While we should expect our functions to finish long within that time, it’s still a good idea to split out the responsibility of each part of the pipeline where we can so we can run our API calls asynchronously and allow them to be replayable and eventually consistent.
By splitting our requests into commands within the CQRS ecosystem, we have ensured that we will have eventual consistency and be able to see a historical audit of all calls to the third party APIs for all users, which also enables us to replay certain events if they failed for any reason.
For our third-party pension provider integration, this meant having events that defined a user behaviour, such as createAccount, updateAccount or createTransferIn.
Caveats of Using SOAP APIs
Alphabetical Ordering
A pitfall of using a SOAP API is that the request body is required to be in alphabetical order, this tripped me up a few times when writing the integration as JSON structures are not ordered so you can pick a more logical ordering based on the object. This got confusing for things like an address where a more logical ordering would be to go from most to least specific:
contact: {
address1: "Test House",
address2: "Test Road",
address3: "Test Town",
country: "United Testington",
houseName: "1",
postCode: "T3 5TT"
}Within the services, a model class was passed around so it’s only when reviewing the logs that this ordering was particularly a pain, but our server that we were sending data to would fail silently if a request wasn’t in alphabetical ordering so it was quite hard to debug.
Request Body Formatting
Request bodies have to be declared in a string, which can lead to a very ugly function, where we’re sending a CreateUserRequest which contains some additional fields. Unfortunately for us, there is no nice way to go from a UserRequest object to a request body string in XML, so we end up with a function like:
const createUserRequestBody = (userRequest: UserRequest, messageId: string) => `
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
<soap:Header>
<a:MessageID>${messageId}</a:MessageID>
</soap:Header>
<soap:Body>
<b:CreateUserRequest xmlns:b="http://tempuri.org/">
<b:Address>
<b:Address1>${userRequest.street}</b:Address1>
<b:Address2>${userRequest.town}</b:Address2>
<b:Address3>${userRequest.city}</b:Address3>
<b:County>${userRequest.county}</b:County>
<b:HouseNumber>${userRequest.number}</b:HouseNumber>
<b:PostCode>${userRequest.postCode}</b:PostCode>
</b:Address>
<b:Contact>
<b:Email>${userRequest.email}</b:Email>
<b:PhoneNumber>${userRequest.phoneNumber}</b:PhoneNumber>
</b:Contact>
<b:Name>${userRequest.name}</b:Name>
</b:CreateUserRequest>
</soap:Body>
</soap:Envelope>
`;The outcome of this is that you can try to write some more generic functions for templating strings, for example breaking this out to a generic createRequestEnvelope and createRequestHeader but ultimately it’s the same logic.
Error Handling
This may be more implementation-specific, but for the integration I was working on a successful and failed response both had 200 OK status codes where the response body contained an error message if applicable. This differed from the usual approach to HTTP APIs as it means parsing the whole response to know if the call was successful, rather than determining from the response status.
In our case, we used the XPath response to determine if the error was present, in our example above we could add extra code to determine if the userId value is falsy, such as:
// ...as before
const doc = this.domParser.parseFromString(xml, "applicatton/xml");
const userId = this.namespacedSelect(
"string(//m:UserId)",
doc,
true
)?.toString();
if (!userId) {
const message = this.namespacedSelect(
"string(//m:ErrorMessage)",
doc,
true
)?.toString();
console.log(`Failed due to: ${message}`);
} else {
console.log(`Successfully created user: ${userId}`);
}This approach is far from the ideal scenario, but I found this quite a lot when writing out functions as the lack of library support and implementation of the third party meant making sacrifices in design in order to produce a working integration.
Overall, integrating with a SOAP API isn’t hugely different from any other third-party, the same flows will exist and the management of the data going in and out of the system is managed by similar services. The main difference is the connection of sending requests and parsing responses in the client. Hopefully, the caveats listed above will save some headaches and the use of event sourcing provides some insights into how events were handled internally within the integration.