A detailed incident summary from the Ventrata Engineering team is now available here:

https://docs.google.com/document/d/1wt6qwaDZDBHEf8J2FMv_XFjgUOqVyI_63SmmttAe-ck

Executive Summary of the Report:

On the morning of January 17, our primary Foreign Exchange provider, OpenExchangeRates (OXR), experienced a data integrity issue where they continued publishing rates for the majority of currency pairs but with zero/null values. This resulted in zero/null values being pushed to our internal caching layer, causing failures across sales channels requiring currency conversion.

Service was fully restored by reverting our internal cache to a state captured prior to the corruption. A permanent code fix has already been deployed to prevent bad data from overwriting our cache in the future.

All services should now be operating normally following the latest update.

While we are still awaiting full resolution from openexchangerates.org, the incident is considered resolved from the Ventrata side. We have implemented mitigating data sources, including the European Central Bank service, and restored exchange rates from yesterday to ensure continuity.

We apologise for the disruption caused and appreciate our clients’ understanding and trust.

If you are still experiencing any issues, please contact Ventrata Support.

We have restored a snapshot of yesterday’s exchange rates as a temporary solution.

For the changes to apply on the Terminal (Ventrata app and hardware), a device restart is required.

According to an update from openexchangerates.org, they are continuing to investigate the incident and working towards a resolution. The issue was caused by a failure in their cache and data supply pipeline, although the root cause has not yet been confirmed:

https://status.openexchangerates.org/

As this issue continues to impact our clients and the European Central Bank service does not cover all currencies, we are proceeding with the restoration of a previous database snapshot to assess whether cached exchange rates from before the openexchangerates.org incident can be used as a temporary solution.

We expect this solution to be available within the next few minutes.

We have implemented an alternative FX source using the European Central Bank service, which covers the most common currency pairs and should restore functionality for most clients.

As this feed does not provide exchange rates for all currencies (for example, AED), we are also restoring a previous database snapshot to assess whether cached exchange rates from before the openexchangerates.org issue can be used as a temporary solution.

If you continue to experience this issue, please reach out to Ventrata Support for assistance.

Some clients operating with multiple currencies may experience an “undefined method '*' for nil” error across certain sales channels.

This is caused by an issue with our exchange rate provider, openexchangerates.org, which is currently not returning currency symbols in their responses.

You can track their incident updates here:
https://status.openexchangerates.org/

The Ventrata team is actively working on alternative solutions and workarounds to resolve this as quickly as possible.