Elevated instant quoting latency
Resolved
Jul 30 at 08:26pm EDT
Incident Post-Mortem Report: Elevated instant quoting latency (2024-07-25)
On July 25, 2024, Arta experienced a period of degraded performance in Create Quote Request operations for Self Ship and Parcel quotes. This was caused by a period of abnormally high latency in responses from UPS, one of our third-party carriers. Arta's Create Quote Request workflows typically generate reference rates for Parcel and Self Ship from multiple carriers in parallel to reduce overall latency and to increase the probability that rates are returned even if one provider or another is offline. However, in the case where a provider is unusually slow to respond but does not fail outright, the impact on Arta's Create Quote Request operations can be significant.
As a result, during this incident Arta's Create Quote Request API response times increased significantly above our typical baseline. The issue was particularly impactful for clients using our configurable request timeout feature set below the increased response time, as they received no Parcel or Self Ship rates during this period.
Our team promptly detected the issue through our automated monitoring systems and immediately began root cause analysis. We continuously monitored the situation until resolution, communicating the degraded service externally via our status page and directly to impacted clients through their account managers.
After UPS's API recovered, and after Arta's engineers completed extensive validation, we posted a resolution notice. Throughout the incident, all other Arta services, including read and write operations, Dashboard functionality, communication systems, search, and public shipment views, continued to operate normally.
Moving forward, we are implementing several measures to mitigate similar incidents in the future. These include:
- Enhancing our monitoring and alert systems for third-party API response times.
- Developing fallback mechanisms for providing estimated quotes during carrier API issues.
- Reviewing and optimizing timeout settings throughout the system.
- Exploring ways to reduce the impact on our overall instant quoting API latency when individual carriers experience high latency.
We appreciate your understanding during this incident and remain committed to providing reliable and efficient service. Arta's team is dedicated to continuous improvement and ensuring the resilience of our systems in the face of external dependencies.
Affected services
Overall API platform
Hosted Tracking and Booking views
Updated
Jul 25 at 03:28pm EDT
API response times for the impacted third-party provider, UPS, are returning to normal thresholds. Arta's instant quoting response times are also recovering.
Engineering is continuing to monitor the performance recovery.
Affected services
Overall API platform
Hosted Tracking and Booking views
Created
Jul 25 at 01:12pm EDT
Arta's automated monitoring has detected elevated latency from UPS, one of the parcel transport rate provider APIs in Arta's network.
Instant quoted rates are returning but outside of expected latency thresholds.
Arta has contacted the carrier and our engineers are monitoring the situation.
Affected services
Overall API platform
Hosted Tracking and Booking views