February 2023 - Performance and Order Failure Updates

Last Updated: 2/10/23 10:00AM MT

Some larger Aspenware Commerce resorts have seen a large number of order failure errors due to connection issues since the start of the Winter 22/23 season. These errors can be resolved by reprocessing the order, and will typically process successfully into the POS on the first retry.

After extensive research, we have come to the conclusion that these errors are not caused by a particular bug or single cause. We have identified that these failures are a result of code performance issues that are manifesting in different types of order failures.

All of our resorts are seeing very high traffic on their online stores (up ~50% on average) and some areas of our code have hit a code performance threshold. Orders are failing as a result of too many API calls to Aspenware Commerce.

In the last 2 weeks, we have identified and have been working on improvements that will help our resorts manage the order failures and prevent these order failures in the future, plus significantly improve Aspenware Commerce.

Solutions

Following is a summary of the solutions we have been working on and details about when these improvements will be available to you.

1. Triggers (Available now)

When orders in the Aspenware Commerce Order Queue fail to process to the POS due to connectivity or similar issues, those orders can be manually flagged to reprocess and typically will be successfully sent to the POS. Aspenware has developed 3 database triggers that will automatically flag such orders to reprocess.

These triggers are designed to reprocess failed orders while we work on preventing these failed orders from happening. These triggers are available for your resort to implement immediately and do not require a release or a store restart. They have been thoroughly tested and have been in production at 2 large Aspenware Commerce resorts for several days and are successful at reprocessing failed orders due to connection issues.

The goal of these triggers is to automatically reprocess the orders so that a site administrator does not have to monitor the order queue checking for failures. Admins are notified via email only when an order fails to reprocess after 2 retries.

We highly recommend you implement the triggers immediately. For more information about the triggers, please see Trigger Details.

2. Hotfix 2.38.1 (Available and being monitored in 2 stores)

Aspenware has identified areas of our code where we can optimize API calls that add load to the system without affecting functionality. In this hotfix, we are removing unnecessary API calls in the Order Processor and Inventory Sync functions. We are also adding caching to some of these functions to reduce the number of API calls needed.

With these improvements, we expect to further reduce roughly 15% of the API calls made to Aspenware Commerce.

Note: This hotfix is complete and being monitored in 2 stores currently. It is only an update to the Order Processor and Inventory Sync function and not Aspenware Commerce itself. It requires no downtime.

3. 2.39 Release (Available early March)

In the March release, we will continue our work on reducing the unnecessary API calls to Aspenware Commerce and making updates to the API calls related to Payment Plans. For resorts using payment plans, roughly 25% of the server calls are related to payment plans. The planned optimization to the Payment Plan plugin will significantly decrease the number of API calls made related to Payment Plans.

For ResortOS, we are refactoring a pricing update made in checkout. These calls are made with every order and can be optimized.

4. Ongoing improvements

In 2.38.1 and 2.39, we focused on optimizing functions that generate a large volume of API calls. In the upcoming releases, we will continue to focus on other areas where we can reduce API calls and improve our code performance. We will also continue to investigate opportunities for more caching, and as we improve the code performance, the need for the triggers will decrease. Once we identify that these triggers are no longer needed, we can turn them off.

Outstanding Order Failure Scenario

There is one order failure scenario for which we do not yet have a solution.

In very rare occurrences, there is an issue in which orders do not appear in the order processor. When this happens, it is likely that the guest received an error on the site and recreated the order. For this reason, we do not want to write a trigger to create the order in the order queue table for processing as it may result in duplicate line items or orders.

For this scenario, we are considering email alerting and/or a report that resorts can run to identify these orders and follow up to determine if the order was recreated by the guest, or if it needs to be manually created.

We will keep you updated as we continue to research solutions to this scenario.

Recommendation for Immediate Action

As of February 10, our recommendation is that your resort immediately implement the Triggers mentioned above. With the President's Day weekend fast approaching, a combination of the Triggers and appropriate scaling is recommended until the week of Feb 21 and we recommend that you upgrade to 2.38.1 after Presidents Day weekend.

If you have any questions please reach out to your Service Representative.