CIM Timeout Issue as of 11 PM EST

[jmagaro88]

I am experiencing a timeout issue on CIM as of 11 PM EST. Our last successful transaction and payment profile was created at 10:55 PM EST. Profile response, shipping address response, and payment profile response all looked normal and time for API call was 2 seconds max. However, on our next transaction at 11:41 PM EST, I am receiving time out errors as follows:

[2014-09-19 03:33:57][----Request----]
MASKED
----CURL ERROR----
Operation timed out after 45000 milliseconds with 0 bytes received

[2014-09-19 03:34:42][----Response----]

[ Time Taken for the API CALL : 45seconds ]

[2014-09-19 03:34:42][----Request----]
MASKED
----CURL ERROR----
Operation timed out after 45000 milliseconds with 0 bytes received

[2014-09-19 03:35:27][----Response----]

[ Time Taken for the API CALL : 45seconds ]

[2014-09-19 03:37:20][----Request----]
MASKED
[2014-09-19 03:37:58][----Response----]
<?xml version="1.0" encoding="utf-8"?><updateCustomerProfileResponse xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" ><messages><resultCode>Error</resultCode><message><code>E00014</code><text>Customer Profile ID is required.</text></message></messages></updateCustomerProfileResponse>
[ Time Taken for the API CALL : 38seconds ]

[1411097878]:Error Creating Customer profile 23454: Error: Error
Message: Customer Profile ID is required.
E00014
[2014-09-19 03:39:41][----Request----]
MASKED
----CURL ERROR----
Operation timed out after 45000 milliseconds with 0 bytes received

[2014-09-19 03:40:26][----Response----]

[ Time Taken for the API CALL : 45seconds ]

[2014-09-19 03:40:26][----Request----]
MASKED
[2014-09-19 03:41:09][----Response----]
<?xml version="1.0" encoding="utf-8"?><ErrorResponse xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" ><messages><resultCode>Error</resultCode><message><code>E00003</code><text>The element 'profileTransAuthCapture' in namespace 'AnetApi/xml/v1/schema/AnetApiSchema.xsd' has invalid child element 'order' in namespace 'AnetApi/xml/v1/schema/AnetApiSchema.xsd'. List of possible elements expected: 'lineItems, customerProfileId' in namespace 'AnetApi/xml/v1/schema/AnetApiSchema.xsd'.</text></message></messages></ErrorResponse>
[ Time Taken for the API CALL : 43seconds ]

The same errors occurred for 2 other users under different IP addresses, as well as several of my own tests. Our system has been working solidly for weeks now, and I have not uploaded any changes in over 2 weeks. Is anyone else experiencing this CIM timeout issue?

[voidix]

It makes you realize how chinsy this operation is. A payment processor going down for 9 hours is unheard of. Honestly it's hard to believe that when their offices close for the day, there is no system that monitors uptime and alerts staff in the case of emergency.

[npiasecki]

It's getting unbelievable how many outages they've had this year. And there's never any response, no status page, no root cause analysis.

It's gotten so bad that I've set our server monitoring service to monitor the CIM API just so I can know whether the problem is them or me:

Pingdom Outage Graph for Last 7 Days

Since I started monitoring since 7/29, the Authorize.NET CIM API has been down at least 5 hours 5 minutes, or 99.59% uptime. That's just the IsAlive() call, not reflecting actual payment processing downtime. It also doesn't even count the outage in early July.

Where's is Authorize.Net's status page?

Where is the root cause analysis?

Why has uptime and quality tanked in 2014?

[moklett]

I can confirm we saw the timeouts beginning at 11:02pm EST and ending at 5:49am EST.  We run a lot of transactions through their system, and so we usually have a pretty good gauge on exactly when things begin and end.

BEWARE that, in the past with this kind of timeout, we have seen SOME of the transactions actually go through, even though a response was not received within a 60 second timeout window.  So, make sure you check the Authorize.Net transaction list before you retry any failed transactions, lest you double charge.

Some of you may want to chime in on my post from June, asking Authorize.Net to address these issues in some way: http://community.developer.authorize.net/t5/Community-Feedback/Service-Status-Transparency-Please/m-...

We have communicated with personnel at Authorize.Net who assured us they had completed maintenance that should have "fixed" these timeout issues several months ago.

I know running servers can be hard.  But communicating with your customers in times of trouble should be easy.  Unfortunately, they are failing at both.

[npiasecki]

Just spoke with Liz on the phone at Authorize.Net. She helpfully insisted that Authorize.Net didn't go down, only CIM did, and apparently that doesn't count. I told her it was the same thing, and she said it wasn't. I said "if you aren't storing credit card numbers through the use of CIM, and CIM is down, then how is that not an outage?"

The arrogance of these people....

[RichardH]

We are aware of an issue with CIM transaction processing last night that has since been resolved. We are also currently looking into an ARB issue causing delays with some subscriptions. Any CIM transactions that did not go through last night should be run again. We apologize for these issues and any disruptions they have caused.

I'd recommend subscribing to this topic so that you'll be alerted via email if there are updates. To subscribe, click Topic Options at the top of this thread and then select Subscribe. You'll then receive an email once anyone replies to your post.

Thanks,

Richard

[RichardH]

The ARB issue has been resolved as well. Any delayed subscriptions should now have been processed and be visible in the Merchant Interface.

Richard

[daking13]

Will you be posting a root cause analysis?  Also, why didn't authorize.net acknowledge this issue until business hours this morning?  was your technical support team working on this last night or did it resolve itself?

[daking13]

I called Customer Support right when they opened at 5am, and had a similar experience that npiasecki mentioned a few posts ago.  The first rep i talked to was not even aware of the issue and initially said "it is probably related to a change we released on Tuesday" - total conjecture with no additional facts to support it.  He was unable to get any more information and unwilling to connect me with the technical team.  I then spoke to a supervisor, Ben D., who acknowledged the outage, but said the same thing, that since this just affected CIM, they don't communicate status because they "don't want to worry their other customers"

My biggest concern is that we were sitting and troubleshooting on our end between 9pm and midnight PT, sure that it was the CIM API, but had no way of knowing if anyone was even working on restoring service.

Even if customer support is unavailable, this is still a 24/7 service and it doesn't seem to be treated as such.

[anooshr]

I second these thoughts. 

[jmagaro88]

---

@daking13 wrote:

My biggest concern is that we were sitting and troubleshooting on our end between 9pm and midnight PT, sure that it was the CIM API, but had no way of knowing if anyone was even working on restoring service.

Even if customer support is unavailable, this is still a 24/7 service and it doesn't seem to be treated as such.

---

I completely agree. I was in panic mode from 11 PM EST until 2:30 AM EST trying to diagnose what the problem was. I had to spend approximately 45 mins on the phone with RackSpace for them to run diagnostics on my server to find out what was happening, only to end the call with no understanding of the problem other than my packets were being filtered by authorize.net's servers.

How can a company that probably handles a majority of the eCommerce transactions across the internet have absolutely no warning system to their customers in the event of an outage or even a status page to let us know what the problem is and an ETA on fixing it? I lost approximately $1,000 in revenue because of our inability to accept orders last night, as we pay for PPC traffic around the clock. At the very least, if I was notified of some kind of outage and when the service was fixed, I could have paused my advertising until it was fixed so I wasn't paying to drive paying customers to our website who couldn't pay.