Reply
Contributor
Posts: 19
Registered: ‎01-05-2017

Webhook notifications not being sent consistently?

I recently integrated the Accept Hosted method using the full page redirect approach and webhooks to record the payment result.  For the most part this has been working fine.  

 

Just the other day, Sunday night 2/19, I noticed that the payments showing in my system were not reconciling with what authorize.net showed.  A handful of transactions were showing as unpaid on my end even though the payment was approved on auth.net.

 

Because there is no way that I'm aware of to check if a particular transaction actually triggered the webhook and what response it received back (please add this!), I could not immediately tell if the issue was on the auth.net side or mine.  So I added some logging to my webhook script that instantly logs all incoming requests to a new file before doing anything else, just in case the issue was due to the script failing or timing out during processing.  

 

Since doing that, log files are being created for the "good" transactions, the ones that are successfully updated in my system.  However, I just saw a new transaction come through on auth.net but did not get reflected as paid in my system.  So I checked for the log file and there was none.  This leads me to believe that in certain cases the webhook script is not being called, otherwise there would be a log file.

 

Can someone check this out deeper to see if you can shed any light?  One example of a transaction that I do not show a notification or log file for is 20442699149.  (FYI this is for a hosted application so the merchant account related to that transaction is not mine.)

 

Please let me know what you can find and if you need any more info from me.

 

Thanks

Posts: 2,765
Topics: 57
Kudos: 248
Blog Posts: 67
Registered: ‎12-05-2011

Re: Webhook notifications not being sent consistently?

Hello @wp

 

I would suggest contacting customer support and request escalation to a support specialist for your issue since it is a problem with a production account.

 

Richard

Contributor
Posts: 19
Registered: ‎01-05-2017

Re: Webhook notifications not being sent consistently?

Did that, they directed me here...
Posts: 2,765
Topics: 57
Kudos: 248
Blog Posts: 67
Registered: ‎12-05-2011

Re: Webhook notifications not being sent consistently?

@wp  :(

 

If you could PM me your case number, I'll request escalation.

 

Richard

Contributor
Posts: 19
Registered: ‎01-05-2017

Re: Webhook notifications not being sent consistently?

Just sent it, thanks
All Star
Posts: 699
Registered: ‎11-03-2016

Re: Webhook notifications not being sent consistently?

[ Edited ]

Hi @wp,

 

Let me chime in with a couple of things. First of all, is what you're seeing sporadic, or following a pattern like being tied to a specific merchant? I'm assuming you've already ruled out things like having it just not work for a specific merchant, right? If it was something simple like that, obviously we'd have an easier starting point for troubleshooting, so I like to start there.

 

Next, how is your setup working? Do you have a single Webhook set up for multiple event types, or a separate Webhook set up for each event type you're listening for? Again, just seeing if we can narrow down the situations we see this in.

 

Third, have you tried any analysis of the Webhooks functions to retrieve the notification history from our server? That's spelled out in our documentation, but I'm wondering if the history that we report matches what you see on your end. If you didn't receive a notification and we don't show that we sent it, that's a different troubleshooting path than if we did say we sent it and you didn't log it as received.

 

Fourth, if you check the status of any of the relevant Webhooks (by just doing the "Get A Webhook" request from our documentation, do any of them show they've been marked "inactive"?

 

Fifth, the log creation should be happening before any of your other processing, but is there any possibility that that part is failing somehow? Sort of grasping at straws, but again, we're looking at everything here.

 

Our system is designed to ensure that every notification is received once. If we don't get a 200 response from your server, we'll keep trying at defined intervals, but if you don't show a delayed delivery like that, the only possibilities are that we didn't send the notification but acted like we did send it and got the 200 response, or that we did send the notification, we got a 200 response from you, but it fell apart on your end. We want to help you either way, of course, but we're trying to rule any possibilities out.

 

Is your listener running through another web server that would keep its own logs, or is it just a program attached to a port on your end? Are you currently doing any firewalling or anything that would log incoming network traffic itself? Any chance you could log at the TCP level, even temporarily.

 

Since you're getting escalated through CS as well, I don't want to duplicate effort, but if you can let me know who you end up dealing with there or tell the CS folks "hey, check with Aaron Wright on this", we can team up on our side and avoid any redundant redundancies.

Posts: 2,765
Topics: 57
Kudos: 248
Blog Posts: 67
Registered: ‎12-05-2011

Re: Webhook notifications not being sent consistently?

@wp

 

Our support specialists asked if you check your notification history?

 

Richard

 

 

Contributor
Posts: 19
Registered: ‎01-05-2017

Re: Webhook notifications not being sent consistently?

@RichardH

 

How do you mean, through the notifications API endpoint?  If so, yes I have checked that but I don't see how to correlate that back to which transaction each notification belongs to. Is there a way?

All Star
Posts: 699
Registered: ‎11-03-2016

Re: Webhook notifications not being sent consistently?

It's tricky, because when you do a request for notification history, what you're getting back is something like:

      },
      "notificationId": "e35d5ede-27c5-46cc-aabb-131f10154ed3",
      "deliveryStatus": "Delivered",
      "eventType": "net.authorize.payment.authcapture.created",
      "eventDate": "2017-01-09T19:18:42.167"
    },

The "notificationId" is the piece of information that correlates this history entry with the notification as originally sent, but of course if you didn't receive that notification...

 

I'm pushing hard for some improvements here in the future like including the payload from the original notification (with things like transaction ID) or at least specifying the Webhook ID in the history entry.

 

Until then, the way to make this information work for you would be to retrieve the notification history for a specific date range, and compare it to your logs of what you've actually received. If there's something in the history you retrieved that's not in your logs, that means we think we sent it, but you think or know that you didn't receive it. So that's a more specific troubleshooting path for us. Of course, this only works if your logs of Webhook notifications received are logging the notificationId, because if not, you've really got no way to correlate since we're not providing the transaction IDs or Webhook IDs.

 

If the entries for notificationId all match up to everything you've received, but you still have transactions unaccounted for, that's a different kind of thing we'd investigate on our end. Either way, any further information you can give us here (or in response to my other post above) would be HUGELY helpful to us.

 

By the way, you can restrict the notification history query to a certain date range by structuring the URL like so:

https://apitest.authorize.net/rest/v1/notifications?from_date=2017-02-01&to_date=2017-02-13

 

Thanks again for the time you're spending helping us with this.

 

Contributor
Posts: 19
Registered: ‎01-05-2017

Re: Webhook notifications not being sent consistently?

@Aaron Sorry, was not ignoring this post, just did not see it until yesterday.

 

I should start by saying that the last couple of days I have seen no missing notifications.  Every transaction showing in the daily batch list is reflected on my end.  But I have done nothing on my side to "fix" the issue - just added the additional logging as I mentioned. So I am not confident that whatever was broke is now resolved.

 

So to answer your questions:

 

 

First of all, is what you're seeing sporadic, or following a pattern like being tied to a specific merchant?

I only have a single merchant right now using this integration method.  So in that sense, yes it is limited to a single merchant, but also that means it is happening for all my merchants!

 

Do you have a single Webhook set up for multiple event types, or a separate Webhook set up for each event type you're listening for?

A single webhook listening for priorAuthCapture.created, authcapture.created and authorization.created.  Because there are no custom fields in the notification to identify which client/merchant the notification relates to, I use the webhookID to check my database for which merchant it belongs to and then proceed from there. 

 

Third, have you tried any analysis of the Webhooks functions to retrieve the notification history from our server?

This is exactly what I'd like to do but it doesn't appear there is a mechanism to look up the notifications, see what transaction each relates to and what the response status was.  Even if I just logged every incoming notification to a database, if the notification never came then there would be no record.  So it would look the same whether the notification came and my logging failed, or the notification never came at all but you guys showed that it did.

 

Fourth, if you check the status of any of the relevant Webhooks (by just doing the "Get A Webhook" request from our documentation, do any of them show they've been marked "inactive"?

No. I ran into that surprise a couple weeks back where it got silently disabled, which was not ideal behavior IMO.  So I've checked that regularly and it has always shown active since.

 

Fifth, the log creation should be happening before any of your other processing, but is there any possibility that that part is failing somehow?

Definitely something I considered but a) the extra logging was added in reaction to these failures so it could not have been the cause, and b) I have no system errors on the server and have run it many times in many test scenarios and it is performing as expected.  Using the file system for logging mostly eliminates it failing due to latency or connectivity issues.

 

If you don't show a delayed delivery like that, the only possibilities are that we didn't send the notification but acted like we did send it and got the 200 response, or that we did send the notification, we got a 200 response from you, but it fell apart on your end.

Here's why I don't think it could be the latter.  There are a lot of validations happening in the script, making sure the incoming data is from auth.net, contains valid json, pertains to our app and not some other app, etc.  If we encounter a validation error that is permanent, i.e. the incoming request is blank, then we just return a 200, delete the text log file (since we don't care about this transaction) and silently exit.  But if we hit something that is a temporary issue, let's say our database connection fails, then we return a non-200 so that the notification will be tried again.  We keep the text log file around showing that we received the notification and exactly what happened.  THen a few seconds/minutes later you guys resend the notification, the temporary issue is gone, so it runs through successfully and we are done.  This logic has been working pretty flawlessly.  But in these recent cases, there is no log file at all, which is why I question whether the notification is being sent.

 

Is your listener running through another web server that would keep its own logs, or is it just a program attached to a port on your end?

No logging in between.

 

Are you currently doing any firewalling or anything that would log incoming network traffic itself? Any chance you could log at the TCP level, even temporarily.

Not doing that currently, no.  *Could* theoretically do that but hoping not to have to go that route and dig through loads of network traffic to find a needle that may not be there.  That's why I was hoping you guys had more detail of the send/receive status on your end but were just not exposing it in the API.

 

 

So as I said, things have been pretty much back to normal the last few days, with no changes on my end.  But naturally I'm not too confident in that.  So where do we go from here?  Does the transaction number I provided allow you to bring up any more detail on your end about what specifically transpired in that case?