Alright, since no one at this company cares about potential customers and their problems at all, I did trial-and-error research myself. I'm posting my results in hopes that it will help people in the future who have questions about their API. Though I would highly suggest against this unintuitive headache and just use PayPal instead.
Renaissance was right in that it looks like it's including the entire JSON in the max length validation. Why? No idea. That is a really poor way of handling custom text by including other hardcoded text that is out of their control. So when you send your custom text like this, it's including these entire lines for the length validations:
"show":true,"merchantName":"Greater International Airport School for the Deaf and Blind"
"text":"Pay Now With Venmo, PayPal, Or My Tears"
To confirm this I set up a name that was just barely on the text limit but still passed. If I had added another character to the name it would have failed. Then I took that good name and changed the 'true' to a 'false', thus adding one more character to the entire JSON but not touching the merchantName string. That failed. So the entire JSON is affecting it.
Which means that I will want to impose a maximum length of about 60 for the merchantName and 30 for the payButtonText if I want to avoid the API returning an error any of my customers.
Oh and apparnetly there is zero limit on the hostedPaymentReturnOptions. My text was blown way off the page without failing to get a nonce. Just more wild inconsistencies with the API.
Hopefully this helps any future people who run into this problem.
Problem resolved.
Good job, lightwave365 (and Renaissance) .
Edit: Seriously man, you delete my post because I talk bad about your customer service? Who is running this place? Do you care about people getting help at all?