unfortunately due to all this A.I noise we are drowning a lot of good things happening in tech specially around DBs
migrating services is easier if your DB supports those things and is more oriented around AP
now a lot of people just jump straight to microservices without considering what their DB can do for them, or if at times they need a relational DB, or if their use case can be served by a hybrid Db.
if you see most of the talks at Monster Scale [0]: They're around DBs
Every major high-throughput database now runs as microservices, not sure why people still act like things just grind to a halt when the network is involved.
Since when were payment networks latency sensitive? It’s usually 2 or more seconds to even get a payment up on the card terminal from the merchant POST system, then 2-5 seconds more from card presentation to getting approval back.
It's not simple though. In that 140ms the network is checking fraud rules, validating the card, checking available credit, applying rewards logic, and routing across multiple parties. The actual subtract-one-number-from-another takes microseconds. The rest is trust verification across organizational boundaries — which is the hard part of any payment system.
At best it’s checking available credit. All the other stuff is done after the fact. The idea that any banking transaction involves “subtracting one number from another” is so wrong it’s barely worth engaging with.
>Since when were payment networks latency sensitive?
Apple Pay is extremely fast from my experience (at least the web version). There is a high percentage of market loss if payments take long or fail. Im sure there must be a graph for where it plateaus with diminishing returns when it comes to speed but faster payments definitely help with sales.
> Since when were payment networks latency sensitive?
Since the advent of e-commerce, POS-networking and fraud detection systems in 1990's-2000's.
User-facing and authorisation path are highly latency sensitive. It includes tap-to-pay, online checkout, issuer authorisation, fraud decisioning, and instant payment confirmation – even moreso for EFT payments.
> […] 2-5 seconds more from card presentation to getting approval back.
This is the mid-1990's level QoS when smaller merchants connected the acquirer bank via a modem connection, and larger ones via ISDN.
Today, payments are nearly instant in most cases, with longer than one-second card payment flows falling into the exceptions territory or inadequate condition of the payment infrastructure.
> Maybe Amex being a closed-loop network helps with latency?
Yes, this is a huge deal. VisaNet and friends have to wait on the actual bank cores in order to perform online authorization. Amex can guarantee end to end latency.
This is cool, but a part of me has to be flip, and humanity is kind of hilarious to me. All of this amazing infrastructure and hard work to essentially just very carefully and accurately subtract one number from another in a way that makes auditing easy.
"Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling."
Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring that all those things are correctly unwound in the case of refunds being issued.
As someone who recently got an Amex card (primarily for Air Canada lounge access), I've been impressed at how nice their app is compared to the five previous Canadian bank apps I've been exposed to in recent years (Scotia, BMO, RBC, Tangerine, CIBC). Some nice things I noticed in the Amex CA app that I haven't previously experienced:
- instant alerts on use, even when it was a non mobile pay transaction
- up to the minute transaction history in-app, including Aeroplan point accrual; all my other credit cards have a delay before new items appear.
- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
Apple Card has the same features, currently issued and operated by Goldman Sachs but to be transferred to Chase within the next two years. Hopefully the features and functionality will survive. I don't think Chase has them today, so I wonder.
It's atrocious how bad most bank and card apps are. I'm planning a switch to a new bank, and mobile app quality is a huge criterion. Bank of America and Wells Fargo get zero points from me.
>- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
?!? Are you saying that Amex is the first Canadian credit card you've used to let you make the payment from on its app, with Amex pulling the funds from your bank? Other Canadian credit cards all require you to "push" payments to them via your bank's billpay? This astounds me.
unfortunately due to all this A.I noise we are drowning a lot of good things happening in tech specially around DBs
migrating services is easier if your DB supports those things and is more oriented around AP
now a lot of people just jump straight to microservices without considering what their DB can do for them, or if at times they need a relational DB, or if their use case can be served by a hybrid Db.
if you see most of the talks at Monster Scale [0]: They're around DBs
https://www.scylladb.com/monster-scale-summit/on-demand/
NextDNS has blacklisted americanexpress.io as a threat.
Re: transition to micro services (from monolith).
I’m surprised a network so sensitive to latency (as are payment networks), was able to achieve their latency SLAs with micro services.
Maybe Amex being a closed-loop network helps with latency?
Every major high-throughput database now runs as microservices, not sure why people still act like things just grind to a halt when the network is involved.
High-throughput is not the same thing as low latency. In fact, they're often at odds with each other
Since when were payment networks latency sensitive? It’s usually 2 or more seconds to even get a payment up on the card terminal from the merchant POST system, then 2-5 seconds more from card presentation to getting approval back.
I’m fairly certain that’s a POS issue, not the payment network.
I’ve heard anecdotally that it’s < 140 ms for payment networks.
Anyone, please correct me if you know better.
It's definitely not 140ms round-trip. Issuer processors typically have a ~2.5s window for approval.
In practice, the POS sends a message to the acquirer processor -> hits the network -> is sent to the issuer processor, and back again.
https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-...
140ms is still a ton of time to do a simple transaction
It's not simple though. In that 140ms the network is checking fraud rules, validating the card, checking available credit, applying rewards logic, and routing across multiple parties. The actual subtract-one-number-from-another takes microseconds. The rest is trust verification across organizational boundaries — which is the hard part of any payment system.
At best it’s checking available credit. All the other stuff is done after the fact. The idea that any banking transaction involves “subtracting one number from another” is so wrong it’s barely worth engaging with.
Some (rewards processing) is probably done later, but the fraud check definitely isn’t.
>Since when were payment networks latency sensitive?
Apple Pay is extremely fast from my experience (at least the web version). There is a high percentage of market loss if payments take long or fail. Im sure there must be a graph for where it plateaus with diminishing returns when it comes to speed but faster payments definitely help with sales.
> Since when were payment networks latency sensitive?
Since the advent of e-commerce, POS-networking and fraud detection systems in 1990's-2000's.
User-facing and authorisation path are highly latency sensitive. It includes tap-to-pay, online checkout, issuer authorisation, fraud decisioning, and instant payment confirmation – even moreso for EFT payments.
> […] 2-5 seconds more from card presentation to getting approval back.
This is the mid-1990's level QoS when smaller merchants connected the acquirer bank via a modem connection, and larger ones via ISDN.
Today, payments are nearly instant in most cases, with longer than one-second card payment flows falling into the exceptions territory or inadequate condition of the payment infrastructure.
> Maybe Amex being a closed-loop network helps with latency?
Yes, this is a huge deal. VisaNet and friends have to wait on the actual bank cores in order to perform online authorization. Amex can guarantee end to end latency.
Norton Antivirus for Android doesn't allow me to access the site. Hmm.
My ISP (Spectrum) is also blocking it as suspicious...it's kind of funny that they haven't migrated the blog to SSL
why would you trust those charlatans
> Norton Antivirus for Android
I had no idea that was a thing
> If anomalies were detected
but also
> Latency, throughput, and resiliency characteristics had to remain consistent, and in some cases improve.
> Payment requests could not be dropped, delayed, or left unanswered.
what else would an "anomaly" be?
I wonder what IaC tooling they used. Interesting that “Kubernetes” and “Envoy” are the only product names mentioned that I noticed.
A fascinating en dash, used as an em dash, followed by two em dashes.
Hmmm.
Em dashes, twice!
This is cool, but a part of me has to be flip, and humanity is kind of hilarious to me. All of this amazing infrastructure and hard work to essentially just very carefully and accurately subtract one number from another in a way that makes auditing easy.
"Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling."
Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring that all those things are correctly unwound in the case of refunds being issued.
As someone who recently got an Amex card (primarily for Air Canada lounge access), I've been impressed at how nice their app is compared to the five previous Canadian bank apps I've been exposed to in recent years (Scotia, BMO, RBC, Tangerine, CIBC). Some nice things I noticed in the Amex CA app that I haven't previously experienced:
- instant alerts on use, even when it was a non mobile pay transaction
- up to the minute transaction history in-app, including Aeroplan point accrual; all my other credit cards have a delay before new items appear.
- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
Apple Card has the same features, currently issued and operated by Goldman Sachs but to be transferred to Chase within the next two years. Hopefully the features and functionality will survive. I don't think Chase has them today, so I wonder.
It's atrocious how bad most bank and card apps are. I'm planning a switch to a new bank, and mobile app quality is a huge criterion. Bank of America and Wells Fargo get zero points from me.
>- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
?!? Are you saying that Amex is the first Canadian credit card you've used to let you make the payment from on its app, with Amex pulling the funds from your bank? Other Canadian credit cards all require you to "push" payments to them via your bank's billpay? This astounds me.
Cmd+F "Kubernetes".
Oh Jesus Christ.