Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.
While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per day service.
Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.
I think it’s like advertising - 50% of my code is wildly over complicated - I just don’t know which 50%
But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner.
So you could not actually make 80% of them happy - they would ever buy it.
> showstopping technical debt is unavoidable is very new
No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.
I feel you are asking for perfection from humans rather than building a system that takes it into account and defends against it.
I think that the best you can hope for is a CI process that verifies a minimum quality standard. There are I am sure plenty of CI funnels that measure cyclometric complexity, but “rewrite this because it works but is over complex” is a hard sell, compared to “we need to rethink the whole architecture, give me three months”
stmw · · focus · HN ↗
Anon1096 · · focus · HN ↗
leptons · · focus · HN ↗
Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.
pixl97 · · focus · HN ↗
It turns out business environments are wildly overly complex.
lifeisstillgood · · focus · HN ↗
But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner.
So you could not actually make 80% of them happy - they would ever buy it.
DANmode · · focus · HN ↗
Yes, you largely do - they’re the commits that get rushed to, and through.
This take that showstopping technical debt is unavoidable is very new, and will age like milk.
fragmede · · focus · HN ↗
No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.
DANmode · · focus · HN ↗
My point is the technical debt is stopping the show way more often.
Not “this never existed before selling”,
but “we never had the team in place who could do this right in the first place”.
You can quibble about who is responsible, but the fact remains.
lifeisstillgood · · focus · HN ↗
I think that the best you can hope for is a CI process that verifies a minimum quality standard. There are I am sure plenty of CI funnels that measure cyclometric complexity, but “rewrite this because it works but is over complex” is a hard sell, compared to “we need to rethink the whole architecture, give me three months”