Thread

GrapheneOS has fixed the Android 17 QPR1 kernel performance regression

142 points · 102 comments · Cider9986

mmooss · 1790974594 · focus · HN ↗
[dead]
1. Dezvous · 1790976821 · focus · HN ↗
  >Just zip it, be professional, focus on what you are doing and not on others.
  How do you propose they "focus on what [they're] doing and not on others" when their entire project is dependent on other's (Google's) work that directly impacts their own users? There's nothing about this blog post that is unprofessional. GrapheneOS released a patch for a rather severe performance bug, which is outside the scope of their normal work, so they provided context into why they made the decision to do so.
  1. andrewaylett · 1790978533 · focus · HN ↗
    Look to "HugOps" as inspiration: when someone else is having a bad day, offer them a hug rather than running them down.
    The article uses phrases like "completely broken", "awful", and "<delay> wouldn't be a surprise". Which might be true but it's not helping.
    Dezvous · 1790995951 · focus · HN ↗
    Those phrases are true and they helped by making a patch, which Google will most likely end up merging. So what's your point? Should GrapheneOS blogs treat Google as if they're an individual with feelings having a bad day? The same Google that routinely and deliberately hampers the development of GrapheneOS and other similar projects?
    Google pushed out bugged code ostensibly breaking millions of phones, mine included. They have the means to fix it in a short amount of time like GrapheneOS devs did, but they won't do it, so GrapheneOS took the initiative.
    andrewaylett · 1791026356 · focus · HN ↗
    No, we shouldn't treat Google like an individual, because it's not. But we can (and should) be mindful of the individuals who make up the various teams within Google.
    That's part of the motivation behind HugOps: the people working on fixing a thing are only rarely the people who are responsible for the system that caused the breakage.
    grapheneos · 1791058776 · focus · HN ↗
    Individuals at Google working on Android are ultimately responsible for Google's years of attempts to harm projects based on the Android Open Source Project. That includes Google coercing their OEM partners to sign illegal anti-competitive agreements, pushing the illegal anti-competitive Play Integrity API and breaking many public commitments to open source along with the promises they made to sell Pixel phones.
    South Korea and multiple other countries have already found the Google Mobile Services licensing agreements to be illegal, so it's an objectively true statement to call it illegal. Samsung has been largely freed from the restrictions due to the court decisions in South Korea combined with their market share, which is good, but also incredibly unfair to other Android OEMs.
    GrapheneOS was initially collateral damage for Google's anti-competitive behavior but in the past couple years they've shown they feel threatened by us and are very actively trying to hold us back. It's not everyone at Google doing it but it's enough of them including the people in control.
    andrewaylett · 1791067767 · focus · HN ↗
    Right. But riffing off a response that's a cousin of this one — what are you actually trying to achieve with your writing? And I should also note that I agree with you, in substance if not in style.
    I'd suggest that Google aren't your primary audience for this kind of post: it's folk like me, who are quite keen on the idea of GrapheneOS but don't actually use it yet (I actually bought a second-hand spare phone last week with the ulterior motive of trying GrapheneOS on it, we'll see...). What do you want your average reader to think about you, from reading what you've written?
    I should add that this post (that I'm directly replying to) comes across to me as very much more appropriate than the original blog post, even as it's in many ways a lot harsher. It's putting Google's behaviour in context, rather than drawing comparisons between your own abilities and their poor performance.
    You don't need to drag Google down in order to look good; there are many reasons why what Google (and especially their management) have done is bad for society (and not just for you) and also many reasons why what you're doing is excellent on its own merits, not merely compared to Google.
    grapheneos · 1791074452 · focus · HN ↗
    It was an announcement on our forum for our userbase to explain why the last update introduced major issues with lag, how that happened, that it wasn't specific to GrapheneOS and that we were highly prioritizing working on it. Our release fixing it was pushed out to our Alpha channel yesterday, reached Beta early today and is now in Stable. We had to make an announcement about this because it was heavily impacting perhaps 1/5 people and was noticeable by many more people once they were made aware of it even though they weren't heavily impacted. Many users were complaining about this after there had been several days for memory pressure to start getting triggered for a much higher proportion of users.
    There were a small number of complaints prior to us moving the release introducing the issue to Stable. There were very important security updates in the updated firmware/driver code and we decided we needed to release it despite that. It took several days for most people's setups to build up enough memory usage to start triggering it. It started impacting a lot more people after a while and we realized it was a severe issue we had to address ourselves instead of waiting for Google to do it.
    Google likely had the same reasoning for pushing it out to people to get many other issues including security bugs fixed despite a major known regression. We did not have time to avoid the situation in advance. We had to either delay driver/firmware security updates or ship it and then fix this afterwards. The problem is that it typically takes them at least around 4 to 6 weeks to ship a fix for even a critical issue. They've sometimes managed to do it in 3 weeks in extremely critical cases. It's unusual for them to move that quickly and it takes them longer to do kernel changes userspace ones. Once we figured out the issue was, it was quickly fixed and we started builds for a release in the same day.
    It takes a long time for us to build a release for 21 different devices despite having build machines for it due to each having a specialized build with only what they used and their CPU architecture, etc. and we actually need 42 builds due to security preview releases. We plan to make incremental builds reliable enough to use those for production but it requires great care and potentially ongoing CI testing of reproducible builds to make sure incremental builds haven't regressed. If memory wasn't so expensive, we could buy a bunch of hardware to upgrade or replace our existing 4 local build machines for official releases.
    mmooss · 1791091546 · focus · HN ↗
    > It was an announcement on our forum for our userbase to explain why the last update introduced major issues with lag, how that happened, that it wasn't specific to GrapheneOS and that we were highly prioritizing working on it.
    Whatever your 'reasons', the consequences are the same. Of course you know that "announcement on our forum for our userbase" can spread much more widely than that.
    To lead successfully, we can't just do what we want and ignore the consequences. A leader's job is to get the best consequences.
    notpushkin · 1790997837 · focus · HN ↗
    “HugOps” is great (I’ll be stealing this term :-), but Google has been downright hostile to custom ROM maintainers lately. They deserve no hugs here.
    microtonal · 1791008233 · focus · HN ↗
    We can hug the engineers that have to fix this while at the same time criticizing the process that led to this.
    Google has betas for QPRs and yet this was not caught. It makes you wonder if Google is dogfooding the devices themselves or at least the devices that are not the latest gen and have 16GB memory.
    The other issue is that it often takes weeks or months to fix show-stopping issues. Compare to e.g. Apple who rolled out a fix for Apple Watch 12/Ultra reboots or the iPhone 18 Pro Face ID issues within a week.
    There are seriously things wrong in their process if they don’t catch these bugs and/or fix them quickly. (Or the other possibility is that they are understaffed.)
    grapheneos · 1791060102 · focus · HN ↗
    Google caused many of these problems with mass layoffs, buyouts and ending remote work which drove away many of their best engineers. The hostility of Google's business side towards projects based on AOSP has also led to many additional issues for the Pixel OS which could have been avoided. If they hadn't made it nearly impossible to contribute and been so hostile towards many projects including GrapheneOS then they could have a lot of help with these issues and could get them fixed far easier. The Beta releases aren't released as open source and there's no way to contribute.
    grapheneos · 1791058090 · focus · HN ↗
    Google has been increasingly hostile towards GrapheneOS over the past several years despite our substantial contributions upstream. We've spent years helping Google with vulnerability reports, improvements submitted upstream and multiple proposals for security protections which were implemented for both Android and Pixels. We've been treated incredibly poorly by Google.
    They're trying to limit how many devices can be sold with GrapheneOS by Android OEMs, reduced the number of AOSP releases from 12 to 2 per year with 10 of those now being Pixel exclusive, removed Pixel support from AOSP, unsuccessfully tried to cut us off from early access to security patches which are now allowed to be shipped early by OEMs while under embargo (which we do), are trying to convince app developers to ban using GrapheneOS via the Play Integrity API and even recently funded an AI slop paper with misinformation about GrapheneOS falsely claiming it missed security patches that it did not based on an entirely false premise that it has diverged from AOSP in a way that it hasn't and has to port patches to a diverged fork of the OS which isn't how it works. More info at <a href="https://news.ycombinator.com/edit?id=49946698">https://news.ycombinator.com/edit?id=49946698.
    Many of the things Google has done to hinder open source projects based on Android are illegal and they continue doubling down on it despite a lot of regulatory and legal actions already taken against them.
    mmooss · 1791059530 · focus · HN ↗
    You're missing an essential point, it seems, about public communication (and behavior): It doesn't matter what the other guy does. Google's conduct is irrelevant here. I care about GOS, not Google. The question is, what is best for GrapheneOS? GOS may be competely justified in their criticisms of Google, but that doesn't matter at all:
    * If GOS is justified in their criticisms and tone, and expresses them, and it harms GOS's relationship with Google (problematic as it is), then it causes GOS harm.
    * If GOS is not justified in their criticisms and tone, and expresses them, and it harms GOS's relationship with Google (problematic as it is), then it causes GOS the exact same harm.
    So if you're thinking about justification, you are distracted - it doesn't matter. 'What is best for GOS' is the only question, and what is best is very likely to not antagonize the company and its employees when GOS is at their mercy entirely. What is best is to find a way to work with them and form as solid a bond as possible (and have a plan B if they screw you).
    We all have to work with people we don't like and who treat us poorly; that's life when you have higher priorities.
    grapheneos · 1791075418 · focus · HN ↗
    We no longer have any ongoing collaboration or communication with Google due to their hostility towards open source projects based on Android. We aren't completely at their mercy. Google are facing major regulatory and legal action around the world. The future of Android is as an independent company forcefully split away from Google. Once Google has to compete in the market on fair terms rather than exerting coercive control over Android OEMs through multiple monopolies, they'll want all of their apps and services to run on GrapheneOS. We look forward to once again working with the upstream Android engineers once they're no longer held hostage by bean counters laying off their coworkers and ruining the project they work on.
    The Android team is barely allowed to talk to us or the general public anymore, so what relationship is there to preserve? They'll very likely handle this whole issue with near complete silence towards the media and everyone else. A fix will eventually ship and they'll very likely never acknowledge it beyond a sentence in the release notes, will never explain what went wrong and won't commit to doing better in the future. They likely won't do better but rather worse since they keep doing more cost cutting including layoffs.
    mmooss · 1791078869 · focus · HN ↗
    > We no longer have any ongoing collaboration or communication with Google due to their hostility towards open source projects based on Android.
    Maybe that's partly due to GOS's behavior toward Google and its engineers. For example, maybe there would be more workarounds and back channels otherwise.
    > The future of Android is as an independent company forcefully split away from Google.
    You are putting a lot of eggs in that quite uncertain and unlikely basket (given how impossible it is to predict the future and also the very wide range of possibilities). Splitting Google hasn't happened yet, anywhere, despite many attempts.
    If they do split, there's no reason to think Android will be independent: It could be part of a larger subset of Google, even 'old' Google. It could be sold to another large company - many would love to control half the world's phones, if only to guarantee their applications are easiest to use and competitors can be disrupted (e.g., Meta or Microsoft could benefit enormously - and Microsoft could finally extend its OS business to phones).
    Even if independent, why would Android act the way you hope? Who will run it - who will its shareholders and managers be? Likely current many Google shareholders and managers would be part of the spin-off. There's no reason to think they - or even completely new people - will be more supportive of FOSS; they could go the other way.
    GOS is amazing. I don't know you so likely this is off-target to some degree - my apologies: Many engineers have learned, through defeat or success, that they need more than great engineering to get their passion into people's hands and change the world. Just as essential - more essential (lots of poor engineering is widely used!) - are great business skills. There's no shame in it: very few people can do it all (who is both a great engineer and great business manager?), and none have time to do both.
    I hope someday the world knows 'Graphene' and a billion use it. It's one of the most important projects around - how else can typical end users have a chance at privacy? Good luck!

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.