Thread

1. MichaelNolan · 1789565768 · focus · HN ↗

   Maybe I missed it, but it look like this has just a single metric. Maybe instead of making a new project, you could try to get this metric added to a existing tool like <a href="https:&#x2F;&#x2F;dekobon.github.io&#x2F;big-code-analysis&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;dekobon.github.io&#x2F;big-code-analysis&#x2F;index.html which already has dozens of metrics.

   1. stingraycharles · 1789566126 · focus · HN ↗

      That project in itself looks very interesting. How are people using it, any examples of how people get this into an actual report &#x2F; CI test &#x2F; benchmark &#x2F; whatever ?

      1. MichaelNolan · 1789567940 · focus · HN ↗

         Code metrics in general aren’t that widely used. I’ve only ever worked at one place (a bank) that tracked it, and that was only because sonarcube had it built in.

         While a lot of metrics make intuitive sense, we don’t have that much hard evidence to prove or disprove their value. Part of it is the whole “if a metric becomes a target, it ceases to be a good metric” thing. Adding the checks to a large existing project probably has negative value. But I think it’s worth doing for greenfield projects.

         For humans, these should just be advisory. But for LLMs I’m happy enough to make it a blocking check.

         I keep thinking of doing an experiment where I give the same LLM the same problem, and only change which metric is enforced. And then see if any of them have a noticeable effect on correctness&#x2F;maintainability.

         &gt; any examples of how people get this into an actual report &#x2F; CI test &#x2F; benchmark &#x2F; whatever ?

         Yeah they have examples of adding it to CI, or local checks, generate html reports, etc in their docs.

         1. paimapi · 1789568885 · focus · HN ↗

            I think the future of development will be a lot of automated quality checks like these on AI-drafted code that humans review and ensures that it doesn&#x27;t muck up the business logic and actually fulfills the acceptance criteria. that said, I don&#x27;t think existing toolsets are really great at actually measuring code quality

            I&#x27;ve been in the process of reviewing and validating a lot of tools like this (qlty, Sonarqube, fallow, etc) and the false positive rate is anywhere from 20% to 80% for a lot of our sniff tests (zizmor produces an overwhelming majority of false positives here for what feels like arbitrary and very context-dependent GHA requirements)

            the last thing I want to do is to annoy the hell out of our devs by requiring checks like these to pass especially since it&#x27;s only a small percentage of them who vibe code everything and then also vibe response to code reviews. I feel like that&#x27;s the anti-pattern that we&#x27;d push people towards by requiring checks like these to pass

            another avenue of exploration has been requiring test coverage but also good test quality metrics (eg are there negative tests? mutation testing? empty asserts?) something that seems quite easy to spin up into a skill and pair with a deterministic harness. trash-tests is a neat little project that incorporates some of this: <a href="https:&#x2F;&#x2F;github.com&#x2F;frangelbarrera&#x2F;trash-tests" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;frangelbarrera&#x2F;trash-tests (disclosure: I am not the repo owner or even a contributor, just a quality nerd who loves underdogs lol)

            all in all, it really does feel like we&#x27;ll need a revamp of the SDLC with our current expected velocities

            1. sagenschneider · 1789570289 · focus · HN ↗

               Yep, agree on annoying developers. So just to cover usability it can run in warn and block mode to address this.

               Though to the bigger point of your comment, yes SDLC are becoming faster. We can churn out code at a ridiculous rate. However, doesn&#x27;t mean it&#x27;s good code. And hence, there are studies showing things actually slowing down because reviews pile up.

               I guess I look at the Change Impact formula (and <a href="https:&#x2F;&#x2F;impactgate.officefloor.net" rel="nofollow">https:&#x2F;&#x2F;impactgate.officefloor.net implementation of it) as threshold tool. Small changes that aren&#x27;t contributing to god classes, just let through. When things start to smell, the files involved get marked for review.

               Ideally this then can cut down on review time and allow overall increased velocity.

               But yes relies on trusting the AI to do &quot;simple&quot; things

Unofficial Hacker News client; not affiliated with Y Combinator.