My employer pays for Claude, and my approach is to use it as a better Google search. It's often not better.
Just today it made three glaring mistakes in one session:
1. It read a file in the wrong directory, because that file had the same name as the file in the right directory. It apologized when I challenged it, promising me that it would remember to "read import statements" in the future.
2. It miscounted the number of times a function was called in my repo. It said 20, while my built-in IDE search accurately showed 17. Again, it apologized when I corrected it.
3. It referred to a variable by name that does not exist anywhere in my code. It apologized, and said it was referring to a variable used internally by one of the third-party packages installed in my repo.
So many apologies.
It's the little things like this that remind me on a regular basis just how little I can trust artificial "intelligence."
Are you using the best models? I feel like my experience is completely different. I am an expert in a small part of a huge monolithic codebase that I’ve worked in for years. When customers report issues that would take me days or weeks to debug, AI can figure it out on the first try.
Yes, but it’s usually only hours to validate. I understand the codebase so I have the intuition to know almost immediately if the theory makes sense. Validating and reproducing the root cause is often an order of magnitude easier than identifying a production-reported issue.
stack_framer · · focus · HN ↗
Just today it made three glaring mistakes in one session:
1. It read a file in the wrong directory, because that file had the same name as the file in the right directory. It apologized when I challenged it, promising me that it would remember to "read import statements" in the future.
2. It miscounted the number of times a function was called in my repo. It said 20, while my built-in IDE search accurately showed 17. Again, it apologized when I corrected it.
3. It referred to a variable by name that does not exist anywhere in my code. It apologized, and said it was referring to a variable used internally by one of the third-party packages installed in my repo.
So many apologies.
It's the little things like this that remind me on a regular basis just how little I can trust artificial "intelligence."
iloveoof · · focus · HN ↗
Applejinx · · focus · HN ↗
Could you work out that it was wrong, given weeks to go and check its work? If so, your trust is misplaced.
You ARE taking days or weeks to go and check, yes?
iloveoof · · focus · HN ↗