We recently started using the ChatGPT bot from OpenAI to perform AI automated assist while writing our applications. Here’s what I found out.
Chat GPT: The Good Results
Boilerplate and Tests
The first and greatest thing about the AI is its ability to quickly generate boilerplate code based on simple narrative requests. This is primarily useful for repetitive tasks such as unit test validations and CRUD endpoints in multi-layer systems. Simple requests like ‘Using this simple Entity Framework object, generate the CRUD operations’ saves a good bit of time from an otherwise mind-numbing task. This, in turn allows me to spend my time more efficiently handling security issues and making sure that all failure cases are properly handled.
The system has yet to generate perfect code, mind you, but it’s far easier to make the small edits than type the entire set by hand.
The other half of this is the system’s ability to, given a specific function, rapidly generate automated tests for it. You’ll want to manually touch these up, but touch ups are much quicker than typing it all from scratch. For reasons you might not want to do this, see the section on Big Brother, below.
Purpose-built examples
Reading the documentation is always the first and best way to learn any framework or library, but a strong second is reading examples. Being able to ask the AI for a generic example of exactly the scenario I’m looking for has been a powerful tool. Even if I still need to go look up specific details in the documentation, the AI can take the description of what I’m trying to do and turn it into a recommended approach and set of function calls. This gives me a huge head start in jumping straight to the pieces I’m looking for.
Instant, hype-less deep dives
When researching a new technical subject, whether it’s to understand my client’s business or grok the latest hotness in the developer world, I frequently come across technical terms that I’m either not familiar with, or, more importantly, mostly understand but wish to refresh my memory on the details. ChatGPT gives me a quick rundown on the term. Because there is such a low resistance to finding a good answer, I find that I’m looking up things more often, and can follow my interests down a rabbit hole similar to using successive Wikipedia pages.
Rapid iteration
It’s implied in the deep dives above, but ChatGPT remembers what you’ve told it before, and you can rapidly iterate on any examples or add new caveats. ‘Let me see that file transfer example using a different library’ or ‘Add input validation to the Create and Update functions to ensure the reference data exists’
Error code examination
Obscure errors are the bane of every developer’s existence. Being able to simple paste a stack trace into the chat window and have ChatGPT decompose it for you is a huge help. Here again, Language AI assistance isn’t a silver bullet that will solve all, or even most of your problems, but it’s a good first step that generally has some insight into the issue, or worse case, is able to isolate and tease out where the application went wrong and pull out relevant data from the massive chaff that is a modern stack trace.
The Bad News About Chat GPT
Dated content
When using any reference material, it’s important to note the last time it was updated. Any newer information isn’t going to be included. This is especially relevant to using ChatGPT for automated code generation because of the lightning speed at which libraries change and evolve. You go in knowing that what you’re looking at is based on the versions and best practices of two years ago.
While this is perfectly acceptable for more staid systems like C# or JavaScript, where you just know you’re not getting the latest language features, using it for new and rapidly evolving releases such as .NET MAUI can leave you missing the developments such as core feature rewrites, workarounds to known issues, or even knowledge of newly added components.
This is no deal breaker, but it’s important to remember when you’re asking it questions about newer technologies.
Forgetfulness
The Chat algorithm can sometimes be a little forgetful about what you’ve told it when you add additional information. Usually this is an easy fix ‘Hey I wanted that example in C#, remember?’ This creates a wall on the complexity of what you can use it for, however, as it becomes impossible to add additional constraints to your request.
The Ugly Truth About Chat GPT
Flaky and overburdened
Today, the free tier of ChatGPT is a heavily utilized system, and it just doesn’t have the resources to keep up with the overwhelming demand. The system is sometimes down or loses active connection, requiring a reload after a period of not using the app actively. The system has had trouble reloading conversations, meaning that old questions and answers are unavailable.
Eventually, I expect we’ll all be pushed to the paid tier, so this is likely a short-term concern. It doesn’t take very much saved time for me to justify a 20$ a month bill, though.
That’s just wrong
Sometimes the system just gives a wrong answer. Even when it does, it’s very confident about being correct, at least in the initial response. I’ve found that asking a simple question, like ‘are you sure?’ can frequently get any number of apologetic responses.
When using ChatGPT to help design logical application patterns, it’s useful to remember that it might be very wrong, and that you need to check your answers before going too far. Recently I had ChatGPT help design a data management service, and it gleefully built out all the crud operations for the database and several microservices. While the code was generally useful and well organized, the object types it was using to pass data around were completely inappropriate for the JSON conversion I was using, and I had to update each of the files to a slightly different pattern as part of implementation.
This was a simple problem with an easy and quick fix, but it’s important to remember that using AI language possessors for automated code generation don’t have so much as a built-in syntax validator for the language in question, much less any assurance that code will do what the AI says it will. If you ever want an example of this lack of understanding, ask it for the current time in a foreign city given your local time. The AI will happily tell you the time zone for your city, the time zone for the foreign city, and then remember the time you say it is, and then promptly whiff on giving you the accurate time in the foreign city.
Big Brother is watching you
Just like any other web application, anything you type (or copy) into ChatGPT or any other Language AI (or any search engine for that matter) is retained and available to be seen by at least the developers of ChatGPT, if not broadcast to the entire world, or used as recycled training material for future answers. It’s important to make sure to never ask questions or have it do automated processing on private or privileged data.
Chat GPT’s Usefulness In Review
ChatGPT has been a great tool and rapidly earned a place in my toolbelt for rapid prototyping and boilerplate generation. Additionally, I frequently use it to see in-place examples of various libraries or frameworks I’m evaluating. This allows me to rapidly zoom in on the keywords and functions I’m currently interested in, as most documentation better supports searches for DoTheThingAsync() over How does language X support doing the thing?