AI, especially LLMs, has become ubiquitous in the tech industry. Although it’s only been two years since I started using tools like ChatGPT or Co-Pilot, I can’t imagine working without them. And I don’t know many developers who haven’t used AI tools daily to help them write code, documentation, or even blog posts.
That’s not all the current state of AI has to offer. The LLMs on the market provide APIs and interfaces to build context-aware generative pre-trained transformers, (GPTs) fine-tuned to your data. Many indie hackers and small startups have already built businesses around these APIs, catering to various niches and industries.
The rapid adoption of AI tools and the possibility of building custom models with relatively little effort has raised a question: “What if we could build an interactive experience into our documentation platforms? We’ll just train a model on our documentation and let it answer questions. What could go wrong?”
Well, among others, GitHub tried that with Copilot for docs, and there may be good reasons why it got sunset.
But let’s not get ahead of ourselves. Let’s first take a look at what added value we would expect the AI to provide in the context of documentation.
I believe that only after identifying how developers interact with documentation can we improve their experience and assess the value of AI chatbots in this context.
As developers, we often don’t read the documentation. We skim it, search for keywords, look for relevant examples. We’re looking for answers to our questions and want them fast.
This seems to be the perfect use case for an AI chatbot. It could provide instant answers to our questions without us having to search for them. It could even provide answers to questions we didn’t know we had by suggesting related topics or examples.
Speaking of searching, it’s not always easy to find the right documentation page or section. Sometimes, we don’t know the exact keywords to search for or the right terminology. In other cases, the documentation site’s search functionality isn’t very good and we end up with irrelevant results or no results at all.
AI chatbots could synthesize the answer to our question from the documentation or at least help us develop a better search query. AI tools could also improve the search functionality itself by understanding the intent behind the query and providing more relevant results.
Sometimes, we look for a configuration parameter or method signature. Other times, we try to figure out how to use the technology to solve a specific problem.
In cases like this, we have to read through many bits and pieces of the documentation and then try to stitch them together, as the docs don’t provide an example of such a use case or simply don’t contain feature-complex examples.
A properly trained GPT could help with that experience by providing a complete example or a step-by-step guide based on the context of the question and the documentation.
Any public API or library maintainer will tell you how easy it is for documentation to become outdated. It’s a never-ending battle to keep the documentation up to date with the latest changes in the codebase, and you need to have good processes in place to ensure that the documentation is updated whenever the code changes.
Updating documentation takes effort and time. What if the AI had access to the latest codebase and could always synthesize results based on the latest code? That would save the maintainers time and help ensure that the users are accessing up-to-date information.
Sometimes, the documentation could be easier to understand. It can be too verbose or abstract, or the reader might be too distracted to fully comprehend the text at a glance.
An integrated AI chatbot that the user could ask to rephrase, summarize, or simplify a certain part of the documentation could help.
To go a step further, if the user struggles with some specific functionality, they could paste their code into the chatbot and ask it to explain what’s wrong with it or how to use it properly.
Finally, the AI chatbot could monitor and analyze users’ interactions with the documentation. It could provide insights into what parts of the documentation are difficult to understand, what parts are missing, or what parts are outdated.
This feedback loop could be used to improve the documentation itself and to provide better answers to users’ questions in the future, either by learning the AI model or by updating the documentation by the maintainers.
We’ve identified several areas where integrating GPTs into documentation seems beneficial at first glance, but some risks and caveats need to be considered.
The first and most obvious risk is the quality of the answers provided by the AI chatbot.
Fine-tuning a GPT on the documentation data is a challenging task. It requires a lot of quality data and testing to ensure the model provides accurate and relevant answers.
Even then, the model can and will hallucinate. In fact, AI chatbots hallucinate a lot.
You don’t want your documentation to confidently provide wrong answers, which will eventually result in user distrust and many open issues in your repository or ticketing system.
The quality of your model will be related to the quality of the data on which it was trained. If the documentation suffers from poor structure, lack of examples, or outdated information, the AI chatbot will suffer from the same issues, effectively amplifying the problems the documentation had in the first place.
Although I’ve stated that developers are already using AI tools on a daily basis and that the adoption of AI tools is rapid, it doesn’t necessarily mean that developers want to interact with AI chatbots in the context of documentation.
While the AI chatbot could provide instant answers and search assistance, it may do so in a different manner than developers are comfortable with.
As developers, we already have an M.O. for finding answers in the documentation, and it’s not by prompt engineering questions to get a chatbot to provide us with a wordy answer. We are used to certain affordances, and we often just want to search for what we need, copy-paste an example, and move on. If I want to discuss a coding problem, I’ll ask in a forum or huddle with my colleagues.
Developers come from different backgrounds and have different levels of expertise. If I had just started learning the technology, my interaction with the documentation would have been very different from that of someone who has been using it for years and just needs to look up a specific method signature.
An AI chatbot isn’t able to recognize that by default unless we applied some clever settings, which would then be translated to additional prompts injected into the model. Even then, it would be very hard to get that right.
Another significant risk is the security and privacy of the data.
Unless the AI chatbot is built on a private, self-hosted model, the model provider will log and analyze interactions with it. This could be a problem for companies with strict data privacy policies or for developers working on proprietary code.
Another aspect is analyzing the user interactions and feedback loop. Unless clearly communicated, this may be a legal issue. Even then, users might not be comfortable with the fact that their interactions with the documentation are being monitored and analyzed by an AI model, and some companies might even have policies against that.
We have a list of pros and cons, but we still don’t have a clear answer as to whether the benefits of integrating AI chatbots outweigh the caveats and bring real value to the documentation’s users and maintainers.
One thing speaks clearly from the points above: Integrating AI into your documentation won’t fix the problems it had in the first place. If the documentation is hard to navigate, lacks examples, or is outdated, the AI chatbot will suffer from the same issues, and even worse, it may amplify them.
On the other hand, if the documentation is well-structured, has plenty of examples, and is kept up to date, the AI chatbot could provide real value to the users if properly integrated as a complementary tool.
The main focus should be on providing the right information in the right format, available at hand, and easily understood. AI chatbots could be part of that, but they shouldn’t be the main focus. I suggest that you first focus on improving the documentation itself, with emphasis on the structure, search functionality, and examples. Maybe you can prepare the ground for the AI chatbot along the way, but you may also find that you don’t need it at all.
