Docs, LLMs, agents - using AI to create a content feedback loop

Justin Chavez: Hey, everybody. My name is Justin, and I'm an applied AI engineer at InKeep. So we started as a simple chatbot that wrapped ChatGPT, and we've grown that into so much more. So at this conference, I'm gonna be showing you what we've learned over the past two years two years and what we've seen when you use the AI to really listen to what the users are asking your documentation and how that can create a content feedback loop, which we've been implementing in some of our customers today, and that we'll be releasing at the end of the month. So what do we do at Inkeep?

We answer over a million questions monthly, developer questions about how does she use their products and who trusts us with their voice. It's like the teams building the future, Anthropic, Cursor, and Posthog. We integrate all of their sources of knowledge across their docs, their blog posts, their support tickets, and we deploy that across anywhere that the users might be interacting with the product, across their docs, their help centers, their apps, and their community. We've been yeah. We've doing this for the last past two years, and we've I have a lot of experience to show.

So that is what Inkeep is. It's a AI platform that powers the people that are building the future. So show of hands, for those of you that are that have, like, a docs platform, how many of you have Ask AI in your docs? Okay. It's like half.

Should should be should be more because, Ask AI right now is the norm. Developers have come to expect an Ask AI experience in your docs because they wanna go straight to getting answers to to get building. They don't wanna navigate, like, a documentation tree, and they don't wanna talk to any human unless it's absolutely necessary. Support, sales, anybody. So and and if you think about it, developers also judge your product based on how self serve it is.

If they're able to get what they need without interfacing with someone else, then it's it's like a sign of a badge of acceptance if your product is doable or not. So, if you don't have Ask AI, you're kind of already behind. But that's not the point of this conversation. It's, what is what is the future? What is beyond Ask AI?

Implementing AI for Documentation Improvement

And to start with that, let me share an inconvenient truth that AI is only as good as your docs. So in the last Stack Overflow survey, forty one percent of developers cited that outdated docs was their main pain point when interacting with AI systems, because it only And let me show you why. The AI would read your outdated docs, and then it would confidently give the wrong answer to your users. And the tragedy there is that your users would your developers would lose trust lose trust in AI, in your product, in you. So there is a solution, and that's what I'm here to show you with our experience integrating AI into developer documentations.

So the knowledge to fix that problem is already there. It's just not in your docs. And let me walk you through, two scenarios. So in, like, support conversations, developer, like, hits a problem, and they create a support ticket. And then your team solves that support ticket, but it gets trapped.

It's, like, lost in Slack. It gets buried in Zendesk, and it's lost in email threads. Same with your code base and engineers. They, your engineering team makes a change to your product, and it's great, but it's not well publicized. And it just stays hidden in the code base and your PRs.

So the result is your your your docs never improve. The same questions keep coming, and your teams keep solving the same problems over and over again. So what we're doing with AI now is to solve this problem. So same same scenario, your developer creates a support ticket. Your team solves it, but then AI can then analyze that solve ticket and create a update to your knowledge base in or FAQ.

Same with engineering team. It can read the feature that was uploaded and create documentation updates. This is something that we've been implementing in the past, couple months, and I wanted to show you how it works. So here and and this is still, like, more of, like, a human in the loop process. It's not automatically done yet.

So I'll show you, like, some of the guardrails that we found going along with this process. So there's, three agents that we've built to handle this process of automatic documentation updates. It's the, Insight Finder that monitors all the user interactions and what is the new source of knowledge. It, determines if this is new. Like, does this warrant a documentation update or not?

The next one is the content architect, which takes those insights and figures out where in your documentation do you place that update. Is it in the existing docs, or maybe it's like a new page, or maybe it's removal of a page if it's something that got removed. And then the third is the smart publisher. It's, that's what handles the workflow of routing your content for review, approval, and finally, publication. So these three agents that we've pulled together, work as a pipeline, and each one has a specific job and passes the information down the knowledge chain.

And I'll show you how each of those can be built, starting with the Insight Finder. So the Insight Finder, again, it determines, the critical question is if this is new information. And what it does is it compares the new information that it has from the context against your existing documentation, your support ticket, and it determines if there are gaps where your docs are silent or if there's outdated content. And to build this today, we you just generally need, like, to embed your knowledge base into LLM embedding. So OpenAI and Linkchain provide these out of the box pretty easily, and then you have to store those in a vector database.

We use we use Pinecone or Supabase to do so. And then to to compare the new knowledge against the existing, you'll just need existing, a vector retrieval algorithm in combination with, like, Claude or GPT four. The key insight when building the Insight Finder is just to make sure that you clearly define the source of the trigger that started the process. If it's from support tickets, you just need to make sure that you categorize them for first. If it's chat logs, you're gonna just make sure to filter it for the actual question that was asked and and and relate that directly to the answer that was provided.

This helps the agent focus on the knowledge that matters when it's searching your existing content for for updates. So with the content architect, this is where the insight that this is something that's new becomes actual documentation. It first understands your existing documentation structure, where knowledge belongs, and then your formatting, and then it maps the update to the actual page and and goes in forward with that. So to build this today, you'll need to just have, like, read access to your content management system or GitHub. We use Nango internally to manage all these different integrations and making sure the API maps to the right to the right downstream source.

And since this is now getting in more into, like, multi agent, like, handoffs, we use Pydantic to create the agents and, like, map their output and TypeScript and everything like that and orchestrate it all with LingQraph. And the key consideration here is to make sure that you're making making the right diffs. So we're doing, like, diffs, like, line by line updates to the documentation, not full page outputs from each of the agents. Just as simpler for the agent to, like, minimize the context it's needing to update. And also, another consideration would be to, since this is, like, the actual update that's being considered, to run, like, multiple agents to have, like, a, like, a multi score voting system to approve the the change that's being made.

Then finally is a smart publisher. So the smart publisher we created, is the thing that actually goes and makes the pull requests and changes to your docs. And the thing that makes it smart is that it knows your existing workflows of approval and how to work with that, because it's not gonna make it automatically up to date. You it's still gonna embed in your existing workflow approval system and so forth. So to build it today, same Nango implementation.

Just make sure it has right access to the same APIs that you're using. And when you the key insight here is if your downstream system doesn't approve or doesn't enable drafting of content, then you just have to figure out, like, what would the rollback strategy be in the case that it it does go forth and makes a mistake. So that's that's it there. So we'll see it in action. I actually have, like, a little code snippet of actual Slack channel support requests that we had at InKeep, and I'll just show you how how exactly the agent might work.

So let me I wanna share my screen. Okay. I can make it bigger, but so the actual support question that we received was we do all our support in Slack right now, but we also integrate with others for our customers. But someone had a question about if it's possible to update the knowledge base with well, if we go down, we're gonna paste the whole thing. If they're wondering if they could query a specific source and get the relevant document chunks.

So they're wondering if they can, like, use the API that we provide to our customers to filter the sources that it's considering when answering a customer question. And then Robert, our CTO, answered yes, but this is not documented. But the endpoint is here, so then we just need to follow this endpoint. And then they said, cool. I'll try that.

So what I did is I took this conversation and I placed it in our code base for the the entire agent framework that I've been going over. I'll show you how it works. So let me restart the script. So it has the Slack conversation in context, and what it's doing is it's running a couple of the agents that we already, like, set up to, one, fetch, like, the context of the conversation, and then it's gonna also read, like, that link that the our CTO sent to understand where it might apply the change and what what the change should be. So it's gonna go through and do that.

Okay. And then it's given us update describing that this documentation need to needs to be updated on how to filter a request by the source ID, and it's giving us the source of where it's thinking to do the update in. And since I've run this demo multiple times before this clock, I know that that's, like, the the right source to update for this. So right now, I'm, like, doing the role of the router agent. You would typically have a router agent that assesses the output of each agent and, like, determines if it should move forward or not.

But for a live demo, I'm just proceeding as the router agent, the human router agent. So now it's it understands that this is new knowledge. That was just the Insight Finder, and it's gonna figure out what is the exact changes it needs to make from the documentation that it just received. So it's going forth right now, and it's reading the that same a file about how to do APIs and Rag in our API, and it's running a couple different tools to figure out what the exact change should be. So it's going forth and it's running.

We use our Clotsonnet four. We find that this is like the best right now in terms of code reviewing and just like technical knowledge. So that's a nice tip. So it's going through. It's still running.

Okay. And it found that it wants to update the source with the following, and just like reviewing it in my knowledge of like how the API is formatted. This is this is pretty correct. So then I'm gonna proceed after it finalizes. Still running a couple tools to validate that this is the right to run, And give it a second.

Should finish up soon. So it's just redoing itself because it found an error real quick. It should should validate soon. Okay. Okay.

What else? Yes. This is how long it takes usually. Okay. Finally.

So it found it that was, like, something that was, like, printing in the logs, but it's the same documentation update. It's saying that it's we need to include this specific source and update the documentation with this, And I say that looks good to go, so then I'm executing the smart publisher. And it's gonna run the GitHub agent that we have, and it's gonna create a pull request for the documentation update that adds that specific source ID filtering information in our Rag API. So if we go to our docs. If we get it right.

Docs and then go to our pull requests. Yeah. We can see that this this is the docs writer test that I should just comment to now, like, what the changes that were made that is now adding the filtering by source section to the Rag API documentation and so forth, and it made the actual code changes. And this respects our existing documentation flow. We generally just like wanna open a PR and have that approved by a maintainer.

So, yeah, that's how the that's how it would work in in real life. Thank you. Yep. So just wrapping up. Two more slides left.

If you were thinking about yeah. So that's that's just like the the three flow. Like, so there's three agents here, like Insight Finder, the Content Architect, and the Smart Publisher. You can think about using this in different triggers, like a support ticket that's resolved, or maybe, like, a question from your internal sales team. You can trigger this from any, like, insight that might be, like, a source of knowledge or leaving, like, a GitHub issue close.

And the outputs don't just have to be documentation updates. It could be, like, drafts of blog posts for your marketing site, or a video script if you run video for your online team, or even social media posts. And it doesn't just have to be those three agents. It could be something that works alongside, like, outside after the content architect. It could be a style guide enforcer that just understands, like, how you like to format your docs, or after the draft gets updated, maybe SEO optimizer.

I've seen a couple agents that are doing this that just understands what's, like, the hot topic of the week, what might get prioritized if you include these keywords in your online blog post, or even if you have, like, multiple downstream, like, countries that have different languages that they read their docs in, you can throw in translation engines on top of the output, to move forward. So, yeah, with that, just the knowledge to improve your docs is already already exists, in your everyday user interactions. The and the AI right now is up to date and is ready to, like, take those interactions and make it applicable to your end users. The question is is just, are you ready to start experimenting with these changes or not? And if you are, just let me know.

We power these for many, many customers. And yeah. Thank you for every for paying attention.