My name is Allison and I’m in FTC team 12808 RevAmped Robotics. A few weeks ago, I built my first version of a chatbot to answer FTC rule questions using GPTs. I have received a lot of feedback on how to improve it (details discussed later). Yesterday, I launched a new version, a standalone web site https://first-chat.com/ that can answer REEFSCAPE rule questions with improved accuracy, and most importantly, it does not require sign up or log in.
Here’s why I decided to abandon my old version and completely make a new one from scratch:
- My old chatbot was built using GPTs on the ChatGPT platform. What I liked about GPTs is that it’s straightforward to make a bot. All I needed was to upload the game manuals and add a short instruction. However, once more people started using it, I found that it has pretty serious hallucination problems which leads to wrong answers. For example, when I asked a question about FTC: “how many points if a neutral sample is made into a specimen and hung?” (which should be 0 points since neutral specimens are against the rules), it said “10 points if hung on high chamber and 6 points if hung on low chamber”. (Thanks to u/xyzzzzy on reddit for sharing this)
- I then spent lots of time trying to improve it. I tried to add more instructions and have it use web search tools, and none of them worked for me. I guess there are 2 main reasons why: my GPT may inherit the lengthy system prompt from ChatGPT, so my instructions are less effective; although I provided the game manuals for my GPT to ground answers on, the GPT does not comprehend the manuals very well.
- I decided that I can’t fix any of these problems if I continue to use ChatGPT, so I designed my own web app powered by a state of the art language model.
- I tested it out on FTC trick questions and I see accuracy improvement .
I will keep improving it on FRC questions too. I’m excited for you to try it out, and let me know how it works for you. If you’re also interested in AI, feel free to comment below too, we can share learnings!
