Tuesday, June 26, 2007

Making it easy for users to write quality bug reports

Computerkatze by avatar-1 from flickr (CC-SA)

One of the coolest things about making your programs public is the user feedback you will get. Some is going to be "Awsum thx, your program just saved my life" and "I tried to run it on Amiga 500 and it crashed, you suck", but in my experience most of the feedback consists of very helpful suggestions and bug reports.

This is very valuable, and as a developer you have a great influence on quality of the feedback. For most Unix console programs and Ruby libraries nothing needed to be done, as normal stack traces get printed to the terminal, and Unix users tend to know how to write good bug reports, but for jrpg it wasn't that simple - many bugs were highly nondeterministic, and as most users ran it on Windows there was no console to print stack traces to. At first I kept telling users to retry with some sort of console turned on, but after two or three such cases I wrote the following code. In util.py:
def save_errormsg(trace_back):
(tp,v,tb) = trace_back
tbf = traceback.format_exception(tp,v,tb)
f = open("errormsg.txt", "a")
f.write("== ")
f.write(time.asctime(time.localtime()))
f.write(" ==\n")
for line in tbf:
f.write(line)
f.write("\n")
f.close()

And in jrpg.py:
try:
mistakes = Mistakes()
book = demonsoul.Book_of_demons()
mhc = Main_Hero_Controller()
wm = World_model()
wv = World_view()
ui = UI()

main_hero = Chara("female-blue", position=(0, 0))
main_hero.is_main = True

wm.switch_map("world", (14,25))

ui.change_text([U"Welcome to the 日本語RPG", U"", U"Press F1 for quick help"])

ui.main_loop()
except SystemExit:
pass
except Exception:
util.save_errormsg(sys.exc_info())
raise


All exceptions get saved to errormsg.txt. When user reports a bug, I can ask them to attach this file, and thanks to stack backtraces bugs are much easier to fix. For boring technical reasons we don't really want to capture SystemExit, so we let it through.

Another thing that increases feedback quality (and also user count) is good packaging and really good documentation on how to get started - that's where most of the problems seem to be. Listening to users also helps - as you wrote the program it's too easy for you to convince them that they're wrong, and very few users are going to argue with that. Often you're right and their ideas wouldn't work for some reason, but not always. What I found very helpful was explaining the rationale behind the things being the way they are in detail every time an user suggests a change. A couple of times it let me find out that the user was actually right. For example backspace on jrpg repeated significantly too fast - I did some calculations which showed the speed to be just right, and I got used to the way it worked so I didn't notice, but when I tried to explain that to an user I found out that there was a mistake in my calculations and it should really be slowed down a bit.

Being nice to users and replying quickly also improves feedback.

4 comments:

  1. You probably know this, but for the record, if you use threads, this won't catch exceptions outside the "main" one.

    ReplyDelete
  2. Jonathan Ellis: True, but I try to avoid threads unless I really have to use them. They're not really of much use in Python anyway due to the Global Interpreter Lock.

    ReplyDelete
  3. Anonymous01:04

    You can automate this further by including code that sends the bug report to a CGI script on your web server or perhaps uses an smtp server to send an email.

    How difficult would that be?

    ReplyDelete
  4. Rudolf: It might work for some programs, and technically it wouldn't be difficult, but I think it's generally a bad idea to send any user data anywhere without a prior explicit user request.

    ReplyDelete