sixtyPercent: Cochlear Implants, Aviation, Technlology, and Philosophy 2006/03
Contnuous Software Refactoring
After my visit to Pycon 2006, I've started to (re) get that old-time software testing religion, not only as an end in itself, but also to support my habit of continuous refactoring. I've developed a habit of trying to improve on every source file I visit during the course of a day's work. Each visit usually involves just a small change -- sometimes merely fixing the formatting or leaving a comment or two. But on occasion I will find something that needs a lot of work, and wind up pulling several modules apart in a manic fit of refactoring. This approach works well for my brain -- I can't seem to work on any one thing for too long, but I also can't stand clumsy, confusing, or ugly code. During the course of a day I typically spend about two-thirds of my time on my main task, and perhaps one-third on my "software cleaning" habit.
Automated testing nicely supports this approach to development -- I just seem to need to re-learn the lesson every year or so. Having once again climbed back on the wagon, I installed buildbot, wrote some unittests for many of the lower level system components, and then attacked the higher level integration tests with Twill. Twill generally works quite well, but automated web testing in general, and Twill in particular seems a bit finicky. Sometimes our web pages don't seem to be parsed quite right (even though we are careful to always be XHTML String or Transitional. Naming of links to follow also seems a bit brittle -- I can specify the URL or the link text I want Twill to follow, but thanks to Routes and a new UI designer, I am changing both of these a lot right now. Perhaps a mechanism to specify links via an id attribute might help.
One of the biggest hurdles to get past was just starting. I began with a large-ish application with almost no tests. I'm not done yet, but I can see the light at the end of the tunnel. Here was my plan: First, get buildbot running. The "build" at first was just an import all of the top level components to check for simple errors. Next, I wrote the simple test harness and test runners for the subcomponents of the application, each with a very minimal set of tests. In our top level application directory, I can now just type "python test/test.py" and all of the tests of all of the components will run. Next, I integrated running these tests, and the Twill integration tests into buildbot. This last part involved reading way too much about buildbot and building custom steps to start and stop our application server during the course of the build. The last step is easy and the most important part: write a few tests every day. Since the automated build and test running framework is in place, the benefit of any new test is immediately realized.
One final challenge has been our data model tests. At first I tried to create a snapshot of a testable set of data from Postgres using pg_dump, and then before running the tests, automatically restore the database using pg_load. This works, but had two problems -- it's slow (we need a big data set), and it needs to be constantly tweaked and re-snapshotted as we evolve our application. My current seems to work ok, though it has it's own problems. Instead of creating a new DB instance every time, we use the same persistent testing DB for all tests. Each data model test is responsible for cleaning up after itself, and returning the database to a reasonably pristine state. Broken or failed tests can mess this up of course, but in practice that has been rare (and manageable).
So armed with these tools and a slightly obsessive desire for clean, readable, understandable implementation, I am happily tweaking away on the entire system. I'm sure my coworkers will soon tell me how much they appreciate me mucking with their code... ;-)
by David Creemer : 2006/03/15 : Categories python : 0 trackbacks : 0 comments (permalink)
The story I wanted to read about Jason McElwain
Jason McElwain, the high school senior from Rochester, NY has been all over the news recently. (See for example this story at ESPN). He is the kid who for two years was the "team manager" (i.e. helper) for the school basketball team, and who gets put into the last game of his last year for the unimportant last four minutes. It's news of course because he happens to be autistic, and he proceeded to score 20 points in those four minutes, including six three-point shots. In case you don't know, that's an absolutely amazing accomplishment, even for someone who plays regularly.
Of course everyone has been crying tears of joy, and I suppose the McElwain family is busy reviewing very lucrative movie proposals. Good for them -- they deserve no less.
Here's what I want to know about this story: How come the fricking coach assumed that since this kid is autistic he can't play basketball, or needs a pity opportunity in an inconsequential game!? Why was someone who is so obviously talented at basketball relegated to fetching towels and water for two years?! Did the coach never in two years see him even shoot a ball? How come we assume that a label of "autistic" or "deaf" or whatever equals pitiful or incapable?
I'm sure they all feel quite good about themselves at that high school -- they should feel shame instead. Both the "deaf" and "autistic" labels happen to apply to my son too, and guess what? -- I'm sure he's much, much smarter than that coach, and probably sings better too. He does suck at basketball though :-) .
by David Creemer : 2006/03/09 : Categories autism life : 0 trackbacks : 0 comments (permalink)
PyCon 2006 Day Three
Notes from Day three of the PyCon2006 Python Conference. *Still no coffee*! I spent most of the day today focusing on testing tools and methodologies for Python web applications. Grig Gheorghiu and Titus Brown gave at least three talks (one lightning talk and two regular sessions) on this topic, and I learned a lot of practical knowledge. Both of these guys are really passionate about testing, and have done much through their writing and development to advance the field. At work, we are already doing unit testing (though I have switched to using nose as the test driver).
I left these talks enthusiastic about setting up buildbot, twill, and perhaps FITness and Selenium for build automation, functional, and GUI testing. Getting buildbot to periodically run the unit tests -- and setting up a maintainable strategy for loading PostgreSQL test data sets before running our data model tests -- are the first tasks I'll tackle.
Other random thoughts: Anyone not using the web seemed to be building GUI applications with wxPython. Beautiful Soup seems to be the most popular choice for doing web screen-scraping.
After the sessions, I took a bit of time on the first "Sprints" day to make some small contributions to the vobject library. I've never participated in a sprint before, and the experience was quite interesting. I have no experience with the vobject library, though I do know something about vCard and iCalendar from my Palm days. Armed with that and some Python skills, I was only able to get a tiny bit done, as much of the time was spent getting my head around the vobject design, etc. Next time I'd like to attend the sprints for at least two days.
by David Creemer : 2006/03/01 : Categories python : 0 trackbacks : 0 comments (permalink)
