On every page load drupal_load() is run which loads all enabled modules files which get stored in the php memory. With Drupal 6 ability to attach files to specific paths allows for limited loading of code dependent on the path, very cool.

Solutions for Drupal 5 other than increasing the memory limit is to install APC or something like it, this could very well not be a possible solution for those in virtual servers hosting. Once APC (or something like it) is installed, you will see your memory usage drop dramatically. If you are use to seeing 30M - 40M, you would very likely be seeing 2M - 5M as a result of APC.