You should not normally need to run TestCaddy with Administrator privileges on Vista. It will need to be installed by a user with the rights to install applications, and if it is most convenient to use the Administrator to do that then that will be fine. End users on the box will not need to re-run the Configuration Wizard because there is one common INI file for all users on a box.
Multiple users can run TestCaddy on one box at the same time (e.g. using Vista's switch login feature). In that case they share an INI file so only one will need to run the Configuration Wizard.