Basically, ~/.bash_profile is loaded when you login. It is read only once.
~/.bashrc is loaded everytime you start a shell (⁖ starting a terminal), but is not loaded when you login the first time.
Ideally, you should set important stuff such as $PATH in ~/.bash_profile.
And ~/.bashrc can contain things like aliases, prompt coloring, etc.
Here's a excerpt from man bash:
/etc/profile
The systemwide initialization file, executed for login
shells
~/.bash_profile
The personal initialization file, executed for login
shells
~/.bashrc
The individual per-interactive-shell startup file
~/.bash_logout
The individual login shell cleanup file, executed when
a login shell exits
~/.inputrc
Individual readline initialization file
There's also {.login, .profile, …}. These are basically legacy that bash may also read.
Note: on Mac OS X, the Terminal app starts as a login shell, so it runs ~/.bash_profile. Different unixes have different setup. You should not worry about “correct” init file too much; just try and test on each system.