This page tells you which characters are not allowed in Windows or Mac.
Mac OS X uses HFS+ file system, Windows use NTFS. Both encode file names using UTF-16, although the encoding scheme is a bit different. Both also allow a max of 255 Unicode chars in file name.
NTFS does not allow the following chars:
/ \ : * ? " < > |
In Mac OS X, you cannot use the colon “:”. That's it.
In practice, this means, when you have files with a lot of chars that contains any of / \ : ? * + | < >,
you'll have problems transfer it from Mac to Windows. Depending on what tool you use to transfer the file, the tool may stop dead, or change the file name in different ways.
You may think that these weird chars doesn't happen in practice. Actually, they do.
For examples, Mac OS X Adium chat client, save its chat log's filenames like this:
233598025 (2004|10|29).html
Note that the VERTICAL LINE | (U+007C) char is not allowed on Windows.
The chars
? / "" *
may also happen often, such as arbitrary web pages you
drag'n'dropped to your desktop over the years, some math files names may use the
asterisk “*” and the slash “/” as part of the math formula in the file
name, your mom may saved files with question mark and slashes in them,
etc.
You may think a few file name screw-up is ok. True. However, some critical places matters. For example, saved HTML files have local links that relies on correct file names. And, for programing systems to work (database, language libraries, code repositories, etc), correct file names is critical.
For some other practical issues such as Mac resource fork, icon files, “.DS_Store” files, files with Chinese characters, see: Mac ＆ Windows File Conversion ◇ Unicode Support in File Names: Windows, Mac, Emacs, Unison, Rsync, USB, Zip.
For unix file name issues, see: On Unix Filename Characters Problem.
For Perl Scripts that lets you know which files have bad names, see: Perl Scripts For Mac/Windows File Moving.