As the official NYC guide says, don't bother flagging down a cab without its vacancy light on (that's the light on top of a cab's roof) – it means someone else got to it first and it won't stop for you no matter how frantic you seem. Do not, please do not, whistle – only the most naïve tourists do that, and it hurts our ears.
Once it starts to rain, even lightly, vacant cabs will vanish quickly. Also, in some neighborhoods, many of them go off duty between the hours of 4 and 6 pm. This is especially true in midtown, near, for example, the Morgan Library on Madison Avenue and 36th Street or RILM/CUNY Graduate Center on Fifth Avenue and 34th Street. If you find yourself in this situation, probably best to get on one of the buses that go uptown on Madison Avenue.
NYC is a very easy-to-navigate grid. Here's a quick guide:
Streets run east-west and are numbered from 1st Street (in the south) to 216th (near the north end of Manhattan). The grid breaks down a bit below 14th, and the lowest part of the island is not numbered.
Avenues run north-south:
On the east side (east of 5th Avenue, that is):
You get the idea. So, if there are no vacant cabs going in the direction in which you are headed, try finding one that's going in the opposite direction – it can turn around to take you where you need to go.