Water Music and Music for Royal Fireworks. Bang.
The Messiah.
Clock Symphony.

Me, I like Hayden partly for sentimental reasons, and partly because of his influence on Beethoven (who I kind of like more). For that, look at his later symphonies (not the very last, more toward the middle-end) and his late string quartets. But that's just me.

As to performances, I'm cheap and easy, but generally, anything pre-late-70s is going to take liberties with - oh, how do you say - trills, ornamentation and fancy notes, and anything after is going to avoid interpretation and go straight to what's on the page. Me, I like the ornamentation. But for "purity," go for the later recordings, post-Hogwood St. Martins-in-the fields stuff.

HI!
There are some "immortal" performances - most of which are not period instrument performances.
The London Symphonies with Beecham conducting. (EMI)
Symphonies 92 - 99 with Szell conducting (Sony)
Oxford Symphony with Bohm (DGG)
Symphony 31(Hornsignal) and 45 (Farewell) with Mackeras (Telarc)
The Season's with Beecham conducting (EMI)
Horn Concerto with Halstead (I forget the label)
The Creation with Karajan conducting (Janowitz and Wunderlich singing) (DGG)
Any of the String quartets - I like Qt. Italiano, Kodaly, Lindsey... Favorites of mine are opus 33, opus 74 and opus 76, also "The Lark"
Bernstein's Haydn is generally very good (I especially like the masses)
Although I am an operaphile, I really don't know Haydn's operas even though he wrote quite a few. Can't recommend anything there...
There is really a wealth of Haydn out there for the listening. Good luck! hope this was helpful.

What, you mean you really don't want to listen to all his Baryton works? : )