I have read only HP's published stories. Where can I get his letters? I'm really interested.
The "standard" sets are the Arkham House
Selected Letters Volumes 1-5. Some of those volumes are now out of print but can be found in used book stores.
If you don't want to jump right in with all five volumes, there's a good "autobiography" in letters that was put together by Lovecraft scholar ST Joshi
Lord of a Visible World. I would recommend this one highly as not only is it a good example of his letters but it also is a good look at his life and thoughts.
Lovecraft at Last is a loving look at HPL through the eyes of a young boy who corresponded with him just before he died.
Hippocampus Press has published
Letters to Alfred Galpin, which I have not yet read.
Night Shade Books is starting a collection of Lovecraft letters. The first was
Mysteries of Time and Spirit which collects the correspondence between HPL and Donald Wandrei. I wasn't too interested in Wandrei's letters, to be honest. The next volume looks more interesting as it contains letters written while HPL was in New York.
That's all I can think of now, except for a few chapbooks put out by Necronomicon Press. I'd recommend starting with
Lord of a Visible World and then trying
Lovecraft at Last. If you like what you've read, then by all means track down the full set of
Selected Letters.