This photo was taken during the construction of Michael Hartigan's house. This is the large, unpainted, house with the mansard roof in the background. In front of that house there is a smaller house (in white) and an even smaller shed in front of that. The house in the middle is believed to be Patrick and Johanna's house.
You'll notice that the wharf actually extends beyond the sides of the image, and was large enough to berth at least two schooners. The Hartigan business was in the herring trade. Herring was used as a bait fish for other fishermen.
The wharf is located on the "Little Harbor" side of Hartigan's Point. Today little remains of the wharf, sheds or houses. Michael's house has changed hands a few times and looks very little like it did when this image was taken.