I guess technically his wife owned them until 1865, if you want to quibble. So much for the "holier than thou attitude".
Um, that's not a quibble, that's a fact. Grant owned one slave. He freed him in 1859. The others did belong to Julia which came to her from her father. And that was in Missouri, a slave state, where she was allowed to keep them despite her husband's wishes. (all sorts of family politics enter here. Julia's father was a pro-slavery SOB, a real piece of work.)
There were a handful of slaves still in "the north". New Jersey had implemented a slow gradual emancipation and by 1860 there were maybe a dozen or so left. There were also a few slaves in the north that belonged to southern residents who had moved north temporarily. This included some military officers who took their slaves with them (usually a man-servant). The northern states had begin to pass laws freeing slaves that were brought into their states in transit. This was a real problem for southern slave owners trying to travel. One reason why the Dred Scott decision was such a bombshell was because it threw a spanner in the works with such laws.
The discussion about slavery often sidetracks, understandably, into ideas of white supremacy. The two ideas are, of course, closely connected. You could not have the slave system as it existed without white supremacy, but you could have the latter without the former. The question is not whether most northern whites believed whites were superior to non-whites. Of course they did. It was a given assumption in society across the whole country (and in other nations too). But one could believe in a hierarchy of races and still think slavery was evil and should be abolished. Southern defenders (and I am
not labeling Reheat as such) often say "well, the Civil war could not be about slavery because Lincoln was a racist." Yeah? so? He was a white man in 1860. That's like saying he was right-handed. So were most people.
The southern system was based upon the idea that superior people (whites, regardless of actual status) were entitled to the profits created by the labor of their inferiors (black slaves). The idea of free labor which was growing in the north said that even if a person was "inferior" (politically or socially) they still had the right to the profits of their own labor. Equality in the right to keep one's own labor did not necessarily mean equality in everything.
In practice there are all sorts of contradictions and hypocrisies here. Northern industrialists had no problem profiting from the labor of workers, a contradiction the south loved to point out. And did the southern planters really consider some dirt-poor white farmers their equals? of course not. But people put their faith in all sorts of belief systems that are full of contradictions and that don't really work that way in practice. The important point is that the different regions were evolving in radically different ways in their operating assumptions about owning labor, and the central part of that debate was slavery.