As history continues to prove...German's knew more about the holocaust than many were willing to admit.
The fact that they didn't do much, collectively, to stop it does not mean that they didn't know. It means they approved or were scared or were indifferent, etc. However, troops serving on the Eastern Front, people who worked for the railroads, hundreds of thousands of troops (ss, police, auxilaries, etc.) all knew and many told what they saw...even if in whispers.
Even Hitler was once confronted with it...sometime in 1943, the wife of Baulder Von Sirach (sp?) who was part of the Hitler inner-circle -- asked Hitler directly about what she had seen on a train coming from Beligium or France (i.e. people being transported East in cattle cars). He ignored her and cut her and her husband out of the inner-circle, but she asked (this is reported by Sareny (sp?) in her book on Speer and others).
And, if not only the German's, the Book, Hitler's Pope, suggests that Pius and the Vatican knew a lot about what was going on and shut their eyes because of a percieved threat to themselves were they to raise the issue openly, as well as the fact that Hitler was taking care of the Russians (at least early on). So if the Pope -- confined to a 100 square acres in Rome -- could know, I feel pretty confident that millions of German's had some inkling.
Strikes me, the argument that German's didn't know is a little like saying "Jim Crow" wasn't obvious...everyone saw it and closed their eyes to the greater implications of Jim Crow...but knew about it nonetheless.
A side point...but you can call it a "conspiracy"... that is what the Nuremberg Court charged....but remember, they had a problem. They were the winners. The innovation of Nuremberg was that they had to try and figure out how to try these bastards...they were the government of a soverign country. Goering, et. al., were trying to argue the Court had no jurisdiction because they were acting under German Law and the Court had no power to try them as the government of germany following their own laws. In addition, they had to figure some way to get to mass murder with the Russians participating on the court. The Anglo-American concept of "conspiracy" was a device for getting there, bringing the Russians along and avoiding the larger question of trying people who were acting under their own law. So, it isn't quite, IMO, like 19 guys plotting to bring down 5 planes, or Mafiosos plotting to undermine and dominate a construction business, and, also IMO, "secret" has a very different meaning when it is done by the entire leadership of a country than when plotting to bomb planes in hotel rooms and strip bars.