If you can find lots of different "simple phrases" to use, you can produce thousands or millions of variations, and put a different number in each one. From a large enough sample, there is a fair chance that two of these samples will have the same hash (through the "Birthday Paradox"). Thus, the hash would not necessarily be evidence of any particular number.
Maybe you could tell us the fifth word of the phrase. Had you done this trick, it is unlikely that you thought ahead of time to make the fifth word the same in all samples, and would probably be unfeasable to create a new phrase that has the hash you already provided.