The Stephen Plamquist article linked from Wikipedia makes a similar case (and far better than I can) -
http://www.hkbu.edu.hk/~ppp/ksp1/KSP6A.html
" Let us now look more closely at some of these positions, beginning with Allison's. His assertion that the
noumenon 'has a basis in transcendental reflection' [A10:55], though in a sense true, is misleading. Of course, all the basic elements in each of Kant's systems are transcen*den*tal in the broad sense that the entire Critical philosophy adopts the Transcendental Perspective [see II.4 and III.4]. In the narrower sense, Kant's discussion of the transcendental set of object-terms does form the basis for his
discussion of the noumenon [cf. 56-8]; neverthe*less, 'noumenon' itself is an
empirical object-term, so its function is more closely related to empirical reflection.
Kant himself explicitly denies that we possess 'a transcendental perspective' from which we can view 'the noumenon as an object' [Kt1:313].
"That Allison himself does not fully appreciate the empirical status of the noumenon is evident at several points in his discussion. For instance, in a rather obscure passage [A10:58] he quotes from Kt1:A253 as evidence of Kant's intent to distinguish between the noumenon and the transcendental object; yet he then plays down the importance of this distinction. After hinting at Kant's recognition, even in the first edition, of the correspondence between the transcendental object and the negative noumenon, he passes this off as insignificant, claiming that 'the noumenon in the negative sense is not really a noumenon' [A10:59-60]. His failure to grasp the perspectival character of this distinction leads him to regard the first edition version of the chapter on Phenomena and Noumena as more adequate than the second. By contrast, I have argued in note VI.23 that the second edition is indeed an improvement for just the reason Allison overlooks:
the positive-negative noumenon distinction allows for a clearer explanation of the difference between the object as viewed from the transcendental and empirical perspectives. Even the passage Allison quotes [Kt1:A253] emphasizes that 'the transcendental object ... cannot be entitled the
noumenon', because the former is related to 'appearance in general' (via the
transcendental perspective), so 'I know nothing of what it is in itself', whereas the latter would require such knowledge (via the
empirical perspective)."
The object in the world is not the same thing as the noumena. That's why Kant creates the separate term in the first place. An if you look at the etymology (from
nous, meaning 'mind'), it seems even clearer that the noumena is an object of cognition and not a "real" object.
In any case, for your framework to account for inter-subjectivity, you need to account for the objects in the world. Materialism does this, because it posits that objects are material. How does your philosophy?