Not really because these oscillations occur both outside and between the plates. There are simply more of these oscillations going on outside the plates than between them.
The modes outside the plate don't contribute to the attraction between the plates. They either have no effect at all, or actually inhibit the attraction.
Imagine the Casimir plates being contained within a perfectly conducting box. What's of concern are the forces acting along the axis perpendicular to the plates, so imagine this axis going from left to right. To the far left is the left box wall; to the far right is the right box wall; and in the middle are the two perfectly conducting plates. The distance between the plates is assumed to be much smaller than the distance from either plate to the walls.
The scenario is divided into three relevant regions. (The walls are assumed to be immovable, so the regions outside the box are irrelevant.) There is the left region, between the left wall and the left plate; there is the right region, between the right wall and the right plate; and there is the central region, between the two plates.
The allowed EM modes along the left-right axis are considered independently for each region. The boundary to each region is perfectly conducting, so the EM field must be zero at these points. This limits the allowed modes to sinusoidal waves with nodes at the boundaries.
In the left region, fewer modes are available the closer the left plate is to the left wall. Thus the EM field in the left region will have less energy if the left plate is further left. The force exerted is proportional to the distance between the plate and the wall to the negative fourth power. Call the distance L; the force is proportional to L^-4.
Replace "left" in the paragraph above with "right", and we've covered another region. Call the relevant distance here R; the force is proportional to R^-4.
In the central region, fewer modes are available the closer the plates are together. The EM field in the central region will have less energy if the plates are closer together--that is, if the left plate is further right and the right plate is further left. Call the distance D. The force on each plate is proportional to D^-4.
There is a force acting on the left plate proportional to D^-4 - L^-4. (Positive sign for right-ward force.) As L is much larger than D, the D-term dominates and the net force is to the right. The force acting on the right plate is proportional to R^-4 - D^-4. (Positive sign for right-ward force.) As R is much larger than D, the D-term dominates and the net force is to the left. In the limit that L and R are infinitely large (i.e. in the limit that there is no box containing the experiment), the only force between the plates is that due to the EM modes between the plates---the D-term. The influence of the EM fields outside the plates vanishes.
Another way of looking at it, perhaps, is that (assuming the experiment is not contained within a box) the allowed modes outside the plates are the same regardless of the plates' position, so the positions of the plates has no bearing on the energy of the EM field outside them.