spring actuated slider mounted
Your problem is basically, that the camera will stick out of the allowable space in some configurations. Such is usually due to the mounting point of the camera holder forcing a position of the components in relation to the backend of the camera.
This can be solved by mounting the components on some kind of sliding gantry and actuating it upwards with a spring. This will automatically align the camera position to the most lipside place the camera standout allows by pressing the lens protector against the lid. For adjustability of the spring pressure, absolute top position and easy swapping of springs (if needed), a grub screw could be used to adjust the position of the holder to the spring's maximum length - green in the image. By choosing a proper spring and having it at or even in the gantry, you can use somewhat standard parts, like the springs from pens, and don't need to work in as much space under the camera but instead can use the more likely more ample space outside of the camera's moving box. Also, the gantry base plate could mount all other components.
For maximum constrained movement and least strain on the parts, it would be advisable to use a pair of guide rails on two opposing sides of the camera plate, and place the springs either on the center of the gantry or use a pair on either side of it.
This can be simplified with the sensor chosen by mounting the whole board on small spring standoffs that allow them to be compressed by the maximum difference, but such a setup might tilt the camera if no guiding is included. That could be facilitated by adding an extension plate bottom or top of the sensor which constrains the tilt of the sensor board. It might be required to shorten springs quite a lot for this.
mounting point on the camera
Currently, you explain to us, that you hold the camera on the mounting plate of the optical sensor. However, you also could mount the camera by the lens holder. As an Upside, the camera will always be at the same position relative to the lid. As a downside, you will need to account for the whole board of the optical sensor to lift with the lid and if you fix the barrel even to turn the whole board of the optical sensor to adjust focus length. Both in turn will move the whole cabling too - you might need to unplug and re-plug it for some adjustments. Also, adding or removing filters will be more complicated.