I have done this.
I built a dedicated emulator machine out of an original xbox, and I wanted an Atari mouse to use with it.
The original Atari mouse is not compatible with modern hardware, so I decided to gut it and replace the innards with those from a modern optical mouse.
I was going to use the hole where the ball used to be as the opening for the optical pickup, but this placed the pickup too high off the mouse mat and the mouse didn't function.
Obviously, 'lift off distance' isn't a problem with the particular mouse I used, it only needs to be lifted a fraction before it stops tracking.
I ended up cutting a rectangular 'chassis' out of the optical mouse to hold the pcb/optical assembly, and then cut a corresponding 'window' out of the bottom of the Atari mouse for the chassis to sit in.
I then made some 'goop' out of acetone and scrap pieces of the shell of the optical mouse and used this to cement the chassis into the Atari mouse.
I also used the goop to form a thin layer over the entire bottom of the inside of the mouse to give it some extra weight and rigidity.
Happily, it just so happened that the microswitches for the buttons were spaced just the right distance apart, so by being careful to place the chassis in exactly the right position, I was able to get them to function with no modification.
Had the buttons not lined up, it would have been a simple matter to use the abs/acetone cement and scrap pieces of plastic to make something that would work.
The result isn't pretty if you look at the underside (mainly because of the difference in color of the plastics) but when it's right side up, the only indication that the mouse is not stock is the color of the lead.
Another problem that I need to fix is that the red led that's now inside glows through the plastic slightly, so I need to make some kind of shield to prevent this.
EDIT: There are actually four photos, but I don't seem to be able to post them, presumably because I'm a newly-registered user. I'll try to get them on here eventually.