The original mustang starter did not work for me (I took it with me from the junkyard). It wouldn't even engage the teeth. I had to go back and check what I used and I found it:
Using the original 3.8L bell with a 28oz 164T flywheel, 11" clutch, and 92-96 F-150/bronco/full size van with a V8 & manual mini starter.
That was with my '97 T5 swap into a 65 with a 289. I wouldn't be surprised if you had the wrong starter out of the box. I seem to get "returned" stuff all the time. I recently bought a Pontiac 400 starter and some asshat took out the new solenoid guts and swapped their old crap in and returned it. Some guy could have come in and looked at 5 starters and they didn't get them all back in the right boxes...
I would remove the starter and use a straight edge and check for depth of engagement at the front and back of the flywheel, then measure from the mounting face of the starter (with the gear pulled out) to the front and back of the gear to see if they're mating up properly (and with the gear in to make sure it's not engaging with it all the way in). It almost sounds like yours might be engaged with the gear all the way in? I'd check a couple more starters at the parts counter with your new measurement info and go from there.
All I know is I used everything I referenced above and my car has been starting beautifully for 5 or so years. I love the sound the mini-starters make.
2011 Mustang 3.7L Premium Pony Package
2015 Silverado 5.3L 2LT DC Z71
1965 Mustang C Code Coupe 289/T5/3.25
1967 Chevrolet C10 350/700R/3.73
1968 Pontiac Firebird 400