Starter drive ??

Some times, not always, when I try to start my car the starter drive does not engage. The starter is new and I have checked all the idler gears and the gear on the flywheel but sometimes it will not turn over the engine.
Other times it will engage, turn the engine over and start it as normal.
The idler shaft has been replaced and it is tight in the engine case.

This has me stumped as to what is causing this.
Any help is greatly appreciated.