Well, yes, "About", "Load", and "Save" are on the bottom of the screen, but "Play New Game" should be in the upper left-hand part of the intro screen that you see every time you launch the game. So, if you're running the game, once you press the Home button on your iPad and then relaunch the Machinarium app, doesn't this screen show up?
Or does it just go to the screen/level you'd left off on? If it's the latter, you may have to quit the Machinarium app and then restart it. Aargh, this would be easier if I owned an iPad. =p
And if you're unsure how to quit an app on the iPad, here's how to do it:
http://surgeworks.com/blog/how-to-close-quit-and-restart-apps-on-ipad-and-iphone