(Disclaimer: I am mostly ignorant of auto tech.)
Why must pistons be round? I'm guessing that it's because it's easy to machine something really accurately if its round, and there's probably also something to be said for even pressure distribution. However, I'm thinking that if you used a square piston with rounded corners, you could get a larger "cylinder" to fit in the same block without compromising the thickness of the walls.
Also, why must ports be round? I can imagine ports that are triangles with rounded corners. This could be used to tune how much air is allowed in or out as the piston is going up and down. This would be a tunable, just like a camshaft.