Simple demo of the problem - http://www.babylonjs-playground.com/#MJRGPB
Try to move to the box from the front or side - camera's ellipsoid (with size 1.5) works as expected. Now try to move to the box from below - you can go a little inside the box and see the internals. You can notice that camera's ellipsoid from above is missing or it's size is 0.
According to this - https://doc.babylonjs.com/babylon101/cameras,_mesh_collisions_and_gravity - ellipsoid should be symmetrical.