It looks like I messed up the preference for the HitRecords. The geometry bounces correctly, but the record that sticks is not necessarily the one closest to the camera.