This paper reviews the scientific motivation and challenges, development, and use of underwater robotic vehicles designed for use in ice-covered waters, with special attention paid to the navigation systems employed for under-ice deployments. Scientific needs for routine access under fixed and moving ice by underwater robotic vehicles are reviewed in the contexts of geology and geophysics, biology, sea ice and climate, ice shelves, and seafloor mapping. The challenges of under-ice vehicle design and navigation are summarized. The paper reviews all known under-ice robotic vehicles and their associated navigation systems, categorizing them by vehicle type (tethered, untethered, hybrid, and glider) and by the type of ice they were designed for (fixed glacial or sea ice and moving sea ice).