Despite a need for accurate weather forecasts for societal and economic interests in the U.S. Arctic, thorough evaluations of operational numerical weather prediction in the region have been limited. In particular, the Rapid Refresh Model (RAP), which plays a key role in short-term forecasting and decision-making, has seen very limited assessment in northern Alaska, with most evaluation efforts focused on lower latitudes. In the present study, we verify forecasts from version 4 of the RAP against radiosonde, surface meteorological, and radiative flux observations from two Arctic sites on the northern Alaskan coastline, with a focus on boundary layer thermodynamic and dynamic biases, model representation of surface inversions, and cloud characteristics. We find persistent seasonal thermodynamic biases near the surface that vary with wind direction, and may be related to the RAP’s handling of sea ice and ocean interactions. These biases seem to have diminished in the latest version of the RAP (version 5), which includes refined handling of sea ice, among other improvements. In addition, we find that despite capturing boundary layer temperature profiles well overall, the RAP struggles to consistently represent strong, shallow surface inversions. Further, while the RAP seems to forecast the presence of clouds accurately in most cases, there are errors in the simulated characteristics of these clouds, which we hypothesize may be related to the RAP’s treatment of mixed-phase clouds.