"Some 3.5 million didn't get a single day" of school last year from among the 6.4 million children aged between five and 17 who were under the care of the UNHCR last year, the agency said in a report.
It was only a slight improvement over the previous year, when the figure was 3.7 million, said the report titled "Left Behind: Refugee Education in Crisis".
"The education of these young refugees is crucial to the peaceful and sustainable development of the places that have welcomed them, and to the future prosperity of their own countries," UNHCR chief Filippo Grandi said in the report.
The agency urged donor governments to increase investment in refugee education and to commit to consistent funding "from the emergency phase onwards".
Half of the 17.2 million refugees under the care of the UNHCR are children, Grandi said.
"Compared to other children and youth around the world, the gap in opportunity for the 6.4 million school-age refugees under UNHCR's mandate is growing ever wider," he said.
Ninety-one percent of the world's children attend primary school, a figure that drops to 61% for refugee children - though this is up from 50% in 2015.
Some 84% of children worldwide are in secondary school, but only 23% of refugee kids.
The report attributes the slight rise in education rates "largely to measures taken by Syria's neighbours to enrol more refugee children... as well as increased refugee enrolment in European countries".
But one in three refugees live in low-income countries where they are six times less likely to go to school than children globally, the report said.
"In low-income countries, which host 28% of the world's refugees, the number in secondary education is disturbingly low, at a mere 9%," it said.
The report included a message from Nobel Peace Prize laureate Malala Yousafzai, the 20-year-old Pakistani education activist who was shot in the head by the Taliban for demanding girls' right to go to school.
"We should not ask a child forced to flee her home to also give up her education and her dreams for the future," Yousafzai said.