Obama pledged when he became president eight years ago to close the facility in Cuba in his first year, but lawmakers in both political parties blocked the effort.
Obama significantly reduced the number of terrorist suspects being detained on the island from several hundred to fewer than 50 in the hope of making a financial case for closing the Bush-era facility.
But that argument has failed to sway lawmakers, many of whom object to moving inmates to US prisons on the mainland.
Congress ultimately banned the transfer of prisoners to US soil for any reason.
The prison population fell to 45 earlier this week after Oman accepted 10 detainees whom officials had cleared for release.
In his letter to House Speaker Paul Ryan, Obama said the population stood at 41 as of Thursday.
A senior administration official confirmed that an additional four inmates had been transferred out of the
prison.
"The restrictions imposed by the Congress that prevent us from imprisoning detainees, even to prosecute and secure a life sentence, in the United States make no sense," Obama said in the letter dated Thursday, the day before his presidency ends.
He noted that no-one has ever escaped from a US super-max or military prison.
"There is simply no justification beyond politics for the Congress' insistence on keeping the facility open," Obama added, reiterating his argument that continuing to operate the prison jeopardises US relations with its allies and poses a continued threat to US national security.
Donald Trump, who will be sworn in as president on Friday, said during the presidential campaign that he wants to keep Guantanamo open and "load it up with some bad dudes".
That put pressure on the Obama administration to find countries willing to accept as many of the prisoners as possible this week.