As Shinzo Abe bade farewell to sunny Florida after another chummy round of golf and cheeseburgers with President Donald Trump, the Japanese prime minister had little to take home to distract voters from the scandals encircling his administration.

On the surface, the pair got on well over two days of discussions at Trump’s Mar-a-Lago estate. They played 18 holes of golf on the same team (and won), donned almost identical ties at one meeting, and traded jokes—Abe poked fun at how much Americans ate. He also won Trump’s support for securing the return of Japanese citizens abducted by North Korea.

But on trade—an issue Trump has used to reward the U.S.’s closest allies—their differences appeared more entrenched than ever. The president criticized Japan both on Twitter and at a joint press briefing with Abe, who failed to win an exemption from U.S. tariffs on metals or convince Trump to rejoin the Trans-Pacific Partnership trade deal.

Abe’s future is increasingly in doubt as his cabinet’s approval rating slides to its lowest levels since he took office in 2012. Until recently seen as a shoo-in to win a third term as president of the ruling Liberal Democratic Party in September—giving him a shot at becoming Japan’s longest serving premier—Abe is now facing predictions he may step down.

Things seem to be getting worse by the day. The resignation of Japan’s top Finance Ministry official over sexual harassment allegations was splashed across Thursday’s front pages in Tokyo, the latest blow to the administration after a string of accusations of cronyism and cover-ups.

Some members of Abe’s ruling party are now saying Finance Minister Taro Aso should consider his position to take responsibility for the harassment case and a separate land-deal scandal involving the ministry. Kyodo News reported that former defense minister and Abe rival Shigeru Ishiba said Aso “had a decision to make.”

“There’s simply too much bad news to hope that the summit would be so successful as to drown out the clamor,” said Tobias Harris, a vice-president at Teneo Intelligence in Washington DC. “Those issues will surely continue to consume media and public attention.”

The first day of talks between the leaders began with the usual show of hand-holding and mutual compliments. But Abe’s arrival was swiftly overshadowed by news that Trump had sent CIA director Mike Pompeo to Pyongyang for talks with Kim Jong Un in preparation for a summit, a reminder of fears in Tokyo that Japan could be left out of talks key to its own national security.

Surprise Tweet

Trump also surprised Abe with a tweet ahead of talks focused on trade. The president wrote that he didn’t like the Trans-Pacific Partnership regional deal, instead favoring bilateral talks, which Japan prefers.

When asked why Japan wasn’t exempted from steel and aluminum tariffs, Trump repeatedly alluded to the U.S. trade deficit with Japan and barriers to U.S. exports. For his part, Abe reiterated that the metal shipments posed no threat to U.S. national security and he’d continue to seek an exception.

“I am aware that the U.S. is interested in a bilateral deal,” Abe told reporters. “But we want to approach the discussions from the point of view that the TPP is best for both of the countries.”

Abe won some initial credit in Japan by rushing to the U.S. as soon as Trump was elected in late 2016. While his charm offensive was previously lauded for bolstering the military alliance and staving off trade barbs, his struggles with Trump since then have raised questions about his strategy.

On the flight from Tokyo to Florida, the inflight movie on Abe’s jet was “Darkest Hour”—an Oscar-winning film depicting the beleaguered Winston Churchill in the run-up to World War II. Abe may need all the inspiration he can get for the political battles he’s facing in the months ahead.