MPs called a vote of confidence in Mr Zeidan after learning that the North Korean-flagged ship had escaped to sea.
Defence Minister Abdullah al-Thinni was named interim prime minister.
Earlier, Libyan officials had said they had “complete control” of the tanker as it tried to leave Sidra port. However, rebel fighters rejected the assertion.
Separatist militants have occupied three major eastern ports since August.
They are seeking a greater share of the country’s oil revenues, as well as autonomy for eastern Libya.
The oil tanker – named Morning Glory – was reported to have taken on at least 234,000 barrels of crude oil at the Sidra terminal.
It was the first vessel to have loaded oil from a rebel-held port since the separatist revolt against the authorities in Tripoli erupted in July.
Libya’s government is still struggling to assert its authority over armed groups and tribesmen who helped topple Muammar Gaddafi in 2011.