Another worker is critically ill but stable with burns after the blast at the Chevron refinery, Pembroke, at 1820 BST on Thursday.
Chevron said the explosion happened during routine maintenance of one of its 730 cubic metre storage tanks.
Non-essential work was suspended on Friday.
Chevron said it had launched its own investigation, alongside the Health and Safety Executive (HSE) with one storage tank destroyed and another damaged in the incident.
Production was able to continue but the company had decided to suspend all non-essential work on Friday.
A spokeswoman said Chevron was a "close family" and counselling was being offered to the 1,400-strong workforce.
Dyfed-Powys Police said the bodies would remain at the scene and indentification would be protracted because of the severity of the accident.
Their recovery is being made more difficulty by the unstable nature of the building.
Chief Supt Dean Richards said: "Early indications suggest that this is a tragic industrial incident.
"Sadly four people have lost their lives, and a fifth person remains in hospital, critically ill.
"We are not currently in a position to release the names of the victims, as they have not yet been formally identified, and this process may take some time."
Continue reading the main story Hywel Griffith BBC Wales correspondent at the Chevron refineryI arrived late on Thursday night to find the road leading to the refinery blocked off and signs that the emergency services were at the site.
As the refinery is such a huge complex it was hard initially to assess the scale of the damage.
On Friday morning the scene surrounding the refinery appeared to be fairly normal, but our news helicopter was able to reveal the scale of the damage around the exploded tank.
Although people here are clearly used to living next door to a refinery and the potential hazard that that entails, this event has clearly shaken them.
Most people in the community know at least one person working at the plant and are worried about the identities of those killed and injured.
BBC Wales correspondent Hywel Griffith said it was understood one of those who died was a woman in her 50s.Greg Hanggi, refinery general manager, said: "The loss of our co-workers has come as a huge shock to us all.
"Our thoughts and deepest sympathy go out to their families. We will ensure that all employees and contractors are fully supported throughout this difficult time."
He added that Chevron would take every step possible to determine the series of events leading to the tragic incident and ensure that any lessons would be learned.
Five Mid and West Wales fire crews had joined the refinery's own fire fighters in tackling the blaze, while specialist fire units, including foam vehicles and a chemical incident unit were sent to the scene.
Chevron spokeswoman Jane James said the company was "hugely saddened"The tank at the centre of the incident contained a component which refiners "routinely used," said a Chevron spokeswoman.
It had been left "out of action," while another tank was damaged.
First Minister Carwyn Jones offered his sympathy to bereaved families and workers, adding: "I am shocked to learn of the accident at the Chevron refinery in Pembroke".
Welsh Secretary Cheryl Gillan added her condolences to families and colleagues of the dead.
"We've been in contact with Chevron and Milford Haven Port Authority and asked to be kept fully informed about the investigation into this tragic event," she said.
"It is essential to understand how this tragedy occurred."
'No ongoing risk'
Assistant Chief Fire Officer Chris Davies said there was no ongoing risk to health to members of the public as a result of the incident.
Continue reading the main story The UK has eight major operating refineriesThere were 19 oil refineries in the UK in 1975The UK has the fourth largest total refining capacity in the EUMost of the UK's refineries are based in coastal areas for ease of crude tanker accessUK refineries supply over 90% of the inland market demand for petroleum productsAbout 130m litres of petrol and diesel are sold in the UK each daySource: United Kingdom Petroleum Industry Association"We can confirm that any material released into the atmosphere as a result of the blast was immediately dispersed. The wind was blowing off shore, away from residential areas," he said.The Health and Safety Executive said it was too early speculate about the cause of the blast.
Milford Haven coastguards reported a "blast large enough to shake the windows" with "black smoke in the air" shortly after 1820 BST.
The Mid and West Wales Fire and Rescue Service sent 10 engines to the scene and brought the incident under control within an hour and a half.
Wales Air Ambulance said a man had been airlifted to Morriston Hospital in Swansea with severe burns.
The refinery was formerly known as Texaco, later rebranded Chevron-Texaco and known as Chevron since around 2005.
In March, Chevron confirmed Texan oil company Valero Energy had agreed to buy the refinery for $730m (?446m) and pay a further $1bn (?611m) for the stocks of oil, petrol and other products on site.
Workers laid floral tributes at the refinery on Friday The site, which can refine 220,000 barrels of crude oil a day into petrol and other products, has yet to change hands.
Earlier this year the sale of the refinery, which is one of the largest in western Europe and employs 1,400 people, was secured.
Valero agreed to buy the refinery site for $730m (?458m) and another $1bn (?611m) for assets including Chevron's petrol stations in the UK and Ireland.
There have been previous incidents at the refinery.
The most notable was in July 1994, with an explosion and fire at refinery, when 26 workers were slightly injured and homes as far away as Milford Haven damaged.
However Simon Hart, Conservative MP for Carmarthen West and South Pembrokeshire, praised the plant's safety record.
"It's a really important business but it is a dangerous one and it's had a fantastic safety record," he said.
The refinery is close to the Pembrokeshire coast 
President Ahmadinejad was unhurt in the blast At least one person has been killed and 20 injured in a blast at an Iranian refinery around the time of a visit by Iran's president, local media say.