The bus, which was stationary on a busy road at the time, was buried under the rubble in Gwangju, some 270 kilometers (168 miles) southwest of the capital Seoul.
Another 25 people were injured, eight of them seriously, the ministry added.
The 16 demolition workers on the site are safe and did not suffer any injuries when the building collapsed.
About 190 firefighters were deployed for search and rescue, and police joined the rescue effort, the ministry said.
The reason for the collapse was unclear, Reuters reported, citing officials at the Gwangju Fire and Safety Headquarters. The Ministry of Land, Infrastructure and Transport will investigate the cause of the collapse.
South Korea struggles with a poor infrastructure record: a collapse of a department store killed more than 500 people in 1995 and a collapse of a bridge in 1994 killed 49 people. In recent years, the country has tried to improve its safety record.