Mr Kostunica said the government no longer had a united policy over Kosovo – which proclaimed independence from Serbia on 17 February – and on whether Serbia could join the EU without the breakaway province being part of the country.
He suggested that early elections be held on 11 May, when local elections are also scheduled to take place.
The Serbian president, the pro-Western moderate Boris Tadic, accepted that the government could no longer function, but said EU integration rather than the future of Kosovo was the issue on which the coalition could not find a common stance, adding that he too opposed Kosovo's independence.
"I believe that the government has no united policy over the European and economic perspective of Serbia and her citizens. I am also convinced that we will best defend our Kosovo from independence precisely with membership in the European Union," Mr Tadic said in a statement, reported by Serbian media.
"If we join the EU, then we can make sure that this outlaw state [Kosovo] never becomes an EU member," he also said in a TV show, according to Reuters.
Following Mr Kostunica's resignation, Serbia is heading for another election likely to be decisive for its future, in particular concerning its path towards the EU.
It will be a race primarily between Mr Tadic's Democrats and the nationalists of the Serbian Radical Party (SRS) – currently the largest party in parliament.
In the January 2007 elections, the SRS won but did not gain a sufficient majority to form a government.
Consequently, the main political parties' leaders became deadlocked in coalition talks for almost four months until the current government was formed. It was lead by Mr Kostunica and contained his Democratic Party of Serbia (DSS), Mr Tadic's Democrats (DS), as well as the moderate nationalist New Serbia (NS), and the liberal conservative G17 Plus party.
But last month, the Radical candidate Tomislav Nikolic was defeated by Mr Tadic during presidential elections which were overshadowed by Kosovo's imminent declaration of independence and seen as a referendum on the country's EU future.
The next vote, expected to take place in May, will also be "a referendum on Serbia's European perspective — or on its isolation," Serbian defence minister Dragan Sutanovac, vice president of Mr Tadic's party was reported as saying by the Associated Press.
Meanwhile, EU's foreign ministers will today in Brussels discuss the regional situation in the western Balkans and are set to reaffirm the countries' "European perspective".