The academics say there is no evidence the move would raise maths standards.
The warning comes as thousands of Year 6 pupils (aged 10 and 11 years) sit maths tests on Wednesday and Thursday.
But Schools Minister Elizabeth Truss said children needed to be confident with maths skills "before they pick up a calculator".
Until this year, the Key Stage 2 national curriculum tests, often known as Sats, taken at the end of primary school included a mental arithmetic paper, one maths paper where calculators were used and one where they were not allowed.
But in November 2012, Ms Truss announced that calculators would be banned in maths tests at Levels 3 to 5 from this summer.
However, calculators can be used for Level 6 papers which are sat by a small number of high-achieving pupils. The limit on the use of calculators was intended to make sure that children learned maths skills for themselves rather than relying on the help of a calculator. The government has prioritised improving maths skills - with the UK having been ranked in 26th place in the most recent international Pisa tests.
But academics at Oxford University, Cambridge University and King's College London have challenged the principle that stopping the use of calculators will be beneficial in maths lessons. Anne Watson, emeritus professor of mathematics education at Oxford University, said: "There is a substantial amount of good evidence on calculators in schools, mainly from the US, and none of it shows their use is detrimental to pupils' learning.
"In fact, students who use calculators regularly in lessons score as high or higher in tests, taken without calculators, compared to those who do not.
"On the whole, the use of calculators as an integral mathematical tool has been shown to be beneficial, particularly in the development of mathematical problem solving.
"It is a pity that current policy is retrogressive in this respect."
Jeremy Hodgen, professor of maths education at King's College, London, said: "The evidence suggests that in primary school the use of calculators is beneficial provided children are taught to use calculators alongside other methods.
"Indeed, when taught with calculators, children's understanding and fluency increased and they used calculators less." Terezinha Nunes, professor of educational studies also at Oxford University said: "Removing national tests where pupils can use calculators will place greater emphasis on the testing of calculation skills and less on the assessment of mathematical reasoning. I think one can safely say that is a step backwards.
"Research shows children's achievement in maths is influenced by both their ability to do calculations, that is sums, and their competence in mathematical reasoning, that is knowing how to solve problems to do with relative quantities, such as weight, volume and distance."
But the government has rejected the argument - saying that in high-performing education systems such as Singapore, Hong Kong and Massachusetts in the US there is a recognition that "calculators should not be used as a replacement for basic understanding and skills".
"All children should be confident with addition, subtraction, times tables and division before they pick up a calculator," said Ms Truss.
"It is vital that children have a solid grounding in the basics so they can grow up to be comfortable with the maths they need in their adult lives. Banning calculators in primary school tests will help end the culture of reaching for a calculator at the first sign of a tricky sum.
"Some of the world's top education systems already do this and there is no reason why children in England can't compete with the best. Ensuring children leave primary school with a strong grasp of mathematics is a vital part of a long-term economic plan to safeguard this country's future."