Factory workers in industries that sell a large amount to the EU are particularly at risk, the IFS warned.
A number of studies have indicated that people with lower or no qualifications were more likely to vote Leave.
A government spokesperson said it had proposed frictionless trade between the UK and the EU.
“We have put forward an ambitious proposal that will allow frictionless trade between the UK and the EU, protect jobs across all sectors and deliver a successful Brexit that works in the interests of the whole UK,” the spokesperson said.
Read more here