Officials say it was probably caused by a soft raw milk cheese called Ouleout from Vulto Creamery in New York state.
The cheese was stocked by a Whole Foods shop in Fairfield, Connecticut, and may also have been available in specialised cheese shops.
The creamery recalled several soft cheeses on Tuesday.
Six cases of listeria have been recorded in Connecticut and Vermont, where the deaths occurred, as well as in New York and Florida.
The people infected range in age from less than a year to 89, and five of them are female, the US Centers for Disease Control and Prevention (CDC) says. All six were taken to hospital and the identity of those who died has not been released.
Listeria bacteria can occur in raw milk and foods made with it, and can survive refrigeration. The bacteria are killed by cooking and pasteurisation.
Vulto Creamery said it was working on recalling the affected cheese.
Raw milk is milk from cows, goats, sheep or other animals that has not been pasteurised - the process of heating the milk to a specific temperature for a specific period of time to kill bacteria.
the CDC says raw milk presents one of the biggest risks to consumers, who may face "many days of diarrhoea, stomach cramping and vomiting" and in rare cases kidney failure, paralysis, chronic disorders and even death.
The bacteria in raw milk can be especially dangerous to those with weak immune systems, older people, pregnant women and children, the CDC says.
Raw milk products are illegal in 20 US states, can be obtained from farms in 25 states and are available in shops in 13 states. EU countries make their own laws but products made with raw milk must be labelled. About a fifth of French cheese is made using raw milk.
In the UK, the sale of raw cow's milk is banned in Scotland but products made from it can be bought. Raw cow's milk can be bought from producers in the rest of the country.