After labouring to a 0-0 draw in 90 minutes, they lost 4-3 on penalties with Christian Eriksen and Lucas Moura missing.
Tom Lapslie scored the decisive kick for the U’s as they added another Premier League scalp to the collection following their beating of Crystal Palace – also on penalties – in the last round.
Much of Pochettino’s dialogue in the early part of the campaign has been about how his squad were unsettled, with the likes of Eriksen, Jan Vertonghen, Danny Rose, Toby Alderweireld and Victor Wanyama all uncertain of their future.
Pochettino was hoping that would correct itself after the European transfer window shut at the start of the month, but he says his side are still playing catch up.
“When you have an unsettled squad always it’s difficult and you lose time and then you need time to recover the time you lose,” the Argentinian said.
“That’s where we are. Maybe our performances are good but you need this extra, which is mental, connection, it’s energy to be all together, not to have different agendas in the squad.
“We need time again to build that togetherness that you need when you are competing at this level.
“We are in a period where it’s a bit tough for us but we keeping working to find a solution.
“When this type of thing happens it’s about staying clear and fresh and calm. We’re trying to find solutions and we only need time.”
That is no excuse for being unable to get past a side 71 places below them in the footballing pyramid as they rarely looked like scoring.
They had only four shots on target, despite ending the game with Eriksen, Moura, Dele Alli, Erik Lamela and Son Heung-min on the pitch.
“They were tough, they defended well,” Pochettino added. “Of course the difference of the level is there and if you analyse the circumstances, yes we need to create more and be more aggressive in the final third.”
It was one of Colchester’s most famous nights, even if they did not manage a shot on goal in the 90 minutes.
Goalkeeper Dean Gerken, who saved Eriksen’s spot-kick, and Lapslie were the heroes.
Boss John McGreal said: “It’s a great night under the lights, absolutely fantastic.
“It;s a fantastic night for the players and the club and even the staff.
“When you are coming up against the players of Tottenham’s calibre, sometimes on these type of nights you need a bit of luck.
“We have had a run around, we have got some tired legs in there. It’s a great occasion for us.”