Josh Scowen scored from the penalty spot early in the second half before Paddy Madden levelled matters with his spot-kick shortly after.
Over 120 minutes were played before a penalty shoot-out decided the evening.
The Iron went into the game having made four changes to the side that lost their League One opener last Saturday at Burton Albion. Sean McAllister, David Mirfin, Darius Henderson and Joe Anyon replaced Luke Daniels, Luke Williams, Niall Canavan and Tom Hopper.
The first half was one of missed opportunities for both sides, including a missed penalty for the visitors. Madden had the best opportunities for the Iron, but Nick Townsend was equal to each of the United attempts.
The Iron forged a half chance with four minutes on the clock, as good link-up play down the right wing between Scott Wiseman and Jordan Clarke saw the latter fire in a dangerous cross, but Barnsley dealt with it well.
Moments later Scott Laird whipped in a dangerous cross-cum-shot but the Barnsley 'stopper Townsend did superbly well to hold on to the ball, before clearing upfield.
The first major chance came with 11 minutes on the clock, as Stephen Dawson laid on a perfect defence cutting ball to Henderson – whose curled shot was palmed away smartly.
Barnsley were awarded a penalty with 14 minutes of the match gone, as Jack King tripped Scowen on his surge into the United box. The Iron centre-half had no complaints.
But Anyon, who was making his first competitive start for Scunthorpe, was to be the hero of the moment – as he saved Marley Watkins’ shot low to his left and parried it away for a corner.
With 24 minutes gone, Paddy Madden worked himself an opening as he fired from 30 yards out – but his low shot never troubled Townsend on this occasion.
15 minutes before the break Barnsley had a half chance, as a great ball was whipped in from the right and met on the volley by Watkins – but he couldn’t keep his shot down.
Straight up the other the end was a chance for United from a free kick in a dangerous position. Laird’s first shot hit the wall before his follow up was deflected for a corner. The resulting corner saw McAllister fire high and wide.
No more than two minutes later Madden had a wonderful opportunity gifted to him by the Barnsley defence, but his slow shot only stung the palms of Townsend, who once more did well to hold on to the initial shot.
Another superb chance fell for United not long after the last. This time Henderson’s low shot was palmed away and somehow managed to evade the oncoming Madden. After a mad scramble, Barnsley cleared.
Bree then went close for the visitors as his surging run resulted in a powerful shot, but the ball kept rising.
With the break drawing closer, the Iron had a goal disallowed for offside, as Laird’s low cross was put away by Madden.
Townsend was then called into action to superbly tip away a long-range effort from Madden. The Barnsley back line dealt with the resulting corner once more.
The scores remained goalless at the half time interval, but United would count themselves unlucky to have not taken the lead.
Straight after the break Barnsley had their second penalty, as Anyon brought down the surging frontman. Scowen would not make the same mistake as his teammate had in the first half, firing low into Anyon’s left hand corner, and the score was 1-0 to the Tykes.
No more than two minutes later, the Iron had a penalty of their own as two United bodies looked to go down in the box and was deemed to be shoved by one of the Barnsley defenders.
Madden stepped up to take it, and he smashed it straight down the middle – scoring his first goal of the new campaign in the process to level the scores at 1-1.
With 64 minutes on the clock, Sam Winnall made his first return to Glanford Park as he replaced Wilkinson for the visitors. Henderson was then replaced by Tom Hopper for the Iron.
Laird would force another smart save out of the Barnsley keeper a few minutes later, as a cleared corner fell perfectly to him.
With 15 minutes to go another huge opportunity fell to the Iron, courtesy of a perfectly whipped ball from the left, which somehow evaded everyone in the box.
Next it was Barnsley who were on the attack, but a smart chipped ball towards the back post had just a bit too much on it for the oncoming striker. Hakeeb Adelakun would come off the bench to replace Wiseman.
Somehow the Iron were to avoid going ahead once more with just over ten minutes to go, as Dawson crossed, Tom Hopper’s slip and eventual shot squirmed into the middle towards the oncoming Madden – but he mis-hit the shot and let Barnsley off the hook.
Kevin van Veen came on to replace Dawson with six minutes to go, and he almost found the net not long after his introduction, as he powered through the middle, taking on a Barnsley defender before forcing Townsend to make another vital stop.
United nearly won it in added time, but Madden’s low drive was dragged agonisingly wide, and the scores remained at one a piece.
That would be the last chance of the match, and with the scores level at 1-1 the match went into extra time.
The first chance of the first half of extra time fell to van Veen, but his driving run and eventual shot was blocked and cleared comfortably by the Barnsley defence.
Barnsley then went up the other end and won themselves a corner, but the Iron cleared with ease before McAllister was felled on the edge of the United box.
Just before the end of the first period of extra time, Hourihane broke the offside trap and ran through one-on-one – but his low shot was weak and gave Anyon in the United goal a chance to get down and gratefully get a grasp on the ball.
And that would be the best, and last chance of the first half of extra time – with the scores remaining at 1-1.
Hopped created the Iron’s first opportunity of the second period of extra time, but his curled cross from the left had too much on it for Madden, with Adelakun failing to keep the ball in at the by-line.
A blocked shot from Mirfin gave the visitors a corner with five extra time minutes to go, but it was defended well and out for another corner. This time it was cleared properly, and the Iron went on the hunt for a winner themselves – but Laird’s cross evaded everyone once more, before Adelakun whipped in, but once more it had just a little too much on it.
With two minutes left Clarke let rip from 40-yards out and it had Townsend scrambling, but the keeper managed to parry it out for a corner. Sadly for United, the corner came to no avail.
Despite late pressure from the visitors, the Iron stood strong at the back and the match needed to be decided on penalties.
Madden stepped up first for United, and made no mistake – firing hard and down the middle, making it 1-0 to the Iron on penalties.
Winnall was the first for Barnsley, and he dispatched with ease – low to the keeper's left.
Hopper stepped up next, and scored – sending the keeper the wrong way.
Hourihane then scored for Barnsley to level it once more, this time at 2-2.
Adelakun was the third United player to take a penalty, and he left Townsend with no chance – despite the keeper guessing the right way.
Scowen then did what he did in the second half and scored his penalty.
Laird then sent the keeper the wrong way to give the Iron the lead.
Mawson was next for the visitors, and he too left Anyon with no chance, and it was 4-4.
Bishop then coolly slotted away the fifth penalty, and Barnsley needed to score to stay in the tie.
And the man to keep his tie was Roberts, and he made no mistake too.
van Veen stepped up next, and he missed, leaving the Iron on the brink of elimination from the Capital One Cup.
Pearson was the man to send his team through, and he missed also – Anyon saved to his right, with a superb stop.
Mirfin then left the Barnsley stopper with no chance to restore the Iron’s lead.
Nyatanga then scored to keep Barnsley in it, and despite Anyon getting close it went in.
McAllister then fired over for United, and Barnsley had another chance to send United out.
This time, Barnsley scored – and United were knocked out of the Capital One Cup in the first round.
SCUNTHORPE: Anyon, Clarke, King, Mirfin, Wiseman (Adelakun 70), Dawson (van Veen 83), McAllister, Bishop, Laird, Henderson (Hopper 69), Madden. Subs not used: Watson, Goode, McSheffrey, Wotton.
BARNSLEY: Townsend, Roberts, Mawson, Nyatanga, Bree, Scowen, Pearson, Rothwell (Hourihane 89), Smith (Templeton 100), Watkins, Wilkinson (Winnall 64). Subs not used: Davies, Digby, Maris, Williams.
ATTENDANCE: 3,003 (734 away).