In mid-1980s South Boston, Irish Mob boss Frank Costello introduces himself to a young Colin Sullivan. Twenty years later, Sullivan has been groomed as a spy inside the Massachusetts State Police (MSP) and joins the Special Investigation Unit (SIU), which is building a case against Costello. A police academy recruit, Billy Costigan Jr., is selected by Captain Queenan and Staff Sergeant Dignam to go undercover and infiltrate Costello's crew due to his family ties to organized crime, with his deceased uncle Jackie having been connected with Costello's outfit. SIU Captain George Ellerby informs Special Investigations that the identities of undercovers will be known only to Queenan and Dignam to prevent leaks.
Costigan serves a term in prison to set up his cover and further commits a series of crimes in an effort to draw the attention of Costello and his enforcer Arnold French, who later recruit Costigan into the organization due to their affection for his uncle. Over the next year, Costigan becomes increasingly involved in the operation. Due to the stress of having to maintain his cover, his emotional and mental state declines, and he threatens to quit, but Queenan and Dignam convince him to continue. Costigan's only other contact in the MSP is his court-ordered psychiatrist, Madolyn Madden, who is coincidentally Sullivan's girlfriend. Costigan and Madden later have an affair.

Queenan and Dignam tell Costigan that Costello's crew is selling stolen computer microprocessors to China. The MSP sets up surveillance, but Sullivan tips off Costello, allowing everyone to escape. The MSP and Costello realize there is a mole in their respective ranks and task Costigan and Sullivan to find them. Meanwhile, Costigan learns that Costello is a protected FBI informant. Costigan shares his discovery with Queenan and Dignam while warning them that Costello is aware of a spy in his organization.

Late one night, Costigan follows Costello into an adult theater and witnesses him giving Sullivan an envelope containing information of his crew for Sullivan to cross reference with police records. Queenan instructs Costigan to get a visual ID of Sullivan but he is unable to get a good look at his face. When Sullivan realizes that he is being followed, he mistakenly stabs a restaurant worker and flees. Later, Sullivan tries to cross-reference Costigan's picture, captured by nearby security footage, against police officer databases but cannot identify him. Queenan advises Sullivan to follow Costello to find the MSP mole. Fearing that he will soon be discovered and killed, Costigan calls Queenan and sets up a meeting to abort the operation, but Sullivan has Queenan followed, lying to the other officers that Queenan may be the spy. Sullivan also calls in Costello's gang to the meeting location.

When Costello's men arrive, Queenan helps Costigan escape before being thrown from the building's roof to his death. This causes a firefight between SIU agents and Costello's men. Angered by Queenan's murder, Dignam attacks Sullivan and is suspended by Ellerby. Timothy Delahunt, one of Costello's henchmen who was mortally wounded in the gunfight, tells Costigan that he knows he is the rat, but just before he succumbs to his wounds, Delahunt implies that he is also working for law enforcement. A news report reveals that Delahunt was an undercover officer for the Boston Police Department, but Costello suspects the police department made up the claim so he would stop looking for the mole. Sullivan looks through Queenan's belongings and learns that Costello is an FBI informant after reading Queenan's notebook. Deciding to turn on him, Sullivan directs the MSP to tail Costello to a cocaine drop-off, where a gunfight erupts, killing most of Costello's crew. Sullivan confronts a wounded Costello, who admits to being an FBI informant. They exchange gunfire, and Sullivan kills him.

His assignment finished, Costigan goes to Sullivan to reveal his undercover status, still unaware of the other's true identity. After Sullivan goes to another room, Costigan notices the same envelope from the theater on his desk. Costigan ascertains that Sullivan was Costello's mole, and escapes. Shortly after, Sullivan returns to find Costigan gone, and realizes that he now knows his true identity. He erases Costigan's records from police computers. Costigan visits Madolyn and hands her an envelope, instructing her to open it if something happens to him. Later, she opens a package Costigan sent to Sullivan and listens to tapes Costello made of himself with Sullivan, causing her to leave Sullivan. In order to retrieve his records, Costigan arranges to meet Sullivan on the rooftop where Queenan was killed, then arrests him. Costigan calls Trooper Brown, a friend from the police academy, to substantiate his identity, but Brown pulls a gun on Costigan when he arrives, unsure who is telling the truth.
Costigan says that he has evidence tying Sullivan to Costello, and Brown lets him go down the elevator. Upon reaching the lobby, Costigan and Brown are killed by Trooper Barrigan, a friend of Sullivan's who reveals himself to be another spy working for Costello. Sullivan shoots Barrigan dead, allowing him to out Barrigan as the mole while removing suspicion from himself. When Sullivan arrives home, Dignam is waiting for him inside and shoots him in the head before he leaves. A rat runs across the rail of the balcony with a view of the Massachusetts State House in the background.