It can seem like Hollywood is all glitz and glamour, but while the idea of being famous is appealing to many of us, a lot of real, hard work goes into movies behind the scenes. That work can test a person’s limits not just as an actor but as a human being. Here are some times when actors almost bailed on their plum movie or TV roles simply because they just couldn’t take it anymore.

Lauren Cohan couldn’t ‘c’ the point

The Walking Dead nearly suffered what would have been a devastating loss in Season 3, when Lauren Cohan, who plays Maggie, contemplated quitting over a particularly stressful scene. After reading the script and realizing Maggie was supposed to perform an emergency c-section on Lori Grimes (while the character’s son looked on), she took her concerns to co-star Steven Yeun, explaining she was worried it would be too intense—yet after further consideration, that’s exactly why she decided to stay. Recognizing that TWD is a show about people pushed to grueling extremes, she drew on her palpable discomfort to deliver an affectingly honest performance.

Gandalf the Grey couldn’t take the green

It’s almost impossible to imagine the role of Gandalf being played by anyone other than Sir Ian McKellen, who put his indelible stamp on the character in all six Lord of the Rings and Hobbit films. But if things had worked out a little differently during the filming of The Hobbit: An Unexpected Journey, director Peter Jackson might have had to find a different actor for the part.The problem? McKellen was acting almost exclusively to a green screen, owing to the fact that his character towered over the hobbits and their interactions had to be stitched together digitally in production. The frustration of acting alone eventually got to McKellen, who broke down on the set at one point, but Jackson did a little emergency outreach to convince him to stick around: The crew held a "Gandalf Appreciation Day," and decorated his tent with Lord of the Rings mementos.

They made Al Pacino an offer they wanted him to refuse

Al Pacino earned a Best Supporting Actor Oscar nomination for his work as Michael Corleone in The Godfather and a Best Actor nomination for the sequel, so it’s laughable to think studio execs initially wanted him removed from the role—yet according to Pacino, that’s exactly what happened. He felt resistance so acutely, in fact, that he later claimed he would have walked off the set if director Francis Ford Coppola hadn’t been so supportive during those early days on the Godfather shoot. "I was so confused at that time, and Francis was so supportive, you know, and so helping me in it, all of it," Pacino told Nightline. "If it wasn’t for Francis, I would’ve just not showed up one day and said, ‘Hey, look man, I don’t want to be where I’m not wanted.’"

How a military trainer saved The Grinch

Jim Carrey nabbed the role of the Grinch in Ron Howard’s live-action adaptation of Dr. Seuss’s How the Grinch Stole Christmas after making it through an audition process that included earning the approval of Audrey Geisel, Dr. Seuss’ widow and defender of his estate, but those obstacles were minor compared with what he faced on the set. He had a particularly hard time in the makeup chair, where he spent three and a half hours a day during the lengthy shoot—and had to be fitted with what producer Brian Grazer called "green, Frisbee-like contact lenses."

At one point, Carrey told Grazer, "I can’t do this any longer, I’m gonna give all the money back" because the experience was "torturous." Grazer didn’t let him quit. Instead, he called in a friend who trained Green Berets and CIA operatives to spend a weekend with Carrey. After his session with the military trainer, Carrey was on board to make The Grinch.

Jessica Alba didn’t think the Fantastic Four movies were so fantastic

You think you hated Fox’s first couple of Fantastic Four movies? Try being Jessica Alba, whose experiences playing Sue "Invisible Woman" Storm nearly made her walk away from acting altogether. Alba’s frustrations stemmed from working with director Tim Story, who she later claimed issued absurd demands like telling her to keep her facial expression flat during a crying scene so she’d stay "pretty." After 2007’s Rise of the Silver Surfer, she realized she didn’t care about the movie business anymore, and has been more selective about acting projects. But things have still worked out pretty well for her—she went on to co-found the Honest Company, currently valued around $1 billion.

The vampire movie that was the pits for Pitt

Interview with the Vampire united Tom Cruise and Brad Pitt in a visually sumptuous adaptation of the bestselling Anne Rice novel, but being part of 1994’s most eagerly anticipated beefcake summit wasn’t all it was cracked up to be for Pitt. Years after the movie came and went, Pitt admitted he was so miserable during filming that he actually approached producer David Geffen about buying out his contract—only to reconsider after Geffen informed him it’d cost $40 million.

Not Nottingham

Nine times out of 10, an actor who signs up to play a major role in a movie can at least be counted on to identify with the basic crux of the story. For the exception to that rule, we turn to Russell Crowe, who agreed to star in Nottingham even though he thought the concept—a role-reversal on the Robin Hood story, with the Sheriff of Nottingham turning out to be the real good guy—was fundamentally flawed. Demanding a complete overhaul of the script, Crowe ended up costing the studio millions in extra development, and the result was the retitled Robin Hood, yet another standard re-telling of the Robin Hood story.

Daisy Ridley’s career was almost over before it even began

Starring in one of the most anticipated movies of the century is an opportunity most actors would kill for. But the pressure of filming 2015’s Star Wars: The Force Awakens nearly proved too much for Daisy Ridley. The British newcomer didn’t have much experience before J.J. Abrams cast her as Rey, and on the first day the director told her that her performance felt "wooden." That note, along with bearing the weight of the biggest franchise in movie history, triggered a panic attack. "I honestly wanted to die," Ridley later admitted. "I thought I was gonna cry, I couldn’t breathe."

No ‘Bohemian Rhapsody’? Denied!

Mike Myers created the character of Wayne Campbell for Canadian TV, later bringing Wayne with him to Saturday Night Live. The recurring sketch became so popular that Paramount wanted a Wayne’s World movie. Myers wrote the screenplay, and he was especially excited about the scene where Wayne and the boys pack into the Mirthmobile and headbang to Queen’s "Bohemian Rhapsody." But the studio wanted to replace the song with something fresher and cheaper to license like a Guns n’ Roses song. Myers thought the Queen song was crucial, and revealed on WTF with Marc Maron that he threatened to quit unless the song stayed. Both did.

The shoot of Hitchcock’s The Birds was, as they say, for the birds

Alfred Hitchcock had a reputation for being unkind to actresses, but Tippi Hedren signed on to 1963’s The Birds under the guarantee that she’d only be attacked with mechanical birds during the film’s climactic scene. But Hitchcock is the king of twists, and he wanted Hedren’s terror to feel real, so he instructed crew members to throw real, live birds at her. This real-life game of Angry Birds went on for five full days and Hedren had a breakdown when it was over. She only finished the movie when a doctor ordered Hitchcock to give her a week off to recuperate.

Even John Cena has his limits

WWE produced 12 Rounds and cast some of wrestling’s biggest names in the 2009 direct-to-video cop thriller – and few were bigger than star John Cena. Although his day job is to battle hulking beasts in the ring, Cena apparently isn’t fearless. One scene called for his character to rappel down a skyscraper and dangle mid-air from a rope. But Cena revealed in a DVD extra that his severe fear of heights triggered intense anxiety and he considered walking away before he ever got up there. Somehow, he found the strength to put on his game face, and he shot the scene without incident.

Stanley Kubrick just about broke Shelley Duvall’s spirit

Shelley Duvall set a Guinness World Record for her portrayal of Wendy in Stanley Kubrick’s 1980 horror classic The Shining – but she didn’t know she was doing it. The perfectionist Kubrick insisted on 127 takes of the scene during which Duvall’s hysterical character uses a bat to fend off her deranged husband (Jack Nicholson). About 100 takes in, she was literally drained: she was exhausted and dehydrated from crying so much. Although she stuck around, the stress didn’t subside. In a 2016 interview on Dr. Phil about her mental health battles, Duvall said that the stress from shooting The Shining made her hair fall out.

Carl Weathers didn’t appreciate all the violence in a boxing movie

A big part of the fourth Rocky movie involves Rocky’s mentor, Apollo Creed, staging a comeback. Played by former NFL linebacker Carl Weathers, Creed gets roughed up in the ring pretty bad by Communist pugilist Ivan Drago, played by Dolph Lundgren. But one of the boxing scenes got a little too real when Lundgren, according to co-star and director Sylvester Stallone, picked up Weathers and heaved him "three feet into the corner." Weathers was so rattled, Stallone says he climbed out of the ring and said, "I’m calling my agent, I quit!" (He eventually came back.)

Zoe Saldana’s stormy voyage

Before she landed leading roles in huge blockbusters like Avatar and Guardians of the Galaxy, Saldana was just another struggling actress getting by on bit parts. She had a small one as Anamaria in 2003’s Pirates of the Caribbean: The Curse of the Black Pearl, and she said she was treated so badly that she nearly quit. "It was very elitist," she told THR . "People disrespecting me because they look at my number on a call sheet and they think I’m not important." It was so demoralizing, Saldana said she even considered quitting acting altogether.

When The Abyss went too low for Mary Elizabeth Mastrantonio

The dark, claustrophobic sets and constantly gushing water made for some difficult days on the set of James Cameron’s 1989 sci-fi flick The Abyss. But one stands out for Mary Elizabeth Mastrantonio. While filming a scene in which Ed Harris, playing Mastrantonio’s estranged husband, administers her CPR, Cameron directed Harris to actually slap Mastrantonio — and then he had to get physical with her again when the take was ruined by Harris’ arm blocking a shot. After the next take was botched when the camera ran out of film, Mastrantonio actually got up and left the set — and she stayed away for hours, leaving Harris to play make-believe slap-and-CPR with a sandbag until she finally calmed down enough to return and finish filming the scene. Cameron’s demeanor with actors has occasionally been depicted as less than friendly, but this might be the only case where a star walked away with the bruises to prove his commitment to pursue his single-minded vision.

Michelle Rodriguez was equal parts fast and furious

During production on the first Fast and Furious movie, series regular and all-around badass Michelle Rodriguez actually considered walking away over concerns with the way her character, Letty Ortiz, was written. Specifically, Rodriguez took issue with the way the script called for Letty to betray her true love, Dom (Vin Diesel), and fall for undercover cop Brian (Paul Walker)—a betrayal she felt made no sense for a woman who’d clearly prefer, in her words, "the alpha-est of alpha males" to a pretty boy. Refusing to betray her vision of Letty, Rodriguez put her foot down, telling producers she’d quit the movie if she had to. Fortunately for Fast and Furious fans, the storyline was tweaked to Rodriguez’s liking, and the rest is blockbuster history.

What was eating Johnny Depp?

In the 1993 film What’s Eating Gilbert Grape?, Johnny Depp starred as the titular Gilbert, a guy stuck in a small town caring for his developmentally disabled younger brother (Leonardo DiCaprio in a breakout, Oscar-nominated role) and their mother, housebound due to extreme obesity. Cast as the mom: Darlene Cates, an acting novice that director Lasse Hallström spotted on The Sally Jessy Raphael Show discussing her own struggle with obesity.

Depp was going through some personal stuff during the film’s shoot—he had just split up with his fiancée, Winona Ryder, and he was using alcohol to cope. Hallström later told Entertainment Weekly that Depp also empathized with his character a bit too much: "It was more that he could relate to the troubled family life and the desire to get out of a small-town situation."

All told, Depp was in such a sensitive place that he had trouble filming scenes that called for his character to yell cruel insults at his mother. Cates recalled that each night, after those scenes were shot, Depp called her to apologize. "Here he is the professional; he’s done all this stuff before. But I thought, ‘What a tribute to his kindness that he would take the time to consider that I might be hurt by what he said.’"

The one thing scarier than the Zodiac killer was the director of a movie about the Zodiac killer

While on the set of his 2002 thriller Panic Room, director David Fincher told The New York Times that he felt he "needed to be more attentive to watching the actors." He put that theory into motion on his next film, Zodiac, a period piece about the search for the "Zodiac Killer" who murdered their way through Northern California in the 1970s.

Jake Gyllenhaal, who starred as cartoonist-turned-crime reporter Robert Graysmith, bore the brunt of Fincher’s commitment to putting his actors through the ringer. On one scene, Fincher required 70 takes; on another, he asked for 90, according to Gyllenhaal. Plus, because the film was shot digitally, Fincher could spontaneously delete a slew of takes all at once. "But there is a stopping point. There’s a point at which you go, ‘That’s what we have to work with,’" Gyllenhaal said. "But we would reshoot things."

Sorry Thanos – Jeremy Renner tried to kill Hawkeye already

There’s a lot to like about Marvel’s first two Avengers movies. Certainly the most ambitious and crowded superhero movies ever attempted, they boast Earth-destroying action sequences, amazing special effects, romance, and the funny and charming Robert Downey Jr. pretty much playing himself playing Tony Stark playing Iron Man.

Marvel’s The Avengers was such a huge, ambitious project that it attracted actors who usually don’t do comic book movies, including two-time Oscar nominee Jeremy Renner as Clint Barton, a.k.a. Hawkeye. While some of the more famous heroes in the movie (Iron Man, the Hulk, Captain America) got to do more interesting things, Hawkeye spent most of The Avengers as the brainwashed servant of villain Loki (Tom Hiddleston). Renner was displeased. "I was just getting to know who Hawkeye was, and then zap, I go round like a zombie," he said at a Q&A event. "I’m like Loki’s minion." Renner took it like a champ — just kidding, he sulked and whined. "I just had a heart attack in every scene," Renner explained. "I’d just be walking with Scarlett Johansson and just be like ‘Ugh!’" Renner said he told the crew that he was providing an easy out for Hawkeye. "Just you know, at any given moment, if you wanna kill me off, daddy’s gonna be having a heart attack."

Working 9 to 5 was not how Lily Tomlin wished to make a living

Before Horrible Bosses, there was 9 to 5. This 1980 dark comedy centered on three overworked, overlooked female secretaries enacting brutal revenge on a male chauvinistic pig of a manager (Dabney Coleman). Dolly Parton, Lily Tomlin, and Jane Fonda headed up the cast, with Fonda also producing. A week after filming began, Tomlin tried to quit. "She asked my producing partner to let her go and she’d give the week’s money back," Fonda said during a 2017 joint interview with Tomlin on The Late Show with Stephen Colbert.

What set off the legendary comic actress? Tomlin watched the raw footage of herself and was embarrassed by her performance, particularly the scenes for a Snow White-inspired animated fantasy sequence where he acted opposite a yet-to-be-created cartoon bird. "I was talking to the birds that weren’t there and I was not doing a good job," Tomlin said. "And I thought, ‘Oh I’m going to be horrible in this.’" Fortunately, she changed her mind — and the rest was cinematic history.

Roger Moore hated guns as much as James Bond loved them

Shooting a gun is one of James Bond’s favorite activities, trailing only drinking shaken (and not stirred) martinis, wearing tuxedos, and bedding suggestively named women. In his duties in service of queen and country, killing bad guys and foiling terrorist plots, 007 needs to use a firearm on occasion, and one of the handful of actors who portrayed Bond onscreen had a really hard time with those moments. Roger Moore starred in seven 007 movies, including The Spy Who Loved Me, Moonraker, and, ironically, The Man with the Golden Gun. While he doesn’t regret playing James Bond, Moore once said that what he really regretted is how "heroes in general are depicted with guns in their hands, and to tell the truth I have always hated guns and what they represent."

It all goes back to his childhood. In his memoir My Word is My Bond, Moore said that he was rendered deaf for a few days as a boy when a gun "blew up" in his hands. A few years later, a friend shot a BB gun into the future star’s leg. When he’d have to use a firearm onscreen, Moore would squeeze his eyes tightly in discomfort as he’d pull the trigger on his prop Walther PPK firearm. By the time he filmed his final Bond movie, 1985’s A View to a Kill, he publicly lamented that the film series had "turned so violent."