Prince Williamis reportedly restarting conversation with his father,Prince Charles, aboutPrince Andrewas renewed scandal over alleged sexual abuse has ushered in fresh scrutiny of the Duke of York.

“First and foremost, Charles and William are united; there will never be a return to public life for Andrew,” a source told theMirroron Thursday.

The source added that the direct heirs to the throne, “along with members [of the royal family], are undoubtedly increasingly concerned about the effect all of this turmoil could have on the Queen.”

William, 39, was previouslyinvolved in discussions around Andrew’s step back from royal duties, so it’s no surprise the Duke of Cambridge — who is third in line to the throne behind Charles — would play a key role in decisions impacting the monarchy.

Samir Hussein/WireImage

Prince William, Prince Andrew, Prince Charles

“Senior members of the family were involved” in Andrew’s decision to"step back" from public dutiesin 2019 when the scandal first emerged, one palace source told PEOPLE at the time.

And since Virginia Roberts Giuffre filed alawsuitin New York on Monday claiming Andrew had repeatedly forced her to participate in non-consensual sex when she was under the age of 18, senior royals are increasingly concerned the fallout will create “unwelcome reputational damage to the institution” of the monarchy, a source close to Charles told theTimes of Londonon Thursday.

For his part, Andrew, 61, has consistentlyclaimed Giuffre’s accusations could not be true.

Prince Andrew; Virginia Roberts Giuffre.Alexander Koerner/Getty; Emily Michot/Miami Herald/Tribune News Service via Getty

Prince Andrew, Virginia Roberts

Prince Charles, Prince William and Prince Andrew.TOLGA AKMEN/AFP/Getty Images

Trooping the Colour

On Thursday, it was reported that Charles, 72, wasgrowing more resolvedin the decision that Andrew would not return to life as a working royal.

The Prince of Wales “long ago concluded that it is probably an unsolvable problem,” a source close to the future king told theTimes of London.

“This will probably further strengthen in the prince’s mind that a way back for the duke is demonstrably not possible, because the specter of this [accusation] raises its head with hideous regularity,” added the source.

Spokespeople for Prince Andrew have repeatedly declined PEOPLE’s request for comment, though Andrew did tell the BBC in 2019: “I have no recollection of ever meeting this lady, none whatsoever. It just never happened.”

source: people.com