Stars, elegance, wit return to Broadway Aren't We All? Comedy by Frederick Lonsdale. Starring Rex Harrison, Claudette Colbert, Lynn Redgrave, George Rose, Jeremy Brett. Directed by Clifford Williams.

At long last, a rare theatrical treat for Broadway playgoers -- a star-spangled pi`ece de r'esistance. Rex Harrison and Claudette Colbert, who made comedy `a la carte of ``The Kingfisher'' in the 1978-79 season, are reunited in Frederick Lonsdale's ``Aren't We All?'' It's a reunion to celebrate. Lynn Redgrave, George Rose, and Jeremy Brett complete the galaxy in the choice company directed by Clifford Williams, who performed the same office for the London hit revival. The production at the Brooks Atkinson Theatre is a nostalgia trip on gossamer wings to the airy comedy fancies of the 1920s. (``Aren't We All?'' opened in both London and New York in 1923.)

Lonsdale's delicately balanced plot begins unfolding when Margot Tatham (Miss Redgrave) arrives home unexpectedly from four months in Egypt to discover husband Willie Tatham (Mr. Brett) in an ardent clinch with Kitty Lake (Leslie O'Hara). By the time the comedy has moved from the Tathams' Mayfair drawing room to the stately country home of Willie's father, Lord Grenham (Mr. Harrison), his lordship has learned that Margot came close to a wifely indiscretion in the land of the Nile.

Lonsdale's slight tissue of plots provides the pretext for a genially satiric view of life amid the rich and titled. Principal among them are the wise but cynical Grenham, a widower with a past, and Lady Frinton (Miss Colbert), a widow whose cap is gently but firmly set for the old rascal.

Mr. Harrison and Miss Colbert play these delightful elders with autumnal wisdom and springtime freshness. Mr. Harrison, whose range was demonstrated most recently hereabouts as Shaw's Captain Shotover in 1983, is in top light-comedy form as Lord Grenham. An actor who can get a laugh from even the syllabic intonations of ``ab-so-lute-ly,'' Mr. Harrison handles the Lonsdale aphorisms with a witty and deceptive offhandedness. He accentuates Grenham's joie de vivre with an occasional skipping step or hand wave, insouciant touches to the comic portrait. Miss Colbert matches him all the way. Her beauty, mischievous style, and burnished contralto are ideally suited to the future Lady Grenham.

The mellow maturity of ``Aren't We All?'' is counterpointed in the Tathams' emotional ordeal -- painful enough for them but treated for purposes of laughter. Mr. Brett is a dab hand at the Lonsdale style, combining the nuances of vocal precision with the conviction that is at the heart of all true comedy. The strikingly handsome Miss Redgrave makes a believable transition from the righteous wifely outrage of Act I to panic at the threatened exposure of her regretted momentary fling.

From stars to supporting players, style is the essence of the glistening revival staged by Mr. Williams. Mr. Rose's petulant Rev. Ernest Lynton personifies the sanctimony that prompts his kindly wife (Brenda Forbes) to observe: ``If Ernest could only humbug others as he humbugs himself, he would be a bishop.'' Ned Schmidtke stalwartly fulfills Lord Grenham's definition of a gentleman (in an era when the term meant something). The aides-de-camp of drawing-room comedy are well but not campily played by Miss O'Hara as the temptress, Peter Pagan and George Ede as the indispensable butlers of the occasion, and Steven Sutherland and Patrick Hurley as a pair of Mayfair masqueraders.

The production has been handsomely designed by Finlay James, whose paneled scrim curtain pictures the excursion from London to a stately English home of which ``Masterpiece Theatre'' could be proud. Judith Bland's costumes include some smashing creations for the ladies and Savile Row tailoring for the gentlemen. The lighting is by Natasha Katz.

The matter of ``Aren't We All?'' is slight. The manner is sleight of hand. But there is something beyond the rippling small talk and brittle trivia. It is warmhearted as well as lighthearted, a comedy about forgiveness and understanding, tolerance and generosity. In a season of Broadway's discontent, one welcomes Miss Colbert, Mr. Harrison, and company. For this comic relief, much thanks!