New trend in mysteries: crumbling of the small world of murder; Death of a Minor Character, by E. X. Ferrars. Garden City, N.Y.: Doubleday & Co. (published for the Crime Club). 1983. 183 pp. $11.95.; In at the Kill, by E. X. Ferrars. Garden City, N.Y.: Doubleday & Co. (published for the Crime Club). 1979.; Last Will and Testament, by E. X. Ferrars. Garden City, N.Y.: Doubleday & Co. (published for the Crime Club). 1978.; Frog in the Throat, by E. X. Ferrars. Garden City, N.Y.: Doubleday & Co. (published for the Crime Club). 1981.; Thinner Than Water, by E. X. Ferrars. Garden City, N.Y.: Doubleday & Co. (published for the Crime Club). 1982.; The Small World of Murder, by E. X. Ferrars. Garden City, N.Y.: Doubleday & Co. (published for the Crime Club). 1973.; Innocent Blood, by P. D. James. New York: Charles Scribner's Sons. 1980.; Master of the Moor, by Ruth Rendall. New York: Pantheon Books. 1982.

The cozy little world of the classic British mystery novel is breaking down and may eventually collapse altogether. Ruth Rendall and P.D. James, both of whom have been compared to Agatha Christie, have been moving away from what their fellow mystery writer E. X. Ferrars has called ''The Small World of Murder ,'' the title of one of her novels. This small world comprises little English towns and villages and ordinary people from all walks of life whose lives are placed under the microscope of investigation when someone they know is murdered. And the sleuths - the policemen, the private detectives, and the amateurs - are themselves ordinary people.

P. D. James sets up the little world of Detective Chief-Inspector Adam Dalgliesh of Scotland Yard and Cordelia Gray, girl private detective, in seven excellent mystery novels. In ''Innocent Blood'' she abandoned the mystery format and its little world and wrote a psychological novel about incest, revenge, and madness. In her most recent novel, ''The Skull Beneath the Skin,'' she returned to the mystery format and to her young sleuth, Cordelia Gray. This novel is different from the earlier mysteries, however. It is gruesome and macabre in its subject matter, tinged with the themes of ''Innocent Blood.'' Cordelia Gray, that heroine of the small world of murder, seems out of place in the distasteful plot.

Ruth Rendall has written more than 20 novels, 12 of them constituting her Chief Inspector Wexford series. The Wexford novels are among the best in the mystery genre. Miss Rendall portrays the little world and its inhabitants shrewdly and adeptly and constructs complicated, twisting plots. The Wexford characters are allowed to change and grow while enriching, not damaging, their little world. The Rendall hallmark is exposing the hidden guilt in every suspect , and this psychological slant is evident in all her mystery novels to some degree. Lately it seems to be taking over the non-Wexford novels, and, like P. D. James, Rendall is delving into the distasteful aspects of the human psyche.

E. X. Ferrars has written more than 40 mystery novels and has been honored by the British Crime Writers Association for continuing excellence in the mystery field. In her latest novel, ''Death of a Minor Character,'' I sense that she, too, is beginning to give up on the small world of murder.

In ''Death of a Minor Character'' she returns to the English town of Allingford, home of Virginia Freer, a typical Ferrars character. In Allingford, about an hour from London by car, Virginia lives in a pleasant house with a garden and works part-time in a private clinic as a physiotherapist. She is an attractive, intelligent, capable, woman of 41 with a highly developed sense of duty - almost like a younger Margaret Thatcher.

The most interesting aspect of the character, and the one most out of character, is her attraction to her estranged husband, Felix. She and Felix separated six years earlier after three years of marriage. Felix is also 41, attractive, and intelligent. An amoral, irresponsible, charming egotist, he is not a typical Ferrars character. He could be villain material if he were not so likable and innocent of malice. There is always a certain amount of mystery about his past, his motives, his sources of income, and his feelings for Virginia. Virginia herself is uncertain about her feelings for Felix. Although they have not divorced, Virginia is certain that she can never go back to him. Their relationship is in limbo, although a very cozy, settled limbo. They are friends, but not lovers, although it is clear that they were, and still are to some extent, very attracted to each other. Their relationship is full of possibilities. So far Miss Ferrars has not even begun to develop these possibilities.

Virginia and Felix have appeared in four other Ferrars mysteries, ''Last Will and Testament,'' ''In at the Kill,'' ''Frog in the Throat,'' and ''Thinner Than Water.'' In each of these novels, as in ''Death of a Minor Character,'' Miss Ferrars leaves her characters exactly where they were when the story opened, both settled in their cozy homes and routines, solving mysteries together, and making no decisions about their relationship. Time has stood still in all five mysteries. Virginia and Felix are always 41, always separated for six years. It appears that Miss Ferrars has no intention of allowing Virginia and Felix to grow or change. Perhaps she feels that allowing this would upset the small world of murder she has created. Ruth Rendall has proved in the Wexford novels, however, that continuing characters can change without upsetting their little world.

In the plotting of ''Death of a Minor Character'' Miss Ferrars has, intentionally or not, chipped away at the small world of murder. The typical Ferrars character plays amateur sleuth and helps the police track down the murderer when one or more members of his circle of friends and acquaintances are murdered. The plot can become very complicated, because in that circle everyone knows everyone else or is connected to everyone else in some way. When not working or sleuthing, the typical Ferrars character spends his time drinking tea , gardening, and chatting with and entertaining his friends. In ''Death of a Minor Character,'' Virginia and Felix spend quite a bit of time in these pursuits, as well as driving back and forth between London and Allingford in Virginia's car. They spend little time sleuthing.

Virginia and Felix are pulled into the investigations of two apparently unrelated murders by their friendships with the victims. Yet Felix and Virginia are almost insignificant in the solution of the murders. Has Miss Ferrars given up on the little, moral person? The police know what is going on, but not the amateur sleuths. It seems as if the evils of the world have become too much for the Freers, and only the police are equipped to handle them. This is a significant and disappointing departure from the rules of the small world of murder.

It would be easy to blame this departure on Miss Ferrars's advanced age and the great number of novels behind her. But because the same breaking down of the small world of murder can be seen in the work of P. D. James and Ruth Rendall, who are much younger women, could it be that the breaking down is attributable to outside causes? Could this crumbling of the cozy little world in fiction be a reaction to the current role of Margaret Thatcher and her Conservative government in Britain? Do these authors feel the need to depart from their little fictional world when they see a person so like a character from one of their novels running their country? Is reality causing the breaking down of the small world of murder?

Ruth Rendall and P. D. James could be going along with the current popularity of psychology and the supernatural. Perhaps the moral safety of the little world of the mystery novel is now being viewed, even by those who helped create it, as boring, predictable, and restrained.