Novel of man in the Ice Age; Dance of the Tiger (A Novel of the Ice Age), by Bjorn Kurten. Introduction by Stephen Jay Gould. New York: Pantheon Books. $10.95.

Bjorn Kurten is one of Europe's foremost evolutionary paleontologists, a professor at the University of Helsinki, widely known for his scientific writings, an expert in particular on mammalian life of the Pleistocene epoch.

His book is an odd chimerical creature, part roman a clef,m part novel of ideas, part mystery-riddle, part treatise on Ice Age anthropology.

It includes a helpful, scene-setting introduction by the evolutionary biologist Stephen Jay Gould, a prefatory "Challenge to the Reader" from the author himself, and finally, by way of epilogue, an "Author's Note," in which Kurten recapitulates the critical clues and describes the proper solution to his riddle, for any of us who may have missed it.

Kurten is an ethnic Swede native to Finland, and "Dance of the Tiger" has been translated from the Swedish. The novel is set in Scandinavia, during a great episode of warm weather and thawing that occurred late in the Ice Age, roughly 30,000 years ago, just about the time when the Neanderthal subspecies of humans was being supplanted across Europe by a new subspecies of immigrants, Homo sapiens sapiensm (more commonly known as Cro-Magnon man), considered to be our own direct and immediate ancestors. And the central mystery of Kurten's story is this: Why did the Neanderthals disappear so suddenly and so completely just after the arrival of modern man? Kurten's answer, played out through the pageant of his story, more compactly stated in his afterword, is both interesting and persuasive.

But that is not enough to make "Dance of the Tiger" a wholly interesting and persuasive novel. Gould (who is himself a writer of wonderful scientific essays) says in his introduction: "This book is, first and foremost, just what a fine novel should be -- a good story filled with insight into human character and the ways of nature." Well, no, actually not. First and foremost it is a conceptual model, a framework of paleoanthropological ideas, with the clay of fictionalized detail packed in around that framework. Insight into "the ways of nature," yes certainly, and the book even contains some passages of very nice writing on that theme; but Kurten's plot, with more coincidences per chapter than Charles Dickens at his most shameless, contorts itself unstintingly to reflect Kurten's theoretical brainstorming. And his portrayals of the human principals more often suggest a biologist's grasp of the manifestations of genotype than a novelist's insight into character. With two exceptions, as fairness requires noting: His troubled villain Shelk, and his venial buffoon Goshawk, are admirable literary creations.

This problem over characterization was built into the premise of the story: two genetically distinct subspecies of the human come into territorial collision , and what are the demographic consequences, based on what differentiae of adaptability? Forced to act out an answer to that, no wonder Tiger and Silverbirch and Fox and Baywillow seem two-dimensional; no wonder the plot is suffused with genetic determinism. If it weren't, Kurten could not have used it to illustrate his riddle and solution.

For a paleontologist, Bjorn Kurten is a rather skilled storyteller. Judged as a novelist, he is better than Spiro Agnew but not so good as Bill Buckley.