Britain's Ulster strategy hangs in the balance

The future of the Northern Ireland Assembly and of the British government's current Ulster strategy hang in the balance following the murder of three Protestants at worship in an isolated Irish border area.

Members of the Official Unionist Party, the largest group representing the North's 1 million Protestants, have withdrawn from the Assembly. They say they will not return until there is an improvement in security, coupled with more power from Westminster for local political representatives.

James Prior, who as Northern Ireland secretary is the government's chief minister in Ulster, is attempting to talk the Unionists into reconsidering their position. It is an unenviable task.

Described recently by a House of Commons observer as ''a decent man afflicted with a notable measure of pessimism,'' Mr. Prior is aware the government cannot provide full security for Ulster's 11/2 million people. Nor can he prevent a small group of gunmen from attacking an isolated building and shooting its occupants.

Such an attack occurred Sunday. Armed men burst into the Mountain Lodge Pentecostal Assembly at Darkley, in the ''bandit country'' of South Armagh, and raked the congregation of men, women, and children with gunfire.

Three elders of the sect were killed. Several other people were injured. The attackers made their escape, possibly across the nearby Irish border.

The outburst of community fury and grief after the attack threatens to make a grave situation even worse. Tomas Cardinal O'Fiaich has expressed revulsion at the killings and disgust that the attackers claimed to be the Catholic Reaction Force.

Police believe this is a contrived name for a small group from the Irish National Liberation Army. They claim one of the weapons used in the shooting had been used in earlier INLA attacks.

The INLA is the paramilitary wing of the Marxist Irish Republican Workers' Party, and it is thought to be recruiting former members of the Provisional wing of the illegal Irish Republican Army who are disenchanted with the increasing political momentum of the Provisionals' political wing, Sinn Fein.

Despite the plea for calm by Cardinal O'Fiaich and other church leaders, Unionist politicians and many of their constituents are grimly determined that terrorism be brought to heel. There have been calls for the resignation of Mr. Prior and of the chief constable of the Royal Ulster Constabulary, Sir John Hermon.

Their critics claim the current security policy is not firm enough, although they do not specify how police and soldiers on both sides of the border can track down terrorists experienced in going to ground.

But both Prior and Hermon repudiate such criticism. Hermon is giving protection to a number of border churches that appear particularly vulnerable, but he is resisting political pressure to pour a large force into the South Armagh area ''as a cosmetic exercise'' to allay public fears.

''Our emphasis is turning more and more toward providing security that cannot be seen and that cannot be used as a target for terrorists,'' he said.

This approach does not impress those who want a dramatic drive against terrorism. Hence Prior's difficulty in convincing them that the best way of tackling terrorism is through informers and the slow erosion of terrorism on the ground.

Unless Prior can persuade the Official Unionists to stay in the Assembly, its future looks limited. Already the mainly Catholic Social Democratic and Labor Party is staying away; its members say it is a waste of time to talk about power-sharing with the Unionists.

Only the moderate Alliance Party and the Democratic Unionists are still backing the Assembly, partly because it provides a local forum for their views and partly because it offers an opportunity for dialogue, however imperfectly.

The obvious fall-back would be direct rule from Westminster.

This has been often described as ''the least unacceptable option'' for most people in Ulster.