There has been a quiet revolution in the notion of what prisons are for. The criminal justice community has finally conceded that serving time does not rehabilitate criminals and that the threat of lengthy sentences does not deter crime.
Nonetheless, the prisons are being overpopulated to the point of overflow. The Federal Bureau of Justice Statistics announced last week that the prison population rose 6.9 percent during the first half of this year. It was the highest rate of increase ever recorded by the bureau, whose prison statistics date back to 1926. A year and a half ago, Associate United States Attorney General Rudolph Giuliani called the shortage of cells ''an emergency situation.'' Since then, the problem has worsened.
Criminologists have thus devised a new theory of justice, tailormade to relieve prison crowding and to redefine the purpose prisons should serve. The theory has an Orwellian label - selective incapacitation. The idea is to identify the worst career criminals and end their careers by keeping them -and few others - in prison. The strategy offers no guidance for handling the lesser offenders who make up the majority of those who commit crimes. According to a recent Rand Corporation study, this approach could reduce both the prison population and the crime rate. That makes it attractive to many politicians and judges. ''Selective incapacitation would attempt to allocate overburdened prison space by emphasizing the incarceration of the most dangerous felons,'' said Chief Judge Lawrence H. Cooke of the New York Court of Appeals last month, addressing a two-day bar association retreat devoted to selective incapacitation.
But the theory obviously raises serious concern for civil liberties. It is a radical departure from the precepts of criminal law, verging perilously close to preventive detention. Instead of punishing criminals for what they have done, the justice system would base punishment on a prediction of what they will do. That could mean throwing the book at a relatively harmless first offender who fits the crime-predictive profile.
Selective incapacitation rests on two assumptions: that there is a class of career criminals ressponsible for the bulk of serious crimes, and that the members of the class can be identified. There is some support for the career criminal concept. President Reagan believes in it. In a speech to the Justice Department last month he said New York City's experience attests to the existence of a ''new privileged class of repeat offenders.'' The city's transit police, he said, estimate that 500 habitual offenders accounted for nearly half the crimes committed in the subways last year. And the Rand study, using data supplied by prisoners in three states, found that ''among active burglars 50 percent committed fewer than 6 per year, while 10 percent committed more than 230 per year.''
Even if this is so - and the reliability of prisoner self-reporting is questionable - there remains the problem of identifying that hardcore 10 percent. The Rand study, a six-year effort by Peter W. Greenwood, found a strong correlation between repeated criminal behavior and factors such as juvenile convictions, heroin or barbiturate use, unemployment and prior imprisonment. Supporters of selective incapacitation admit better predictions could probably be made by including more controversial factors, such as race and marital status.
The predictions will never be perfect. There will always be ''false positives,'' people who match the hard-core offender profile but who will not repeatedly commit crimes. Punishing them, the Rand study recognized, doesn't mesh with the concept of being innocent until proven guilty. ''As long as our ability to discriminate between highand low-rate offenders is imprecise, there will be legitimate concern about those who are improperly classified,'' the study said. ''Furthermore, there will be differences of opinion as to the legitimacy of using some of the factors that are correlated with rates of offending (e.g. juvenile record, drug use, employment) for sentencing purposes.''
There will also be concern for the other 90 percent of criminals. Should a man convicted of his first crime be sent to prison if he doesn't match the career criminal profile? Releasing him may not please his victim, but it is already common practice today to avoid sending first offenders to jail.
Other tenets of selective incapacitation are now in place. Many states have repeat-offender laws, prosecutorial units devoted to career criminals and sentencing policies that consider prior offenses, job stability and other personal data. Most experts at the bar association retreat considered these practices sensible, given the justice system's limited resources. But almost all balked at a wholesale application of selective incapacitation.
Few believed that the worst recidivists could be identified and imprisoned early enough in their criminal careers to affect the crime rate. Fewer still were comfortable with a plan that relied on noncriminal traits to identify them. As Norval Morris of the University of Chicago law school put it, ''The danger of selective incapacitation is that it will be taken by the barbarians and misused.'' (More Ideas & Trends, page 11.)
Illustrations: photo of Maryland House of Correction