I just found an interesting paper: “Windows Access Control Demystified,” by Sudhakar Govindavajhala and Andrew W. Appel. Basically, they show that companies like Adobe, Macromedia, etc., have mistakes in their Access Control Programming that open security holes in Windows XP.
In the Secure Internet Programming laboratory at Princeton University, we have been investigating network security management by using logic programming. We developed a rule based framework — Multihost, Multistage, Vulnerability Analysis(MulVAL) — to perform end-to-end, automatic analysis of multi-host, multi-stage attacks on a large network where hosts run different operating systems. The tool finds attack paths where the adversary will have to use one or more than one weaknesses (buffer overflows) in multiple software to attack the network. The MulVAL framework has been demonstrated to be modular, flexible, scalable and efficient . We applied these techniques to perform security analysis of a single host with commonly used software.
We have constructed a logical model of Windows XP access control, in a declarative but executable (Datalog) format. We have built a scanner that reads access-control conguration information from the Windows registry, file system, and service control manager database, and feeds raw conguration data to the model. Therefore we can reason about such things as the existence of privilege-escalation attacks, and indeed we have found several user-to-administrator vulnerabilities caused by misconfigurations of the access-control lists of commercial software from several major vendors. We propose tools such as ours as a vehicle for software developers and system administrators to model and debug the complex interactions of access control on installations under Windows.
EDITED TO ADD (2/13): Ed Felten has some good commentary about the paper on his blog. [Schneier on Security]