I would like to mention that ContentDocument object only supports delete(), describeSObjects(), query(), retrieve(), undelete(), update() calls. And ContentVersion Object supports create(), describeLayout(), describeSObjects(), query(), retrieve(), search(), update(), upsert() calls.
Documentation can be found here:
http://www.salesforce.com/us/developer/docs/api/Content/sforce_api_objects_contentdocument.htm
http://www.salesforce.com/us/developer/docs/api/Content/sforce_api_objects_contentversion.htm
Now to restrict the files I tried working with before insert and update triggers which did not work well as before the insertion the fileType fields came as null.
So Using After insert I have achieved the required. Use the below code at your own discretion.
public without sharing class FileService {
public static void RestrictFiles(List<ContentVersion> ContentList){
for(ContentVersion doc : ContentList){
if(doc.FileType != null && doc.FileType =='EXCEL_X' ){
doc.addError('File having this extension could not be attached,Please try some other extension.');
}
}
}
}