Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has “Full Read” permissions on the SharePoint Web Application being crawled. ( HttpStatusCode Unauthorized The request failed with HTTP status 401: Unauthorized. )
Due to the fully qualified domain name that it’s trying to crawl… it handles it with a loop back request.
1. Go to command window and type regedit.exe
2. Once opened the registry editor, just navigate to “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa”
3. Right click on “Lsa” and create a new 32bit DWORD value
4. Rename it as “DisableLoopbackCheck” (Note: you cannot rename it actually. so, once created the DWORD value, directly paste/type as “DisableLoopbackCheck”.
5. Then again modify the value of “DisableLoopbackCheck” as “1″
6. Close the registry editor.
7. Now start to crawl the content.