c# & Fizzler to crawl web page in a certain website domain

使用fizzler [HtmlAgilityPackExtension]和c#进行网页数据提取;fizzler是HtmlAgilityPack的一个扩展,支持jQuery Selector;

提取数据一般都是有规律url拼凑,然后挨个儿发request得到response进行解析:

1.假如一个website下的所有xxx.sample.com/contactus.html里边存在邮箱字段(准备提取的数据)

  a)当有子域名的时候,比如:a.sample.com, aadr.sample.com, 135dj.sample.com,随机性比较强;

   解决方法:bing search engine中使用 site:b2b.sample.com搜索得到的result页面可以提取所有子域名,然后拼凑成xxx.sample.com/contactus.html,继而发送请求到这个url,得        到response进行解析;

   NOTE:关于site:b2b.sample.com的搜索url拼凑如下,

       http://www.bing.com/search?q=site%3A{b2b.sample.com}&go=Submit&qs=n&form=QBRE&pq=site%3A{b2b.sample.com}&sc=1-19&sp=-1&sk=&cvid=6165a189f5354b1982fb8cd6933abb6f&first={pageIndex}&FORM=PERE

2.像www.sample.com/1456.html的页面可以直接平凑1456.html/1457.html/1458.html etc.此处不列举;

Fizzler使用方法:

1.从nuget上安装Fizzler

2.使用方法参考code.google.com

3.使用bing提取website下的所有子域:

c# & Fizzler to crawl web page in a certain website 
domain
private static List<string> GetSubdomains(string websiteDomain, int startPageIndex = 1, int pageCount = 999, int pageSize = 14)
        {
            var list = new List<string>();
            //using bind to search subdomains in a certain website
            var bingSearchUrlFormat = "http://www.bing.com/search?q=site%3a{0}&go=Submit&qs=n&pq=site%3a{0}&sc=1-100&sp=-1&sk=&cvid=a9b36439006f4b05b09f9202c5b784bd&first={1}&FORM=PQRE";

            WebClient client = new WebClient();
            client.Encoding = Encoding.UTF8;
            var doc = new HtmlDocument();

            var first = (startPageIndex / 10) * 140 + 1;
            var stopIndex = first + pageCount*pageSize;
            var currentPageIndex = startPageIndex;
            for (var startItemSquenceNumber = first; startItemSquenceNumber < stopIndex; startItemSquenceNumber = startItemSquenceNumber + pageSize)
            {
                var response = client.DownloadString(string.Format(bingSearchUrlFormat, websiteDomain, startItemSquenceNumber));
                HtmlDocumentExtensions.LoadHtml2(doc, response);
                var docNode = doc.DocumentNode;
                var subDomains = docNode.QuerySelectorAll(".sb_meta cite");foreach (var subDomain in subDomains)
                {
                    list.Add(subDomain.InnerText);
                }
            }return list;
        }
c# & Fizzler to crawl web page in a certain website 
domain

4.获取网页节点:

c# & Fizzler to crawl web page in a certain website 
domain
        private static List<HtmlNode> GetWebPageNodes(string url, string elementSelector, string attributeNameContained, string attributeNameContainedValueLike)
        {
            var client = new WebClient();
            client.Encoding = Encoding.UTF8;
            var response = client.DownloadString(url);
            var doc = new HtmlDocument();
            HtmlDocumentExtensions.LoadHtml2(doc, response);
            var docNode = doc.DocumentNode;
            var emailNode = docNode.QuerySelectorAll(elementSelector).Where(node => node.Attributes.Where(attr => attr.Name == attributeNameContained).FirstOrDefault().Value.Contains(attributeNameContainedValueLike)).FirstOrDefault();

            var nodes = (from node in docNode.QuerySelectorAll(elementSelector)
                         where node.HasAttributes && node.GetAttributeValue(attributeNameContained, string.Empty).Contains(attributeNameContainedValueLike)
                         select node).ToList();

            return nodes;
        }
c# & Fizzler to crawl web page in a certain website 
domain

5.获取某个网页中邮箱的方法:

c# & Fizzler to crawl web page in a certain website 
domain
var subdomains = GetSubdomains("b2b.sample.com", stopPageIndex, 10);
var urlFormat = "http://{0}/contactus.html";
GetWebPageNodes(string.Format(urlFormat, item), "body table a", "href", "mailto").FirstOrDefault();
c# & Fizzler to crawl web page in a certain website 
domain

 

最后的问题:当通过bing搜索子域时会有限制,发送100~150个请求后获取到的response就不是我想要的页面,而是要求输入验证码防止攻击的html;此问题暂时未解决,望大神指点!

c# & Fizzler to crawl web page in a certain website domain,布布扣,bubuko.com

c# & Fizzler to crawl web page in a certain website domain

上一篇:Web前端开发-选择器


下一篇:css和div