.NET Core(C#) 使用IHttpClientFactory实现爬虫执行GET和POST请求

1､.NET Core项目中注册HttpClientFactory

1) ASP .NET Core项目

在Startup.cs中ConfigureService方法中注册HttpClient，代码如下，

services.AddHttpClient();

2) .NET Core项目

.NET Core项目中需要安装引用Microsoft.Extensions.Http和Microsoft.Extensions.Hosting，注册代码如下，

static void Main(string[] args)
   {

      var builder = Host.CreateDefaultBuilder(args);

      builder.ConfigureServices(x =>

      {

          x.AddHttpClient();

      });

      Console.ReadKey();

  }

2､配置HTTP请求头Header

1) 注册HttpClient时配置

services.AddHttpClient();
services.AddHttpClient("cjavapy", c =>
{
    c.BaseAddress = new Uri("https://api.cjavapy.com/");
    c.DefaultRequestHeaders.Add("content-type","application/json");
    c.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
    c.DefaultRequestHeaders.Add("Referer","");
});

或者

   static void Main(string[] args)
        {
            var builder = Host.CreateDefaultBuilder(args);
            builder.ConfigureServices(x =>
            {
                x..AddHttpClient("cjavapy", c =>
{
    c.BaseAddress = new Uri("https://api.cjavapy.com/");
    c.DefaultRequestHeaders.Add("content-type","application/json");
    c.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
    c.DefaultRequestHeaders.Add("Referer","");
});
            });
            Console.ReadKey();
        }

2) 执行请求之前配置

var httpClient = context.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
httpClient.BaseAddress = new Uri("https://api.cjavapy.com/");
httpClient.DefaultRequestHeaders.Add("content-type","application/json");
httpClient.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
httpClient.DefaultRequestHeaders.Add("Referer","");

3､使用HttpClientFactory(IHttpClientFactory)执行GET和POST请求

命名空间：

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Reflection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

1) GET请求获取网页代码

static void Main(string[] args)
  {
      var builder = Host.CreateDefaultBuilder(args);
      builder.ConfigureServices(x =>
      {
          x.AddHttpClient();
      });
      using (var scope = builder.Build().Services.CreateScope())
      {
          var httpClient = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
          httpClient.BaseAddress = new Uri("https://www.baidu.com");
          httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
          httpClient.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com");
          Console.WriteLine(httpClient.GetAsync("/").Result.Content.ReadAsStringAsync().Result);
      }
      Console.ReadKey();
  }

2) POST请求提交数据

static void Main(string[] args)
   {
       var builder = Host.CreateDefaultBuilder(args);
       builder.ConfigureServices(x =>
       {
           x.AddHttpClient();
       });
       using (var scope = builder.Build().Services.CreateScope())
       {
           var httpClient = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
           httpClient.BaseAddress = new Uri("apiUrl");
           httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
           //httpClient.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com");
           var result="";
           try
           {
               //2 minute timeout on wait for response
               httpClient.Timeout = new TimeSpan(0, 2, 0);
               //Create an HttpRequestMessage object and pass it into SendAsync()
               HttpRequestMessage message = new HttpRequestMessage();
               message.Headers.Add("Accept", "application/json");
               message.Content = new StringContent("{\"user\":\"11\"}", System.Text.Encoding.UTF8, "application/json");
               message.Method = HttpMethod.Post;
               message.RequestUri = new Uri(httpClient.BaseAddress.ToString() + "someapiendpoint");
               HttpResponseMessage response = httpClient.SendAsync(message).Result;
               result = response.Content.ReadAsStringAsync().Result;
               //deserialize the result into proper object type
           }
           catch (Exception ex)
           {
               //Log exception
           }
       }
       Console.ReadKey();
   }

相关文档：

ASP.NET Core 2.1 HttpClientFactory使用的3种方法

.NET(C#)使用HttpClient请求JSON数据的示例代码

.NET Core(C#) RestSharp GET和POST请求、下载大文件及cookie管理

.NET(C#) Fluent HTTP (Flurl Get和Post请求)使用方法及示例代码

.NET Core 使用HttpClient通过配置Proxy(代理)执行Get和Post请求数据操作

ASP.NET Core 使用HttpClient PostAsync POST Json数据