Alex小新

模式

ElasticSearch数据查询 ​

基础查询 ​

GET单条数据查询 ​

查询meat数据

bash
GET /{index_name}/_doc/{id}
bash
GET /gupaoedu-logstash-2023.07.31/_source/bpEnq4kB5X9NTM0seYU2
json
{
  "_index" : "gupaoedu-logstash-2023.07.31",
  "_type" : "_doc",
  "_id" : "bpEnq4kB5X9NTM0seYU2",
  "_version" : 1,
  "_seq_no" : 14412,
  "_primary_term" : 2,
  "found" : true,
  "_source" : {
    "@version" : "1",
    "level_value" : 20000,
    "level" : "INFO",
    "transaction.id" : "7e56fac7c650655e",
    "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
    "port" : 35926,
    "@timestamp" : "2023-07-31T08:53:58.067Z",
    "logger_name" : "com.gponline.service.course.teacher.controller.TeacherInfoApi",
    "message" : "通过ids获取讲师信息--Api-入参：57",
    "host" : "gupaoedu-course-service.open-net",
    "thread_name" : "http-nio-7009-exec-10"
  }
}

只查询原始数据

bash
GET /{index_name}/_source/{id}
bash
GET /gupaoedu-logstash-2023.07.31/_source/bpEnq4kB5X9NTM0seYU2
json
{
  "@version" : "1",
  "level_value" : 20000,
  "level" : "INFO",
  "transaction.id" : "7e56fac7c650655e",
  "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
  "port" : 35926,
  "@timestamp" : "2023-07-31T08:53:58.067Z",
  "logger_name" : "com.gponline.service.course.teacher.controller.TeacherInfoApi",
  "message" : "通过ids获取讲师信息--Api-入参：57",
  "host" : "gupaoedu-course-service.open-net",
  "thread_name" : "http-nio-7009-exec-10"
}

MGET一次查询多条数据 ​

指定索引和ID

bash
GET /_mget 
{
  "docs": [
   {
     "_index":  "{index_name}",
     "_id": "{id}"
   },
   {
     "_index":  "{index_name}",
     "_id": "{id}"
   } 
  ]
}
bash
GET /_mget 
{
  "docs": [
   {
     "_index": "gupaoedu-logstash-2023.07.31",
     "_id": "bpEnq4kB5X9NTM0seYU2"
   },
   {
     "_index": "gupaoedu-logstash-2023.07.31",
     "_id": "eJEnq4kB5X9NTM0seYU3"
   } 
  ]
}
json
{
  "docs" : [
    {
      "_index" : "gupaoedu-logstash-2023.07.31",
      "_type" : "_doc",
      "_id" : "bpEnq4kB5X9NTM0seYU2",
      "_version" : 1,
      "_seq_no" : 14412,
      "_primary_term" : 2,
      "found" : true,
      "_source" : {
        "@version" : "1",
        "level_value" : 20000,
        "level" : "INFO",
        "transaction.id" : "7e56fac7c650655e",
        "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
        "port" : 35926,
        "@timestamp" : "2023-07-31T08:53:58.067Z",
        "logger_name" : "com.gponline.service.course.teacher.controller.TeacherInfoApi",
        "message" : "通过ids获取讲师信息--Api-入参：57",
        "host" : "gupaoedu-course-service.open-net",
        "thread_name" : "http-nio-7009-exec-10"
      }
    },
    {
      "_index" : "gupaoedu-logstash-2023.07.31",
      "_type" : "_doc",
      "_id" : "eJEnq4kB5X9NTM0seYU3",
      "_version" : 1,
      "_seq_no" : 14421,
      "_primary_term" : 2,
      "_ignored" : [
        "message.keyword"
      ],
      "found" : true,
      "_source" : {
        "@version" : "1",
        "level_value" : 20000,
        "level" : "INFO",
        "transaction.id" : "32a7925d6c93d8b2",
        "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
        "port" : 35926,
        "@timestamp" : "2023-07-31T08:53:58.089Z",
        "logger_name" : "com.gponline.service.course.teacher.service.impl.TeacherInfoServiceImpl",
        "message" : "通过ids获取讲师信息--Service-出参：[TeacherInfoDTO(id=59, version=0, headImg=https://cdn.manage.gupaoedu.cn/fe-website-crm/manage/permanent/2021-8/24FAB3AE-9203-489C-A5D1-119434716B24.png, intro=, nickName=测试老师, subId=null, createTime=null, updateTime=null, uuId=f0e43d7b-bfad-4659-9556-ba45a2ca8a34, teacherTitle=null, teacherType=null, status=1, isHide=0)]",
        "host" : "gupaoedu-course-service.open-net",
        "thread_name" : "http-nio-7009-exec-6"
      }
    }
  ]
}

指定ID

bash
GET /{index_name}/_mget 
{
  "ids": ["{id}", "{id}"]
}
bash
GET /gupaoedu-logstash-2023.07.31/_mget 
{
  "ids": ["bpEnq4kB5X9NTM0seYU2","eJEnq4kB5X9NTM0seYU3"]
}
json
{
  "docs" : [
    {
      "_index" : "gupaoedu-logstash-2023.07.31",
      "_type" : "_doc",
      "_id" : "bpEnq4kB5X9NTM0seYU2",
      "_version" : 1,
      "_seq_no" : 14412,
      "_primary_term" : 2,
      "found" : true,
      "_source" : {
        "@version" : "1",
        "level_value" : 20000,
        "level" : "INFO",
        "transaction.id" : "7e56fac7c650655e",
        "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
        "port" : 35926,
        "@timestamp" : "2023-07-31T08:53:58.067Z",
        "logger_name" : "com.gponline.service.course.teacher.controller.TeacherInfoApi",
        "message" : "通过ids获取讲师信息--Api-入参：57",
        "host" : "gupaoedu-course-service.open-net",
        "thread_name" : "http-nio-7009-exec-10"
      }
    },
    {
      "_index" : "gupaoedu-logstash-2023.07.31",
      "_type" : "_doc",
      "_id" : "eJEnq4kB5X9NTM0seYU3",
      "_version" : 1,
      "_seq_no" : 14421,
      "_primary_term" : 2,
      "_ignored" : [
        "message.keyword"
      ],
      "found" : true,
      "_source" : {
        "@version" : "1",
        "level_value" : 20000,
        "level" : "INFO",
        "transaction.id" : "32a7925d6c93d8b2",
        "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
        "port" : 35926,
        "@timestamp" : "2023-07-31T08:53:58.089Z",
        "logger_name" : "com.gponline.service.course.teacher.service.impl.TeacherInfoServiceImpl",
        "message" : "通过ids获取讲师信息--Service-出参：[TeacherInfoDTO(id=59, version=0, headImg=https://cdn.manage.gupaoedu.cn/fe-website-crm/manage/permanent/2021-8/24FAB3AE-9203-489C-A5D1-119434716B24.png, intro=, nickName=测试老师, subId=null, createTime=null, updateTime=null, uuId=f0e43d7b-bfad-4659-9556-ba45a2ca8a34, teacherTitle=null, teacherType=null, status=1, isHide=0)]",
        "host" : "gupaoedu-course-service.open-net",
        "thread_name" : "http-nio-7009-exec-6"
      }
    }
  ]
}

Search查询模型 ​

查询所有数据 ​

基础语法

bash
GET /{index_name}/_search 
{
  "size": 1
}
bash
GET /gupaoedu-logstash-2023.07.31_shard3/_search
{
  "size": 1
}
json
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "gupaoedu-logstash-2023.07.31_shard3",
        "_type" : "_doc",
        "_id" : "bpEnq4kB5X9NTM0seYU2",
        "_score" : 1.0,
        "_source" : {
          "@version" : "1",
          "level_value" : 20000,
          "level" : "INFO",
          "transaction.id" : "7e56fac7c650655e",
          "trace.id" : "f050515a3e76826c5104257fbbb79bdc",
          "port" : 35926,
          "@timestamp" : "2023-07-31T08:53:58.067Z",
          "logger_name" : "com.gponline.service.course.teacher.controller.TeacherInfoApi",
          "message" : "通过ids获取讲师信息--Api-入参：57",
          "host" : "gupaoedu-course-service.open-net",
          "thread_name" : "http-nio-7009-exec-10"
        }
      }
    ]
  }
}

URL查询参数 ​

基础语法

参数名称描述示例
size每页大小20
from开始位置0
sort排序字段
scroll滚动快照查询
version是否返回数据的版本号
track_total_hits是否返回总数量 7.0z 之后最多返回10000true

组合查询 ​

组合查询分类 ​

参数名称描述
bool布尔组合
boost加权
constant固定分值
dis_max单字符多字段组合
function函数脚本组合 慎用

Boolean组合查询 ​

关键字说明是否计算分值
must必须包含，类似 and是
should可选包含，类似 or否
filter必须包含，类似 and否
must_not不包含，等同于 !=
mixed水平组合混合以上，类似多条件平级组合
mixed嵌套组合混合以上，类似多条件平级组合

must查询 ​

并且关系，类似and

bash
GET /gupaoedu-logstash-2023.07.31/_search
{
  "size": 5, 
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "message": "课程"
          }
        },
        {
          "match": {
            "message": "参"
          }
        }
      ]
    }
  }
}
json
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 291,
      "relation" : "eq"
    },
    "max_score" : 11.651195,
    "hits" : [
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "ZK_pqokBQn-49qJVob5I",
        "_score" : 11.651195,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "03ed03dbcf053d8d",
          "trace.id" : "1a9ee9244d9784a17ec3acb36f72dcac",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:27:28.255Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1443]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-10"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "56_pqokBQn-49qJVob51",
        "_score" : 11.651195,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "84e68104ebad87df",
          "trace.id" : "9f4aeada7bfc48f912c213a25780c2bd",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:28:15.669Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1298]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-7"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "zq_pqokBQn-49qJVob-Y",
        "_score" : 11.651195,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "7d889a10b0da2e36",
          "trace.id" : "6bc9956e0ff1b62136f63e7e74a8a3c4",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:31:29.382Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1095]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-9"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "Ja_pqokBQn-49qJVosEs",
        "_score" : 11.651195,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "acc52eec4ab7cc7c",
          "trace.id" : "b1e63348c2dc848553a675db61f33f57",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:39:59.138Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1095]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-8"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "yLDrqokBQn-49qJV5y9b",
        "_score" : 11.651195,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "8d187300d4f2b7b2",
          "trace.id" : "ff2943703829b31eac669200d4a0dc46",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:48:53.932Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1501]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-4"
        }
      }
    ]
  }
}

filter查询 ​

并且关系，类似and，和 must 等效，只是没有评分 score 字段，有助于提升性能。

bash
GET /gupaoedu-logstash-2023.07.31/_search
{
  "size": 5, 
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "message": "课程"
          }
        },
        {
          "match": {
            "message": "参"
          }
        }
      ]
    }
  }
}
json
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 291,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "GJEkq4kB5X9NTM0sqms_",
        "_score" : 0.0,
        "_source" : {
          "@version" : "1",
          "level_value" : 20000,
          "level" : "INFO",
          "transaction.id" : "cfc2ac68c90094e9",
          "trace.id" : "0c1d2975f2b4ac3ba2c9964c8d996a3a",
          "port" : 35926,
          "@timestamp" : "2023-07-31T08:47:34.912Z",
          "logger_name" : "com.gponline.service.course.curriculum.service.impl.CurriculumEnrollStatisticServiceImpl",
          "message" : "获取课程报名人数--Service-入参:288",
          "host" : "gupaoedu-course-service.open-net",
          "thread_name" : "http-nio-7009-exec-2"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "GpEkq4kB5X9NTM0sqms_",
        "_score" : 0.0,
        "_source" : {
          "@version" : "1",
          "level_value" : 20000,
          "level" : "INFO",
          "transaction.id" : "8d70486f9084f180",
          "trace.id" : "76aeea2747d05733aa51c3c7df4990e7",
          "port" : 35926,
          "@timestamp" : "2023-07-31T08:47:34.914Z",
          "logger_name" : "com.gponline.service.course.curriculum.service.impl.CurriculumEnrollStatisticServiceImpl",
          "message" : "获取课程报名人数--Service-入参:288",
          "host" : "gupaoedu-course-service.open-net",
          "thread_name" : "http-nio-7009-exec-4"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "F5Ekq4kB5X9NTM0sqms_",
        "_score" : 0.0,
        "_source" : {
          "@version" : "1",
          "level_value" : 20000,
          "level" : "INFO",
          "transaction.id" : "cfc2ac68c90094e9",
          "trace.id" : "0c1d2975f2b4ac3ba2c9964c8d996a3a",
          "port" : 35926,
          "@timestamp" : "2023-07-31T08:47:34.912Z",
          "logger_name" : "com.gponline.service.course.curriculum.controller.CurriculumEnrollStatisticApi",
          "message" : "获取课程报名人数--Api-入参:288",
          "host" : "gupaoedu-course-service.open-net",
          "thread_name" : "http-nio-7009-exec-2"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "GZEkq4kB5X9NTM0sqms_",
        "_score" : 0.0,
        "_source" : {
          "@version" : "1",
          "level_value" : 20000,
          "level" : "INFO",
          "transaction.id" : "8d70486f9084f180",
          "trace.id" : "76aeea2747d05733aa51c3c7df4990e7",
          "port" : 35926,
          "@timestamp" : "2023-07-31T08:47:34.913Z",
          "logger_name" : "com.gponline.service.course.curriculum.controller.CurriculumEnrollStatisticApi",
          "message" : "获取课程报名人数--Api-入参:288",
          "host" : "gupaoedu-course-service.open-net",
          "thread_name" : "http-nio-7009-exec-4"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "ZK_pqokBQn-49qJVob5I",
        "_score" : 0.0,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "03ed03dbcf053d8d",
          "trace.id" : "1a9ee9244d9784a17ec3acb36f72dcac",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:27:28.255Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1443]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-10"
        }
      }
    ]
  }
}

should查询 ​

并且关系，类似or，查看返回的总数量即可判断出来差异。

bash
GET /gupaoedu-logstash-2023.07.31/_search
{
  "size": 0, 
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "message": "课程"
          }
        },
        {
          "match": {
            "message": "参"
          }
        }
      ]
    }
  }
}
json
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2246,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

boost 加权组合查询 ​

通过boost可以设置排序对应的分值，score对应的分值。

bash

GET /gupaoedu-logstash-2023.07.31/_search
{
  "size": 2, 
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "message": {
              "query": "课程",
              "boost": 0.5
            }
          }
        }
      ]
    }
  }
}
json
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 878,
      "relation" : "eq"
    },
    "max_score" : 4.726747,
    "hits" : [
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "ZK_pqokBQn-49qJVob5I",
        "_score" : 4.726747,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "03ed03dbcf053d8d",
          "trace.id" : "1a9ee9244d9784a17ec3acb36f72dcac",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:27:28.255Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1443]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-10"
        }
      },
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_type" : "_doc",
        "_id" : "56_pqokBQn-49qJVob51",
        "_score" : 4.726747,
        "_source" : {
          "@version" : "1",
          "level_value" : 40000,
          "level" : "ERROR",
          "transaction.id" : "84e68104ebad87df",
          "trace.id" : "9f4aeada7bfc48f912c213a25780c2bd",
          "port" : 51730,
          "@timestamp" : "2023-07-31T07:28:15.669Z",
          "logger_name" : "com.gponline.crm.process.order.machine.state.OrderBaseState",
          "message" : "课程[1298]参数错误,无法找到对应的课程或未关联云课堂课程",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-7"
        }
      }
    ]
  }
}

EQL ​

安全示例

简单EQL示例 ​

bash
GET /gupaoedu-logstash-2023.07.31/_eql/search
{
  "size": 1,
  "query": """
  any where level == "WARN"
  """
}
json
{
  "is_partial" : false,
  "is_running" : false,
  "took" : 0,
  "timed_out" : false,
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "events" : [
      {
        "_index" : "gupaoedu-logstash-2023.07.31",
        "_id" : "MpEwq4kB5X9NTM0suqkH",
        "_source" : {
          "@version" : "1",
          "level_value" : 30000,
          "level" : "WARN",
          "transaction.id" : "5e5dd38f46f1f837",
          "trace.id" : "a4ee33dc4d2d648385a63664f9fbbbf3",
          "port" : 36248,
          "@timestamp" : "2023-07-31T09:04:04.462Z",
          "logger_name" : "com.gponline.crm.process.order.OrderProcess",
          "message" : "课程信息不存在: 1289",
          "host" : "gupaoedu-crm-service.open-net",
          "thread_name" : "http-nio-7002-exec-9"
        }
      }
    ]
  }
}

人生感悟

Copyright © 2019~2024 由vitepress强力驱动