我们通过之前得学习，得知kafka一些特性和python操作kafka得方法，但是我们还没有学习过查看kafka数据方法，然后在这一篇笔记中我们来学习下使用命令来查看kafka数据

我们首先查看kafka主题分布情况

/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic nginx-logs
Topic:nginx-logs        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: nginx-logs       Partition: 0    Leader: 0       Replicas: 0     Isr: 0

Leader：指定主分区的broker id
Replicas： 副本在那些机器上
Isr：可以做为主分区的broker id

接着我们计入kafka存储得topic目录下，我的目录为/usr/local/kafka/logs/kafka/nginx-logs-0

可以发现数据文件由.index文件、.log文件、.timeindex文件组成

可以通过kafka安装目录bin目录下的kafka-run-class.sh查看这些文件的内容

查看.index文件的方法

../../../bin/kafka-run-class.sh kafka.tools.DumpLogSegments -files 00000000000000000000.index --print-data-log

返回结果我们可以看到偏移量和定位

......
offset: 1546 position: 1547281
offset: 1551 position: 1552092
offset: 1556 position: 1557085
offset: 1560 position: 1561398
offset: 1564 position: 1565669
offset: 1568 position: 1569935
offset: 1573 position: 1574858
offset: 1578 position: 1579745
offset: 1582 position: 1584156
offset: 1586 position: 1588553
offset: 1590 position: 1592966
offset: 1594 position: 1597382
offset: 1598 position: 1601877
offset: 1603 position: 1606706
offset: 1608 position: 1611517

查看.log文件的方法

../../../bin/kafka-run-class.sh kafka.tools.DumpLogSegments -files 00000000000000000000.log --print-data-log

返回的都是生产者发送的消息

| offset: 1607 CreateTime: 1573205403831 keysize: -1 valuesize: 900 sequence: -1 headerKeys: [] payload: {"timestamp":"08/Nov/2019:17:30:02 +0800","log":{"offset":3002136,"file":{"path":"/usr/local/nginx/logs/access.log"}},"response_code":200,"ecs":{"version":"1.1.0"},"request":"/","@version":"1","input":{"type":"log"},"auth":"-","method":"GET","httpversion":"1.1","client":"49.233.176.23","response_time":17589,"tags":["hk-nginx-logs"],"user_agent":"\"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0)\"","agent":{"type":"filebeat","version":"7.4.0","id":"fcf18d27-a7d6-48d1-8749-7eaabc70d6d9","ephemeral_id":"173f7435-086b-43ae-b953-cc5fa8925e9b","hostname":"iZj6c1iqplyxdp3km5vse6Z"},"host":{"os":{"version":"6.10 (Final)","family":"redhat","name":"CentOS","platform":"centos","kernel":"2.6.32-754.12.1.el6.x86_64","codename":"Final"},"containerized":false,"architecture":"x86_64","hostname":"iZj6c1iqplyxdp3km5vse6Z","name":"iZj6c1iqplyxdp3km5vse6Z"},"@timestamp":"2019-11-08T09:30:02.620Z"}
baseOffset: 1608 lastOffset: 1608 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 1611517 CreateTime: 1573205419161 size: 1003 magic: 2 compresscodec: NONE crc: 1720564619 isvalid: true
| offset: 1608 CreateTime: 1573205419161 keysize: -1 valuesize: 933 sequence: -1 headerKeys: [] payload: {"timestamp":"08/Nov/2019:17:30:14 +0800","log":{"file":{"path":"/usr/local/nginx/logs/access.log"},"offset":3002278},"response_code":200,"ecs":{"version":"1.1.0"},"request":"/retrieve___fd3d_1991_1.html","@version":"1","input":{"type":"log"},"auth":"-","method":"GET","httpversion":"1.1","client":"54.36.148.0","response_time":4986,"tags":["hk-nginx-logs"],"user_agent":"\"Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)\"","agent":{"type":"filebeat","id":"fcf18d27-a7d6-48d1-8749-7eaabc70d6d9","version":"7.4.0","ephemeral_id":"173f7435-086b-43ae-b953-cc5fa8925e9b","hostname":"iZj6c1iqplyxdp3km5vse6Z"},"host":{"os":{"version":"6.10 (Final)","family":"redhat","name":"CentOS","platform":"centos","kernel":"2.6.32-754.12.1.el6.x86_64","codename":"Final"},"containerized":false,"architecture":"x86_64","hostname":"iZj6c1iqplyxdp3km5vse6Z","name":"iZj6c1iqplyxdp3km5vse6Z"},"@timestamp":"2019-11-08T09:30:17.626Z"}

查看.timeindex文件的方法

../../../bin/kafka-run-class.sh kafka.tools.DumpLogSegments -files 00000000000000000000.timeindex --print-data-log

返回时间戳和偏移量

......
timestamp: 1573204946730 offset: 1541
timestamp: 1573205029744 offset: 1546
timestamp: 1573205147762 offset: 1551
timestamp: 1573205186786 offset: 1556
timestamp: 1573205186787 offset: 1558
timestamp: 1573205188782 offset: 1563
timestamp: 1573205188792 offset: 1565
timestamp: 1573205217794 offset: 1572
timestamp: 1573205220814 offset: 1578
timestamp: 1573205221801 offset: 1582
timestamp: 1573205223960 offset: 1584
timestamp: 1573205223983 offset: 1587
timestamp: 1573205342820 offset: 1603
timestamp: 1573205419161 offset: 1608

index件和log文件组成segment，segment文件的命名规则是，partion全局的第一个segment从0开始，后续每个segment文件名为上一个全局partion的最大offset(偏移message数)。数值最大为64位long大小，19位数字字符长度，没有数字用0填充。log.segment.bytes参数配置了一个log文件的大小，文件大小超过这个值就会生成新的文件