如何清理ogg日志文件

进入目标端ogg文件目录，/usr/local/ogg/dirdat，删除hr000009674之前的文件，不要都全部删掉，最好保留5-6个最新文件。ogg分为源端和目标端，由于目标端和kafka部署在一机器上，ogg日志文件使这台机器磁盘频繁的被占满，导致kafka 进程被杀死。之前配置的是3天就把已经完成推送的日志给删除，现在改成1小时就把已经完成推送的日志给删除。接下来的话 ./dirda

为爱停留

2742人浏览 · 2022-11-09 18:48:58

为爱停留 · 2022-11-09 18:48:58 发布

ogg分为源端和目标端，由于目标端和kafka部署在一机器上，ogg日志文件使这台机器磁盘频繁的被占满，导致kafka 进程被杀死

[root@qhtx_kafka_001 ogg]# ./ggsci 

Oracle GoldenGate for Big Data
Version 19.1.0.0.5 (Build 007)

Oracle GoldenGate Command Interpreter
Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141
Linux, x64, 64bit (optimized), Generic on Jun 28 2020 23:01:58
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.



GGSCI (qhtx_kafka_001) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
REPLICAT    RUNNING     REKAFKA     00:00:05      00:00:02    
REPLICAT    RUNNING     REKAFKA2    07:02:21      00:00:01    


GGSCI (qhtx_kafka_001) 2> view report REKAFKA

会查看到此时这个进程已经报错了：


2022-11-09 06:27:51  INFO    OGG-02232  Switching to next trail file /usr/local/ogg/dirdat/hr000016218 at 2022-11-09 06:27:51.705193 due to EOF. with current RBA 499,999,791.

2022-11-09 06:28:00  INFO    OGG-02232  Switching to next trail file /usr/local/ogg/dirdat/hr000016219 at 2022-11-09 06:28:00.318853 due to EOF. with current RBA 499,999,659.
2022-11-09 06:29:08,035 pool-2-thread-1 ERROR Unable to write to stream dirrpt/REKAFKA.log for appender rollingAppender org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream dirrpt/REKAFKA.log
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:252)
	at org.apache.logging.log4j.core.appender.FileManager.writeToDestination(FileManager.java:273)
	at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.writeToDestination(RollingFileManager.java:240)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:282)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:291)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:199)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
	at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:312)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)

自动删除

查看mgr进程配置：

GGSCI (qhtx_kafka_001) 2> view param mgr

port 7809
dynamicportlist 7810-7890
autorestart REPLICAT *,retries 5,waitminutes 3,RESETMINUTES 30
PURGEOLDEXTRACTS ./dirdat/*,USECHECKPOINTS, MINKEEPDAYS 3
lagreporthours 1
laginfominutes 30
lagcriticalminutes 45

PURGEOLDEXTRACTS ./dirdat/*,USECHECKPOINTS, MINKEEPDAYS 3

之前配置的是3天就把已经完成推送的日志给删除，现在改成1小时就把已经完成推送的日志给删除

则需要执行以下操作

stop mgr

edit param mgr

其中把 PURGEOLDEXTRACTS ./dirdat/*,USECHECKPOINTS, MINKEEPDAYS 3 修改成PURGEOLDEXTRACTS ./dirdat/*,USECHECKPOINTS, MINKEEPHOURS 1

port 7809
dynamicportlist 7810-7890
autorestart REPLICAT *,retries 5,waitminutes 3,RESETMINUTES 30
PURGEOLDEXTRACTS ./dirdat/*,USECHECKPOINTS, MINKEEPHOURS 1
lagreporthours 1
laginfominutes 30
lagcriticalminutes 45

start mgr

接下来的话 ./dirdat/* 日志文件就不会频繁的把磁盘占满了

[root@qhtx_kafka_001 dirdat]# ls
hr000016625  hr000016646  hr000016667  hr000016688  hr000016709  hr000016730  hr000016751  hr000016772  hr000016793  hr000016814  hr000016835  hr000016856  hr000016877  hr000016898  hr000016919  hr000016940  hr000016961  hr000016982
hr000016626  hr000016647  hr000016668  hr000016689  hr000016710  hr000016731  hr000016752  hr000016773  hr000016794  hr000016815  hr000016836  hr000016857  hr000016878  hr000016899  hr000016920  hr000016941  hr000016962  hr000016983
hr000016627  hr000016648  hr000016669  hr000016690  hr000016711  hr000016732  hr000016753  hr000016774  hr000016795  hr000016816  hr000016837  hr000016858  hr000016879  hr000016900  hr000016921  hr000016942  hr000016963  hr000016984
hr000016628  hr000016649  hr000016670  hr000016691  hr000016712  hr000016733  hr000016754  hr000016775  hr000016796  hr000016817  hr000016838  hr000016859  hr000016880  hr000016901  hr000016922  hr000016943  hr000016964  hr000016985
hr000016629  hr000016650  hr000016671  hr000016692  hr000016713  hr000016734  hr000016755  hr000016776  hr000016797  hr000016818  hr000016839  hr000016860  hr000016881  hr000016902  hr000016923  hr000016944  hr000016965  hr000016986
hr000016630  hr000016651  hr000016672  hr000016693  hr000016714  hr000016735  hr000016756  hr000016777  hr000016798  hr000016819  hr000016840  hr000016861  hr000016882  hr000016903  hr000016924  hr000016945  hr000016966  hr000016987
hr000016631  hr000016652  hr000016673  hr000016694  hr000016715  hr000016736  hr000016757  hr000016778  hr000016799  hr000016820  hr000016841  hr000016862  hr000016883  hr000016904  hr000016925  hr000016946  hr000016967  hr000016988
hr000016632  hr000016653  hr000016674  hr000016695  hr000016716  hr000016737  hr000016758  hr000016779  hr000016800  hr000016821  hr000016842  hr000016863  hr000016884  hr000016905  hr000016926  hr000016947  hr000016968  hr000016989
hr000016633  hr000016654  hr000016675  hr000016696  hr000016717  hr000016738  hr000016759  hr000016780  hr000016801  hr000016822  hr000016843  hr000016864  hr000016885  hr000016906  hr000016927  hr000016948  hr000016969  hr000016990
hr000016634  hr000016655  hr000016676  hr000016697  hr000016718  hr000016739  hr000016760  hr000016781  hr000016802  hr000016823  hr000016844  hr000016865  hr000016886  hr000016907  hr000016928  hr000016949  hr000016970  hr000016991
hr000016635  hr000016656  hr000016677  hr000016698  hr000016719  hr000016740  hr000016761  hr000016782  hr000016803  hr000016824  hr000016845  hr000016866  hr000016887  hr000016908  hr000016929  hr000016950  hr000016971  hr000016992
hr000016636  hr000016657  hr000016678  hr000016699  hr000016720  hr000016741  hr000016762  hr000016783  hr000016804  hr000016825  hr000016846  hr000016867  hr000016888  hr000016909  hr000016930  hr000016951  hr000016972  hr000016993
hr000016637  hr000016658  hr000016679  hr000016700  hr000016721  hr000016742  hr000016763  hr000016784  hr000016805  hr000016826  hr000016847  hr000016868  hr000016889  hr000016910  hr000016931  hr000016952  hr000016973  hr000016994
hr000016638  hr000016659  hr000016680  hr000016701  hr000016722  hr000016743  hr000016764  hr000016785  hr000016806  hr000016827  hr000016848  hr000016869  hr000016890  hr000016911  hr000016932  hr000016953  hr000016974  hr000016995
hr000016639  hr000016660  hr000016681  hr000016702  hr000016723  hr000016744  hr000016765  hr000016786  hr000016807  hr000016828  hr000016849  hr000016870  hr000016891  hr000016912  hr000016933  hr000016954  hr000016975  hr000016996
hr000016640  hr000016661  hr000016682  hr000016703  hr000016724  hr000016745  hr000016766  hr000016787  hr000016808  hr000016829  hr000016850  hr000016871  hr000016892  hr000016913  hr000016934  hr000016955  hr000016976  hr000016997
hr000016641  hr000016662  hr000016683  hr000016704  hr000016725  hr000016746  hr000016767  hr000016788  hr000016809  hr000016830  hr000016851  hr000016872  hr000016893  hr000016914  hr000016935  hr000016956  hr000016977  hr000016998
hr000016642  hr000016663  hr000016684  hr000016705  hr000016726  hr000016747  hr000016768  hr000016789  hr000016810  hr000016831  hr000016852  hr000016873  hr000016894  hr000016915  hr000016936  hr000016957  hr000016978  hr000016999
hr000016643  hr000016664  hr000016685  hr000016706  hr000016727  hr000016748  hr000016769  hr000016790  hr000016811  hr000016832  hr000016853  hr000016874  hr000016895  hr000016916  hr000016937  hr000016958  hr000016979  hr000017000
hr000016644  hr000016665  hr000016686  hr000016707  hr000016728  hr000016749  hr000016770  hr000016791  hr000016812  hr000016833  hr000016854  hr000016875  hr000016896  hr000016917  hr000016938  hr000016959  hr000016980
hr000016645  hr000016666  hr000016687  hr000016708  hr000016729  hr000016750  hr000016771  hr000016792  hr000016813  hr000016834  hr000016855  hr000016876  hr000016897  hr000016918  hr000016939  hr000016960  hr000016981
[root@qhtx_kafka_001 dirdat]#

其中一个文件500M

手动删除

./ggsci

查看命令

view report REKAFKA

2022-10-25 16:32:32  INFO    OGG-02232  Switching to next trail file /usr/local/ogg/dirdat/hr000009674 at 2022-10-25 16:32:32.208037 due to EOF. with current RBA 499,999,672.

可以看到消费到哪个文件
进入目标端ogg文件目录，/usr/local/ogg/dirdat，删除hr000009674之前的文件，不要都全部删掉，最好保留5-6个最新文件
rm -rf hr000009600
……
rm -rf hr000009670

Kafka开源项目指南

Kafka开源项目指南提供详尽教程，助开发者掌握其架构、配置和使用，实现高效数据流管理和实时处理。它高性能、可扩展，适合日志收集和实时数据处理，通过持久化保障数据安全，是企业大数据生态系统的核心。

更多推荐

Kafka入门（一）概述、部署与API的简单使用

Kafka概述、部署与API的简单使用

Kafka开源项目指南

基于canal和kafka同步，实现binlog同步ElasticSearch

文章目录前言elasticsearch 安装canal安装canal-adapter 安装及配置mysql 安装zk及kafaka安装查看效果注意事项前言中间件版本elasticsearch7.5.2canal1.1.4client-adapter1.1.5-alpha-1zookeeper3.4.13kafka2.6.0mysql5.7.31elasticsearch 安装{"settings"

Kafka开源项目指南

基于 Iceberg 的湖仓一体架构在 B 站的实践

背景在B站，每天都有PB级的数据注入到大数据平台，经过离线或实时的ETL建模后，提供给下游的分析、推荐及预测等场景使用。面对如此大规模的数据，如何高效低成本地满足下游数据的分析需求，一直是我们重点的工作方向。我们之前的数据处理流程基本上是这样的：采集端将客户端埋点、服务端埋点、日志、业务数据库等数据收集到HDFS、Kafka等存储系统中，然后通过Hive、Spark、Fl...