Skip to main content

Data Management

Data Storage​

In stand-lone deployment mode, the storage components are already built into the mirror, so users do not need to install them separately. The generated data is stored in a specified directory through mounting.

In HAP, the default data path is /data/mingdao, but if you have a custom data path, use the actual path.

The data generated by storage components is by default mounted in the /data/mingdao/script/volume/data/ directory on the server. The directory structure is as follows:

/data/mingdao/script/volume/data/
├── elasticsearch
├── elasticsearch-8
├── kafka
├── logs
├── mongodb
├── mysql
├── redis
├── storage
└── zookeeper

Data stored in each directory:

  • elasticsearch, for elasticsearch data prior to version 3.9.0

  • elasticsearch-8, for elasticsearch data starting from version 3.9.0, mainly supporting super search and application logs

  • kafka, for storing message queue data, expired data is automatically cleaned up by default

  • log, for logs of component service

  • mongodb, for storing business data (such as applications, worksheets, workflows, etc.)

  • mysql, for storing system core data such as users, organizations, etc.

  • redis, for caching data

  • storage, for storing attachment data

  • zookeeper, to provide services for kafka

Data Backup​

Data Cleaning​

In general, the data in logs, mongodb, and storage occupy relatively large disk space.

  • For logs stored in the logs directory for component services, if you do not need this log data, you can clean it up.

  • For business data stored in mongodb, currently only log data archiving and cleaning deleted application data are supported.

  • For attachment data stored in storage, if the disk capacity cannot be supported, it is recommended to limit the file size on the upload page.

If your environment has been upgraded, mirrors of history versions that have not been cleaned up will also take up some disk space.

Example of Cleaning up History Mirrors

  1. View the list of mirror files on the current server.

    [root@localhost ~]# docker images
    REPOSITORY                                                        TAG                   IMAGE ID       CREATED         SIZE
    registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-community   4.4.2                 cacc79998cbd   5 days ago      9.33GB
    registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-community   4.3.0                 e0c6e370de63   9 days ago      9.26GB
    registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-doc         1.2.0                 cc4d668f289b   4 months ago    2.66GB
    • For example, if the current version is 4.4.2, then the mirrors of version 4.3.0 can be cleaned up.

  2. Clean up the mirrors of history versions.

    docker image rm registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-community:4.3.0
    or
    docker image rm e0c6e370de63

If the above methods fail to free up more disk space, consider expanding the disk capacity.

Data Migration​