V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Emptycyx
V2EX  ›  程序员

分布式环境下使用 MongoDB 作为数据库异常的卡

  •  
  •   Emptycyx · 2022-01-19 11:00:19 +08:00 · 1407 次点击
    这是一个创建于 799 天前的主题,其中的信息可能已经有所发展或是发生改变。

    这个问题困扰了我许久,我在写一个分布式项目时,使用 MongoDB 作为数据库,单独架设在一台 4H8G 的服务器上,并供另外两台 worker 服务器作数据存储使用,然而不知道是什么原因,这种情况下我的 MongoDB 服务器异常卡顿,经常会在执行一些诸如 find_one 、insert_one 方法时 timeout 。实在是不清楚什么原因了,望解答。

    下面是 MongoDB 操作的 Util ,不太清除是不是 Util 部分出了问题...

    from pymongo import MongoClient
    
    
    class mongo_util:
        def __init__(self, host, port, username, password, db_name):
            try:
                self.client = MongoClient(host=host, port=port, serverSelectionTimeoutMS=10000, socketTimeoutMS=10000)
                if username != '' and password != '':
                    self.client['admin'].authenticate(name=username, password=password)
                self.db = self.client[db_name]
            except Exception as e:
                print('[-] 连接 MongoDB 服务器失败:{}'.format(e))
                exit(0)
    
        def find(self, col_name, conditions, keys=None):
            i = 0
            results = None
            while True:
                try:
                    col = self.db[col_name]
                    results = col.find(conditions)
                    break
                except Exception as e:
                    i += 1
                    print(f'[-] Fail to find {conditions} --> {e} , current retry time is {i}.')
                    continue
    
            if keys is None:
                return results
            else:
                new_results = []
                for result in results:
                    if len(keys) == 1:
                        if keys[0] not in result.keys(): break
                        new_results.append(result[keys[0]])
                    else:
                        temp = []
                        for key in keys:
                            if key in result.keys():
                                temp.append(result[key])
                        new_results.append(temp)
                return new_results
    
        def find_one(self, col_name, conditions, keys=None):
            i = 0
            result = None
            while True:
                try:
                    col = self.db[col_name]
                    result = col.find_one(conditions)
                    break
                except Exception as e:
                    i += 1
                    print(f'[-] Fail to find_one {conditions} --> {e} , current retry time is {i}.')
                    continue
    
            if not result: return
            if keys is None:
                return result
            elif len(keys) == 1 and keys[0] in result.keys():
                return result[keys[0]]
            else:
                temp = {}
                for key in keys:
                    if key in result.keys():
                        temp[key] = result[key]
                return result
    
        def insert(self, col_name, data_list):
            i = 0
            while True:
                try:
                    col = self.db[col_name]
                    return col.insert_many(data_list, bypass_document_validation=True).inserted_ids
                except Exception as e:
                    if 'E11000 duplicate key' in str(e):
                        print(f'[-] Fail to insert_one {data_list} --> {e}')
                        return True
                    i += 1
                    print(f'[-] Fail to insert {data_list} --> {e} , current retry time is {i}.')
                    continue
            return False
    
        def insert_one(self, col_name, data):
            i = 0
            while True:
                try:
                    col = self.db[col_name]
                    return col.insert_one(data, bypass_document_validation=True).inserted_id
                except Exception as e:
                    if 'E11000 duplicate key' in str(e):
                        print(f'[-] Fail to insert_one {data} --> {e}')
                        return True
                    i += 1
                    print(f'[-] Fail to insert_one {data} --> {e} , current retry time is {i}.')
                    continue
            return False
    
        def update(self, col_name, conditions, update_data):
            i = 0
            while True:
                try:
                    col = self.db[col_name]
                    return col.update_many(conditions, {'$set': update_data}).modified_count
                except Exception as e:
                    i += 1
                    print(f'[-] Fail to update {conditions} --> {e} , current retry time is {i}.')
                    continue
            return False
    
        def origin_update(self, col_name, conditions, update_data):
            i = 0
            while True:
                try:
                    col = self.db[col_name]
                    return col.update_many(conditions, update_data).modified_count
                except Exception as e:
                    i += 1
                    print(f'[-] Fail to origin_update {conditions} --> {e} , current retry time is {i}.')
                    continue
            return False
    
        def delete(self, col_name, conditions):
            i = 0
            while True:
                try:
                    col = self.db[col_name]
                    return col.delete_many(conditions).deleted_count
                except Exception as e:
                    i += 1
                    print(f'[-] Fail to delete {conditions} --> {e} , current retry time is {i}.')
                    continue
    
        def count(self, col_name, conditions):
            i = 0
            while True:
                try:
                    col = self.db[col_name]
                    return col.count(conditions)
                except Exception as e:
                    i += 1
                    print(f'[-] Fail to count {conditions} --> {e} , current retry time is {i}.')
                    continue
            return False
    
        def __del__(self):
            self.client.close()
    
    
    if __name__ == '__main__':
        from configs.db_config import mongo_host
        from configs.db_config import mongo_port
        from configs.db_config import mongo_user
        from configs.db_config import mongo_pass
        from configs.db_config import mongo_db_name
    
        mongo_instance = mongo_util(host=mongo_host, port=mongo_port, username=mongo_user, password=mongo_pass,
                                    db_name=mongo_db_name)
    
    9 条回复    2022-01-20 11:13:18 +08:00
    liprais
        1
    liprais  
       2022-01-19 11:13:29 +08:00
    秘诀就是没事别用 mongodb....
    hunk
        2
    hunk  
       2022-01-19 13:08:59 +08:00
    @liprais 类似 json 数据的存储用啥? mysql 可以,但确实不如 mongodb 粗暴
    qW7bo2FbzbC0
        3
    qW7bo2FbzbC0  
       2022-01-19 13:49:02 +08:00
    1.看起来像是英文机翻出来的中文
    2.无法理解 while true 的使用方式
    hotcool100
        4
    hotcool100  
       2022-01-19 13:53:05 +08:00
    MongoDB 支持集群
    hidemyself
        5
    hidemyself  
       2022-01-19 13:55:01 +08:00
    @hunk postgresql
    liprais
        6
    liprais  
       2022-01-19 13:56:28 +08:00
    @hunk 用你会用的
    starsky007
        7
    starsky007  
       2022-01-19 14:02:52 +08:00 via Android
    @liprais 我最近的一个项目就用的 MongoDB ,语言也是 Python ,数据量的话,有几个表近百万,不算大,并发也不高,没遇见啥问题。
    Emptycyx
        8
    Emptycyx  
    OP
       2022-01-19 14:59:22 +08:00
    @hjahgdthab750 while true 的原因就是因为某些时候会 timeout 所以不断重试,本人英文较差,不过是自己写的而非机翻哈哈

    @hotcool100 个人小项目啊,这用得上集群吗,就是比较卡,不太清除到底什么原因导致的,难道 4H8G 做服务器真的支撑不了我一个人使用的分布式小项目嘛
    dream4ever
        9
    dream4ever  
       2022-01-20 11:13:18 +08:00
    搜索一下性能方面的资料,排查一下是哪个环节出了问题。我之前解决过同事弄出来的 CPU 100% 的问题,最后发现是 MySQL 所有表都没加索引,加了索引之后立刻解决问题。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   我们的愿景   ·   实用小工具   ·   975 人在线   最高记录 6543   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 27ms · UTC 21:54 · PVG 05:54 · LAX 14:54 · JFK 17:54
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.