Uploaded image for project: 'Document Management System'
  1. Document Management System
  2. DMS-2468

Deadlocks in Communication EmailQueueExtractor service

    Details

    • Type: Maintenance
    • Status: To Do
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Company:
      All Clients/Multiple Clients

      Description

      Deadlocks in Communication EmailQueueExtractor service:

       

      =====================================
      2021-04-19 01:47:17 0x2410 INNODB MONITOR OUTPUT
      =====================================
      Per second averages calculated from the last 14 seconds
      -----------------
      BACKGROUND THREAD
      -----------------
      srv_master_thread loops: 785061 srv_active, 0 srv_shutdown, 3955412 srv_idle
      srv_master_thread log flush and writes: 0
      ----------
      SEMAPHORES
      ----------
      OS WAIT ARRAY INFO: reservation count 10759217813
      OS WAIT ARRAY INFO: signal count 5704025318
      RW-shared spins 1205697749, rounds 1281209273, OS waits 82128657
      RW-excl spins 58978312, rounds 134463004, OS waits 3166376
      RW-sx spins 2098, rounds 60202, OS waits 1834
      Spin rounds per wait: 1.06 RW-shared, 2.28 RW-excl, 28.69 RW-sx
      ------------------------
      LATEST DETECTED DEADLOCK
      ------------------------
      2021-04-15 13:12:46 0x1ce8

          • (1) TRANSACTION:
            TRANSACTION 12908311, ACTIVE 0 sec fetching rows, thread declared inside InnoDB 3338
            mysql tables in use 1, locked 1
            LOCK WAIT 1059 lock struct(s), heap size 123088, 13335 row lock(s)
            MySQL thread id 43645, OS thread handle 3184, query id 40684577 172.16.196.5 dmsuser updating
            UPDATE emaillog SET TaskProcessorId = '71be92f1-732e-4994-bf5d-4bfa9fb432a5', TaskPickedAt = timestamp('2021-04-15 20:12:46.424101'), SendTaskPickCount = SendTaskPickCount + 1
            WHERE SendAttempted = false AND SendAt <= timestamp('2021-04-15 20:12:46.424101') AND SendTaskPickCount < 3 AND (TaskProcessorId IS NULL OR (TaskProcessorId IS NOT NULL AND TaskPickedAt < timestamp('2021-04-15 19:52:46.424101')))
            limit 100
          • (1) WAITING FOR THIS LOCK TO BE GRANTED:
            RECORD LOCKS space id 452 page no 27920 n bits 80 index PRIMARY of table `dms_com_prod`.`emaillog` trx id 12908311 lock_mode X locks rec but not gap waiting
            Record lock, heap no 14 PHYSICAL RECORD: n_fields 29; compact format; info bits 0
            0: len 8; hex 8000000000042844; asc (D;;
            1: len 6; hex 000000c4f71f; asc ;;
            2: len 7; hex 02000016231994; asc # ;;
            3: len 30; hex 62363430376262332d363062622d346261652d616535662d343833343135; asc b6407bb3-60bb-4bae-ae5f-483415; (total 36 bytes);
            4: len 22; hex 6e594e58653150375463616c764e6a374b5159625a77; asc nYNXe1P7TcalvNj7KQYbZw;;
            5: SQL NULL;
            6: len 30; hex 42656e5863656c20322e302050656e64696e672057616976656420456c65; asc BenXcel 2.0 Pending Waived Ele; (total 45 bytes);
            7: len 30; hex 3c703e44656172204e6963686f6c6173204420477265656e652c3c2f703e; asc <p>Dear Nicholas D Greene,</p>; (total 407 bytes);
            8: SQL NULL;
            9: len 5; hex 99a95f4329; asc _C);;
            10: len 5; hex 99a95f432f; asc _C/;;
            11: len 9; hex 416e6f6e796d6f7573; asc Anonymous;;
            12: len 9; hex 416e6f6e796d6f7573; asc Anonymous;;
            13: len 5; hex 3134323438; asc 14248;;
            14: len 9; hex 776f726b7465727261; asc workterra;;
            15: len 2; hex 3537; asc 57;;
            16: len 2; hex 3135; asc 15;;
            17: len 1; hex 01; asc ;;
            18: len 5; hex 99a95f4326; asc _C&;;
            19: SQL NULL;
            20: len 1; hex 00; asc ;;
            21: len 8; hex 8000000000000000; asc ;;
            22: len 1; hex 00; asc ;;
            23: len 0; hex ; asc ;;
            24: len 4; hex 80000000; asc ;;
            25: SQL NULL;
            26: len 5; hex 99a95f432a; asc _C*;;
            27: len 4; hex 80000001; asc ;;
            28: len 4; hex 80000000; asc ;;
          • (2) TRANSACTION:
            TRANSACTION 12908319, ACTIVE 0 sec updating or deleting, thread declared inside InnoDB 4999
            mysql tables in use 1, locked 1
            3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
            MySQL thread id 43651, OS thread handle 7400, query id 40684668 172.16.196.4 dmsuser updating
            UPDATE `emaillog` SET `AppModuleId` = '15', `Body` = '<p>Dear Nicholas D Greene,</p>
            <p>This is to notify you that your pending waived election for Benefit/Plan Flexible Spending Account 2021 is accepted by the Administrator.</p>
            <p>Please contact your HR Department for more details.</p>
            <p> </p>
            <p>Regards,</p>
            <p>BenXcel Support</p>
            <p> </p>
            <p><em>Please Note: This is an auto generated message. Please do not reply back to this message.</em></p>', `CategoryId` = '57', `ConversationUUID` = 'b6407bb3-60bb-4bae-ae5f-4834150aa46d', `ErrorMessage` = '', `ErrorRetryPickCount` = 0, `FromTemplate` = false, `IsError` = false, `IsOutbound` = NULL, `IsTransactional` = NULL, `MessageID` = 'nYNXe1P7TcalvNj7KQYbZw', `MetaAppId` = 'workterra', `MetaCompanyId` = '14248', `MetaDateCreated` = timestamp('2021-04-15 20:12:41.000000'), `MetaDateModified` = timestamp('2021-04-15 20:12:46.726236'), `MetaUserCreated` = 'Anonymo
          • (2) HOLDS THE LOCK(S):
            RECORD LOCKS space id 452 page no 27920 n bits 80 index PRIMARY of table `dms_com_prod`.`emaillog` trx id 12908319 lock_mode X locks rec but not gap
            Record lock, heap no 14 PHYSICAL RECORD: n_fields 29; compact format; info bits 0
            0: len 8; hex 8000000000042844; asc (D;;
            1: len 6; hex 000000c4f71f; asc ;;
            2: len 7; hex 02000016231994; asc # ;;
            3: len 30; hex 62363430376262332d363062622d346261652d616535662d343833343135; asc b6407bb3-60bb-4bae-ae5f-483415; (total 36 bytes);
            4: len 22; hex 6e594e58653150375463616c764e6a374b5159625a77; asc nYNXe1P7TcalvNj7KQYbZw;;
            5: SQL NULL;
            6: len 30; hex 42656e5863656c20322e302050656e64696e672057616976656420456c65; asc BenXcel 2.0 Pending Waived Ele; (total 45 bytes);
            7: len 30; hex 3c703e44656172204e6963686f6c6173204420477265656e652c3c2f703e; asc <p>Dear Nicholas D Greene,</p>; (total 407 bytes);
            8: SQL NULL;
            9: len 5; hex 99a95f4329; asc _C);;
            10: len 5; hex 99a95f432f; asc _C/;;
            11: len 9; hex 416e6f6e796d6f7573; asc Anonymous;;
            12: len 9; hex 416e6f6e796d6f7573; asc Anonymous;;
            13: len 5; hex 3134323438; asc 14248;;
            14: len 9; hex 776f726b7465727261; asc workterra;;
            15: len 2; hex 3537; asc 57;;
            16: len 2; hex 3135; asc 15;;
            17: len 1; hex 01; asc ;;
            18: len 5; hex 99a95f4326; asc _C&;;
            19: SQL NULL;
            20: len 1; hex 00; asc ;;
            21: len 8; hex 8000000000000000; asc ;;
            22: len 1; hex 00; asc ;;
            23: len 0; hex ; asc ;;
            24: len 4; hex 80000000; asc ;;
            25: SQL NULL;
            26: len 5; hex 99a95f432a; asc _C*;;
            27: len 4; hex 80000001; asc ;;
            28: len 4; hex 80000000; asc ;;
          • (2) WAITING FOR THIS LOCK TO BE GRANTED:
            RECORD LOCKS space id 452 page no 27840 n bits 600 index SendAttempt of table `dms_com_prod`.`emaillog` trx id 12908319 lock_mode X locks rec but not gap waiting
            Record lock, heap no 528 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
            0: len 1; hex 00; asc ;;
            1: len 8; hex 8000000000042844; asc (D;;
          • WE ROLL BACK TRANSACTION (2)
            ------------
            TRANSACTIONS
            ------------
            Trx id counter 13057387
            Purge done for trx's n:o < 13057376 undo n:o < 0 state: running but idle
            History list length 16
            LIST OF TRANSACTIONS FOR EACH SESSION:
            ---TRANSACTION 283403308663144, not started
            0 lock struct(s), heap size 1136, 0 row lock(s)
            ---TRANSACTION 283403308662272, not started
            0 lock struct(s), heap size 1136, 0 row lock(s)
            ---TRANSACTION 283403308660528, not started
            0 lock struct(s), heap size 1136, 0 row lock(s)
            ---TRANSACTION 283403308664016, not started
            0 lock struct(s), heap size 1136, 0 row lock(s)
            ---TRANSACTION 283403308661400, not started
            0 lock struct(s), heap size 1136, 0 row lock(s)
            --------
            FILE I/O
            --------
            I/O thread 0 state: wait Windows aio (insert buffer thread)
            I/O thread 1 state: wait Windows aio (log thread)
            I/O thread 2 state: wait Windows aio (read thread)
            I/O thread 3 state: wait Windows aio (read thread)
            I/O thread 4 state: wait Windows aio (read thread)
            I/O thread 5 state: wait Windows aio (read thread)
            I/O thread 6 state: wait Windows aio (write thread)
            I/O thread 7 state: wait Windows aio (write thread)
            I/O thread 8 state: wait Windows aio (write thread)
            I/O thread 9 state: wait Windows aio (write thread)
            Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
            ibuf aio reads:, log i/o's:, sync i/o's:
            Pending flushes (fsync) log: 0; buffer pool: 0
            29433417550 OS file reads, 16057398 OS file writes, 8785599 OS fsyncs
            0.07 reads/s, 16384 avg bytes/read, 0.86 writes/s, 0.71 fsyncs/s
            -------------------------------------
            INSERT BUFFER AND ADAPTIVE HASH INDEX
            -------------------------------------
            Ibuf: size 1, free list len 195, seg size 197, 216225 merges
            merged operations:
            insert 491112, delete mark 12489, delete 822
            discarded operations:
            insert 0, delete mark 0, delete 0
            Hash table size 2267, node heap has 1 buffer(s)
            Hash table size 2267, node heap has 1 buffer(s)
            Hash table size 2267, node heap has 1 buffer(s)
            Hash table size 2267, node heap has 0 buffer(s)
            Hash table size 2267, node heap has 1 buffer(s)
            Hash table size 2267, node heap has 1 buffer(s)
            Hash table size 2267, node heap has 1 buffer(s)
            Hash table size 2267, node heap has 1 buffer(s)
            0.00 hash searches/s, 0.14 non-hash searches/s

            LOG

            Log sequence number 11399975757
            Log buffer assigned up to 11399975757
            Log buffer completed up to 11399975757
            Log written up to 11399975757
            Log flushed up to 11399975757
            Added dirty pages up to 11399975757
            Pages flushed up to 11399975757
            Last checkpoint at 11399975757
            5257583 log i/o's done, 0.29 log i/o's/second
            ----------------------
            BUFFER POOL AND MEMORY
            ----------------------
            Total large memory allocated 8585216
            Dictionary memory allocated 616968
            Buffer pool size 512
            Free buffers 249
            Database pages 256
            Old database pages 0
            Modified db pages 0
            Pending reads 0
            Pending writes: LRU 0, flush list 0, single page 0
            Pages made young 0, not young 2
            0.00 youngs/s, 0.00 non-youngs/s
            Pages read 29444695611, created 107483, written 8974523
            0.07 reads/s, 0.00 creates/s, 0.29 writes/s
            Buffer pool hit rate 955 / 1000, young-making rate 0 / 1000 not 0 / 1000
            Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
            LRU len: 256, unzip_LRU len: 0
            I/O sum[5]:cur[0], unzip sum[0]:cur[0]
            --------------
            ROW OPERATIONS
            --------------
            0 queries inside InnoDB, 0 queries in queue
            0 read views open inside InnoDB
            Process ID=4280, Main thread ID=0000000000001834 , state=sleeping
            Number of rows inserted 677702, updated 1541229, deleted 39022, read 408503895333
            0.07 inserts/s, 0.00 updates/s, 0.00 deletes/s, 2.21 reads/s
            ----------------------------
            END OF INNODB MONITOR OUTPUT
            ============================

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            harshveer.singh Harshveer Singh (Inactive)
            Reporter:
            harshveer.singh Harshveer Singh (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated: