======================================================== WARNING: possible irq lock inversion dependency detected 6.4.0-rc7-next-20230623 #1 Not tainted -------------------------------------------------------- systemd-udevd/103 just changed the state of lock: ffffffff85815818 (blkg_stat_lock){+.-.}-{2:2}, at: __blkcg_rstat_flush.isra.0+0x11f/0x4e0 but this lock was taken by another, HARDIRQ-safe lock in the past: (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} and interrupts could create inverse lock ordering between them. other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(blkg_stat_lock); local_irq_disable(); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); lock(blkg_stat_lock); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); *** DEADLOCK *** 5 locks held by systemd-udevd/103: #0: ffffffff85687bb0 (dup_mmap_sem){.+.+}-{0:0}, at: dup_mmap+0xf5/0x1770 #1: ffff888016d28ae0 (&mm->mmap_lock){++++}-{3:3}, at: dup_mmap+0x113/0x1770 #2: ffff88800845de20 (&mm->mmap_lock/1){+.+.}-{3:3}, at: dup_mmap+0x163/0x1770 #3: ffffffff8560b5a0 (rcu_callback){....}-{0:0}, at: rcu_core+0x83e/0x28b0 #4: ffffffff8560b6c0 (rcu_read_lock){....}-{1:2}, at: __blkcg_rstat_flush.isra.0+0x93/0x4e0 the shortest dependencies between 2nd lock and 1st lock: -> (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} { IN-HARDIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime_field+0xa0/0x120 account_system_index_time+0x199/0x2c0 update_process_times+0x26/0x150 tick_sched_handle+0x8e/0x170 tick_sched_timer+0xe6/0x110 __hrtimer_run_queues+0x17f/0xb60 hrtimer_interrupt+0x2ef/0x750 __sysvec_apic_timer_interrupt+0xff/0x380 sysvec_apic_timer_interrupt+0x69/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 lock_acquire+0x1c7/0x4c0 kernfs_unbreak_active_protection+0x89/0xa0 cgroup_kn_unlock+0xc4/0x420 __cgroup_procs_write+0x3b6/0x6b0 cgroup_procs_write+0x26/0x60 cgroup_file_write+0x1e5/0x7c0 kernfs_fop_write_iter+0x3f7/0x610 vfs_write+0x97d/0xda0 ksys_write+0x122/0x250 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __mod_memcg_lruvec_state+0x18d/0x2b0 mod_objcg_mlstate+0x119/0x230 drain_obj_stock+0x125/0x580 refill_obj_stock+0x12d/0x5c0 kmem_cache_free+0x444/0x4d0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 run_ksoftirqd+0x29/0x50 smpboot_thread_fn+0x40b/0x930 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_flush_locked+0x131/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] __key.0+0x0/0x40 ... acquired at: _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 mem_cgroup_wb_stats+0x3d8/0x4b0 balance_dirty_pages+0x339/0x2640 balance_dirty_pages_ratelimited_flags+0xbce/0x1130 fault_dirty_shared_page+0x2b3/0x500 do_wp_page+0x3c8/0x3b00 __handle_mm_fault+0xcbc/0x2d20 handle_mm_fault+0x1af/0xba0 do_user_addr_fault+0x58e/0x12f0 exc_page_fault+0x9c/0x1a0 asm_exc_page_fault+0x26/0x30 -> (blkg_stat_lock){+.-.}-{2:2} { HARDIRQ-ON-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_next_frame+0x1aa/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x59/0x70 kmem_cache_alloc+0x16b/0x370 __create_object+0x3c/0xc90 kmem_cache_alloc+0x20b/0x370 __pmd_alloc+0x98/0x690 copy_page_range+0x2570/0x3330 dup_mmap+0xee2/0x1770 copy_process+0x3e68/0x7320 kernel_clone+0xeb/0x7d0 __do_sys_clone+0xba/0x100 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_next_frame+0x1aa/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x59/0x70 kmem_cache_alloc+0x16b/0x370 __create_object+0x3c/0xc90 kmem_cache_alloc+0x20b/0x370 __pmd_alloc+0x98/0x690 copy_page_range+0x2570/0x3330 dup_mmap+0xee2/0x1770 copy_process+0x3e68/0x7320 kernel_clone+0xeb/0x7d0 __do_sys_clone+0xba/0x100 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 mem_cgroup_wb_stats+0x3d8/0x4b0 balance_dirty_pages+0x339/0x2640 balance_dirty_pages_ratelimited_flags+0xbce/0x1130 fault_dirty_shared_page+0x2b3/0x500 do_wp_page+0x3c8/0x3b00 __handle_mm_fault+0xcbc/0x2d20 handle_mm_fault+0x1af/0xba0 do_user_addr_fault+0x58e/0x12f0 exc_page_fault+0x9c/0x1a0 asm_exc_page_fault+0x26/0x30 } ... key at: [] blkg_stat_lock+0x18/0x60 ... acquired at: __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_next_frame+0x1aa/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x59/0x70 kmem_cache_alloc+0x16b/0x370 __create_object+0x3c/0xc90 kmem_cache_alloc+0x20b/0x370 __pmd_alloc+0x98/0x690 copy_page_range+0x2570/0x3330 dup_mmap+0xee2/0x1770 copy_process+0x3e68/0x7320 kernel_clone+0xeb/0x7d0 __do_sys_clone+0xba/0x100 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 stack backtrace: CPU: 0 PID: 103 Comm: systemd-udevd Not tainted 6.4.0-rc7-next-20230623 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 Call Trace: dump_stack_lvl+0x91/0xf0 print_irq_inversion_bug.part.0+0x3d5/0x570 mark_lock.part.0+0x900/0x2f50 __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:unwind_next_frame+0x1aa/0x2490 Code: 42 ff 39 c6 0f 83 c2 16 00 00 48 b8 00 00 00 00 00 fc ff df 89 f2 48 8d 3c 95 30 70 4f 86 49 89 f8 49 c1 e8 03 45 0f b6 04 00 <48> 89 f8 83 e0 07 83 c0 03 44 38 c0 7c 32 45 84 c0 74 2d 48 89 54 RSP: 0018:ffff88800fbd71c8 EFLAGS: 00000213 RAX: dffffc0000000000 RBX: 0000000000000001 RCX: ffffffff81812598 RDX: 0000000000008125 RSI: 0000000000008125 RDI: ffffffff865174c4 RBP: ffff88800fbd7290 R08: 0000000000000000 R09: ffff88800fbd7278 R10: 0000000000038001 R11: 000000000002119e R12: ffff88800fbd7298 R13: ffff88800fbd7280 R14: ffff88800fbd7279 R15: ffff88800fbd7238 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x59/0x70 kmem_cache_alloc+0x16b/0x370 __create_object+0x3c/0xc90 kmem_cache_alloc+0x20b/0x370 __pmd_alloc+0x98/0x690 copy_page_range+0x2570/0x3330 dup_mmap+0xee2/0x1770 copy_process+0x3e68/0x7320 kernel_clone+0xeb/0x7d0 __do_sys_clone+0xba/0x100 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7f54f22a84bb Code: ed 0f 85 f8 00 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 41 89 c5 85 c0 0f 85 9e 00 00 RSP: 002b:00007fffc2ed5a30 EFLAGS: 00000246 ORIG_RAX: 0000000000000038 RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f54f22a84bb RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f54f1e1c8c0 R10: 00007f54f1e1cb90 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fffc2ed5b00 R14: 0000000000000000 R15: 0000000000000000 netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. loop6: detected capacity change from 0 to 512 netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. loop6: detected capacity change from 0 to 512 netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. Bluetooth: hci6: command 0x0406 tx timeout loop6: detected capacity change from 0 to 512 netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. loop3: detected capacity change from 0 to 512 ext4: Unknown parameter 'd*PWVcI+X|cC0#nŤ}ZEQYͿD:' UDC core: USB Raw Gadget: couldn't find an available UDC or it's busy netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. misc raw-gadget: fail, usb_gadget_register_driver returned -16 UDC core: USB Raw Gadget: couldn't find an available UDC or it's busy loop3: detected capacity change from 0 to 512 ext4: Unknown parameter 'RWVcI+X|cC0#nŤ}ZEQYͿD:' misc raw-gadget: fail, usb_gadget_register_driver returned -16 loop6: detected capacity change from 0 to 33 EXT4-fs (loop6): mounting ext2 file system using the ext4 subsystem EXT4-fs (loop6): bad geometry: block count 128 exceeds size of device (8 blocks) netlink: 12 bytes leftover after parsing attributes in process `syz-executor.5'. loop3: detected capacity change from 0 to 512 ext4: Unknown parameter 'ꯇ[WVcI+X|cC0#nŤ}ZEQYͿD:' loop6: detected capacity change from 0 to 33 EXT4-fs (loop6): mounting ext2 file system using the ext4 subsystem EXT4-fs (loop6): bad geometry: block count 128 exceeds size of device (8 blocks) ---------------- Code disassembly (best guess): 0: 42 ff rex.X (bad) 2: 39 c6 cmp %eax,%esi 4: 0f 83 c2 16 00 00 jae 0x16cc a: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax 11: fc ff df 14: 89 f2 mov %esi,%edx 16: 48 8d 3c 95 30 70 4f lea -0x79b08fd0(,%rdx,4),%rdi 1d: 86 1e: 49 89 f8 mov %rdi,%r8 21: 49 c1 e8 03 shr $0x3,%r8 25: 45 0f b6 04 00 movzbl (%r8,%rax,1),%r8d * 2a: 48 89 f8 mov %rdi,%rax <-- trapping instruction 2d: 83 e0 07 and $0x7,%eax 30: 83 c0 03 add $0x3,%eax 33: 44 38 c0 cmp %r8b,%al 36: 7c 32 jl 0x6a 38: 45 84 c0 test %r8b,%r8b 3b: 74 2d je 0x6a 3d: 48 rex.W 3e: 89 .byte 0x89 3f: 54 push %rsp