======================================================== WARNING: possible irq lock inversion dependency detected 6.4.0-rc7-next-20230622 #1 Not tainted -------------------------------------------------------- swapper/0/0 just changed the state of lock: ffffffff85814c18 (blkg_stat_lock){+.-.}-{2:2}, at: __blkcg_rstat_flush.isra.0+0x11f/0x4e0 but this lock was taken by another, HARDIRQ-safe lock in the past: (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} and interrupts could create inverse lock ordering between them. other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(blkg_stat_lock); local_irq_disable(); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); lock(blkg_stat_lock); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); *** DEADLOCK *** 2 locks held by swapper/0/0: #0: ffffffff8560b5a0 (rcu_callback){....}-{0:0}, at: rcu_core+0x83e/0x2860 #1: ffffffff8560b6c0 (rcu_read_lock){....}-{1:2}, at: __blkcg_rstat_flush.isra.0+0x93/0x4e0 the shortest dependencies between 2nd lock and 1st lock: -> (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} { IN-HARDIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime_field+0xa0/0x120 account_system_index_time+0x199/0x2c0 update_process_times+0x26/0x150 tick_sched_handle+0x8e/0x170 tick_sched_timer+0xe6/0x110 __hrtimer_run_queues+0x17f/0xb60 hrtimer_interrupt+0x2ef/0x750 __sysvec_apic_timer_interrupt+0xff/0x380 sysvec_apic_timer_interrupt+0x69/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 lock_acquire+0x1c7/0x4c0 _raw_spin_lock+0x2b/0x40 d_splice_alias+0x98/0xe30 kernfs_iop_lookup+0x283/0x330 lookup_open.isra.0+0x745/0x1400 path_openat+0x96c/0x2710 do_filp_open+0x1ba/0x410 do_sys_openat2+0x164/0x1d0 __x64_sys_openat+0x143/0x200 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime_field+0xa0/0x120 account_system_index_time+0x199/0x2c0 update_process_times+0x26/0x150 tick_sched_handle+0x8e/0x170 tick_sched_timer+0xe6/0x110 __hrtimer_run_queues+0x17f/0xb60 hrtimer_interrupt+0x2ef/0x750 __sysvec_apic_timer_interrupt+0xff/0x380 sysvec_apic_timer_interrupt+0x33/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 __do_softirq+0x177/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_call_function_single+0x6e/0x90 asm_sysvec_call_function_single+0x1a/0x20 _raw_spin_unlock_irqrestore+0x34/0x50 kmemleak_free+0x18a/0x200 __kmem_cache_free+0x95/0x330 kernfs_release_file+0xf7/0x240 kernfs_fop_release+0x193/0x1e0 __fput+0x406/0xad0 task_work_run+0x174/0x280 exit_to_user_mode_prepare+0x17d/0x180 syscall_exit_to_user_mode+0x1d/0x50 do_syscall_64+0x4c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_flush_locked+0x131/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] __key.0+0x0/0x40 ... acquired at: _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 -> (blkg_stat_lock){+.-.}-{2:2} { HARDIRQ-ON-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 default_idle+0xf/0x20 default_idle_call+0x67/0xa0 do_idle+0x305/0x3c0 cpu_startup_entry+0x18/0x20 rest_init+0x168/0x290 arch_call_rest_init+0x13/0x40 start_kernel+0x3c4/0x4e0 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0xcf/0xe0 secondary_startup_64_no_verify+0x179/0x17b IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 default_idle+0xf/0x20 default_idle_call+0x67/0xa0 do_idle+0x305/0x3c0 cpu_startup_entry+0x18/0x20 rest_init+0x168/0x290 arch_call_rest_init+0x13/0x40 start_kernel+0x3c4/0x4e0 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0xcf/0xe0 secondary_startup_64_no_verify+0x179/0x17b INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] blkg_stat_lock+0x18/0x60 ... acquired at: __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 default_idle+0xf/0x20 default_idle_call+0x67/0xa0 do_idle+0x305/0x3c0 cpu_startup_entry+0x18/0x20 rest_init+0x168/0x290 arch_call_rest_init+0x13/0x40 start_kernel+0x3c4/0x4e0 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0xcf/0xe0 secondary_startup_64_no_verify+0x179/0x17b stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0-rc7-next-20230622 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 Call Trace: dump_stack_lvl+0x91/0xf0 print_irq_inversion_bug.part.0+0x3d5/0x570 mark_lock.part.0+0x900/0x2f50 __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:default_idle+0xf/0x20 Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 93 20 31 00 fb f4 e9 ab 55 02 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 RSP: 0018:ffffffff85407e10 EFLAGS: 00000206 RAX: 0000000000099ed1 RBX: 0000000000000000 RCX: ffffffff84514c7e RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8128a1c5 RBP: dffffc0000000000 R08: 0000000000000001 R09: ffffed100d9c69f8 R10: ffff88806ce34fc3 R11: 0000000000000001 R12: ffffffff85d3e0d0 R13: 1ffffffff0a80fc7 R14: 0000000000000000 R15: 0000000000000000 default_idle_call+0x67/0xa0 do_idle+0x305/0x3c0 cpu_startup_entry+0x18/0x20 rest_init+0x168/0x290 arch_call_rest_init+0x13/0x40 start_kernel+0x3c4/0x4e0 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0xcf/0xe0 secondary_startup_64_no_verify+0x179/0x17b kworker/u4:7: attempt to access beyond end of device loop5: rw=1, sector=124, nr_sectors = 4 limit=40 Buffer I/O error on dev loop5, logical block 31, lost async page write loop5: detected capacity change from 0 to 40 kworker/u4:7: attempt to access beyond end of device loop5: rw=1, sector=124, nr_sectors = 4 limit=40 Buffer I/O error on dev loop5, logical block 31, lost async page write loop5: detected capacity change from 0 to 40 kworker/u4:2: attempt to access beyond end of device loop5: rw=1, sector=124, nr_sectors = 4 limit=40 Buffer I/O error on dev loop5, logical block 31, lost async page write autofs4:pid:5063:autofs_fill_super: called with bogus options autofs4:pid:5070:autofs_fill_super: called with bogus options autofs4:pid:5079:autofs_fill_super: called with bogus options autofs4:pid:5083:autofs_fill_super: called with bogus options netlink: 248 bytes leftover after parsing attributes in process `syz-executor.2'. ---------------- Code disassembly (best guess): 0: 4c 01 c7 add %r8,%rdi 3: 4c 29 c2 sub %r8,%rdx 6: e9 72 ff ff ff jmpq 0xffffff7d b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 90 nop 16: 90 nop 17: 90 nop 18: 90 nop 19: 90 nop 1a: 90 nop 1b: f3 0f 1e fa endbr64 1f: eb 07 jmp 0x28 21: 0f 00 2d 93 20 31 00 verw 0x312093(%rip) # 0x3120bb 28: fb sti 29: f4 hlt * 2a: fa cli <-- trapping instruction 2b: e9 ab 55 02 00 jmpq 0x255db 30: 66 66 2e 0f 1f 84 00 data16 nopw %cs:0x0(%rax,%rax,1) 37: 00 00 00 00 3b: 90 nop 3c: 90 nop 3d: 90 nop 3e: 90 nop 3f: 90 nop