======================================================== WARNING: possible irq lock inversion dependency detected 6.4.0-rc7-next-20230622 #1 Not tainted -------------------------------------------------------- ksoftirqd/1/19 just changed the state of lock: ffffffff85814c18 (blkg_stat_lock){+.-.}-{2:2}, at: __blkcg_rstat_flush.isra.0+0x11f/0x4e0 but this lock was taken by another, HARDIRQ-safe lock in the past: (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} and interrupts could create inverse lock ordering between them. other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(blkg_stat_lock); local_irq_disable(); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); lock(blkg_stat_lock); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); *** DEADLOCK *** 2 locks held by ksoftirqd/1/19: #0: ffffffff8560b5a0 (rcu_callback){....}-{0:0}, at: rcu_core+0x83e/0x2860 #1: ffffffff8560b6c0 (rcu_read_lock){....}-{1:2}, at: __blkcg_rstat_flush.isra.0+0x93/0x4e0 the shortest dependencies between 2nd lock and 1st lock: -> (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} { IN-HARDIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime_field+0xa0/0x120 account_system_index_time+0x199/0x2c0 update_process_times+0x26/0x150 tick_sched_handle+0x8e/0x170 tick_sched_timer+0xe6/0x110 __hrtimer_run_queues+0x17f/0xb60 hrtimer_interrupt+0x2ef/0x750 __sysvec_apic_timer_interrupt+0xff/0x380 sysvec_apic_timer_interrupt+0x69/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 lock_acquire+0x33/0x4c0 fs_reclaim_acquire+0x121/0x170 kmem_cache_alloc+0x48/0x370 __create_object+0x3c/0xc90 kmem_cache_alloc+0x20b/0x370 getname_flags.part.0+0x50/0x4f0 getname+0x92/0xe0 do_sys_openat2+0xec/0x1d0 __x64_sys_openat+0x143/0x200 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime+0x75/0xc0 update_curr+0x350/0x6d0 enqueue_task_fair+0x6a9/0x1ca0 activate_task+0xe7/0x250 ttwu_do_activate+0x10d/0x7a0 sched_ttwu_pending+0x1da/0x3b0 __flush_smp_call_function_queue+0x594/0xea0 __sysvec_call_function_single+0x96/0x2a0 sysvec_call_function_single+0x33/0x90 asm_sysvec_call_function_single+0x1a/0x20 unwind_next_frame+0x5c9/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 __kasan_record_aux_stack+0x8e/0xa0 __call_rcu_common.constprop.0+0x6a/0xb90 kmem_cache_free+0xc1/0x4d0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 kernel_text_address+0x0/0xc0 __kernel_text_address+0xd/0x40 unwind_get_return_address+0x59/0xa0 arch_stack_walk+0x9d/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x59/0x70 kmem_cache_alloc+0x16b/0x370 __create_object+0x3c/0xc90 kmem_cache_alloc_bulk+0x3b4/0x6a0 mas_alloc_nodes+0x337/0x890 mas_preallocate+0x1bf/0x370 __split_vma+0x191/0x770 split_vma+0xc6/0x110 mprotect_fixup+0x8d2/0xb30 do_mprotect_pkey+0x87e/0xd40 __x64_sys_mprotect+0x78/0xc0 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_flush_locked+0x131/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] __key.0+0x0/0x40 ... acquired at: _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 -> (blkg_stat_lock){+.-.}-{2:2} { HARDIRQ-ON-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 run_ksoftirqd+0x29/0x50 smpboot_thread_fn+0x40b/0x930 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 run_ksoftirqd+0x29/0x50 smpboot_thread_fn+0x40b/0x930 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] blkg_stat_lock+0x18/0x60 ... acquired at: __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 run_ksoftirqd+0x29/0x50 smpboot_thread_fn+0x40b/0x930 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 stack backtrace: CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 6.4.0-rc7-next-20230622 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 Call Trace: dump_stack_lvl+0x91/0xf0 print_irq_inversion_bug.part.0+0x3d5/0x570 mark_lock.part.0+0x900/0x2f50 __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x2860 __do_softirq+0x1b7/0x7d4 run_ksoftirqd+0x29/0x50 smpboot_thread_fn+0x40b/0x930 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 FAULT_INJECTION: forcing a failure. name failslab, interval 1, probability 0, space 0, times 0 CPU: 1 PID: 5795 Comm: syz-executor.6 Not tainted 6.4.0-rc7-next-20230622 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 Call Trace: dump_stack_lvl+0xc1/0xf0 should_fail_ex+0x4b4/0x5b0 should_failslab+0x9/0x20 __kmem_cache_alloc_node+0x66/0x320 __kmalloc_node_track_caller+0x4b/0x160 kmemdup+0x2c/0x60 sidtab_sid2str_get+0x180/0x700 sidtab_entry_to_string+0x33/0x110 security_sid_to_context_core+0x32e/0x600 selinux_inode_init_security+0x4c4/0x6b0 security_inode_init_security+0x155/0x360 __ext4_new_inode+0x38ed/0x5410 ext4_mkdir+0x33e/0xb20 vfs_mkdir+0x24b/0x470 do_mkdirat+0x163/0x2d0 __x64_sys_mkdir+0xf6/0x150 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7ff1f15ddb19 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ff1eeb53188 EFLAGS: 00000246 ORIG_RAX: 0000000000000053 RAX: ffffffffffffffda RBX: 00007ff1f16f0f60 RCX: 00007ff1f15ddb19 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000000 RBP: 00007ff1eeb531d0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002 R13: 00007ffed50e65bf R14: 00007ff1eeb53300 R15: 0000000000022000 loop2: detected capacity change from 0 to 40 FAULT_INJECTION: forcing a failure. name failslab, interval 1, probability 0, space 0, times 0 CPU: 1 PID: 5811 Comm: syz-executor.5 Not tainted 6.4.0-rc7-next-20230622 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 Call Trace: dump_stack_lvl+0xc1/0xf0 should_fail_ex+0x4b4/0x5b0 should_failslab+0x9/0x20 __kmem_cache_alloc_node+0x66/0x320 __kmalloc_node_track_caller+0x4b/0x160 kmemdup+0x2c/0x60 sidtab_sid2str_get+0x180/0x700 sidtab_entry_to_string+0x33/0x110 security_sid_to_context_core+0x32e/0x600 selinux_inode_init_security+0x4c4/0x6b0 security_inode_init_security+0x155/0x360 __ext4_new_inode+0x38ed/0x5410 ext4_mkdir+0x33e/0xb20 vfs_mkdir+0x24b/0x470 do_mkdirat+0x163/0x2d0 __x64_sys_mkdir+0xf6/0x150 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7fc126c14b19 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fc12418a188 EFLAGS: 00000246 ORIG_RAX: 0000000000000053 RAX: ffffffffffffffda RBX: 00007fc126d27f60 RCX: 00007fc126c14b19 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000000 RBP: 00007fc12418a1d0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002 R13: 00007ffe59456edf R14: 00007fc12418a300 R15: 0000000000022000 loop3: detected capacity change from 0 to 2048 EXT4-fs (loop3): stripe (5) is not aligned with cluster size (16), stripe is disabled EXT4-fs warning (device loop3): read_mmp_block:114: Error -117 while reading MMP block 0 loop3: detected capacity change from 0 to 2048 EXT4-fs (loop3): stripe (5) is not aligned with cluster size (16), stripe is disabled EXT4-fs warning (device loop3): read_mmp_block:114: Error -117 while reading MMP block 0