EXT4-fs (loop3): feature flags set on rev 0 fs, running e2fsck is recommended EXT4-fs (loop3): ext4_check_descriptors: Block bitmap for group 0 not in group (block 0)! EXT4-fs (loop3): group descriptors corrupted! ======================================================== WARNING: possible irq lock inversion dependency detected 6.4.0-rc7-next-20230623 #1 Not tainted -------------------------------------------------------- systemd-udevd/103 just changed the state of lock: ffffffff85815818 (blkg_stat_lock){+.-.}-{2:2}, at: __blkcg_rstat_flush.isra.0+0x11f/0x4e0 but this lock was taken by another, HARDIRQ-safe lock in the past: (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} and interrupts could create inverse lock ordering between them. other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(blkg_stat_lock); local_irq_disable(); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); lock(blkg_stat_lock); lock(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); *** DEADLOCK *** 2 locks held by systemd-udevd/103: #0: ffffffff8560b5a0 (rcu_callback){....}-{0:0}, at: rcu_core+0x83e/0x28b0 #1: ffffffff8560b6c0 (rcu_read_lock){....}-{1:2}, at: __blkcg_rstat_flush.isra.0+0x93/0x4e0 the shortest dependencies between 2nd lock and 1st lock: -> (per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)){-.-.}-{2:2} { IN-HARDIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime_field+0xa0/0x120 account_system_index_time+0x199/0x2c0 update_process_times+0x26/0x150 tick_sched_handle+0x8e/0x170 tick_sched_timer+0xe6/0x110 __hrtimer_run_queues+0x17f/0xb60 hrtimer_interrupt+0x2ef/0x750 __sysvec_apic_timer_interrupt+0xff/0x380 sysvec_apic_timer_interrupt+0x69/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_get_return_address+0x4e/0xa0 arch_stack_walk+0x9d/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x59/0x70 kmem_cache_alloc_lru+0x247/0x680 alloc_inode+0x178/0x240 iget_locked+0x199/0x660 kernfs_get_inode+0x4b/0x520 kernfs_iop_lookup+0x1e9/0x330 lookup_open.isra.0+0x745/0x1400 path_openat+0x96c/0x2710 do_filp_open+0x1ba/0x410 do_sys_openat2+0x164/0x1d0 __x64_sys_openat+0x143/0x200 do_syscall_64+0x3f/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_updated+0xcb/0x2e0 __cgroup_account_cputime+0x75/0xc0 update_curr+0x350/0x6d0 enqueue_task_fair+0x6a9/0x1ca0 activate_task+0xe7/0x250 ttwu_do_activate+0x10d/0x7a0 try_to_wake_up+0x627/0x1b30 wake_bit_function+0xfe/0x240 __wake_up_common+0x14c/0x4c0 __wake_up_common_lock+0xd7/0x140 wake_up_bit+0xdd/0x130 end_buffer_read_sync+0x85/0xe0 end_bio_bh_io_sync+0xde/0x130 bio_endio+0x51a/0x620 blk_update_request+0x397/0x12a0 scsi_end_request+0x7a/0xa20 scsi_io_completion+0xc0/0x1640 scsi_complete+0x114/0x380 blk_complete_reqs+0xb8/0xf0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_call_function_single+0x6e/0x90 asm_sysvec_call_function_single+0x1a/0x20 lock_acquire+0x1c7/0x4c0 __is_insn_slot_addr+0x41/0x290 kernel_text_address+0x5b/0xc0 __kernel_text_address+0xd/0x40 unwind_get_return_address+0x59/0xa0 arch_stack_walk+0x9d/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x10a/0x190 kmem_cache_free+0x10d/0x4d0 security_file_free+0xae/0xe0 __fput+0x57a/0xad0 task_work_run+0x174/0x280 exit_to_user_mode_prepare+0x17d/0x180 syscall_exit_to_user_mode+0x1d/0x50 do_syscall_64+0x4c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock_irqsave+0x3a/0x60 cgroup_rstat_flush_locked+0x131/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 flush_memcg_stats_dwork+0x9/0x50 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] __key.0+0x0/0x40 ... acquired at: _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 mem_cgroup_wb_stats+0x3d8/0x4b0 wb_over_bg_thresh+0x30e/0x6a0 wb_writeback+0x338/0x980 wb_workfn+0x81d/0xed0 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 -> (blkg_stat_lock){+.-.}-{2:2} { HARDIRQ-ON-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_next_frame+0xbf7/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x10a/0x190 kmem_cache_free+0x10d/0x4d0 security_file_free+0xae/0xe0 __fput+0x57a/0xad0 task_work_run+0x174/0x280 exit_to_user_mode_prepare+0x17d/0x180 syscall_exit_to_user_mode+0x1d/0x50 do_syscall_64+0x4c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 IN-SOFTIRQ-W at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_next_frame+0xbf7/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x10a/0x190 kmem_cache_free+0x10d/0x4d0 security_file_free+0xae/0xe0 __fput+0x57a/0xad0 task_work_run+0x174/0x280 exit_to_user_mode_prepare+0x17d/0x180 syscall_exit_to_user_mode+0x1d/0x50 do_syscall_64+0x4c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 INITIAL USE at: lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 blkcg_rstat_flush+0x87/0xb0 cgroup_rstat_flush_locked+0x706/0xd80 cgroup_rstat_flush+0x37/0x50 do_flush_stats+0x97/0xf0 mem_cgroup_wb_stats+0x3d8/0x4b0 wb_over_bg_thresh+0x30e/0x6a0 wb_writeback+0x338/0x980 wb_workfn+0x81d/0xed0 process_one_work+0xabf/0x1770 worker_thread+0x64f/0x12a0 kthread+0x33f/0x440 ret_from_fork+0x2c/0x50 } ... key at: [] blkg_stat_lock+0x18/0x60 ... acquired at: __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 unwind_next_frame+0xbf7/0x2490 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x10a/0x190 kmem_cache_free+0x10d/0x4d0 security_file_free+0xae/0xe0 __fput+0x57a/0xad0 task_work_run+0x174/0x280 exit_to_user_mode_prepare+0x17d/0x180 syscall_exit_to_user_mode+0x1d/0x50 do_syscall_64+0x4c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 stack backtrace: CPU: 1 PID: 103 Comm: systemd-udevd Not tainted 6.4.0-rc7-next-20230623 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 Call Trace: dump_stack_lvl+0x91/0xf0 print_irq_inversion_bug.part.0+0x3d5/0x570 mark_lock.part.0+0x900/0x2f50 __lock_acquire+0x8b8/0x6340 lock_acquire+0x19a/0x4c0 _raw_spin_lock+0x2b/0x40 __blkcg_rstat_flush.isra.0+0x11f/0x4e0 __blkg_release+0xfa/0x3b0 rcu_core+0x8c8/0x28b0 __do_softirq+0x1b7/0x7d4 irq_exit_rcu+0x93/0xc0 sysvec_apic_timer_interrupt+0x6e/0x90 asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:unwind_next_frame+0xbf7/0x2490 Code: 04 01 e8 1c 84 1d 00 4c 8b 4c 24 08 e9 22 f7 ff ff 48 be 00 00 00 00 00 fc ff df 48 8d 79 02 48 89 f8 48 c1 e8 03 0f b6 14 30 <48> 8d 41 03 49 89 c0 49 c1 e8 03 41 0f b6 34 30 49 89 f8 41 83 e0 RSP: 0018:ffff88801423f978 EFLAGS: 00000217 RAX: 1ffffffff0c9815c RBX: 0000000000000001 RCX: ffffffff864c0ae4 RDX: 0000000000000000 RSI: dffffc0000000000 RDI: ffffffff864c0ae6 RBP: ffff88801423fa40 R08: ffffffff864c0ae8 R09: ffff88801423fa28 R10: 0000000000038001 R11: 00000000000b63f3 R12: ffff88801423fa48 R13: ffff88801423fa30 R14: ffff88801423ff40 R15: ffff88801423f9e8 arch_stack_walk+0x87/0xf0 stack_trace_save+0x90/0xd0 kasan_save_stack+0x22/0x50 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x10a/0x190 kmem_cache_free+0x10d/0x4d0 security_file_free+0xae/0xe0 __fput+0x57a/0xad0 task_work_run+0x174/0x280 exit_to_user_mode_prepare+0x17d/0x180 syscall_exit_to_user_mode+0x1d/0x50 do_syscall_64+0x4c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7f9c689a30f3 Code: 8b 15 d9 9e 00 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 RSP: 002b:00007fffb393af28 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 00007f9c6840a6c0 RCX: 00007f9c689a30f3 RDX: 00007fffb393ae98 RSI: 0000000000000000 RDI: 0000000000000011 RBP: 0000000000000011 R08: 000055d3e3eb9a10 R09: 0073656369766564 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000010 R14: 0000000000000010 R15: 000055d3e3eb9a31 syz_tun: entered promiscuous mode syz_tun: left promiscuous mode syz_tun: entered promiscuous mode syz_tun: left promiscuous mode ---------------- Code disassembly (best guess): 0: 04 01 add $0x1,%al 2: e8 1c 84 1d 00 callq 0x1d8423 7: 4c 8b 4c 24 08 mov 0x8(%rsp),%r9 c: e9 22 f7 ff ff jmpq 0xfffff733 11: 48 be 00 00 00 00 00 movabs $0xdffffc0000000000,%rsi 18: fc ff df 1b: 48 8d 79 02 lea 0x2(%rcx),%rdi 1f: 48 89 f8 mov %rdi,%rax 22: 48 c1 e8 03 shr $0x3,%rax 26: 0f b6 14 30 movzbl (%rax,%rsi,1),%edx * 2a: 48 8d 41 03 lea 0x3(%rcx),%rax <-- trapping instruction 2e: 49 89 c0 mov %rax,%r8 31: 49 c1 e8 03 shr $0x3,%r8 35: 41 0f b6 34 30 movzbl (%r8,%rsi,1),%esi 3a: 49 89 f8 mov %rdi,%r8 3d: 41 rex.B 3e: 83 .byte 0x83 3f: e0 .byte 0xe0