- Sep 21, 2022
-
-
Dylan Yudaken authored
Add tracing for io_run_local_task_work Signed-off-by:
Dylan Yudaken <dylany@fb.com> Link: https://lore.kernel.org/r/20220830125013.570060-8-dylany@fb.com Signed-off-by:
Jens Axboe <axboe@kernel.dk>
-
Dylan Yudaken authored
Some workloads rely on a registered eventfd (via io_uring_register_eventfd(3)) in order to wake up and process the io_uring. In the case of a ring setup with IORING_SETUP_DEFER_TASKRUN, that eventfd also needs to be signalled when there are tasks to run. This changes an old behaviour which assumed 1 eventfd signal implied at least 1 CQE, however only when this new flag is set (and so old users will not notice). This should be expected with the IORING_SETUP_DEFER_TASKRUN flag as it is not guaranteed that every task will result in a CQE. Signed-off-by:
-
Dylan Yudaken authored
Non functional change: move this function above io_eventfd_signal so it can be used from there Signed-off-by:
-
Dylan Yudaken authored
Allow deferring async tasks until the user calls io_uring_enter(2) with the IORING_ENTER_GETEVENTS flag. Enable this mode with a flag at io_uring_setup time. This functionality requires that the later io_uring_enter will be called from the same submission task, and therefore restrict this flag to work only when IORING_SETUP_SINGLE_ISSUER is also set. Being able to hand pick when tasks are run prevents the problem where there is current work to be done, however task work runs anyway. For example, a common workload would obtain a batch of CQEs, and process each one. Interrupting this to additional taskwork would add latency but not gain anything. If instead task work is deferred to just before more CQEs are obtained then no additional latency is added. The way this is implemented is by trying to keep task work local to a io_ring_ctx, rather than to the submission task. This is required, as the application will want to wake up only a single io_ring_ctx at a time to process work, and so the lists of work have to be kept separate. This has some other benefits like not having to check the task continually in handle_tw_list (and potentially unlocking/locking those), and reducing locks in the submit & process completions path. There are networking cases where using this option can reduce request latency by 50%. For example a contrived example using [1] where the client sends 2k data and receives the same data back while doing some system calls (to trigger task work) shows this reduction. The reason ends up being that if sending responses is delayed by processing task work, then the client side sits idle. Whereas reordering the sends first means that the client runs it's workload in parallel with the local task work. [1]: Using https://github.com/DylanZA/netbench/tree/defer_run Client: ./netbench --client_only 1 --control_port 10000 --host <host> --tx "epoll --threads 16 --per_thread 1 --size 2048 --resp 2048 --workload 1000" Server: ./netbench --server_only 1 --control_port 10000 --rx "io_uring --defer_taskrun 0 --workload 100" --rx "io_uring --defer_taskrun 1 --workload 100" Signed-off-by:
-
Dylan Yudaken authored
This is not needed, and it is normally better to wait for task work until after submissions. This will allow greater batching if either work arrives in the meanwhile, or if the submissions cause task work to be queued up. For SQPOLL this also no longer runs task work, but this is handled inside the SQPOLL loop anyway. For IOPOLL io_iopoll_check will run task work anyway And otherwise io_cqring_wait will run task work Suggested-by:
Pavel Begunkov <asml.silence@gmail.com> Signed-off-by:
-
Dylan Yudaken authored
This will be used later to know if the ring has outstanding work. Right now just if there is overflow CQEs to copy to the main CQE ring, but later will include deferred tasks Signed-off-by:
-
Dylan Yudaken authored
'running' is set once and read once, so can easily just remove it Signed-off-by:
-
- Sep 18, 2022
-
-
Stefan Metzmacher authored
It's confusing to see the string SENDZC_NOTIF in ftrace output when using IORING_OP_SEND_ZC. Fixes: b48c312b ("io_uring/net: simplify zerocopy send user API") Signed-off-by:
Stefan Metzmacher <metze@samba.org> Cc: Pavel Begunkov <asml.silence@gmail.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: io-uring@vger.kernel.org Reviewed-by:
-
Pavel Begunkov authored
Notifications usually outlive requests, so we need to pin buffers with it by assigning a rsrc to it instead of the request. Fixed: b48c312b ("io_uring/net: simplify zerocopy send user API") Signed-off-by:
-
- Sep 15, 2022
-
-
Jens Axboe authored
If we're invoked with a fixed file, follow the normal rules of not calling io_fput_file(). Fixed files are permanently registered to the ring, and do not need putting separately. Cc: stable@vger.kernel.org Fixes: aa184e86 ("io_uring: don't attempt to IOPOLL for MSG_RING requests") Signed-off-by:
-
- Sep 13, 2022
-
-
Pavel Begunkov authored
Kernel test robot reports that we test negativity of an unsigned in io_fixup_rw_res() after a recent change, which masks error codes and messes up the return value in case I/O is re-retried and failed with an error. Fixes: 4d9cb92c ("io_uring/rw: fix short rw error handling") Reported-by:
kernel test robot <lkp@intel.com> Signed-off-by:
-
- Sep 09, 2022
-
-
Pavel Begunkov authored
We have a couple of problems, first reports of unexpected link breakage for reads when cqe->res indicates that the IO was done in full. The reason here is partial IO with retries. TL;DR; we compare the result in __io_complete_rw_common() against req->cqe.res, but req->cqe.res doesn't store the full length but rather the length left to be done. So, when we pass the full corrected result via kiocb_done() -> __io_complete_rw_common(), it fails. The second problem is that we don't try to correct res in io_complete_rw(), which, for instance, might be a problem for O_DIRECT but when a prefix of data was cached in the page cache. We also definitely don't want to pass a corrected result into io_rw_done(). The fix here is to leave __io_complete_rw_common() alone, always pass not corrected result into it and fix it up as the last step just before actually finishing the I/O. Cc: stable@vger.kernel.org Signed-off-by:
Beld Zhang <beldzhang@gmail.com> Signed-off-by:
-
- Sep 08, 2022
-
-
Pavel Begunkov authored
Every time we return from an issue handler and expect the request to be retried we should also setup it for async exec ourselves. Do that when we return on IORING_RECVSEND_POLL_FIRST in io_sendzc(), otherwise it'll re-read the address, which might be a surprise for the userspace. Fixes: 092aeedb ("io_uring: allow to pass addr into sendzc") Signed-off-by:
-
- Sep 07, 2022
-
-
Pavel Begunkov authored
When we queue a request via tw for execution it's not going to be executed immediately, so when io_queue_async() hits IO_APOLL_READY and queues a tw but doesn't try to recycle/consume the buffer some other request may try to use the the buffer. Fixes: c7fb1942 ("io_uring: add support for ring mapped supplied buffers") Signed-off-by:
-
Pavel Begunkov authored
When we don't recycle a selected ring buffer we should advance the head of the ring, so don't just skip io_kbuf_recycle() for IORING_OP_READV but adjust the ring. Fixes: 934447a6 ("io_uring: do not recycle buffer in READV") Signed-off-by:
-
- Sep 05, 2022
-
-
Jiapeng Chong authored
The function io_notif_complete() is defined in the notif.c file, but not called elsewhere, so delete this unused function. io_uring/notif.c:24:20: warning: unused function 'io_notif_complete' [-Wunused-function]. Link: https://bugzilla.openanolis.cn/show_bug.cgi?id=2047 Reported-by:
Abaci Robot <abaci@linux.alibaba.com> Signed-off-by:
Jiapeng Chong <jiapeng.chong@linux.alibaba.com> Reviewed-by:
-
- Sep 01, 2022
-
-
Pavel Begunkov authored
Following user feedback, this patch simplifies zerocopy send API. One of the main complaints is that the current API is difficult with the userspace managing notification slots, and then send retries with error handling make it even worse. Instead of keeping notification slots change it to the per-request notifications model, which posts both completion and notification CQEs for each request when any data has been sent, and only one CQE if it fails. All notification CQEs will have IORING_CQE_F_NOTIF set and IORING_CQE_F_MORE in completion CQEs indicates whether to wait a notification or not. IOSQE_CQE_SKIP_SUCCESS is disallowed with zerocopy sends for now. This is less flexible, but greatly simplifies the user API and also the kernel implementation. We reuse notif helpers in this patch, but in the future there won't be need for keeping two requests. Signed-off-by:
-
Pavel Begunkov authored
We're going to remove the userspace exposed zerocopy notification API, remove notification registration. Signed-off-by:
-
Pavel Begunkov authored
This reverts commit 4379d5f1. We removed notification flushing, also cleanup uapi preparation changes to not pollute it. Signed-off-by:
-
Pavel Begunkov authored
This reverts commit 492dddb4. Soon we won't have the very notion of notification flushing, so remove notification flushing requests. Signed-off-by:
-
- Aug 26, 2022
-
-
Pavel Begunkov authored
Length parameter of io_sg_from_iter() can be smaller than the iterator's size, as it's with TCP, so when we set from->count at the end of the function we truncate the iterator forcing TCP to return preliminary with a short send. It affects zerocopy sends with large payload sizes and leads to retries and possible request failures. Fixes: 3ff1a0d3 ("io_uring: enable managed frags with register buffers") Signed-off-by:
-
Luis Chamberlain authored
io-uring cmd support was added through ee692a21 ("fs,io_uring: add infrastructure for uring-cmd"), this extended the struct file_operations to allow a new command which each subsystem can use to enable command passthrough. Add an LSM specific for the command passthrough which enables LSMs to inspect the command details. This was discussed long ago without no clear pointer for something conclusive, so this enables LSMs to at least reject this new file operation. [0] https://lkml.kernel.org/r/8adf55db-7bab-f59d-d612-ed906b948d19@schaufler-ca.com Cc: stable@vger.kernel.org Fixes: ee692a21 ("fs,io_uring: add infrastructure for uring-cmd") Signed-off-by:
Luis Chamberlain <mcgrof@kernel.org> Acked-by:
Paul Moore <paul@paul-moore.com>
-
- Aug 25, 2022
-
-
Pavel Begunkov authored
We usually copy all bits that a request needs from the userspace for async execution, so the userspace can keep them on the stack. However, send zerocopy violates this pattern for addresses and may reloads it e.g. from io-wq. Save the address if any in ->async_data as usual. Reported-by:
-
- Aug 24, 2022
-
-
Pavel Begunkov authored
There are opcodes that need ->async_data only in some cases and allocation it unconditionally may hurt performance. Add an option to opdef to make move the allocation part from the core io_uring to opcode specific code. Note, we can't just set opdef->async_size to zero because there are other helpers that rely on it, e.g. io_alloc_async_data(). Signed-off-by:
-
Pavel Begunkov authored
Currently, there is no ordering between notification CQEs and completions of the send flushing it, this quite complicates the userspace, especially since we don't flush notification when the send(+flush) request fails, i.e. there will be only one CQE. What we can do is to make sure that notification completions come only after sends. The easiest way to achieve this is to not try to complete a notification inline from io_sendzc() but defer it to task_work, considering that io-wq sendzc is disallowed CQEs will be naturally ordered because task_works will only be executed after we're done with submission and so inline completion. Signed-off-by:
-
Pavel Begunkov authored
Fix up indentation before we get complaints from tooling. Signed-off-by:
-
Pavel Begunkov authored
Failed requests should be marked with req_set_fail(), so links and cqe skipping work correctly, which is missing in io_sendzc(). Note, io_sendzc() return IOU_OK on failure, so the core code won't do the cleanup for us. Fixes: 06a5464b ("io_uring: wire send zc request type") Signed-off-by:
-
Pavel Begunkov authored
Fix up the io_alloc_notif()'s __must_hold as we don't have a ctx argument there but should get it from the slot instead. Reported-by:
-
- Aug 23, 2022
-
-
Kanchan Joshi authored
If ->uring_cmd returned an error value different from -EAGAIN or -EIOCBQUEUED, it gets overridden with IOU_OK. This invites trouble as caller (io_uring core code) handles IOU_OK differently than other error codes. Fix this by returning the actual error code. Signed-off-by:
Kanchan Joshi <joshi.k@samsung.com> Signed-off-by:
-
Jens Axboe authored
The passed in index should be validated against the number of registered files we have, it needs to be smaller than the index value to avoid going one beyond the end. Fixes: 78a861b9 ("io_uring: add sync cancelation API through io_uring_register()") Reported-by:
Luo Likang <luolikang@nsfocus.com> Signed-off-by:
-
- Aug 18, 2022
-
-
Pavel Begunkov authored
There is another spot where we check ->async_data directly instead of using req_has_async_data(), which is the way to do it, fix it up. Fixes: 43e0bbbd ("io_uring: add netmsg cache") Signed-off-by:
-
- Aug 16, 2022
-
-
Pavel Begunkov authored
1024 notification slots is rather an arbitrary value, raise it up, everything is accounted to memcg. Signed-off-by:
-
Pavel Begunkov authored
We may account memory to a memcg of a request that didn't even got to the network layer. It's not a bug as it'll be routinely cleaned up on flush, but it might be confusing for the userspace. Signed-off-by:
-
Pavel Begunkov authored
We have a helper that checks for whether a request contains anything in ->async_data or not, namely req_has_async_data(). It's better to use it as it might have some extra considerations. Fixes: 43e0bbbd ("io_uring: add netmsg cache") Signed-off-by:
-
- Aug 12, 2022
-
-
Stefan Metzmacher authored
Signed-off-by:
-
Stefan Metzmacher authored
We need to make sure (at build time) that struct io_cmd_data is not casted to a structure that's larger. Signed-off-by:
-
- Aug 11, 2022
-
-
Stefan Metzmacher authored
This makes the assignment typesafe. It prepares changing io_kiocb_to_cmd() in the next commit. Signed-off-by:
-
Anuj Gupta authored
Commit 97b388d7 ("io_uring: handle completions in the core") moved the error handling from handler to core. But for io_uring_cmd handler we end up completing more than once (both in handler and in core) leading to use_after_free. Change io_uring_cmd handler to avoid calling io_uring_cmd_done in case of error. Fixes: 97b388d7 ("io_uring: handle completions in the core") Signed-off-by:
Anuj Gupta <anuj20.g@samsung.com> Signed-off-by:
-
- Aug 05, 2022
-
-
Dylan Yudaken authored
Fix casts missing the __user parts. This seemed to only cause errors on the alpha build, or if checked with sparse, but it was definitely an oversight. Reported-by:
-
- Aug 04, 2022
-
-
Pavel Begunkov authored
io_uring handles short sends/recvs for stream sockets when MSG_WAITALL is set, however new zerocopy send is inconsistent in this regard, which might be confusing. Handle short sends. Signed-off-by:
-
